APFS in Detail: Overview

Apple announced a new file system that will make its way into all of its OS variants (macOS, tvOS, iOS, watchOS) in the coming years. Media coverage to this point has been mostly breathless elongations of Apple’s developer documentation. With a dearth of detail I decided to attend the presentation and Q&A with the APFS team at WWDC. Dominic Giampaolo and Eric Tamura, two members of the APFS team, gave an overview to a packed room; along with other members of the team, they patiently answered questions later in the day. With those data points and some first hand usage I wanted to provide an overview and analysis both as a user of Apple-ecosystem products and as a long-time operating system and file system developer.

I’ve divided my review into several sections that span a few posts. I’d encourage you to jump around to topics of interest or skip right to the conclusion (or to the tweet summary). Highest praise goes to encryption; ire to data integrity.

Basics

APFS, the Apple File System, was itself started in 2014 with Dominic as its lead engineer. It’s a stand-alone, from-scratch implementation (an earlier version of this post noted a dependency on Core Storage, but Dominic set me straight). I asked him about looking for inspiration in other modern file systems such as BSD’s HAMMER, Linux’s btrfs, or OpenZFS (Solaris, illumos, FreeBSD, Mac OS X, Ubuntu Linux, etc.), all of which have features similar to what APFS intends to deliver. (And note that Apple built a fairly complete port of ZFS, though Dominic was not apparently part of the group advocating for it.) Dominic explained that while, as a self-described file system guy (he built the file system in BeOS, unfairly relegated to obscurity when Apple opted to purchase NeXTSTEP instead), he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself.

Dominic praised the APFS testing team as being exemplary. This is absolutely critical. A common adage is that it takes a decade to mature a file system. And my experience with ZFS more or less confirms this. Apple will be delivering APFS broadly with 3-4 years of development so will need to accelerate quickly to maturity.

Paying Down Debt

HFS was introduced in 1985 when the Mac 512K (of memory! Holy smokes!) was Apple’s flagship. HFS+, a significant iteration, shipped in 1998 on the G3 PowerMacs with 4GB hard drives. Since then storage capacities have increased by factors of 1,000,000 and 1,000 respectively. HFS+ has been pulled in a bunch of competing directions with different forks for different devices (e.g. the iOS team created their own HFS variant, working so covertly that not even the Mac OS team knew) and different features (e.g. journaling, case insensitive). It’s old; it’s a mess; and, critically, it’s missing a bunch of features that are really considered the basic cost of doing business for most operating systems. Wikipedia lists nanosecond timestamps, checksums, snapshots, and sparse file support among those missing features. Add to that the obvious gap of large device support and you’ve got a big chunk of the APFS feature list.

APFS first and foremost pays down the unsustainable technical debt that Apple has been carrying in HFS+. (In 2001 ZFS grew from a similar need where UFS had been evolved since 1977.) It unifies the multifarious forks. It introduces the expected features. In general it first brings the derelict building up to code.

Compression is an obvious gap in the APFS feature list that is common in many file systems. It’s conceptually quite easy, I told the development team (we had it in ZFS from the outset), so why not include it? To appeal to Dominic’s BeOS nostalgia I even recalled my job interview with Be in 2000 when they talked about how compression actually improved overall performance since data I/O is far more expensive than computation (obvious now, but novel then). The Apple folks agreed, and—in typical Apple fashion—neither confirmed nor denied while strongly implying that it’s definitely a feature we can expect in APFS. I’ll be surprised if compression isn’t included in its public launch.

 

Next in this series: Encryption, Snapshots, and Backup

Posted on June 19, 2016 at 7:29 pm by ahl · Permalink
In: Software · Tagged with: 

42 Responses

Subscribe to comments via RSS

  1. Written by Patrick
    on June 20, 2016 at 4:19 am
    Permalink

    Let’s hope that they reconsider the user data checksum decision. Claiming it’s not required because of excellent components is kind of a strange thing to say, and makes the entire FS suspect.

  2. [...] announced something inferior to what #linux had for years and media goes wild http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/ Like this:Like [...]

  3. Written by Dominic Giampaolo
    on June 20, 2016 at 1:30 pm
    Permalink

    The statement at the beginning of the article that “APFS, the Apple File System, relies on Core Storage functionality” is not accurate. Apple File System does not use any functionality or code whatsoever from CoreStorage. Apple File System was developed completely from scratch.

    • Written by ahl
      on June 20, 2016 at 1:59 pm
      Permalink

      Hey Dominic, thanks for the comment. That was the my understanding after our conversation at WWDC, my apologies for getting it wrong. I’ll make the appropriate update. Was I wide of the mark on anything else?

      • Written by Dominic Giampaolo
        on June 20, 2016 at 4:10 pm
        Permalink

        Thanks for updating it. Everything else seems correct. Also, thanks for the even handed review.

  4. [...] APFS in Detail: Overview Apple announced a new file system that will make its way into all of its OS variants (macOS, tvOS, iOS, watchOS) in the coming years. Media coverage to this point has been mostly breathless elongations of Apple’s developer documentation. With a dearth of detail I decided to attend the presentation and Q&A with the APFS team at WWDC. Dominic Giampaolo and Eric Tamura, two members of the APFS team, gave an overview to a packed room; along with other members of the team, they patiently answered questions later in the day. With those data points and some first hand usage I wanted to provide an overview and analysis both as a user of Apple-ecosystem products and as a long-time operating system and file system developer. [...]

  5. Written by Michael Tsai - Blog - Apple File System (APFS)
    on June 21, 2016 at 12:55 am
    Permalink

    [...] series of posts based on his talks with Apple engineers Dominic Giampaolo and Eric Tamura at WWDC. Overview (Hacker News): APFS, the Apple File System, was itself started in 2014 with Dominic as its lead [...]

  6. Written by WWDC 2016: Apple File System
    on June 21, 2016 at 3:01 pm
    Permalink

    [...] ) kümmert, dann könnte eventuell der Mac-Finder schneller kopieren. Weiterführend: dtrace.org.Sharen [...]

  7. [...] From Leventhal’s blog: [...]

  8. [...] From Leventhal’s blog: [...]

  9. [...] Leventhal’s blog: APFS first and foremost pays down the unsustainable technical debt that Apple has been carrying in [...]

  10. [...] From Leventhal's blog: [...]

  11. Written by Chris
    on June 22, 2016 at 12:14 am
    Permalink

    One bit flip corruption in a typical file isn’t a big deal. One bit flip corruption in a compressed file can be game over. And if that bit is in the core OS, do not pass go, do not collect $200.

  12. Written by csh
    on June 22, 2016 at 10:00 am
    Permalink

    Thank you Dominic, Eric, and the team. This has been on the top of my feature request list for many years. It will be a breathe of fresh air. Looking forward to it.

  13. Written by A one word review of macOS Sierra - Cool Tech Reviews
    on June 22, 2016 at 9:21 pm
    Permalink

    [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  14. Written by A one word review of macOS Sierra | Gulf News Today
    on June 22, 2016 at 9:22 pm
    Permalink

    [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  15. Written by A one word review of macOS Sierra - Bain Daily
    on June 22, 2016 at 9:25 pm
    Permalink

    [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  16. Written by A one word review of macOS Sierra | EuroMarket News
    on June 22, 2016 at 9:37 pm
    Permalink

    [...] had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  17. Written by A one word review of macOS Sierra | BizTechPartners
    on June 22, 2016 at 9:43 pm
    Permalink

    [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  18. Written by A one word review of macOS SierraTech Giant News
    on June 22, 2016 at 9:58 pm
    Permalink

    [...] had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  19. Written by A one word review of macOS Sierra | Tech News Daily
    on June 22, 2016 at 11:12 pm
    Permalink

    [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  20. Written by A one word review of macOS Sierra | Read-Buy
    on June 23, 2016 at 2:21 am
    Permalink

    [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  21. [...] Adam Leventhal’s blog » APFS in Detail: Overview [...]

  22. Written by A one word review of macOS Sierra | Gadget News
    on June 23, 2016 at 6:48 am
    Permalink

    [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  23. [...] APFS in Detail: Overview APFS in Detail: Encryption, Snapshots, and Backup APFS in Detail: Space Efficiency and Clones APFS [...]

  24. Written by APFS in Detail | We Are 4n6
    on June 26, 2016 at 8:17 am
    Permalink

    [...] Basics [...]

  25. [...] This article was originally published on Adam Leventhal’s blog in multiple parts. [...]

  26. [...] This article was originally published on Adam Leventhal’s blog in multiple parts. [...]

  27. Written by Adam Leventhal's blog » APFS in Detail: Performance
    on June 26, 2016 at 4:18 pm
    Permalink

    [...] series of posts covers APFS, Apple’s new filesystem announced at WWDC 2016. See the first post for the table of [...]

  28. [...] Two hours or so of WWDC keynoting and Tim Cook didn’t mention a new file system once? Andrew Cunningham This article was originally published on Adam Leventhal’s blog in multiple parts. [...]

  29. Written by Tab Sweep – 26 June 2016 | Colin Charles Agenda
    on June 27, 2016 at 2:05 am
    Permalink

    [...] APFS in Detail: Overview [...]

  30. [...] This article was originally published on Adam Leventhal’s blog in multiple parts. [...]

  31. Written by rockhound
    on June 27, 2016 at 7:02 am
    Permalink

    What is the maximum filesize / volume size for APFS?

  32. [...] This article was originally published on Adam Leventhal’s blog in multiple parts. [...]

  33. [...] APFS, el nuevo sistema de archivos de Apple, en detalle — Parte 1 [ENG] Apple anunció hace unos días su nuevo sistema de archivos, APFS, sistema que en el futuro va a ir integrado en todos sus sistemas operativos. Después de su presentación, Dominic Giampaolo y Eric Tamura, dos miembros del equipo de diseño, junto con otros miembros del equipo respondieron pacientemente a nuestras preguntas al final del día. Con estos datos y con el uso de primera mano de este sistema de archivos, voy a hacer un análisis de APFS como usuario y com diseñador de sistemas de archivos. etiquetas: apfs, sistema de archivos, apple, adam leventhal, dominic giampaolo usuarios: 1   anónimos: 0   negativos: 0   compartir: sin comentarios SysDevs karma: 14 (adsbygoogle = window.adsbygoogle || []).push({}); (adsbygoogle = window.adsbygoogle || []).push({}); [...]

  34. [...] a participé à cette session et il en a livré un récit aussi complet que possible sur son blog. Nous avons puisé dans ces observations ainsi que dans les docs d’Apple pour faire le point sur [...]

  35. [...] (an earlier version of this post noted a dependency on Core Storage, but Giampaolo set me straight in this comment). I asked him about looking for inspiration in other modern file systems such as BSD’s [...]

  36. [...] Andrew Cunningham This article was originally published on Adam Leventhal’s blog in multiple parts. [...]

  37. [...] TV had where you can watch two footy games at once but that he never really used. There’s a new file system that is super frickin’ cool and could eliminate data loss forever but is not really even [...]

  38. Written by Mertie Viele
    on July 25, 2016 at 10:00 am
    Permalink

    That’s a nice post. I hope you will have a separate blog post for smart home and can you recommend a smart home builder.

  39. [...] APFS in Detail: Overview [...]

Subscribe to comments via RSS