Mac OS X: Your Weekly Mystery Update

[This article, covering Apple's Mac OS X 10.0.3 update, originally appeared in MDJ last Thursday, 5/10/01. It was reprinted with MDJ's extensive, original coverage of the 10.0.2 update in the current edition of MWJ, which is available through Saturday 5/19/01 as part of a free, no-obligation trial subscription to MWJ.]

1.1 The Two 10.0.3 Updates
1.2 The Unbearable Lightness of Notes
1.3 Question the First
1.4 Question the Second
1.5 Marginal Mac OS X

How Can You Take This Seriously?

For the third time in less than seven weeks, and the second time in just eight days, Apple has released a system-wide update to its new operating system. You can now install Mac OS X 10.0.3, identified as build 4P13. This time, the list of changes is small ("ensures full visibility of file lists in directories that contain a large number of items" - remember when Apple always called those "folders"?), as is the list of changed files. It's a shame the two lists don't correspond very well.

The Two 10.0.3 Updates

The download page for Mac OS X 10.0.3 looks exactly like the one for the now-gone Mac OS X 10.0.2 update: it lists a 14.9MB file, gives the same short list of improvements promised for 10.0.2, and even says version "10.0.2" in the table at the top. However, if you use Mac OS X's built in Software Update mechanism, you're offered an update of 1.6MB. The larger stand-alone update requires only Mac OS X 10.0.1, and includes both the 10.0.2 and 10.0.3 updates in a single package. If you already have 10.0.2 installed, Software Update offers you a differential update that includes only the files changed since last week's revision.

The difference is significant: the larger update contains over 1100 files that touch every aspect of the system from the Mach kernel and Carbon framework up to the Mail application (MDJ_ 2000.05.07). The smaller update contains only 54 files, all of which affect the core of the system.

The changes in Mac OS X 10.0.3 overwhelmingly affect "System.kext", the kernel extension that provides access to system-level services.

The changes in Mac OS X 10.0.3 overwhelmingly affect "System.kext", the kernel extension that provides access to system-level services. The IOKit - Mac OS X's facility for adding third-party hardware drivers - is part of System.kext, called the "System resource Pseudoextension" in its property list. However, "System.kext" is not a typical kernel extension. You see, kernel extensions are really just directories with information and supporting files (like a property list) wrapped around the binary file that contains the actual code intended to run in the kernel. That code is not called a "KEXT," or kernel extension; it's a KMOD, or kernel module. A kernel extension may include its own kernel modules, or it may refer to external kernel modules. The system's kernel modules are actually built into the kernel, but for systemic convenience, have kernel extensions so they can be treated like other device drivers and hardware access point.

For these built-in kernel modules, like "System.kext," the property list (the file "info.plist" in the "System.kext" directory) has the key OSKernelResource with the value true. That tells the system and applications that there is no separate KMOD file with the actual code, but instead that the code is already built into the kernel. Knowing this, the list of changes in Mac OS X 10.0.3 makes more sense. The System kernel extension changed, as did all of its plug-in extensions for hardware and system services: BSD kernel, ADB driver support, CD storage driver support, DVD storage driver support, Graphics drivers, human interface device drivers, IOKit services, NDRV graphics card driver support, networking drivers, PCI driver support, generic storage driver support, I/O System Management support, kernel access for applications, and Mach access for all code.

None of these are plug-in extensions: they represent changes in the Mach kernel itself, the invisible file named "mach_kernel" at the root level of your Mac OS X startup disk. It changed in Mac OS X 10.0.3, as did four libraries in the "/usr/lib/" directory that implement much of the functionality for IOKit and kernel modules: libIOKit.A.dylib, libIOKit.dylib, libkmod.a, and libkmodc++.a. We strongly suspect that the first of these libraries has something to do with the Mac OS X versioning scheme that allows older versions of code to stay around when it might be helpful, but we can't prove it.

The Unbearable Lightness of Notes

If you thought we were up in arms over poorly-documented releases earlier this week (MDJ_ 2001.05.07), hang on - you ain't seen nothing yet.

Because of the intolerably poor state of Mac OS X documentation, only a handful of people in the world know what these files even do, much less how changing them might affect your system.

Because of the intolerably poor state of Mac OS X documentation, only a handful of people in the world know what these files even do, much less how changing them might affect your system. We're just as fond as the next media outlet - all right, we're significantly fonder than most media outlets - of pointing people to documentation on Mac OS issues to debunk conspiracy theories, wild speculation, and misguided feature requests (such as "I should be able to merge all my extensions into one file"). But we've spent hours and hours searching Apple's Mac OS X Kernel documentation, both on the Web and in Mac OS X itself, for any explanation of what some of these files do. There's no mention of "libkmod" that we can find, nor of "System.kext" or the other extensions. We finally found the details of kernel extension implementation in a small section of one of the manuals, but we had to deduce the rest. Try searching Apple's entire Web site or all the Mac OS X documentation you can find for "OSKernelResource" and you'll come up empty-handed.

The esoteric knowledge of how IOKit really works resides today with the Apple engineers who create it, with the handful of top- tier developers who have attended sponsored "kitchens" explaining how to write device drivers first-hand, and with a slightly-larger set of seeded developers that get better access to Apple internal documentation than the developer community at large. If you're not in one of those groups, you have no chance of figuring out what's changed in Mac OS X 10.0.3.

We know from Apple's WWDC sessions on IOKit, and from preliminary documentation, that the changed kernel extensions are what we'd call driver families. In Mac OS 9, a driver that controls a SCSI device calls upon a piece of the Mac OS called the SCSI Manager to do most of heavy lifting. It's the SCSI Manager that actually talks to built-in SCSI hardware to send commands over the bus, because third-party driver authors would have no real idea how to control the hardware itself. Instead of triggering registers on a chip to send SCSI commands, drivers simply ask the SCSI Manager to send a given command to a specified target. There is also an ATA Manager, a USB Manager, a FireWire Manager, a Serial Manager, an ADB Manager (mostly a stub on today's machines that translates calls to appropriate USB commands), a Sound Manager, and much more.

In Mac OS X, these functions are implemented differently, but the concepts are similar. The IOKit provides family drivers for the big classes of peripherals. Third-party drivers that control specific hardware devices link to those family drivers for the heavy lifting. The difference is important only to programmers - pretending the system support is local instead of consciously calling an external function - but the end result is the same. The SCSI driver family talks to the SCSI hardware; plug-in drivers rely on the driver family to provide all that functionality. If you want to see it in action, type the command "sudo kmodstat" in a Terminal window. You'll get a numbered list of all loaded kernel modules along with their reverse-DNS-style names (like "com.apple.kernel"). At the end of some listings, you'll see numbers in angle brackets. Those numbers, if present, refer to other kernel modules that the entry links to. On our system, the Apple USB OHCI driver (for USB implementations based on standard Open Hardware Control Interface chips) links to the IOUSBFamily driver, the IOKit module, the "libkern" module on which IOKit is based, and the Mach module for accessing Mach services.

Even having gotten this far, it begs two serious questions about Mac OS X 10.0.3.

Question the First:

How can changes to the Mach kernel possibly "ensure full visibility of file lists in directories that contain a large number of items?"

When we saw that sole release note, we expected changes to the Finder, to the Mac OS X directory navigation dialog boxes (the new versions of Navigation Services), and maybe to Carbon's DataBrowser control that the Finder uses for multi-column view. We believed that any problem with not reading all files in a directory reside with the code that normally enumerated folder contents and displayed them to you. For example, Finder 9.1 has difficulty with folders that contain thousands of files, and the old Standard File dialog boxes that preceded Navigation Services warn you that they can't display more than about 1500 files in a folder.

These limits aren't because the File Manager can't return all the files in a folder. The File Manager only gets information about files as applications or other code ask for it - if Standard File asks for one file at a time, that's all Standard File gets. The File Manager never has to keep information about every file in a folder in memory at the same time. The Finder, Standard File, and Navigation Services may have to do that, because they have to show you all the entries at once. Therefore, we expected to see revision to some part of Mac OS X that shows you the content of directories, thinking there was some programming error that didn't handle large cases correctly.

Instead, we got an update that leaves all those components alone and changes only the low-level driver model inside the Mac OS X kernel itself. We thought over this for hours, researched possibilities, even thought ourselves into headaches trying to process this, and we keep coming back to the same conclusion: if this update really fixes problems with improperly reading directories, the problem must have been inside the kernel.

That's staggering. It presents the appearance of some kind of kernel-level or driver-level problem preventing disk drivers from doing their tasks.

That's staggering. It presents the appearance of some kind of kernel-level or driver-level problem preventing disk drivers from doing their tasks. But drivers do not interpret file systems - they just return data from the disk. The kernel's file system modules for HFS Plus and UFS interpret the raw data from disk into files and directory entries. It's possible those modules are what changed in the kernel, but then we don't understand why all the kernel extensions changed as well - they don't have actual code in them, nor do they link to addresses in the kernel, so an update that didn't change those kernel modules shouldn't require changing their kernel extensions. Yet fifteen kernel extensions did change - the only fifteen with the OSKernelResource property.

We're hoping there's some kind of dependency that forces Apple to update those specific kernel extensions whenever the kernel changes, for that would mean it's not a driver-level problem. Even so, a kernel-level directory reading problem implies that Mac OS X could have been unreliable until now - omitting information on some files in large directories. We're sorry to be so negative, but operating systems that people rely on don't do that. If it's discovered that they do, the company in charge immediately discloses exactly what was wrong, in what situations it may arise, and exhorts people to upgrade as soon as possible. It's called "respect for the customers." Mac OS X's handlers show none of that. The thought that every part of Mac OS X could have been misbehaving on large directories in every program is truly disheartening - and without a more open policy of admitting flaws and their impact, how can Apple Computer expect anyone to take this operating system seriously?

Question the Second

If the kernel is all that truly changed - and by all appearances, that is the case - then what changed in the kernel? If our file system theory is correct, the kernel modules might have required recompilation, and that in turn might have forced an update to the corresponding kernel extensions. But with such a poor correspondence between "missing files in large directories" and "kernel update," we can't help but wonder what actually changed in there.

The kernel contains so many parts of the OS - the BSD layer, the drivers, the driver families, the IOKit, the file systems, the networking stacks - that a kernel change could mean any of those parts got updates.

The kernel contains so many parts of the OS - the BSD layer, the drivers, the driver families, the IOKit, the file systems, the networking stacks - that a kernel change could mean any of those parts got updates. Given the vagueness of what we do know about Mac OS X 10.0.3, it's more than an idle question. Sure, Darwin developers may see the changes soon, or maybe already, but as programmers we know often say, "source code is not documentation." If someone handed you a new version of an 800-page book, would you want to compare it to the original book to see what had changed, or would you want change notes? Full access is not the same thing as disclosing changes.

Marginal Mac OS X

For all the talk about how Mac OS X brings the "industrial strength" power of UNIX to Macintosh customers, Apple Computer's release policy to date has been both counterproductive and insulting. One of the characteristics of the UNIX community is that developers take their customers seriously. That derives, in part, from an operating system that may require system administrators to recompile a program just to implement a minor change. We don't miss that part of UNIX, but Apple seems to have thrown out the baby with the bath water, thinking that if it doesn't have to tell you how to recompile, it doesn't have to tell you what changes it made because there's no chance you made your own changes you'd have to integrate with the new code. The thought is incorrect - people need to know what's going on.

If Mac OS X 10.0.2 and earlier are really unreliable on large folders, customers need to be told exactly what the problem is. How many files trigger the problem? Does it happen on all disks, or just on HFS Plus disks, or UFS disks? Do the "missing" files appear in some situations but not others? If so, when? This is not like some extra pixel in your menu bar or a GIF file that has bad colors. This is data storage, and it could affect everything from backing up to restoring a hard drive.

If this pattern continues, the entire release notes for Mac OS X 10.0.4 will probably be "makes stuff gooder."

Apple's actions in continuing to withhold even the most basic information about Mac OS X revisions sends a strong message: no matter what the marketing material says about "industrial strength," the operating system is really a plaything. By keeping you in the dark about what's right and wrong with the OS, Apple implicitly assumes you're not doing anything serious with Mac OS X, you don't rely on it, and you don't expect it to stand up in situations critical to your work. At this rate, no one will care when it gets bundled on new machines, because no one who depends on a computer for a living wants to depend on a mystery operating system. If this pattern continues, the entire release notes for Mac OS X 10.0.4 will probably be "makes stuff gooder."

For a company that claims to embrace open-source ideals for the kernel of Mac OS X, and that talks about transitioning customers to Mac OS X, Apple certainly doesn't seem to be taking itself very seriously. Install Mac OS X 10.0.3 if you're running Mac OS X, but if you're not running Mac OS X, you have less reason than ever to switch now.

[Can't remember the last time you saw substantive journalism covering Apple or the Mac OS world? Neither can we, and we've been doing this since 1996, so we should have noticed. We'd love it if you checked us out, with a free, no obligation three-week trial of our daily or weekly editions, or a two-month trial of our monthly. Sign up now and get the scoop on WWDC, written by programmers who sweat the details.]