Our Little Panther FileVault Horror Story

Since Apple claims that the Mac OS X 10.3.1 and 10.3.2 updates fix reported problems with Panther's FileVault, we decided to give it a try. After all we travel a lot and are concerned that our 17" Powerbook G4 just might fall into evil hands during one of our trips. However, we did so with much trepidation after reading about the horrific problems caused by enabling FileVault in the initial Panther release. Apple has had two opportunities to fix the most egregious problems--what could be so bad now? We even created another user Home folder to practice on with perfect results.

FileVault is Apple's new-to-Panther security system which encrypts all files in one's Home folder and decrypts them on the fly while using them. We use a third-party encryption application (SubRosa) on all of our sensitive files but it's not as efficient or broadly effective as we would like. FileVault carries the promise of seamless use of encrypted files. Don't believe it just yet, at least not 100%.

After updating to Mac OS X 10.3.2, we went to the Security preference pane and clicked on the button to turn on FileVault. We had to set a master password, essentially a second password with our login password being the first. After confirming that we wanted to enable FileVault, the dialog box estimated that it would take some 250 minutes, i.e. over four hours, to complete. In contrast, our practice Home folder took only 10-15 minutes to enable Filevault but it was basically empty except for our Keychain file, Safari preferences and Mail database. We decided to wait and allow FileVault to work overnight on our real Home folder. In the morning, our PowerBook displayed only the default background and was unresponsive. We restarted by holding down the power button. When it finished rebooting we discovered that for some reason FileVault had not been enabled. But the PowerBook worked fine so we resolved to try it the next night. Bad decision.

Our second attempt began the same as the first. The dialog box stated once again that it would take some 250 minutes to complete the encryption process. In the morning, our PowerBook was at the login window. After logging in we discovered that enabling FileVault had caused the apparent loss of all of our preferences, Safari bookmarks and Keychain entries. Apple's Mail application was also unusable. This is exactly the problem that the 10.3.1update was supposed to have fixed, no? But we hadn't even chosen the option for FileVault to reclaim unused space, the option that caused the problem in the initial Panther release. Could Panther have decided on its own to go ahead and do it anyway sometime during the night when no one responded to a dialog box?

Well, we could reload our extensive Safari bookmarks from .Mac with iSync and reinstall our Keychain file and our Apple Mail database from backup copies right? Nope-maybe-maybe. In trying to get .Mac to work to recover our Safari preferences, none of the System Preference re-entries could be saved--our PowerBook's short-term memory was essentially brain dead. In fact, it couldn't remember any preference or Keychain-related entry. This appeared to be a total disaster requiring wiping the drive and re-installing everything. For better or worse, we first decided to turn FileVault off to see if that cleared things up.

When we turned FileVault off we were presented with a dialog box stating that it would take another 3 hours to unencrypt our Home folder. Fortunately, it only took about 90 minutes. Unfortunately, when FileVault finished we still had the same unusable Home folder. We discovered that the problem was that our Home folder's permissions, and all those folders and many of the files inside, had been reset such that we could not access them without entering our administrator's password. In retrospect, this was probably the problem with FileVault on as well. How could this have happened?

Disk Utility's "Repair Permissions" doesn work on one's Home folder for good reason. We tried it anyway and, as expected, it did nothing for our problem. There may be a third-party application that does work to set all the Home folder's permissions (There is, we subsequently found out thanks to Andy Fragen. Check out CHOP.) or there may be a specific "chmod" or "chown" Unix command for the Terminal application but we couldn't find one and didn't want to experiment. However, we did find a nice OSXFAQ.com tutorial on Unix permissions.

We manually reset a number of folder/file permissions using Get Info to make the PowerBook operational. In doing so we discovered that the button in Get Info that is supposed to set all the contents of a folder to to the same permissions as that folder does not work, at least not completely. Using Get Info we manually reset the permissions of most of the folders and files in our Home folder. Doing so took the lion's share of four hours during an airline flight home. We also replaced our Keychain and Mail files from a backup copy to get them to work properly. While we still occasionally find a file (mostly a preference file) whose permissions have to be reset, our Home folder and its applications are usable again.

We recommend against using Panther's FileVault on a large Home folder for the time being, largely because we have no idea why our problem occurred. We are sure that there are others using FileVault without problems but we plan to stick with our third-party file encryption application until we can figure out what went wrong. For those inclined to enable FileVault, we recommend following the much more extensive procedure outlined in this MacDevCenter article. We seem to have no problems after adding files to our practice Home folder with FileVault enabled. Just be sure to backup all important files immediately before enabling FileVault.