Mac OS X Server, Little Snitch & Carbon Copy Cloner

May 03, 2015

Carbon Copy Cloner and Little Snitch are awesome tools for anyone running Mac OS X Server. But, if you aren't careful, you can lose all remote (including HTTP/FTP/ARD/SSH) access when booting from a clone (created by Carbon Copy Cloner) of a startup volume (that runs Little Snitch).

The Situation

We run two Mac OS X Servers in a colocation facility. We configure and work on those using Apple Remote Desktop (ARD) and SSH. Much of our peace of mind when it comes to security and backups comes from the use of two great pieces of software:

Little Snitch

A fantastic utility with many applications that we use as primarily as a software firewall.

Carbon Copy Cloner

A fantastic duolication utility that we use to create clones of the startup disk on a regular basis. It is scheduled so that we always have seven daily backups (one for each day) and four weekly backups. This allows us to restore files from these clones, boot from them to fully restore to a cloned backup, or (in the event of damage to the boot volume) clone a cloned backup back to the boot volume. These clones are saved to partitions on an external hard drive. If faced with a server failure we can simply get a new server in place, connect the external drive, and then boot or restore from a clone. This, just part of a bigger backup/recovery strategy, was inspired by a great post on backing up Mac servers from the people at MacMiniColo.net.

So Little Snitch helps protect the machine and Carbon Copy Cloner makes sure we always have lots of backups to work with.

The Problem

Recently we had a serious problem on one of the servers and it looked like the best solution was to restore from a clone. The plan was to start up from the selected clone, confirm that it was good and working as expected, then clone that back to the primary boot volume.

Using Apple Remote Desktop (ARD) we accessed System Preferences, changed the Startup Disk to the clone, hit restart, and waited for the machine to come back online and be available in ARD.

That didn't happen. We could not get ARD, SSH or any other access to the server. This required a call to the colocation facility where the tech on duty sent back this image:

This is a security feature in Little Snitch. When the machine booted from the clone Little Snitch was smart enough to see that it was looking at a different rules file and it went into a kind of safe mode that blocked all external connections. It makes perfect sense – Little Snitch was making every possible effort to secure the machine.

The short-term fix was simple  – we simply had the onsite tech click the "Use Current Version" box). This authorized the cloned rules file which allowed for the remote access and we needed.

Unfortunately it also derailed our backup/recovery plan, which needed to work without physical access to the machine. The Little Snitch feature meant that we would have to be hands-on with the server any time we wanted to be able to boot from a clone.

The Solution

This started a discussion with the Mike from Bombich (Carbon Copy Cloner) and Manfred from Objective Development (Little Snitch), and both were truly outstanding. The two of them also developed a very good workaround.

If you know that you are going to reboot from a clone the solution is to disable this by preventing Little Snitch from starting up (and blocking all remote access) automatically. Mike from Bombich confirmed this can be accomplished by deleting these two files on the volume you will be booting from the (the clone).

/Library/Extensions/LittleSnitch.kext
/Library/Little Snitch

You can see these files highlighted in their respective folders below:

Again - make sure that you remove them not from the current startup volume, but the new startup volume – the one that you just selected in System Preferences > Startup Disk.

Why not simply change the Little Snitch preferences so that the network filter does not automatically restart? Because you would be changing the settings on the current startup volume, not the newly selected startup volume, which would still cause Little Snitch to block access.

This is now part of our recovery bible – never, ever, change the startup disk without also deleting those two files.

Belt & Suspenders...

You can also configure Carbon Copy Cloner to exclude those two files from the clones. This means that no cloned volume should ever launch Little Snitch automatically.

To do that open Carbon Copy Cloner and...

• In the “Tasks” panel on the left select the scheduled backup
• In the first column to the right under “Clone:” choose “Custom”
• Click the magnifying glass icon next to “Custom”
• Add entries for the two files mentioned above.

It is safe to safe you should still always check the new startup disk for the existence of those two files before rebooting, but excluding them from the clone gives you a safety cushion should you forget.

The files would be excluded from the clone like this:

A video of the process appears below this post.

It is very important to note that this will prevent those files from being cloned in the future but it will NOT remove them from existing clones, or prevent them from surfing future clones. Unless you have selected "SafetyNet Off" those files are likely to remain in place. If you do specify SafetyNet Off then they will be deleted on the next scheduled clone. You could also delete them from each of the existing clones manually to be sure.

After Booting From The Clone

One you boot from the clone you will have access (good) but Little Snitch will not be running (bad, but necessary). At this point you simply reinstall Little Snitch, or copy over the  two now missing files from a clone that still has them.

Thank you again to Mike from Bombich (Carbon Copy Cloner) and Manfred from Objective Development (Little Snitch) for their great products and incredible support.

Related Content