System Rollback Using Btrfs Snapshots - TUXEDO Computers

The Open Beta of the new Debian-based TUXEDO OS is now available in its second version and fixes some of the issues you have reported. In this article, we explain how system recovery using Btrfs snapshots works if something goes wrong during an upgrade or the system no longer boots for other reasons.

Subvolumes and snapshots

The new TUXEDO OS uses Btrfs and Snapper by default to automatically create snapshots before and after updates or package installations performed via the package management system. Based on these snapshots, you can restore the system to an earlier, known working state at any time. It does not matter whether the system is encrypted or not.

Snapshots are based on the subvolumes created during installation. Subvolumes are logical subdivisions within a Btrfs partition. From the outside, they behave like normal folders, but each has its own independent file hierarchy. They dynamically share the entire storage space of the main drive and do not need to be assigned a fixed size in advance. They also provide the technical basis for snapshots, which can freeze the state of data in a fraction of a second without requiring twice the storage space.

Btrfs Assistant

The Btrfs Assistant tool installed by default provides a graphical interface for managing Btrfs subvolumes and snapshots. It offers a user interface for tasks that would otherwise have to be performed in the terminal.

A first look at the Subvolumes tab immediately after installation shows the existing subvolumes and an initial snapshot that was created during the first boot.

Snapshots are read-only

The image shows that the snapshot is marked as Read-only. Initially, this applies to all snapshots and serves to preserve data integrity. If you were able to modify a snapshot afterwards, the original state could no longer be reconstructed reliably. The snapshot would therefore lose its purpose as a restore point, for example after a failed system update. If you nevertheless want to work specifically within a snapshot, you can create a writable copy of it at any time.

Factory Image

For the next steps towards a rollback, switch to the Snapper tab in Btrfs Assistant. Immediately after installation, you will initially see snapshot number 1, which is created automatically during the first boot and is referred to here as the Factory Image.

Returning to this snapshot at a later point restores the system to the state it was in immediately after the initial installation of the operating system.

You can also display snapshots in the terminal. Immediately after installation, the command

sudo snapper list

also shows the automatically created snapshot.

If you now perform an upgrade or install a package via the package management system, two snapshots are created automatically: one before and one after the operation. APT already indicates this at the end of the process.

After the operation, click Refresh in Btrfs Assistant and you will see the same result. You can also see the reason why snapshots 2 and 3 were created: the package installation that was performed. In the case of an upgrade, the Description field would instead contain DU, short for Dist Upgrade.

Snapshots 2 and 3 ensure that the state before and after the package installation can be restored.

When things go wrong

A simple example shows how quickly a snapshot can help when something goes wrong: If you remove a central system package such as the kernel, the system will no longer start after the next reboot.

If you check the available snapshots before rebooting, you will see that snapshots 4 and 5 (pre and post) have been added.

Nothing works anymore

The next time you restart the system, you will be greeted by a Kernel Panic. Instead of panicking yourself, simply restart the system again and select Tuxedo OS snapshots from the GRUB menu. This displays all available snapshots.

In this example, the highlighted snapshot takes you back to the point before the kernel was removed and therefore restores a bootable system.

As the final step, click the entry, select the kernel itself, and boot the system.

After restarting via the GRUB menu entry, TUXEDO OS initially boots normally again. However, the system has not yet been permanently restored; it is still running in read-only snapshot mode. This gives you time to verify that everything really works as it did before the problem occurred, without making any permanent changes. Only the final rollback turns this snapshot back into the active, normally writable system.

The rollback

To make the rollback as simple as possible, we have created the tuxedo-rollback script. It takes you from the booted snapshot to a writable system without any further intervention. In our test case, this returns the system directly to the state before the kernel was removed:

sudo tuxedo-rollback

At the end of the process, you are informed that the rollback was successful and prompted to reboot immediately. Once the reboot is complete, the system is operational again. Only a few minutes have passed since the reboot and subsequent Kernel Panic.

Where do the new snapshots come from?

If you look at the result in Btrfs Assistant after the rollback, you will see two new snapshots, numbered 6 and 7. You are now running snapshot 7, even though you originally booted into snapshot 4. Btrfs Assistant also indicates that snapshot 7 is a writable copy of snapshot 4.

Snapshot 6 is a backup of the state from which you initiated the rollback – in other words, the broken system without a kernel. Snapper creates this snapshot automatically before making any changes so that this intermediate state is not lost and could itself be restored if necessary.

Snapshot 7 is the actual new, active system state. The reason you are now in snapshot 7 rather than snapshot 4 is that snapshots are read-only by design and are therefore not intended to serve permanently as a regular system. During a rollback, Snapper does not return directly to snapshot 4. Instead, it creates a new, writable copy containing the same data. This copy becomes the new active system and, accordingly, receives a new sequential number. The original snapshot 4 remains unchanged as a reference point.

Deleting old snapshots

Since you will now remain in snapshot 7 until you deliberately boot another snapshot, you can safely remove older intermediate states such as snapshots 4 and 6. In Btrfs Assistant, select the two snapshots, which will then be highlighted in blue, and click Delete in the upper-right corner. Alternatively, you can use the terminal:

sudo snapper delete 4 6

Snapshot numbers are sequential, unique identifiers. When a snapshot is deleted, the remaining snapshots are not renumbered. Gaps in the numbering are therefore normal and do not indicate a problem.

Be sure to keep the factory image labeled “1.” Immediately after installation, it takes up very little additional storage space because it shares all data with the active system. Over time, however, the amount of storage space it actually occupies will increase as the system deviates from its original state due to updates. As a safety net in case you ever want to revert to the original installation, it’s still a good idea to keep it permanently. Users with TUXEDO hardware can, as an alternative, restore the system to its factory settings at any time via WebFAI.

Translated with DeepL.com (free version)

Previous Articles in This Series

• A new foundation for TUXEDO OS: Switching to Debian
• Security on TUXEDO OS: How we secure Debian Testing
• What the future holds for TUXEDO OS, Ubuntu and Kubuntu
• System updates without worries thanks to Btrfs snapshots and Snapper
• What does TUXEDO OS on Debian Testing offer for developers?
• TUXEDO OS with Debian Base: More Stability and Planning Security for Businesses
• Carefully Selected Productivity Tools in TUXEDO OS
• The Kernel Concept for TUXEDO OS Based on Debian Testing
• The time has come: The open beta for TUXEDO OS is just around the corner