Fedora Silverblue provides a useful workstation build on an immutable operating system. In “What is Silverblue?“, you learned about the benefits that an immutable OS provides. But what pieces go into making it? This article examines some of the technology that powers Silverblue.
Fedora Workstation users may find the idea of an immutable OS to be the most brain-melting part of Silverblue. What does that mean? Find some answers by taking a look at the filesystem.
At first glance, the layout looks pretty much the same as a regular Fedora file system. It has some differences, like making /home a symbolic link to /var/home. And you can get more answers by looking at how libostree works. libostree treats the whole tree like it’s an object, checks it into a code repository, and checks out a copy for your machine to use.
The libostree project supplies the goods for managing Silverblue’s file system. It is an upgrade system that the user can control using rpm-ostree commands.
libostree knows nothing about packages—an upgrade means replacing one complete file system with another complete file system. libostree treats the file system tree as one atomic object (an unbreakable unit). In fact, the forerunner to Silverblue was named Project Atomic.
The libostree project provides a library and set of tools. It’s an upgrade system that carries out these tasks.
- Pull in a new file system
- Store the new file system
- Deploy the new file system
Pull in a new file system
Pulling in a new file system means copying an object (the entire file system) from a remote source to its own store. If you’ve worked with virtual machine image files, you already understand the concept of a file system object that you can copy.
Store the new file system
The libostree store has some source code control qualities—it stores many file system objects, and checks one out to be used as the root file system. libostree’s store has two parts:
- a repository database at /sysroot/ostree/repo/
- file systems in /sysroot/ostree/deploy/fedora/deploy/
libostree keeps track of what’s been checked in using commit IDs. Each commit ID can be found in a directory name, nested deep inside /sysroot .A libostree commit ID is a long checksum, and looks similar to a git commit ID.
$ ls -d /sysroot/ostree/deploy/fedora/deploy/*/ /sysroot/ostree/deploy/fedora/deploy/c4bf7a6339e6be97d0ca48a117a1a35c9c5e3256ae2db9e706b0147c5845fac4.0/
rpm-ostree status gives a little more information about that commit ID. The output is a little confusing; it can take a while to see this file system is Fedora 31.
$ rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: ● ostree://fedora:fedora/31/x86_64/silverblue Version: 31.1.9 (2019-10-23T21:44:48Z) Commit: c4bf7a6339e6be97d0ca48a117a1a35c9c5e3256ae2db9e706b0147c5845fac4 GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
Deploy the new filesystem
libostree deploys a new file system by checking out the new object from its store. libostree doesn’t check out a file system by copying all the files—it uses hard links instead. If you look inside the commit ID directory, you see something that looks suspiciously like the root directory. That’s because it is the root directory. You can see these two directories are pointing to the same place by checking their inodes.
$ ls -di1 / /sysroot/ostree/deploy/fedora/deploy/*/ 260102 / 260102 /sysroot/ostree/deploy/fedora/deploy/c4bf7a6339e6be97d0ca48a117a1a35c9c5e3256ae2db9e706b0147c5845fac4.0/
This is a fresh install, so there’s only one commit ID. After a system update, there will be two. If more copies of the file system are checked into libostree’s repo, more commit IDs appear here.
Putting the pieces together, the update process looks like this:
- libostree checks out a copy of the file system object from the repository
- DNF installs packages into the copy
- libostree checks in the copy as a new object
- libostree checks out the copy to become the new file system
- You reboot to pick up the new system files
In addition to more safety, there is more flexibility. You can do new things with libostree’s repo, like store a few different file systems and check out whichever one you feel like using.
Silverblue’s root file system
Fedora keeps its system files in all the usual Linux places, such as /boot for boot files, /etc for configuration files, and /home for user home directories. The root directory in Silverblue looks much like the root directory in traditional Fedora, but there are some differences.
- The filesystem has been checked out by libostree
- Some directories are now symbolic links to new locations. For example, /home is a symbolic link to /var/home
- /usr is a read-only directory
- There’s a new directory named /sysroot. This is libostree’s new home
Juggling file systems
You can store many file systems and switch between them. This is called rebasing, and it’s similar to git rebasing. In fact, upgrading Silverblue to the next Fedora version is not a big package install—it’s a pull from a remote repository and a rebase.
You could store three copies with three different desktops: one KDE, one GNOME, and one XFCE. Or three different OS versions: how about keeping the current version, the nightly build, and an old classic? Switching between them is a matter of rebasing to the appropriate file system object.
Rebasing is also how you upgrade from one Fedora release to the next. See “How to rebase to Fedora 32 on Silverblue” for more information.
The Flatpak project provides a way of installing applications like LibreOffice. Applications are pulled from remote repositories like Flathub. It’s a kind of package manager, although you won’t find the word package in the docs. Traditional Fedora variants like Fedora Workstation can also use Flatpak, but the sandboxed nature of flatpaks make it particularly good for Silverblue. This way you do not have to do the entire ostree update process every time you wish to install an application.
Flatpak is well-suited to desktop applications, but also works for command line applications. You can install the vim editor with the command flatpak install flathub org.vim.Vim and run it with flatpak run org.vim.Vim.
The toolbox project provides a traditional operating system inside a container. The idea is that you can mess with the mutable OS inside your toolbox (the Fedora container) as much as you like, and leave the immutable OS outside your toolbox untouched. You pack as many toolboxes as you want on your system, so you can keep work separated. Behind the scenes, the executable /usr/bin/toolbox is a shell script that uses podman.
A fresh install does not include a default toolbox. The toolbox create command checks the OS version (by reading /usr/lib/os-release), looks for a matching version at the Fedora container registry, and downloads the container.
$ toolbox create Image required to create toolbox container. Download registry.fedoraproject.org/f31/fedora-toolbox:31 (500MB)? [y/N]: y Created container: fedora-toolbox-31 Enter with: toolbox enter
Hundreds of packages are installed inside the toolbox. The dnf command and the usual Fedora repos are set up, ready to install more. The ostree and rpm-ostree commands are not included – no immutable OS here.
Each user’s home directory is mounted on their toolbox, for storing content files outside the container.
Put the pieces together
Spend some time exploring Fedora Silverblue and it will become clear how these components fit together. Like other Fedora variants, all these of tools come from open source projects. You can get as up close and personal as you want, from reading their docs to contributing code. Or you can contribute to Silverblue itself.
Join the Fedora Silverblue conversations on discussion.fedoraproject.org or in #silverblue on Freenode IRC.