The release of Fedora 34 grows ever closer, and with that, some fun new features! A previous feature focus talked about some changes coming to GNOME version 40. This article is going to go a little further under the hood and talk about data compression and transparent compression in btrfs. A term like that may sound scary at first, but less technical users need not be wary. This change is simple to grasp, and will help many Workstation users in several key areas.
What is transparent compression exactly?
Transparent compression is complex, but at its core it is simple to understand: it makes files take up less space. It is somewhat like a tar or ZIP file. Transparent compression will dynamically optimize your file system’s bits and bytes into a smaller, reversible format. This has many benefits that will be discussed in more depth later on, however, at its core, it makes files smaller. This may leave most computer users with a question: “I can’t just read ZIP files. You need to decompress them. Am I going to need to constantly decompress things when I access them?”. That is where the “transparent” part of this whole concept comes in.
Transparent compression makes a file smaller, but the final version is virtually indistinguishable from the original by the human viewer. If you have ever worked with Audio or Video, you have probably heard of the terms “lossless” and “lossy”. Think of transparent compression like a lossy compressed video file. You want the video to look essentially like the original. Small enough to be streamed over the web. Not so compressed that a human can distinguish it from the original file. Transparent compression works similarly. Your file system will look and behave the same way as before (no ZIP files everywhere, no major speed reductions). Everything will look, feel, and behave the same. However, in the background it is taking up much less disk space.
You can learn more about transparent compression at https://btrfs.wiki.kernel.org/index.php/Compression
Transparent compression sounds cool, but also too good to be true…
I would be lying if I said transparent compression doesn’t slow some things down. It adds extra CPU cycles to pretty much any I/O operation, and can affect performance in certain scenarios. However, Fedora is using the extremely efficient zstd:1 algorithm. Several tests show that relative to the other benefits, the downsides are negligible (as I mentioned in my explanation before). Better disk space usage is the greatest benefit. You may also receive reduction of write amplification (can increase the lifespan of SSDs), and enhanced read/write performance.
Btrfs transparent compression is extremely performant, and chances are you won’t even notice a difference when it’s there.
I’m convinced! How do I get this working?
In fresh installations of Fedora 34 and its corresponding beta, it should be enabled by default. However, it is also straightforward to enable before and after an upgrade from Fedora 33. You can even enable it in Fedora 33, if you aren’t ready to upgrade just yet.
- (Optional) Backup any important data. The process itself is completely safe, but human error isn’t.
- To truly begin you will be editing your fstab. This file tells your computer what file systems exist where, and how they should be handled. You need to be cautious here, but only a few small changes will be made so don’t be intimidated. On an installation of Fedora 33 with the default Btrfs layout the /etc/fstab file will probably look something like this:
<strong>$ $EDITOR /etc/fstab</strong>
UUID=1234 / btrfs subvol=root 0 0
UUID=1234 /boot ext4 defaults 1 2
UUID=1234 /boot/efi vfat umask=0077,shortname=winnt 0 2
UUID=1234 /home btrfs subvol=home 0 0
NOTE: While this guide builds around the standard partition layout, you may be an advanced enough user to partition things yourself. If so, you are probably also advanced enough to extrapolate the info given here onto your existing system. However, comments on this article are always open for any questions.
Disregard the /boot and /boot/efi directories as they aren’t (currently) compressed. You will be adding the argument compress=zstd:1. This tells the computer that it should transparently compress any newly written files if they benefit from it. Add this option in the fourth column, which currently only contains the subvol option for both /home and /:
UUID=1234 / btrfs subvol=root,compress=zstd:1 0 0
UUID=1234 /boot ext4 defaults 1 2
UUID=1234 /boot/efi vfat umask=0077,shortname=winnt 0 2
UUID=1234 /home btrfs subvol=home,compress=zstd:1 0 0
Once complete, simply save and exit (on the default nano editor this is CTRL-X, SHIFT-Y, then ENTER).
3. Now that fstab has been edited, tell the computer to read it again. After this, it will make all the changes required:
$ sudo mount -o remount / /home/
Once you’ve done this, you officially have transparent compression enabled for all newly written files!
Recommended: Retroactively compress old files
Chances are you already have many files on your computer. While the previous configuration will compress all newly written files, those old files will not benefit. I recommend taking this next (but optional) step to receive the full benefits of transparent compression.
- (Optional) Clean out any data you don’t need (empty trash etc.). This will speed things up. However, it’s not required.
- Time to compress your data. One simple command can do this, but its form is dependent on your system. Fedora Workstation (and any other desktop spins using the DNF package manager) should use:
$ sudo btrfs filesystem defrag -czstd -rv / /home/
Fedora Silverblue users should use:
$ sudo btrfs filesystem defrag -czstd -rv / /var/home/
Silverblue users may take note of the immutability of some parts of the file system as described here as well as this Bugzilla entry.
NOTE: You may receive several warnings that say something like “Cannot compress permission denied.”. This is because some files, on Silverblue systems especially, the user cannot easily modify. This is a tiny subset of files. They will most likely compress on their own, in time, as the system upgrades.
Compression can take anywhere from a few minutes to an hour depending on how much data you have. Luckily, since all new writes are compressed, you can continue working while this process completes. Just remember it may partially slow down your work at hand and/or the process itself depending on your hardware.
Once this command completes you are officially fully compressed!
How much file space is used, how big are my files
Due to the nature of transparent compression, utilities like du will only report exact, uncompressed, files space usage. This is not the actual space they take up on the disk. The compsize utility is the best way to see how much space your files are actually taking up on disk. An example of a compsize command is:
$ sudo compsize -x / /home/
This example provides exact information on how the two locations, / and /home/ are currently, transparently, compressed. If not installed, this utility is available in the Fedora Linux repository.
Conclusion:
Transparent compression is a small but powerful change. It should benefit everyone from developers to sysadmin, from writers to artists, from hobbyists to gamers. It is one among many of the changes in Fedora 34. These changes will allow us to take further advantage of our hardware, and of the powerful Fedora Linux operating system. I have only just touched the surface here. I encourage those of you with interest to begin at the Fedora Project Wiki and Btrfs Wiki to learn more!
