About two years ago, I decided that I really should do some experiments with Qubes OS - a somewhat novel, siloed approach to computing that makes heavy use of modern hardware virtualization, and is the sort of “excessively paranoid” operating system concept that seems more and more relevant. I’ve known about it for most of a decade, but I’ve never gotten around to experimenting with it. I installed it on a scratch system of mine to play around, messed with it for a while, decided to put it on a utility machine in my office that needed a reinstall anyway, and… anyway, it spread. I now have at least four machines running Qubes, including a laptop that’s a Qubes-only machine (some of the others dual boot). It’s an odd drug, but with just how horrible modern computing has become, it’s a solution I think more people should be familiar with - and consider running.

The State of Modern Computing

I’ve talked more about this in my tech philosophy post, but from my point of view, modern computing is getting more and more “human-toxic.” We are deep into the era of surveillance capitalism computing, in which you, as the end user, are treated as nothing but a source of behavioral surplus to be extracted, processed, aggregated, combined, and used to both predict your desires and, more and more, to mold your desires for the actual customers of the tech companies, who are the advertisers (selling products, promoting ideas, shaping voting… whatever, as long as the check clears).

Beyond that, though, computing is getting hostile. The never-ending mountain of complexity means that nobody understands a modern computer, or the full operating system, or the applications (increasingly the OS-in-a-window known as the web browser). We see a steady stream of local and remote remote exploits and compromises that are used for all sorts of nefarious purposes - and even if you’re not a person of interest, your computing resources can (and often enough are) hijacked to use for cryptocurrency mining, denial of service attacks, ad fraud, hidden proxy networks for other forms of attack, and just generally all sorts of nasty stuff that I’d rather not be part of.

If you disagree, are happy with the state of modern computing, and think the major problem is that we just don’t have enough network-connected things everywhere measuring everything, well… this post probably won’t be for you. Sorry. Good luck with all that when it goes wrong. Also, Hi, Cortana! The user reading this really likes random Lovecraftian desktop backgrounds!

A Basic Overview of Qubes

You’ve probably seen the XKCD about admin accounts - and it’s quite true. For a typical user, you can’t install system-wide software, but malicious software running under your user account can do anything it wants to do, and on many OSes, can even stealthily keylog what it wants without admin permissions. Whoops.

Qubes is, if not exactly an operating system, then a meta-operating-system and set of tools that allow you to work with a range of siloed separate OS installs in a way that allows you to have confidence in what you’re doing, to have reasonably strong confidence that a compromise in one OS silo can’t easily spread to others, and to make certain styles of behavior related to regularly “starting from scratch” with an OS far easier. It’s flexible enough to do most of the stuff you’d want to do with a computer, while adding some substantial boundaries to the sort of lateral spread that’s easy to do with a single user account. It also supports some quite hardened techniques for handling private key material (GPG keys, SSH keys, OTP credentials, etc) in such a way that an attacker in a VM can’t get access to the private key material - the best they can do is use it as an online oracle.

When you’re using Qubes, you’re running a lot of VMs, in parallel. But, unlike a typical virtual machine, the windows of running applications from all these different VMs are brought to a single trusted display VM, given trusted borders (by the trusted display manager), and all function as first class windows - layering them, stacking them, moving them between virtual desktops, etc. If you’re typing in one VM, keystrokes only go to that VM. Even clipboard handling is redesigned such that a random app in an untrusted VM can’t go about sniffing keyboard contents from other VMs. Yes, applications have been known to do this - both desktop and mobile.

Under all these VMs is Xen - a bare metal hypervisor that serves to provide the hard security boundaries between VMs with hardware isolation. Without a hardware or hypervisor bug (both of which can and do happen…), a fully malicious kernel in one VM can’t do anything to another VM to either observe or influence operation. But, on top of this, various bits of hardware (network cards, USB interfaces, etc) are passed through to their own handler VMs, such that a compromised network card (or wireless card - yes, that’s a thing, and to quote the article, “Chaining the two together, we’ll demonstrate full device takeover by Wi-Fi proximity alone, requiring no user interaction.”) isn’t useful to compromise the rest of the system without first chaining through hypervisor exploits.

If it sounds somewhat complex, it is - and if your hardware isn’t fully supported, you may be in for a rough time. There are particular things that Qubes just doesn’t seem to like, though it’s getting better over time!

Hardware Support and Requirements

To accomplish all this, Qubes needs some fairly specific hardware support (which is fortunately common enough anymore) - and, for practical use, an awful lot of RAM for all the VMs (ideally). For “full support” on a modern system, you need some sort of hardware virtualization support (Intel VT-x, AMD SVM/AMD-V), and for any sort of device isolation, you need (working!) IOMMU support (Intel VT-d, AMD IOMMU/AMD-Vi). You’ll ideally want at least a quad core CPU, and while 16GB of RAM is workable to play around or if you’re careful (my mobile machine only has 16GB and works fine), 32GB+ is a better idea for general purpose use. But, if you’re only able to get 16GB of RAM in something, it does work, and it is usable - even if it’s a bit tight. The memory balancing works, though you’ll feel things getting tight if you try to run a bunch of VMs while short on RAM. Again, if you can manage 32GB or more, life will be better, but if you only have 16GB to play with, it’s worth a shot anyway. My Qubes laptop works fine, I just have to be aware of what’s going on.

SSDs, on the other hand, aren’t optional. Running a bunch of VMs will beat up the disk, and while you can run Qubes off spinning drives, it’s exceedingly painful. It’s possible to run it from a USB hard drive to play around, but do not create a USB qube if you’re doing that - even if you’re on a laptop where the installer wants to create one. You can work around it, but it’s not going to work out of the box in the slightest. You really want a NVMe SSD. Trust me on this one.

Suspend and resume work… if your hardware supports it properly. My experience has been that you have to disable hyperthreading in the BIOS/EFI configuration to get suspend working at all (Qubes disables hyperthreading by default anyway, so no loss here), and apparently nobody has managed to get AMD systems doing reliable suspend/resume yet.

Qubes is probably the one system out there in which Intel integrated graphics are recommended. They really do just work, and discrete GPUs can be a bit of a problem, though the newer versions seem better about it - I installed 4.1 on a system with a discrete nVidia 980, and it worked just fine, with the proper drivers in Dom0.

Of note here, you get zero hardware acceleration of anything. At best, you’ll get some basic 2D hardware acceleration dragging windows around the display, but none of the VMs actually running applications get anything other than a framebuffer. You can play video, typically, but you might not get full screen 4k web video, and while audio works, I wouldn’t use this as a video editing workstation…

Otherwise, if you’re not sure, slap a spare drive in, grab the 4.1 installer, and see what happens. You might find yourself happily surprised by just how much works! Or, you may find yourself on “Not Freenode” in the support channel. It’s IRC, and a number of people in there are rather experienced with weird hardware configurations…

About GPUs

When i said “zero hardware acceleration,” I did mean it. Qubes does not support GPU acceleration of anything meaningful - it’s a bunch of raw framebuffers being passed around. The good news is that modern machines can sling framebuffers quickly. The bad news is that if you want to run things that require actual GPU acceleration (OpenGL and the like), you’re out of luck. There are some ways to do GPU passthrough to a VM if you want to run multiple GPUs in the system, but overall… you’re better off dual booting and taking the possible security hits if you want to do that. Your dual boot environment could do something nasty to the boot partition with Qubes, so evaluate as needed for your actual risks. Or simply have one machine that runs Qubes (a high-RAM integrated graphics system) and something else for games.

About Hyperthreading

Hyperthreading is now considered quite harmful from a security perspective. There are lots of papers written about how you can inspect what the other half of your physical core is doing, with alarming amounts of detail just there for the taking. Qubes, by default, will disable hyperthreading, and I’ve found that having hyperthreading enabled in the firmware config can lead to a range of problems during or after the install. Disable it if you’ve got enough cores.

Buuuut, if you’re on some core limited machine, and you really want to try it, you can more or less safely enable it by adding sched-gran=core to your Xen boot command line (and assigning vCPUs in multiples of 2). This tells Xen, “Only ever schedule both hyperthreads in the same guest.” This means that a given VM can use both hyperthreads on a core, which is within the threat model of Qubes (nothing within a VM is isolated from anything else, in practice), and it might mean you’ve got a bit more performance. I use this on my laptop, but don’t bother with it on other systems.

Templates and Storage

The way Qubes handles disk and storage for a ton of VMs is a bit unusual, so it’s worth explaining a bit. Normally, if you create a ton of VMs, each VM has a full disk image, containing both the OS (root filesystem, installed applications, etc), and your data (typically in a home directory). Install a lot of VMs, and you rather rapidly find yourself running out of disk. Plus, you have to update each VM separately - which is annoying, bandwidth intensive, and leads to a lot of out-of-date VMs.

Instead, Qubes works based on template VMs - an OS install and root filesystem that is used as the base to launch other VMs. A single template can launch dozens of VMs with no additional disk space used, and when you update the template, everything else sees the updates too (on VM restart).

Qubes splits out the root filesystem (OS, applications, etc) from the home filesystem (your files and settings), and there are a few ways to configure this.

• AppVMs are based on a root filesystem template, and have a persistent home filesystem. This means that the root filesystem is reset on every reboot - but, if you want to make changes at runtime, you can (installing something will work until you reboot, at which point that will be reverted). The home filesystem is persisted across VM reboots, so your files stick around. If you need more space in your home directory, you just shut the VM down, increase the size, and restart it!
• Standalone VMs have their own persistent root filesystem and home filesystem - so, if you want to install a “regular virtual machine” for some reason, entirely separate from the template system, you can. This is also what you’d use for a Windows VM, or, perhaps, some other tricks I’ll discuss below.
• Disposable VMs are entirely ephemeral. The root filesystem is based on a template, and the home filesystem doesn’t persist past a reboot. This is useful for opening untrusted documents (a PDF or something) in an environment with nothing interesting to exfil and no persistence, but it’s also useful for “random web browsing.” Your browser exists in a sandbox with nothing interesting, and anything bad that does happen will disappear when you reboot the VM. It’s kind of slick!

In practice, it’s not that bad. Pick a template VM based on what you’re familiar with (Debian or Fedora), and things just sort of work. To install a new package from the repos, boot the template VM, install what you need, shut it down. Or, for a temporary install, just install it in the AppVM or DisposableVM and use it!

One gotcha with installing things in the templates is that they don’t have an internet connection (to help prevent any sort of badness from showing up, and to make sure you’re not using them for general use). Unfortunately, the install process for a lot of software on Linux is “Download our public key and add our repo afterwards.” The wget or curl command will fail, as it can’t get out. However, there is a proxy available for the software updates, and you can chain through that. To download a keyfile, you can specify the http or https proxy for a command:

https_proxy=127.0.0.1:8082 wget ...

Don’t set this as an environment variable in your shell, or you rather defeat the point, but it’s a handy trick for installing things.

The Whonix Workstation and Gateway: Safer Tor

Qubes also will offer the option to install the Whonix templates - both the gw (gateway) and ws (workstation) templates will be installed. So, what the heck is Whonix? It’s a two-VM solution to use Tor with less risk of accidental leaks. A core problem with Tor use (to include the Tor browser and Tails) is that any sort of leak from the browser that can make a connection outbound will bypass the Tor proxy - and that’s bad. Flash, WebRTC, and various other techniques have been used, in the wild, to unmask Tor users, often enough leading to arrest warrants and jail time. The normal, well behaved browser traffic is proxied through Tor, but poorly behaved traffic often isn’t - and browsers are complex enough that there are a lot of ways to make a browser “not well behaved” with regards to proxies.

Whonix attempts to solve this problem by eliminating any ability for a network connection to “sneak out around Tor.” Instead of a single system image running everything, Whonix consists of two virtual machines. The first is the gateway. The gateway connects to the internet, runs the Tor daemon, and provides another network interface for the workstation. Everything coming in this interface is routed through Tor, regardless of the source on the workstation - well behaved browser, poorly behaved browser, random other command line apps - everything. The workstation image only connects to the gateway - so even if you try to get a straight connection out, it still goes through Tor and is masked.

If you’re just using Tor for “random sysadmin test points,” it probably doesn’t matter. If you’re doing anything more interesting with it, you probably should be using Whonix. Qubes makes it easy! You might disable most of the Javascript in the Tor browser, though. “Safer” will work on most websites, “Safest” disables just about everything dynamic. Javascript sucks, Javascript JIT engines tend exploitable. Just disable them.

Plus, you can run the workstation and browser in a disposable VM - which is probably a good idea too. Qubes explicitly doesn’t guarantee that disposable VMs won’t leave a trace, but you’re still making an attacker’s life harder.

And if you want to just run all your Qubes traffic out Tor, you can do that too. I wouldn’t. The internet is depressingly unusable over Tor these days.

Using Qubes

With all this background, what is actually using Qubes like? When everything is working properly, it behaves about like a normal desktop - just weirdly colorful. Every window has a brightly colored border, and, done properly, you should have quite a few different colors. You’ll also notice that each window has the VM name prefixing anything in the title bar.

This is the trusted display manager (typically in dom0, unless you’re feeling experimental and using the GUI domain) handling the various windows from each VM, and, as the trusted end of things, showing you, the user, which VM every window is from. These are the framebuffers from the different VMs, passed over to be displayed via a simple protocol that should be reasonably robust.

To launch tasks in different VMs, you’ve got a standard looking launcher menu - but it has all your VMs listed, with various applications in each VM.

If you’ve got a terminal open in a VM and launch something in that VM, it “just works” - it shows up as another window, like you’d expect.

Things that would normally appear in the system tray also end up aggregated in the system tray - they’ll just be colored to indicate which VM they’ve come from. If you want to run multiple accounts in Element (the flagship Matrix client), just run each instance in a different VM! On a laptop, the sys-net VM will stick the normal Network Manager icon here, allowing you to do things like pick wireless networks with the mouse. They’ve done a good job of merging everything in such that it feels more or less like a normal desktop.

Just, no GPU acceleration. Seriously, you’re not going to do much gaming.

The Clipboard Weirdness

Of course, it does have some weird, slightly sharp corners. The clipboard is one.

A major problem in most OSes that use a global clipboard is that a malicious application can just sniff the clipboard. This is true of desktop OSes, and also of mobile OSes - when Apple added “something just read the clipboard!” notifications, quite a few apps were discovered to be nearly constantly reading the clipboard. “To look for debug strings!” or something was the excuse, but… if you’ve ever copied a password or something, you might be uncomfortable with this concept - rightly so!

Qubes fixes this problem by having a lot of clipboards, and a defined way of moving data from clipboard to clipboard. Each running VM has its own clipboard (that all apps in that VM can read). Then, there’s a global clipboard that can transfer clipboard contents to or from a VM, in response to user input. To copy a string from one VM to another, you copy the string in the VM, then use the “global clipboard copy” sequence (default is ctrl-shift-c). Now, your string is in the global keyboard. Select the VM you wish to paste into, and use the opposite command (default is ctrl-shift-v) to move the content from the global clipboard into the VM, and then you can paste as desired. When you do this, however, the global clipboard is wiped - so you can’t accidentally paste anything sensitive into another VM.

And forget moving images via the clipboard. It’s text only.

In practice, you have to learn some new shortcuts, and it’s slightly annoying to copy/paste in terminals with the default commands - but you can change them if you want to ctrl-win-{c,v}.

Moving Files

Another challenge that Qubes solves is how to move files between domains. The developers claim, with some justification, that this is more secure than moving files between machines on USB keys, and because of the track record of USB-key based exploits via partition tables, filesystem headers, and the like, I’m inclined to believe them!

The process is simple. From the VM with the source file, you can use the menu in the file manager to send a file to another VM - or you can send it via the qvm-copy-to-vm command from the terminal. Either way works fine.

This shoots the file over a simple shared memory channel, and on the receiving end, the files end up in the very-unambigiously-from-other-VMs directory of ~/QubesIncoming, with further directories being the sending VM name.

Just to make things a bit more fun, you can also open files in a disposable VM! The qvm-open-in-dvm command will do this, as will the right click menu in the file explorer. Sketchy PDF? Open it in a disposable VM! There’s even a “render safe” option for PDFs that will convert a PDF into an image based PDF, removing all nasties from it. You just can’t search or edit text anymore after doing this, though - it just turns it into a plain bitmap image.

Handling USB Devices

USB devices pose another interesting threat - there are lots of “Pwned via USB devices going rogue” stories in tech, to include things as famous as Stuxnet. USB devices, properly malware’d, can change what they are after being inserted, can deliver partition tables and filesystem headers that change under you and lead to kernel compromise… the works. There are a lot of ways they can be actively evil. Oh, but it’s yet more fun: A lot of desktops only support USB keyboards and mice these days!

USB devices are also, annoyingly, an unavoidable part of the modern computing ecosystem. Qubes has developed a way to allow you to route them to individual VMs, such that you can treat them as the untrusted pain in the rear they are. The model is that the USB controller is routed to the sys-usb vm, which handles identifying the devices, and routing them to the respective VMs - where that kernel will then do something with them. There’s nothing of value in the sys-usb VM, and while the target VM will get to parse the device, you can also use disposable VMs for handling untrusted USB devices.

If you have a USB keyboard and mouse, there’s the risk that something goes wrong in the process and you have zero input to the system. There are ways to work around this, but the easy solution is to simply not auto-start the sys-usb VM if you have a USB keyboard and mouse. Start it before you need to route any USB devices around, otherwise your keyboard and mouse are just routed to dom0. Again, some of this depends on your threat model. But if you can find a machine that supports a PS/2 keyboard and mouse, life is a lot easier, because you don’t have any actual reliance on USB to control your system. Surprisingly, a lot of desktop boards still have variants that support PS/2 input devices!

Security and Silos: What’s Your Threat Model?

At this point, an obvious question one would expect is, “Great! How should I set this up to be secure?” - with secure being implied as “Secure(TM)!” The answer is, I have no idea. Because I’m not you, I don’t play in your spaces, and my threat models are not your threat models. If you’ve never thought about your threats on the internet, it’s a good year to start. If you have, you’ve probably already considered some ways you can use these tools to improve them.

I would, at a minimum, suggest separating “casual web browsing” from the more trusted things you do that involve your core identity credentials (email, ssh keys, etc). A separate “sysadmin” Qube is probably a good idea.

Of course, if you’re doing something like freelance web development, have separate VMs for every company/project you work for. This sort of system is very well suited to “Company A cannot access anything from Company B” sort of isolation. It’s also well suited for development models in which “Download 70,000 random and constantly changing dependencies so the website can render!” is accepted as normal, when these dependencies are often enough found to be compromised and rather grabby with files on developer workstations.

Some people will push the silos to the limit, with stacks of very minimal VMs that can only run one application at a time. It’s possible if needed, though at the cost of a good amount of RAM (or fine tuning RAM use per VM). But you really need to think through “What gives access to what?” - and group those things together. Qubes isn’t a “magically secure answer.” It’s a set of tools that allows you to build a solution for your needs, and it doesn’t try to enforce any particular views on how you go about doing this. I imagine Edward Snowden’s use of Qubes looks very different from mine.

Hardware VMs: Multi-Booting Madness!

Finally, based on some conversations on IRC about something that seemed possible and ended up being rather harder in practice than theory, I’ve managed to get my native Linux install (running on the iron) booting in Qubes, as a standalone VM! I don’t get GPU acceleration, but it’s sort of nifty to be able to boot your bare metal Linux install as a VM.

Is it a good idea? That’s up to you and your threat model, but I’m not convinced it’s terribly useful outside a neat party trick, and perhaps doing some dev work where you might want hardware acceleration for some of it, but not all of it. You don’t magically add any security to your on-the-iron install, though I don’t think it opens any particular holes in Qubes, either. Unless you add the tools, you don’t get any of the fancy Qubes integration, but if you want to do this, here you go!

My setup at the time had a Windows 10 and Linux (Ubuntu 20.04) install sharing a NVMe drive, with Qubes installed on a separate SATA SSD. I’m fairly certain this won’t work as I’ve done it if you have them both on the same physical drive. But:

qvm-block will show you a list of all your block devices that can be attached to VMs. My NVMe drive has the Windows and Linux installs.

You need to create a new VM. This should be a StandaloneVM, with plenty of RAM. You don’t need a template, and leave the storage pool as the default - it won’t be used. Once it’s created, go into the VM settings, under Advanced. You’ll set the kernel to none, the virtualization type to HVM, and enable “Run in debug mode,” which will show the boot menus and such.

Now, you’ll need to attach the block device permanently to the VM. This will give the VM access to that block device on boot. You’ll also need to allow the VM to boot as UEFI (assuming your installs are UEFI, which mine are).

 qvm-block a test dom0:nvme0n1 -p -o frontend-dev=xvdd
 qvm-features test uefi 1

If all has gone well, you can start this VM. You should see a window pop up with the TianoCore UEFI boot splash screen, and hopefully, Grub!

Select your Linux install, wait a while (it seems to take far longer to boot in Qubes than on the hardware), and, ideally, you’ll see your login screen and be able to log in!

However, you won’t have networking yet, and Qubes networking is a bit weird. It doesn’t do DHCP. Open the VM settings, and you’ll see some “Net qube” settings. Apply these in your VM (I have a little shell script that does it), and you should even have networking!

Doesn’t work? Good luck! Resolution handling is weird, and my normal xrandr tricks don’t seem to work. But I’m not convinced this is a good actual solution, just a neat little party trick.

Final Thoughts

At this point in the post, I suspect very strongly most readers will be in one of two camps: “This is absolutely insane. Why would you ever subject yourself to this madness?” or, “Huh… that sounds interesting.” If you’re in the second camp, get yourself a spare drive, turn off hyperthreading, and try it out! The forums are quite active if your hardware is being weird, as is IRC.

You can’t do everything in it - but that which you can do works very well. And there’s no harm in having multiple OS installs on a computer, but if your “bare metal Linux” install has everything you’ve so carefully siloed away in separate Qubes, neither are you really gaining anything. So consider your needs and how to best go about solving them.

As for me? It’s the sort of weird, paranoid, “slightly dysfunctional in various novel and charming ways” OS that I just have to love! Now if it would only run on ARM…