Truly fast container filesystems on macOS

Since the beginning of Docker and containers on macOS, developers have complained about slow installs, app startup, and much more. There have been equally many attempts to fix it, with varying degrees of success.

We've made it 2–10x faster, reaching 75-95% of native performance.

What makes the file system slow?

On macOS, containers run in a Linux virtual machine. This is not only for namespaces and other Linux-exclusive features; it also makes the entire idea of build-once-run-anywhere containers work by keeping the underlying platform the same.

Unfortunately, while virtual machines are generally fast for CPU code, the cost of the crossing the VM boundary for I/O can be high. This makes the native Linux file system (and hence Docker named volumes) fast, but bind mounts are slow because they live on the macOS side, incurring a lot of overhead when crossing the boundary.

Past attempts at fixing this include:

• Network file systems. NFS and SFTP (via sshfs) were once popular options. The protocols themselves are quite slow, but since they're designed to run over the network, they often have aggressive caching that hide some of the performance issues (at the cost of breaking some use cases).
• VirtioFS. This uses the FUSE protocol, funnels requests through shared memory, and handles them on macOS. It's one of the best options at the moment, but still far from native performance.
• Synchronization. This is fast, but only after waiting for it to copy all files (even if you never use them). Synchronization delays and consistency issues prevent it from being used for every bind mount, so you have to think about when to apply it. It's a heavy solution that requires enough disk space to duplicate all files—problematic for large projects and repositories, along with the initial sync and rescan time.

A new file system for OrbStack

Since synchronization has a lot of fundamental problems that are hard to overcome, we decided to focus on making shared file systems as fast as possible. By reducing per-call overhead by up to 10x, we're seeing 2–5x speedups for real-world use cases, typically within 75-95% of native macOS performance.

Benchmarks show:

• pnpm install: 88% native (12.2 vs. 10.9 sec)
• yarn install: 77% native (9.8 vs. 7.9 sec)
• rm -rf node_modules: 87% native (4.0 vs. 3.6 sec)
• Postgres pgbench: 76% native (8998 vs. 11785 TPS)

All node_modules benchmarks were done with 1614 packages (total 807 MiB installed). Postgres can be over 90% of native speed, but OrbStack prioritizes data integrity and waits for data to be sent to the SSD in order to prevent corruption on hard shutdowns, whereas Postgres on macOS doesn't fully flush data to disk.

Canary testers have had great results on large applications:

• 5x faster database tasks (36x compared to Docker Desktop, according to tester)

BritannicMight🇬🇧

@BritannicMight

I don't know what kind of sorcery you're using to do this but it is incredible. SQL housekeeping tasks are down from 1m09s and 46s to 20s and 10s in 1.6.0. For reference these same tasks were taking 10m+ and 6m+ back on Docker Desktop and constantly locked the db. Great work!

• 16% faster Rails test suite

Rob Zolkos

@robzolkos

My Rails test suites are about 16% faster with this update (it all counts!). I have my accessories (db, redis) on docker.

This is an imperfect solution: the macOS file system stack isn't as optimized as Linux's, and it's incredibly challenging to get overheads low enough. Most workloads are now at near-native performance, but we're not quite there yet for some more challenging workloads. There are many optimizations to explore, but it's a great set of tradeoffs overall:

• No thinking about what to use. Named volumes? Synchronized mount? Watch? Bind mounts should always be a reasonable choice.
• Convenient. Files are easily accessible from macOS.
• Needs no extra disk space. No copies of files on disk.

More speedups

The file system isn't the only bottleneck. Virtualization.framework is generally better than anything that came before it for macOS, but there's still a lot of room for improvement.

In OrbStack 1.6, we've rewritten the virtualization stack for performance, stability, and efficiency. This gives us infinite possibilities when it comes to improving on each piece of the puzzle: CPU, memory, file system, disk, network, reliaility, and more. Aside from improving performance in general, we've fixed many crashes and freezes that affected OrbStack.

For the curious: soon, we'll put out a series of blog posts to help you understand virtualization, and what makes it fast and slow 👀

Try the new file system

OrbStack replaces Docker Desktop with a fast and easy way to work with containers, Linux, and Kubernetes. The new file system is now shipping, and automatic migration and drop-in compatibility make it a breeze to switch.

Follow @OrbStack on Twitter and join the Discord community to stay up to date with OrbStack news!