The Google File System

paper link

https://drive.google.com/file/d/11FFRgc7u8OLKVsLWF6mg8m1Z0XyNvtGv/view?usp=sharing

paper year

2003

thoughts

Thanks for the overwhelming response to my last blog post, I got a lot of emails back from the newsletter about that one. So, I felt like I should cover less on explaining the paper again and spend more time on commentary, on what I liked and what’s interesting about it. I think in that borg paper I spent a bit too much time explaining how the paper itself is written, which I don’t think is needed since you can always just read the paper yourself.

This is one of the very famous, classic papers on file systems, called the Google File System. We should first understand the timeline here: this paper was written in 2003, and that’s a very important consideration, because the whole paper revolves around what Google’s workload looked like back then.

This paper is so influential and well done that a lot of its ideas are still relevant in 2026. If you want to build a file system at any scale, this is a very good starting point, because a lot of the systems available today are built on top of the ideas from this paper, even though Google itself isn’t running GFS anymore. There’s a newer version now, called Colossus, but the implementation details for that one aren’t public. A lot of the design principles and ideas here are still really influential though. HDFS, the Hadoop Distributed File System, is almost completely inspired by this paper. BigTable is also built on top of GFS, and a few of MapReduce’s implementation ideas, even though that’s a completely different kind of system, take some inspiration from GFS as well.

I think in that era of Google, roughly 2003 to 2006, there was a heavy preference for single master designs, because their workloads scaled well enough to fit into a single master. A lot of their systems revolved around this, including Borg, which also has a single master. GFS has the same idea. So if you’re building a system now, I think it’s fine to consider a single master, because this paper also talks about the tradeoffs: all the workload goes through that one master, so you need to make sure it’s fault tolerant, which is possible. What I noticed across these papers from that era is that despite the heavy reliance on a single master, none of them really go into detail about the consensus algorithm behind master replication. They clearly replicate the master somehow, but they don’t say which consensus algorithm they use, and I think that’s actually pretty key. Turns out the original 2003 GFS paper doesn’t actually use Paxos for master failover, the master just replicates its operation log and checkpoints to shadow masters, and an external monitoring system detects a master failure and starts a new one, it’s not an automatic, consensus based election like Borg’s. Paxos style consensus for GFS’s master came later.

What I really like about this paper is how much focus they put on keeping things extremely simple, along with how clear they are about what their actual workload looks like. I think those two things are very important. They understand the limitations they’re working with and implement things as simply as possible while still getting the job done. That’s the constant tradeoff they face, and they’re okay with it, because they clearly understand their workload. A couple of the design principles behind GFS follow from this: they’re very aware that whatever physical hardware they use will definitely fail, and they understand their workload well, large file transfers, lots of appends instead of random writes, and so on. These are the insights they already had, and they built the system around them.

One common pattern I see in single master designs is putting a lot of focus on reducing load at the master level. Systems are built around the master in a way that keeps the load on it as low as possible, which makes sense. So the whole engineering problem becomes: how do I have a master that coordinates everything, while making sure I reach that master as little as possible?

One really wise thing they did in this paper, which I think is pretty revolutionary, is not reinventing the file system wheel. Since Linux already has POSIX style file system interfaces, instead of reinventing the whole file system and handling everything themselves, they built a layer above it, writing their own GFS client and their own file system interfaces that sit on top of the existing Linux file system. That’s a cool idea, because it means they don’t have to go mess around in the kernel, which would make them a lot more prone to errors and edge cases.

Another idea, even though it’s 2003, is that they implemented a log based system, what they call the operation log, which is basically a write-ahead log, to replicate the master’s state. It’s funny, because even in 2026 people act like rediscovering write-ahead logs is some big new trend, but this 2003 paper was already using one to replicate master state. That’s also pretty cool.

The best part of this paper, which I found really interesting, is this scenario: say you have three machines, I’ll call them chunk servers, each holding file chunks. Because of replication, you have one primary and two replicas. So how do you make sure all three servers end up with the same data? The obvious answer is some kind of distributed transaction, you lock everything and make sure the write lands on all three or none at all, like a database transaction. But this paper does something different there’s no locking at all. Say you’re writing to server A, server B, and server C, and the write to server C fails. They just retry the whole operation until it lands on all three. The catch is that because of these retries, server A and server B can end up with duplicate copies of the same write. GFS’s answer to this isn’t some automatic dedup system, checksums here are used to catch corrupted blocks of data, not to spot duplicates. The actual duplicates from retried writes are something the client application has to deal with itself, usually by tagging each record with its own unique ID when it writes it, this comes up specifically with their record append operation. So there’s no distributed lock anywhere, the tradeoff is that you might end up with duplicate or padded data sitting around, and it’s on the application layer to handle that if it cares. Compared to the complexity of a proper transaction, this is a very simple, very pragmatic idea, and I hadn’t thought about it that way until I read this.

I also touched on garbage collection earlier, and it’s a pretty important part of this design. Say a client wants to delete a file, it’s not deleted immediately. First, the master logs the delete and renames the file into a hidden namespace, marking it for the garbage collector, and only later does a background process actually come and clean it up and free the space. A lot of what I appreciated here is how they figured out ways to make the system asynchronous, delegating work to background processes so there’s no overhead in the foreground. The drawback is you don’t get instant deletion, but they’re fine with that tradeoff. For workloads that constantly create and delete files and actually need the space back quickly, they have a separate mechanism to reclaim it sooner. It’s a good example of the general idea that you can push a lot of system work onto the background instead of doing it synchronously.

If I remember correctly, something similar shows up in Golang’s garbage collector too, though I’m not fully sure on that. Anyway, those are the pieces I found most interesting in this paper. There’s also a concept called record append, for handling multiple concurrent writes to the same file.

advantages

The advantage I see in this paper is how simple the design choices are. There were plenty of ways they could have made this more complicated, but they kept it as simple as possible, while clearly understanding their own needs and knowing exactly what tradeoffs they were making. That’s one advantage, and also something I learned from this paper: understanding your exact requirements and being okay with the tradeoffs that come with them, because you can’t satisfy everything at once.

disadvantages

For 2003, this is a breakthrough paper, but let’s be a devil’s advocate and compare it against 2026 and point out the disadvantages. The first one is the obvious one: the single master. The master is basically a mapping of where every chunk lives, and all of that sits in memory. As the system keeps scaling, they could keep adding more memory to the master, but that’s not a real fix, you can’t scale the master infinitely that way.

Another disadvantage is the hotspot problem. Say you have an extremely large file, split into five chunks, and each chunk has three replicas, that’s fifteen chunk copies total. Now if you get n concurrent requests, they’re spread across those fifteen servers, which is decently manageable. But now assume a really small file that fits in just one chunk, with only three replicas. The same number of requests now all land on just those three instances, which becomes a hotspot, because all the load falls on a tiny handful of servers. I think that’s a real disadvantage with this design: smaller files are at a disadvantage, and the whole thing is largely optimized for big files.