A good friend of mine just started a new startup gig. He’ll be their company’s “ops-focused” developer (they aren’t at the size yet to staff a full Engineering Operations team of any flavor). It’s a great opportunity for growth albeit an understandably scary position. His company is starting to hit its stride and he’s expected to keep the wheels on while the team rapidly iterates on the product.
While we caught up, he asked for any tips or advice to take on his new job (or I provided it unsolicited. not sure). I chose to give him some thoughts I’ve kicked around in my head for probably over a year: Don’t Prematurely Scale.
The road to hell is paved with good intentions and the path to Engineering hell is premature optimizations. Usually well-intentioned efforts to make sure “the new thing will scale” causes complicated design and complicated design is the enemy of security, reliability, and (most maybe importantly at this stage) ease of development. Sometimes this causes irreversible harm to the system leading to worse problems — like a Rewrite project.
To better illustrate what I’m talking about let’s take an increasingly common pattern in web services. Most people reading this post are likely familiar with how a traditional website is hosted. You have a computer acting as a server. It’s connected to the network with an IP address and has software running on it to respond to HTTP requests. It might respond with JSON for an API or HTML for a web page. There may also be a database running on the server to store things like usernames, emails, past visits, etc.
If this were an interview, I might ask, “How would you scale this service expecting traffic will increase 5% each month over the next year?”
To which you may reply something about creating multiple copies of the server, shifting the database to its own computer (or VM), setting up a load balancer, add caching, migrate to a cloud provider, set up autoscaling, etc.
What if I told you in a month, 100% of that traffic will never touch the database. Oh, did I forget to mention that we have a new partnership with “ThirdParty LLC” and for $1/month they’ll operate an API for us that will make that database obsolete. Better yet, our business will pivot so hard in 3 months that the entire service is in the dumpster and we’ll focus on something entirely new.
How would you answer the interview question now?
Why doesn’t prematurely scaling work?
Scaling a system implies we know and fully understand the future. In laboratories and interview questions, we set parameters and build closed systems to dissect the scenario under test. In the real world, there are no closed systems and the future is unknowable, volatile, and risky.
Preparing a system for scaling to more stress or more users may seem a net good no matter what ends up happening, but it rarely works out that way. In my experience, when a prematurely scaled system ends up working out is usually a rare, happy accident. All too often, it’s a waste of time, energy, and money.
Even if we have a system or application that won’t dramatically change over time like the earlier example, knowing how to scale is all about knowing the bottlenecks. I contest anyone that claims to know the bottlenecks of a system without running it in production for years. Even then, I seriously doubt they can identify the bottlenecks for the exact same system over the next few years. User patterns and conditions change too often and randomly to be predictable.
If you disagree, I recommend you quit your engineering job and start playing the stock market because you must miraculous divination abilities (and can make much more money in stocks).
What to do instead? Anticipate, Monitor, and more
By now, it may seem like I’m proposing to architect our systems reactively — only making design changes in response to an active scaling issue. In some ways, that’s right. In another, more helpful way, there’s a lot of work to be done to help our companies technology scale without resorting to reactive engineering.
In my next post, I’ll go over the ways that we should approach scaling problems that is both efficient and effective while also providing other DevOps benefits to an organization.