Getting Into Build Engineering – The Case Study Of My Own Path

On getting into things sideways. The path is the destination. Scratching an itch.

Build engineering is one of these numerous things I landed in without really expecting it: To anyone interested in the topic, here’s the backstory to how I ended up co-owning the subject with another engineer at Open-Systems.

Who knows, it might make something click for you.

Where It Began

The words build system first reached my ears during my fresh-out-of-school job at Tiko, circa 2014. That’s where the introduction mentioned in what’s a build system happened, and it was still a relatively long time before I started seriously looking into the concepts underlying the vague notion of build engineering.

Regarding my builds, I was doing what everyone else seemed to be doing: copy-pasting pieces of XML from stackoverflow into my Maven poms until things worked. Copy-pasting code would have been bad, right? But configuration? No one cared.

We had a pretty stable monolithic architecture (it was modular & quite flexible to work with, though!) and the build & release process, albeit manual, only had around three steps. All in all there was no need to tweak the build regularly, except for adding and updating dependencies. With a single release per week that took less than 30 minutes, everyone was pretty happy.

Firmware and front-end were other teams' prerogatives: each one was responsible for everything on their turf, from development to releases. So no need to worry about C, Python or JavaSript.

Life was easy: I only dealt with maven, even if I understood very little about what I was doing, and the copy-pasta was sufficient for a year or so.

Nascent Cardinality

That changed when we started exploring tools like JooQ, using Scala to implement binary parsers or trying Thrift as an RPC framework. These required a bunch of maven plugins and forced me to look a little bit into the build lifecycle and how the whole enchilada worked: albeit under the mentorship of an ex-FB engineer on another team, I was the guy pushing for all of the above on my team, so it was my responsibility to get them to work.

It was my first introduction to on-the-fly code generation by the build system – as opposed to committing boiler-plate to the repository directly. Scala parser combinators used from Java, database schema versioning and backward-forward compatible serialisation protocols were also entirely new to me.

I had not yet uncovered the curse of cardinality, which long-lasting projects eventually run into, but there were suddenly way more moving pieces in the build-specific part of my daily work.

Nostalgic note: This all happened during a window of a few months where my rate of learning was absolutely tremendous. I don’t think I’ve ever experienced something as intense ever since: the combination of challenge (it was a startup after all!), focus and youthful optimism was quite optimal.

Level Two

Roughly a year after Daniel Neuhaus, then CEO of Sqooba AG, had sold me on his company¹, I started a new kind of job in the world of data analytics services.

It’s in the service world that I discovered how much build system configurations are actually blindly copy-pasted. Not that it was too problematic: things worked overall, it was just cumbersome when switching between projects (of which there were a few) that were themselves evolving quickly. That’s around thetime where I started wondering why we should tolerate blind copy-paste of build system configuration when we would not accept it for code.

On top of it all, we had stuff going on in Python, Java and Scala – each with their own build system! – and of course we had something comming that would require Golang.

The moment had finally arrived:

Ok, now it’s probably worth to learn how to use Maven properly!

Why Maven? I knew we could build Scala stuff from the Tiko days, it was no sweat for Java, and, believe it or not, a colleague was already playing with the Python+Maven combo². It looked like all the pre-requisites were there, as we were considering using Maven as the single build system for all our needs³.

In parallel to this, we also started consolidating many libraries and utilities that were quickly spreading across hundreds of repositories into a single internal fat repo.

Build approaches were aligning, mutual understanding and team cognition was improving: life was again good.

Cardinal Explosion

Fast forward a year and my employer gets acquired by a customer: Open Systems AG, which I can succinctly describe as A full-stack company handling your network and security.

Why full stack? Because it managed everything⁴ from the hardware to the backend sucking in the data to the admin systems and dashboards: in theory we could touch any part of the system if it was needed, which is a pretty cool spot to be in.

However, this was obviously implying another increase in cardinality: now on top of everything Sqooba was dealing with, we added some Perl, bash scripts and firewall rules. Cardinality was not improving.

For extra fun, and because the background story of the acquisition also included something about moving more into the cloud, usual suspects such as Terraform also joined the party. Just one more thing to deal with.

Functional Intermezzo

You may have gathered from elsewhere that I’m a fan of functional programming: One (of my many) train of thought that started at Sqooba was what if we applied the concepts of FP to as many things as possible?, and an obvious place to consider was our build system.

Why was it that Maven had to eagerly rebuild so many things? How come that our tooling still required so many manual and dirty steps, fixed by clunky ad-hoc home-made scripts? Could we make it easier for everyone to be able to build something on their machine?

Around this time, I ran into Tweag.io’s blog, specifically this post about Bazel and Nix, and was delighted to discover that all my interrogations already had some answers available.

This was a key moment for what would come after the acquisition.

Enterprise Code Delivery Platform

So, we were standing at this post-merger junction, looking at the future and what it would require from us. One thing quickly stood out:

On one hand, flexibility and speed would always be a huge asset: if your organisation is driven by software, it implies that quick adaptation on that level allows quick adaptation on the business level as well. That’s a good goal to strive for, especially in our uncertain world.

On the other hand, there was no way our stack could be massively simplified. Most things that were present still had a very good reason for being there. Plus the existing codebase was not moving anywhere soon.

This is where we started digging into Bazel as the backbone for building a true code delivery platform: something that would let us develop, build and deploy code to any place in our stack as fast as possible. Be it in the field, our on-premise datacenters, our cloud or our frontend.

At this point, I was both hooked and convinced that it was worth focusing on this subject. And so I did!

Epilogue

By coincidence, we started our first internal playground with Bazel a few months before 1.0.0 came along, and activity around that project has steadily increased since: it turned out to be a good choice!

Transformations of this scale take time and involve a lot of back and forth with stakeholders and teams. In fact, way more than I initially had expected:

The key point you need to consider if you’re interested in build engineering is that you’ll have to deal with many teams and people: most things you will do will be for others, as you are seeking to optimise the output of the entire organisation, meaning it will not always be for the immediate benefit of every individual.

At times it will really feel as exciting as watching grass grow: in my case, it took the biggest part of two intensive years to begin to see the benefits. Don’t get into this if you’re not in it for the long haul.

The roundabout path

I’ve read in a few places that security is a typical field where sideways entries are welcome: the roundabout way to get in has the benefit of giving you experience from other software related fields, which will help a lot. After all, your main customers will be other fellow developers.

It seems that this is pretty similar for build engineering: the field touches upon many different subjects, and the specifics can vary with particular industries. Plus, it changes pretty quickly: whatever you see today will have evolved again next year.

So if you have an itch pulling you towards this area: I wish you an adventure as intense and fulfilling as mine was.