Day Two Cloud 100: Adding Complexity to Simplify

By Published On: June 7, 2021

There’s a tension between consumers and producers created by the competing desires of each party. The producer wants to create something robust that meets the needs of 80% of consumers without being too complicated to produce and maintain. The consumer wants a service that meets 100% of their needs without being too complicated to consume and configure. Too meet the needs of 100% of consumers, the producer needs to add more complexity to a service to cover the remaining 20% of use cases. The new complexity will be unneeded and unwanted by many consumers, which leaves them annoyed that their simple service now is more complicated to consume and configure. The producers, likewise, now need to maintain a more complicated service that does more things. The alternative is to leave the service “incomplete”, frustrating potential consumers with a perceived lack of utility. For producers that choose not to ornament their product with every possible use case, a vacuum is created where a new service might be introduced to fulfill the needs of the 20%. Crossplane is that type of service.

For episode 100 of Day Two Cloud, Ethan and I talked to Daniel Mangum, a maintainer of the Crossplane project. The conversation was spurred by a blog post Daniel wrote comparing Crossplane to LLVM. I highly recommend reading the post. And then reading it again, because you definitely did not get it the first time – at least I didn’t. Dan gets into what Crossplane does and how it provides abstractions to both the platform maintainers (producers) and the development team (consumers).

Without getting into the nitty-gritty details of how Crossplane does things, I wanted to focus a little more on the general concepts behind the project. If you want to dig into the how of Crossplane, give the episode a listen, we get deeply nerdy pretty quickly. Ultimately, Crossplane is a platform providing a service. The service it provides is Layer Two cloud to consumers, controlled by producers. What is a Layer Two cloud? It’s an overlay on existing cloud providers that creates an abstraction of their services for simplified consumption, a platform on a platform if you will.

The classic example is a developer needing a SQL database for their application. They don’t want to know the nuts and bolts of the SQL deployment process, they simply want to request a database with a few parameters and they’re off to the races. Doesn’t the public cloud already provide that? Well… sorta. If you want an Azure SQL database, you need to create an Azure SQL server instance, give it a bunch of parameters around sizing and performance, make sure you set the firewall rules correctly, decide whether you need replication, and maybe something around backups. Microsoft gives consumers a lot of flexibility in how they deploy an Azure SQL instance, and honestly a lot of developers don’t care. They want a database with 50GB of space and a connection string.

Crossplane lets a platform team (producers) create an abstraction of an Azure SQL database, and they can decide how much to expose to the consumer and how much to leave hidden. Better yet, they can create an abstraction of a database that works on multiple cloud platforms. And they can create multiple abstractions for different types of consumers with differing use cases and requirements. Dev team A might want a simple T-shirt size of database: S, M, or L. Dev team B might want to be able to control performance tier, replication, and storage types. Crossplane allows the platform team to create separate abstractions for each team that meets their needs. On the backend, the platform team can ensure that the implementation of each abstraction follows company best practices regrading security and compliance.

Of course, best practices change over time, and thus the abstractions need to evolve over time as well. Simple enough when it comes to updating the templates, but what do you do about resources that have already been deployed? Crossplane has a reconciliation cycle that can keep existing deployments in line with a desired specification. The platform engineers develop a desired state and Crossplane handles the process of remediating the current status to match the desired state.

Admittedly, there is a significant amount of work that needs to be done by the platform team to create all these abstractions and maintain them. Crossplane helps with some of it by including providers and pre-built abstractions the platform team can customize. But still, this is no small undertaking. Your average 250 person company is not going to run Crossplane for their massive team of 10 developers. It just doesn’t make sense. The solution is squarely aimed at large organizations and service providers, which from the right perspective is essentially the same thing. Platform teams at a sufficiently large organization are basically service providers to the rest of the company.

If I were building an MSP and wanted to offer a cloud solution, Crossplane would definitely be worth a look. Slap a GUI on the front and provide self-service for my customers to build infrastructure on a Layer Two cloud offering. Depending on the customer, I could create specialized abstractions that meet their needs, without exposing the raw cloud provider to them. The tension of simplicity vs. complexity still exists, but Crossplane is a tool in my arsenal to balance the tension.

There will always be a need for abstractions in technology, and Crossplane has built a reasonable compromise between what the producers want and the consumers demand. It warrants further examination and a bit of weekend tinkering. Off to the lab!