Managing Infrastructure on a Massive Scale With Backstage #1 — Templating
Photo by Wes Hicks on Unsplash
The Problem
There are many problems with development experience in big organizations. CI/CD, local development, many programming languages, the list goes on and on.
One of such problems is the templating of the microservices, packages and libs.
Let’s take Nodejs for example. Try to imagine that you have over 1000 NodeJs Developers in your company, and each one of them will create a new service from scratch. Sometimes the testing framework will be different, and sometimes the CI and CD tools will change, not to mention shared modules and documentation. Each time someone will need to reinvent the wheel, the code standards will get reduced dramatically. Now multiply it by each programming language you’re using, and you’ll get lost.
“Development Experience Teams” that are responsible for velocity put together scripts to help create such templates. They can do it via makefile or a script in package.json, it all depends on what they are using. It doesn’t matter if your service is Nodejs, Golang, React component or a Terraform module — you need a solution to easily create new components and to follow the company code standards.
Most of the time templates like that are a part of something bigger. Creating a template is just the beginning of the development flow, after that you have CI/CD and second-day operations.
In big companies when you need to handle lots of scenarios with different teams working separately, each using different programming languages, things often get quite disconnected.
Not to mention production code, Infrastructure as code, etc.
One of the best tools out there that can help you standardize the development flow with CI/CD and second-day operation are the developer portals.
But what are developer portals? Glad you asked :)
Developer Portals
Developer portals provide us with a unified UI for all the backend systems in an organization, helping unify the way developers and DevOps work. Let’s look at an example. You create a microservice/lib/package from a template using the UI at hand. The software then creates an automated PR in an existing/new repo using your code standards, that’s in addition to all the things that need to be added in order to take your service up to production (CI/CD yamls, K8s manifests, etc).
Here’s a short video overview of Spotify Backstage, one of the most popular platforms:
With Backstage the code you create is templated, meaning you don’t have to spend any time on boilerplating in your organization.
That’s exactly what we did with our Infrastructure as code, Chaos experiments, Grafana dashboards and everything supporting production readiness.
Terraform and Backstage
Let’s look at Terraform, the most common IaC for creating resources with several cloud providers.
If we think about it, writing the Terraform process code can be broken down into 2 phases:
- Writing modules
- Writing the “live” code (environments, states etc) that uses those modules.
Let’s say we want to automate the module creation process. More than that, we want to add all our concerns and standards during the creation process. Things like “security as code” or “Finops tags”, validations, etc.
All of that can be handled via the template engine of Backstage. We’re creating the aforementioned Terraform modules with all our infra and code standards, thus bringing all this velocity to our DevOps.
Developers helping DevOps that helps Developers, Isn’t it cool :)?
(** Infragod is our Terraform solution in Wix, you can check my meetup on Infragod here).
The thing about backstage is that it’s very pluggable (Based on ReactJS and NodeJS), so you can customize it to fit your needs.
We took those plugin capabilities of Backstage and developed on top of it a fully Terraform E2E flow which can be used to create infra and deploy it to the cloud.
Why do I need all this fancy UI just for templating ?!
Most likely you are asking yourself this question because we described just the beginning of the infra development flow. This is just the dev experience. From here we create an artifact (a thing we called Terraform image) and deploy it via Backstage to cloud providers in a gradual deployment manner. We also create backstage Chaos tests templates, Grafana dashboards etc, Everything that the DevOps need to write their infracode to be perfect.
I’ll elaborate on that in my next meetup in Wix engineering soon. Stay tuned!
