How Skaffold Enabled Me to Deliver Features 10 Times Faster
Photo by Marc Sendra Martorell on Unsplash
How many times have you started a new microservice/consumer/module, and asked yourself: “How am I going to build the environment to test it? Check that it’s actually working?”
And I’m not speaking about the code. I’m sure that you’re great at what you do. Most of the time that’s the easy part. I’m speaking about connecting it all to the ecosystem, to other microservices, Databases, MessageQueues, Kubernetes, etc.
Also, what about those tests? And not simple unit tests, but Integration tests. System tests. The real difficult ones, that provide enormous value. The ones that make your system bulletproof.
Take this example :
You have a WebAPI and it produces a message to Kafka — you begin creating a new consumer (consumer Z) to read that message and then do something with it. The rest of the pipeline (Consumer X and Y) rely on you. And they need to do something with your process of writing things to Redis, Indexing in elastic search, etc.
Testing your code like this can result in quite a headache:
Now take 30 seconds to think about how you are going to set up and run all this infrastructure in your local environment…
But wait — what if I told you that it’s so easy that my team does it in just one minute — and for brand new code or features on a daily basis, no less.
If you share the pains that I’ve had, this article is for you!
In this blog, I will share how my team builds all development and test environments on a local Kubernetes cluster using some amazing new tools.
The VDF Approach
There are a number of ways to test/run code locally. Every company/team has its unique ways. Some teams run their code as binary, others use containers, and even just by running code via fake tests.
If we put this on a scale it will probably look something like this:
Eventually, in my opinion, you need to pay attention to 3 key points, which are defined as VDF:
- Velocity
- Developer experience
- Fast feedback loop
In order to make sure you have all three, you should give your developers an easy way to integrate with other components when writing their code, and wait for CI feedback. If a developer has to wait for 30 minutes for their code to compile, that code lacks velocity. Also, the developer experience is awful, and he/she is very frustrated.
If your team is frustrated because there’s a feedback loop once every 30 minutes, or if you spend 90% of time mocking your local environment, you are probably doing something wrong.
Dockerized development
When developers discovered Docker, they almost immediately understood the benefit of using it for development purposes. I won’t get into all the benefits of it here, but will touch on the main ideas.
Docker gives you the ability to run your code in encapsulated environments. Those “environments” are almost identical on local machines to those running on remote environments, like in production. This minimizes the difference between local development and production environments. So problems like versions of the libraries (like Nodejs or Golang, etc.) simply vanish. You stop hearing “but it works on my computer” comments.
In addition, when developing in containers you don’t need to “install” anything. All comes in encapsulated, just for you.
The next generation of dockerized development is on the local Kubernetes cluster. If you already do docker-compose for orchestration, this is amazing for you.
Docker for Mac/Windows has Kubernetes built in and Linux users have Minikube.
The only thing that is left is to set up a robust development process on those local clusters.
Skaffold gives you exactly that — ability to work with a dockerized environment on Kubernetes, while also providing great tools to do it.
Skaffold
Skaffold is a great tool created by Google to easily develop programs and services in any IDE and automatically deploy them to your Kubernetes cluster (locally and remotely).
It basically watches your dev files and triggers very fast docker image builds. Skaffold is using smart build cache to avoid unnecessary layers to build. Those builds are then deployed automatically to the Kubernetes cluster in your context.
You won’t need to use utilities like nodemon for Nodejs anymore. All is handled for you within Skaffold.
Moreover, Skaffold minimizes your development and deployment time. In the past containers for your local and production environments were identical, but the orchestrators were different. Now you even have the same orchestrators and the same tools. You can use helm, Istio, and all of these other production tools locally (Skaffold integrates with all of them).
Skaffold architecture
Skaffold operates on an iterative cycle:
Detect -> Build -> Push -> Deploy
Skaffold is very Pluggable. Meaning you can choose what to use in every phase. For example, you can deploy your code with kubectl or with helm (everything is declared in the skaffold.yaml file).
If you’re using Bazel as your build tool, it has an amazing integration with Skaffold. I will elaborate further on this in my future blog posts.
for example, you can deploy your code using kubectl. but you can use helm to install all your dependencies like Kafka and Redis.
Skaffold demo:
Conclusion
Skaffold and local Kubernetes development increase my team velocity and give me certainty about my code before it reaches production and other various remote environments.
In the next blog post, I’m going to talk about best practices using Skaffold, and the integration of Skaffold with Helm and Bazel, so stay tuned for more!
