Minikube You Say?
When you begin on a Kubernetes journey, often one of the first questions is; "how do I run locally"? That's where Minikube comes in; you get an easy to spin up single node Kubernetes environment where you can rapidly iterate. As with nearly all tools, there's some rough edges and gotchas to figure out. So in the next 5 minutes I hope to save you hours of frustration.
The blog was written using MacOS but the commands should translate across Linux distributions.
Gotchas
1. What eval $(minikube docker-env)
Does
When you run Minikube it's running in a virtual machine, and in effect you are now running two parallel docker environments, the one on your local machine and the one in the virtual machine. What the eval
command does is set a selection of environment variables such that your current terminal session is pointing to the docker environment in the virtual machine. This means all the images you have locally won't be available and vice versa.
2. Changing Back To Your Local Docker Environment
The simplest way to change back is to create a new terminal session, however if you're in a script or want to continue to use your current session the magic runes to swap you back to your local docker environment is:
unset DOCKER_API_VERSION DOCKER_TLS_VERIFY DOCKER_CERT_PATH DOCKER_API_VERSION DOCKER_HOST
3. Swapping Between Minikube And Other Clusters
Cluster autentication is managed via config
files in ~/.kube/
. When you run minikube start
it overwrites ~/.kube/config
by default, at the moment there is no command I have found for regenerating the config apart from starting the cluster again.
4. Kubernetes Doesn't Come Fast Or Easily
The current thinking is becoming proficient with Kubernetes is a 3 to 6 month journey. If you're not planning on running Kubernetes in production you can get simple local container orchestration with docker compose
.
Patterns
1. Script Your Local Build And Local Run
With microservices you can reduce the context switch penalty by providing a common interface for every repository. I've found having the following 5 scripts in every repository sames time and many headaches:
build.sh
This script builds the artifacts, i.e. this is where you callmvn clean verify
ornpm run build
If you have a lot of artifacts generated from a repository it can be useful to make it so you can pass an artifact name to the script and build that in isolation for quicker feedback.local-build.sh
This script idempotently sets up the local environment, i.e. make sure minikube is running, make sure minikube has required dependent resources configured.local-run.sh
This script deploys Kubernetes manifests to Minikube, either via Helm, Kubectl or some other tooling. Also you can run integration tests here.local-nuke.sh
This script remove the resources from Kubernetes and deletes the Docker image artifactpublish.sh
This script takes your artifacts and publishes them to your artifact repository.
This leads into the core of your continuous integration configuration being build.sh && publish.sh
2. Keep Your Containers Small
Using multi-stage Docker builds, i.e one container to build and one minimal container for runtime, will keep your feedback cycles fast. For example, with Java your build
container can have the JDK installed with all your testing dependencies, but then you can create a JRE only Alpine container that can be about 5-10x smaller.
3. Have Your Container Entrypoint And Command Be Targeted
The containers you are expecting to run in your cluster should be single purpose by design, this means that you can have your entrypoint and command set so you don't have to manage that outside the Dockerfile itself.
4. Exposing Ports
e.g. I want to see my postgres database
kubectl port-forward $(kubectl get pods | grep db | awk {'print $1'}) 5432:5432
This is an example command for accessing a Postgres database on the cluster. If we break it down:
-
kubectl port-forward {pod-name} source-port:destination-port
This command sets up the port forwarding so the given pod port is accessible on
localhost:destination-port
-
kubectl get pods | grep db | aws {'print $1'}
This command lists all the pods, selects the line that has the term
db
and then prints the first column of data, i.e. the pod name.
5. Alias Away Common Command Chains
Minikube, alas, is not perfect which means you have to restart it from scratch on occasion. Also when iterating on the scripts from pattern #1 it's helpful to make sure they work on a fresh machine.
alias reset-minikube='minikube delete && minikube start && helm init
Also for iterating on your integration test feedback loop:
alias warm-reload='./local-nuke.sh && ./local-build.sh && ./local-run.sh'
And for managing the docker environments:
alias local-docker='unset DOCKER_API_VERSION DOCKER_TLS_VERIFY DOCKER_CERT_PATH DOCKER_API_VERSION DOCKER_HOST'
Anti-Patterns
1. Every Project In A Repository Should Be Testable In Isolation
With Docker it is so trivial to spin up a database container that people can make the mistake of making all their component tests require an actual database to function. This complicates matters significantly especially when you add parallel docker environments between minikube and your local machine.
2. Don't Rely On Being Able To Run Your Entire Stack Locally
When you have a small number of services it is possible to run them all locally at once, however this stops being the case quickly and is more difficult to retroactively fix then to stop this at the start.
The need to do this is generally driven by insufficient testing, or testing at the wrong isolation levels. For testing across repositories leave that to Continuous Delivery.
Wrap Up
Hopefully this post will have saved you many frustrations in the road ahead to Kubernetes adoption. Minikube is fantastic for keeping your feedback cycles fast and learning its quirks is well worth the time investment.
The next step is getting to grips with your Kubernetes flavour of choice, be that cloud vendor managed or for those stout of heart a bare metal installation. Stay tuned for posts on adopting EKS, up-skilling the team into Kubernetes, and making your cluster production ready.