Testing microservice architecture locally is difficult. At Nylas, we’ve created a local development environment using Minikube and Tilt. Let’s take a look at how we enable local testing for microservices at Nylas.
The problem of local microservice architecture testing
Microservice architecture is a good design practice, but testing it locally is very difficult. Let’s take a look at why.
A simple microservice architecture may look like this:
You can see several components in this architecture:
An API service which deals with API requests from clients.
A message queue that acts as a message broker.
Some async services which perform tasks asynchronously.
Some data layers that store application data.
However, microservice architecture in reality is a lot more complicated. Here is a more realistic version:
First, there are many more services in the real world than in the simple architecture we saw above. Each business domain has its own microservice ecosystem. Each product can have its own API service, message queue, async services, and databases. Once a company adopts microservice architecture, the number of services grows exponentially.
Second, business domains often cross boundaries. A service in Product A sometimes reads from a database defined for Product B. This is not necessarily a bad design, because business domains always correlate to each other. For example, when you do a Google web search, the search API will inevitably also call Google Maps and YouTube. It is very hard to build a solid wall between products, where services from one product never call beyond business boundaries.
Third, not all services belong to a product. Some servicesare built as“platform services”, or “shared services”. Some examples are:
An authentication service that deals with user login. Most product services rely on this for user authentication/authorization.
An admin service built for the DevOps and support teams. Engineers can use the admin service to delete corrupted data in databases, or remove stale workflows.
An API gateway (or “edge service”): a service that routes client requests to different API services. The API gateway also deals with rate limiting, request logging, DDoS protection, etc.
It is very common for a mid-sized company to have hundreds of backend services. Spinning up every single service locally in a laptop environment becomes simply impossible. Microservice architecture poses some challenges for developers:
How do I test my change locally?
How do I make sure my change doesn’t accidentally takes down the entire system?
To solve these problems, we created a local developer environment for Nylas engineers.
Group microservices by “domain”
The foundation of a developer environment is defining a “domain”. A domain is a collection of closely related services. Service coupling inside a domain is much tighter than services across domains.
A domain is usually defined by product. For example, the Nylas Streams domain is defined by the Nylas Stream product (https://www.nylas.com/capabilities/nylas-streams). Inside this domain, there is a Streams API service, Streams async service, Streams database, and Streams message queue.
In this way, we can group microservices into domains.
Here’s what the architecture might look like before grouping:
And after grouping into domains:
At Nylas, code repositories are defined not by service, but by domain. Services that work closely together are stored in the same repository.
Therefore, our developer environment is built on a domain-by-domain basis. Each time our developers want to run and test a service locally, they will have to start the developer environment for the entire domain.
Let’s look at an example developer environment:
Inside the example above, we have the API Service, Async Service A, and Async Service B. They are the services we want to test.
Don’t forget: services have dependencies outside the domain. Some dependencies are provided by third parties: message queues (Kafka/RabbitMQ/SQS), databases (Mongo/Dynamo/Spanner), Cache (Redis), and so on. Other dependencies are in other domains; for example, an email service may need to call an authentication service, which is outside of the email domain.
The developer environment consists of all the services defined in a domain. It also contains emulators for third-party-dependencies and services outside of the domain.
At this point, we’ve defined the scope of our developer environment, but how are we going to run it locally?
The answer is Minikube and Tilt.
Minikube + Tilt
Minikube provides the infrastructure for our developer environment.
A local Kubernetes cluster is called Minikube. It can simulate Kubernetes running in production. You can create Kubernetes deployments and cron jobs in Minikube. These can be accessed via the Kubernetes CLI kubectl. For more about Minikube, see https://minikube.sigs.k8s.io/docs/.
Let’s have a look at the developer environment for the Nylas Sync domain, which has 14 services and 2 jobs running locally inside of a local Minikube cluster:
Each pod here is running a unique service. Services interact with each other inside of the same cluster.
Developers can see a list of services in the “Deployments” tab like “zoom-notification-server”, “graph-notification-subscriber”, “graph-metadata-listener”, and so on:
All of these are Nylas backend services, running locally inside the same Minikube cluster. But why don’t we use Docker Compose?
First, Nylas backend services interact with Kubernetes APIs. Some microservices create Kubernetes jobs, and some provision Kubernetes pods. There is no way Docker Compose can simulate Kubernetes internal APIs.
Second, Minikube can better simulate a production environment. People use Helm, not Docker Compose, to deploy services into production. We can use the same Helm template for our developer environment and production. The only difference is the Helm values. In this way, people can test not only their code, but also their deployment pipeline.
Besides Minikube, we use a wonderful thing called Tilt. Tilt is a developer tool to run a Kubernetes cluster locally. It offers smart rebuilds and live updates.
To set up developer environment, simply run tilt up. All services will be automatically deployed to Minikube. It also provides a dashboard where you can watch the progress:
Under the hood, Tilt does several things for each service:
Compile a service binary
Build the binary into a Docker image
Deploy service in Minikube via Helm
The most exciting thing Tilt offers is live-update. Every time a developer changes code in their IDE, Tilt will detect the change and automatically restart the service! The service restart only takes several seconds. The write-and-test iteration now is shorter than ever.
Most services have third party dependencies. A service may rely on Redis, PostgreSQL, Kafka, and so on. In order to set up a developer environment, we have to run third party dependencies locally.
If the third party service has an official Helm chart, we can simply install it in our Minikube. However, some third party dependencies cannot be installed directly. Services such as Google PubSub and Amazon DynamoDB do not have Helm charts available.
In regard to those services, we have emulators.
Here are some emulators Nylas uses for our developer environment:
gcloud is a Google command-line tool. It can emulate Google PubSub and Spanner.
LocalStack is a single service. It can emulate most AWS services, including DynamoDB and SQS.
Temporalite is a command-line tool. It can emulate the Temporal workflow engine.
We built these emulators into Docker images and wrote a Helm chart template. We can then deploy them to Minikube.
Here is an example Dockerfile for our Google PubSub emulator:
FROM google/cloud-sdk:alpine AS builder
RUN gcloud components install pubsub-emulator
COPY --from=builder /google-cloud-sdk/platform/pubsub-emulator /pubsub-emulator
RUN apk --update --no-cache add tini bash
ENTRYPOINT ["/sbin/tini", "--"]
CMD /pubsub-emulator/bin/cloud-pubsub-emulator --host=0.0.0.0 --port=8085
Here is a list of third-party dependencies Nylas uses:
Between official Helm charts and emulators, we have everything we need to run our third party dependencies locally.
Let’s take a look at our example developer environment again:
Services often interact with other services outside of their domain. An Email service can send API requests to an authentication service, but the two services are not in the same domain. If services in other domains are not running in Minikube, how do we test them locally?
The answer is: building fake services.
If the email service calls the authentication service, we will build a fake-auth-service just for testing. The fake service maintains the same API contract as the real one, but it is not doing actual authentication. The response of a fake service can be hard-coded.
The answer isn’t: deploy multiple domains locally.
We believe services should be tested locally within their own domain without requiring another one to also be running locally. Deploying multiple domains simultaneously in a Minikube cluster causes more problems than it solves.
The biggest reason is CPU and memory consumption. We experimented with deploying two domains of services on a 2019 Apple MacBook Pro. Our IDE froze, the laptop started heating, and the fans wouldn’t stop. It was quite a bad developer experience.
Integrate the developer environment with CI
With a developer environment running locally, we can write function tests. Here are some example function tests we can implement:
Call an API service, and then check a local database.
Publish an event to the message queue emulator, and then check local Redis.
Furthermore, function tests can be integrated with Continuous Integration (CI) using GitHub Actions. We found two GitHub Actions plugins quite useful: