How I designed a CI/CD setup for a Microservice Architecture at zero cost

Read first:

• PART 1: How I designed and deployed a scalable microservice architecture with limited resources
• PART 2: Setting up CI/CD Pipeline for a Monolithic Service

In the last part we set up our pipeline for a Monolithic Architecture, but that’s not what I promised in the first part of the series.

Let’s get going…

Off to decoupling our services, so that they can live freely once again.

Well, that’s easy. Create a separate repository for each service, copy the workflow files to each of them. You are done! Ok, bye.

No, definitely not.

You can use that setup if all you have to do is run unit tests, but what about integration testing. Can’t do that on the production instance, and you are a struggling startup; you don’t have enough resources to spin up more instances just to run integration tests.

Well?

Ok, we will be putting each service in its own separate remote repository, but we will have one parent repository that refers to all the services’ repositories.

Let’s get started with git submodules…

Git Submodules are a way of adding a git repository inside a git repository. All submodules point to a commit in the remote repository.

Want to read this story later? Save it in Journal.

The original intention behind git submodules was to keep a copy of a certain commit (or release) locally on which our project might depend.

You just need to run:

git submodule add <my-remote> <optional-path>

However, we need them to keep up-to-date with our services’ repository, that’s why it’s a good thing that we can make them point to a certain branch instead too.

git config -f .gitmodules submodule.<submodule-name>.branch <branch-name>

Now, you can keep all your submodules up-to-date with just one command

git submodule update --remote

Now, that git submodules are out of the way, let’s get to the actual “good” stuff.

Ok, let’s talk about the actual workflow that I follow:

• Each service’s repository contains its unit test
• After a commit is pushed to the service’s repository, it runs its unit tests.
• If all the unit test pass, then we commit it to the parent repository.

<submodule>/.github/workflows/test-and-push.yml

• The parent repo lists the submodules where the changes were made since the last push and only deploys those services again.

<parent-repo>/.github/workflows/deploy.yml

This way all the other services keep running without any disturbance, while one of the service is updated.

<parent-repo>/scripts/deploy.sh

I also created separate Deployment Groups for each service of the name: <service-name>-prod, i.e. for nodeauth service, I created a nodeauth-prod deployment group, with rest of the configuration same as we did in the previous part.

We also do need to modify the appspec.yml and our scripts.

• Since each service is a separate deployment, we need to put the appspec.yml in each service's repository.

• Since we have decoupled all the services from each other, we no longer run them using sudo docker-compose up, each service has to be started individually.

<submodule>/scripts/start_app.sh

• I wrote a bit complex script so that we can use the same set of scripts in all our services instead of writing them again and again as new services are added.
• The scripts can also be added as a git submodule to all the services’ repositories, which makes it easier to maintain (but they weren’t in my setup at the moment of writing this blog).

• init.sh contains code to install docker on the instance (if not already present).
• cleanup.sh contains code to remove the previous unused containers.

That’s it. You are finally done. You’ve got your own weird-ass setup to test and deploy a Microservice Architecture at zero cost. You can also keep the previous docker-compose.yml to maintain a local development setup.

(The single instance’s cost is covered under AWS free-tier).

disclaimer: Yes, there is probably a better way of doing this. I hope there is.

📝 Save this story in Journal.