Continuous Integration and Continuous Delivery with CircleCI and NestJS

Photo by Mohammad Rahmani on Unsplash

“Investing in yourself is the best investment you will ever make. It will not only improve your life, it will improve the lives of all those around you.” — Robin S. Sharma

A little storytelling…

As a software developer, I understand the importance of upskilling oneself, especially in a fast moving industry like the tech industry. I believe that once you stop learning, the world just leaves you behind.

After I built my first project with the NestJS framework, I enjoyed what I had achieved to the point of writing an article about my experience. I won’t say the project is a fully developed API, but learning a new language and a framework was a win for me. If you’re interested in the article on the experience, here it is:

I wrote a REST API with TypeScript… here’s the story.

After I was done with the project, I decided I was going to build another one with Nest that was a little bit more complex, and also learn at least one technology on the side. I’m working on that second project at the time of writing this article. So I have been hearing about this thing called CI/CD, even while I was writing the previous NestJS project.

Backend gurus were always talking about it on Twitter, YouTube, blog articles, and everywhere else. I decided I was going to learn it for my next project, which is my current one. I read some articles on the topic and learned that there are different platforms for CI/CD: Jenkins, TravisCI, CircleCI, and GitLab CI/CD.

The first platform I tried to learn was Jenkins. Did I succeed in learning it? Absolutely not. Jenkins looked so complicated to me; different articles told me different things when I started out with it. Some said I should install some packages in order to use it, and some gave different configurations. I was very confused; it was like learning Java, so I had to drop it and try another. I tried to learn Travis CI; I can’t really remember my experience with it, but I remember dropping it because I read somewhere that I had to pay for something. I can’t really remember what that was. For GitLab CI/CD, I figured I had to use GitLab to use its CI/CD feature, and I was using GitHub, so that was the end for GitLab.

CircleCI was my last option then, because I didn’t even think of GitHub Actions. I knew CircleCI was it when I watched their YouTube introductory video at that time. The animation and the voice of the presenter were soothing. Yeah, I judged a book by its cover. I was even more convinced to try CircleCI when I opened its documentation page; it was clean. I learned the basics and applied them to my project, and that’s what I’m about to share with you.

To follow along with this article, here are the requirements.

• CircleCI account
• Git and GitHub account
• Docker (Optional)
• DockerHub account
• NestJs
• Jest

Introduction to CI/CD

Before I give details on how to use CircleCI in your own Nest project, let’s go back to the basics. It all starts with the definition of CI/CD. CI/CD is the acronym for Continuous Integration/Continuous Delivery (or Deployment).

The "CI" part of it is a software development practice that integrates code changes to a repository on a frequent basis, meaning that a developer or group of developers continuously adds or edits code to get the defined software that is needed, to a repository that stores that code, until the “perfect” software is created. This practice involves the automated building and testing of source code once a commit is made to the repository.

For the “CD” part, there are two sides to it: Continuous Delivery and Continuous Deployment, so it depends on which one you choose to work with on your application. Continuous Delivery is simply the manual process of moving your code from the development environment to the staging or production environment, after successful builds and tests. This means that after you’ve added your code for whatever feature, you build and test it with a CI/CD platform. If all goes well, you will then take that software and manually configure it for a different environment that is not the development environment. Continuous Deployment is the exact opposite. While Continuous Delivery involves human interaction with a software for the production release, Continuous Deployment is totally automated. This means that once all tests are passed, it is automatically deployed to the production environment.

Automating Tests with Nest and Jest

sweet rhymes

Automated testing is the type of software testing that is done by running scripts to ensure that software meets its requirements and exhibits the behavior that’s needed from it. There are frameworks that can help one run automated tests, and this is based on the programming language that is being used to build your software. With Python, there are unittest and pytest. With JavaScript/Typescript, we have Jest, Mocha, and so many of them. For the nature of this article, we will be focusing of Jest.

For a framework like Nest, it comes with Jest out of the box, meaning that you do not have to install it in order to run automated tests. Jest automatically runs tests from files that have a suffix of .spec or .test. Running tests with Jest can be done with the command:

npm run test

However, if you want to run your test in watch mode, which automatically runs tests in real time as changes occur, you can use the command:

npm run test:watch

Unit Test Demonstration:

I created a dummy blog API to demonstrate this. The API is hooked up to a PostgreSQL database to create and read a single blog resource.

One thing I love about Nest when it comes to testing, is that it automatically creates test files for your providers and controllers. That way, you don’t have to bother so much about structuring your tests. For the blog’s service and controller, there are “blog.service.spec.ts” and “blog.controller.spec.ts” files, respectively. Running the test command, you will get the following error messages:

This is because both the service and controller classes have dependencies initialized in them, and these dependencies are not found in their respective test files. For the blog.controller.spec.ts file, we see that the blog service dependency is missing, and for the blog.service.spec.ts, we see that the blog repository dependency is missing. One of the best ways to fix this issue is by mocking those dependencies. This way, you provide fake versions of those dependencies without relying on the actual dependencies for your tests.

For the mocked repository dependency of blog.service.spec.ts, you can have something similar to:

And for the mocked service dependency of blog.controller.spec.ts:

Running the test command again, you will see that all tests get passed:

With this little demonstration, one can add custom tests to the foundation already provided by the NestJS framework.

The World of CircleCI

In order to use CircleCI on your project, you should have at least a basic understanding of the platform. It is simply a tool that automates the build, test, and delivery/deployment processes. Yes, you can run your tests on your local machine, but it is also advisable to add tests to your CI/CD pipeline. CircleCI is made up of several components that help execute CI and CD processes. Some of the basic components includes:

• Pipeline
• Jobs
• Steps
• Commands
• Executors
• Workflows

Pipeline: A pipeline is a set of automated development procedures that take place on a CI/CD platform. It is made of build, test, and delivery/deployment processes. The platform compiles the software into executable code so that tests can be run, and then deployed to a remote environment.

Job: A job is a collection of steps that run required commands and scripts within a given executor and are orchestrated using workflows.

Step: A step is simply the collection of executable commands.

Executor: In CircleCI, an executor is an execution environment, in which jobs are executed. Different types of executors include:

• Docker
• MacOS
• Windows
• Machine

With the Docker executor, you can containerize your application and push it to any container registry of your choice, right from CircleCI.

Workflow: The workflow is the component of CircleCI that orchestrates and manages jobs. With it, you can define the order in which you want jobs to be executed. Look at it as the Kubernetes of jobs.

There are a lot of CircleCI components, but with these very few, you can apply the CI/CD principle to your project.

Setting up CircleCI

Like I mentioned earlier, you will need a CircleCI account to use it on your project. This can be done by following this link. It should take you to a page similar to this:

Signing up with your favourite Git hosting service allows the CircleCI server to have access to the repositories it will be working with. This access is necessary for CircleCI to build and test your code, as well as to deploy it to a remote environment.

After creating an account, you should see a list of projects (repositories) from your Git hosting service. Choose the one you want to work with, and set up the project with CircleCI. In my case, it is “Blog-Demo”

To define a CI/CD pipeline on CircleCI, you will need a config.yml file. This can be done in any of the 3 ways as shown in the picture below:

Personally I prefer a manual setup of a config.yml file, because CircleCI said it is the fastest way… yeah. But also because I can choose to set up my pipeline however I want, from scratch. You can choose whatever method you want, and it will work just fine.

If you choose the “fastest” way, you will need to create a config.yml in the root directory of your project. This is how it is done:

1. Create a folder called .circleci in the root directory.
2. Create a file in that folder, and name it config.yml

The file structure should be similar to what is in the picture below:

Here is an example of what your config.yml file can contain:

From the above configuration, we are working with CircleCI version 2.1. A job named “blog_demo_deployment_process” is created, and it uses the Docker executor with a Docker NodeJs image provided by CircleCI. This is similar to running an application in Docker with a NodeJs base image, in your local machine. You can see a list of all CircleCI Docker images by following the link below:

Images

The job then builds your application by git cloning the updated code repository into the CircleCI environment with the “checkout” property, every time you push a commit to the repository. The latest npm version is then installed in the environment with the “npm update” step.

sudo npm install -g npm@latest

All automated tests in the code repository are then executed with the “run automated tests” step.

npm run test

The workflows section contains the workflow named "blog-demo", which lists the jobs it contains. In this case, it only has the “blog_demo_deployment_process” job.

The next step is to push the config file to the Git hosting service. Then go to the CircleCI projects dashboard again and set up the project. Choose the fastest option.

From the image above, I created a CircleCI config file on the testing branch. It can be on whatever branch of your choice. Click set up project button.

You should see a page that looks like the following picture:

One feature CircleCI has is that whenever a pipeline fails, maybe as a result of packages not being installed or failed tests, you get an email notification telling you that the pipeline failed.

Clicking on a failed pipeline, you can see that the process for running tests failed because the Jest framework is not installed in the CircleCI environment.

This may not be the same case for everyone, but to fix this issue, you will have to add the command that installs Jest, to the config.yml file. This is the version of the config.yml file that works out.

Create a commit, push it, and you should see a clean pipeline.

Here is one thing you should know: sometimes your pipeline may fail because of an error in the config file. With the CircleCI CLI, you can validate your pipeline configuration before pushing it to your Git hosting service. This can be done with the command:

circleci config validate

To install the CLI on your local machine, click the following link:

Installing the CircleCI local CLI - CircleCI

So far, our pipeline has only been able to build and test the blog-demo API, but we need it to deploy the code to a remote environment, for it to be complete. In this case, we’ll be delivering the code to DockerHub. Yeah, this is where you need a DockerHub account.

To containerize and push an application to DockerHub, one needs to build a Docker image of it, login to the DockerHub account from Docker, and push the image. But how can you do this? This is where another useful component of CircleCI comes in: Contexts.

Contexts provide a way to use environment variables in the config.yml file. These contexts are defined in the CircleCI web application and then called in the config file by using the context key in the workflow section, to be used in the build process on the CircleCI server. With context environment variables, you can add your DockerHub authentication credentials, which will enable CircleCI to push your image to DockerHub.

Steps in Creating a Context

1. Right on the side navigation panel, you should see an “Organization Settings” label, click it.

2. This should take you to another page that has “Contexts” also on the navigation panel. Click it.

3. Create a context by clicking the “Create Context” button, and use a unique name for it. After creating the context, you should see it in the Contexts list. Click it.

4. Click the Add Environment Variable button to add an environment variable to the context, provide the details, then click the Add Variable button to save. You will need to add two environment variables named DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD, or however you want to name them. These are your DockerHub credentials.

Next you will have to update your CircleCI config.yml file

In the above configuration, we added our DockerHub credentials to the auth property of the NodeJS Docker image. Down at the bottom of the file, you are telling CircleCI the context in which it should read your credentials. Now that the credentials have been provided, you can now deploy to DockerHub.

Update your config file to look like this.

In the above configuration, a new property, setup_remote_docker, was added. This is the property that helps you run Docker commands in the executor. This property has a child property, docker_layer_caching, which is set to true. With this property, your Dockerfile is cached in the job. This means that if there are no changes in the Dockerfile, the image will be rebuilt from the previous build thanks to cache. This minimizes the amount of time needed to build the image.

In the “Build and Push Docker Image” step, the image is built and tagged in the current directory. Docker logs in to your DockerHub account with the credentials you provided, and you push that image to your account.

Create a commit, push it, and you should see a clean pipeline.

Open your DockerHub account, and you should see your newly deployed image.

Following this article, I pushed the image from the testing branch. But what if you only wanted to build and run tests in the branch and then push the image from the main branch when you accepted a pull request? Is that possible? Yes, it is, with conditional branching.

This is how the config file should look, if you wanted to do something like that.

In the above configuration, the image is built and pushed only if it was the main branch that triggered it, with pipeline values and parameters.

Create a commit, push it, and you should see a result similar to the following:

You will see that the pipeline process stopped at running automated tests on the testing branch.

Create a pull request to the main branch and merge.

This triggers the pipeline, and you should see your result, right on the main branch.

In conclusion, Continuous Integration and Continuous Delivery are practices that help teams streamline their software development process by automating the building, testing, and deployment of code changes. By doing so, teams can detect and fix issues early on, reduce the risk of errors in production, and deliver new features and updates faster and more reliably to users. With the help of tools like CircleCI and Docker, implementing these practices can be made easier and more efficient.

Please feel free to reach out to me on LinkedIn or Twitter. See you next time.

Till then, keep coding.