Working at a small, young and frugal non-profit, I do not have the luxury of specialised testers giving new features a run-through before they go live. Nevertheless, because our code runs on the websites of other organisations, we have to make absolutely sure that it will not break. At the same time we want to be able to iterate quickly as we adapt to a rapidly changing environment. We manage to balance these two requirements by a variety of measures. Combined, they result in an excellent developer experience, in which I can regularly and confidently deploy code to production.

In this post, I will discuss our approach, starting with the relatively standard elements, building up to the more exciting cutting-edge techniques. All of this uses freely available open source projects.

## The basics: a build pipeline

Like many projects, we have set up a build pipeline: a set of tasks that get run every time code gets pushed to the central repository. This pipeline runs a few basic checks before and after code gets merged into `master`: does the code build successfully, do all unit tests succeed, and what code is covered by them? If a pipeline fails, I receive an email notification. If there's a related open merge request, it will display a warning before merging.

This pipeline already removes a lot of potential for human error. For example, if I forget to run unit tests locally and I break some, that code will not make it into the main codebase without me noticing. Likewise, I could forget to check in parts of my code. Even though tests would still succeed for me locally, the pipeline would fail, and I will immediately be notified. This is vastly preferable over distributing incomplete and broken code to other developers or, worse, users.

Artifacts like e.g. the code coverage report produced during the pipelines are also visible to all team members.

![Screenshot of a code coverage report, displaying a few lines of code that have been run during tests.](/img/fearless-deployments/coverage-report.png)

Our pipelines are powered by [GitLab CI/CD](https://about.gitlab.com/product/continuous-integration/).

## Review apps

Review Apps are the backbone of everything that is to come. The basic concept of a review app is spinning up an entirely new and disposable production-like environment for every branch, with that branch's code deployed to it. My basic workflow is to create a new branch whenever I start solving an Issue, straight from the GitLab interface.

![Screenshot of the GitLab interface, creating a branch called `98-ci-example`](/img/fearless-deployments/branch-from-issue.png)

These branches are automatically named using the format `<issue number>-<issue description>`, e.g. `98-ci-example`. Upon creating that branch, a new pipeline automatically runs, deploying the code in that branch to https://98-ci-example-review.plaudit.pub.

![Screenshot of the Plaudit website deployed to https://98-ci-example-review.plaudit.pub, next to the Issue it was created for.](/img/fearless-deployments/review-app.png)

The advantages of review apps are numerous. It allows me to easily verify whether and how my code works in a clean environment. It tells me that deployments still work as expected, and hence will likely do so when I deploy my code to production as well. And finally, it enables a variety of automated sanity checks on the live application, such as checking for performance regressions or potential security issues.

The primary tool used to provision these new environments is [TerraForm](https://www.terraform.io). It allows you to define the infrastructure you need using a configuration file, and to pass the relevant branch name as a parameter.

## End-to-end tests

Of course, I _could_ visit the deployed review app to check whether everything's OK... But why go through the effort when that, too, can be automated? After a review app gets deployed, the build pipeline automatically fires up Chrome and Firefox. They will then visit the newly deployed app, try to run through a few basic scenario's, and then report on whether they were able to do so successfully.

![Screencast showing the progress of a single end-to-end test running in Firefox.](/img/fearless-deployments/end-to-end.webm)

These end-to-end tests buy me two things. First, they can warn me when a deployment was unsuccessful. Even a simple test that literally just visits the page and checks whether a single DOM element is present is enough to tell you whether the new infrastructure was successfully set up, the code was deployed successfully, and the server is running as expected. Second, they run some basic tests trying to navigate the app by keyboard - an important accessibility concern that is easy to mess up.

The end-to-end tests use [Selenium](https://docs.seleniumhq.org) for browser automation, and [WebdriverIO](https://webdriver.io) to run the checks.

## Storybook

Our primary product is a widget that can be in many different states, and adapts its looks to the space it's allotted. It is therefore useful to be able to see what the widget looks like in all those different variations, without having to reproduce all the steps required to get the widget into those states. To facilitate this, with every review app we also deploy a storybook.

![Screenshot of a storybook showing our "endorsement" widget, with links on the left-hand side to switch between states like "No endorsements", "One endorsement" or "More than three endorsements".](/img/fearless-deployments/storybook.png)

Using the storybook, it's just a single click in order to inspect the [various states](https://medium.com/swlh/the-nine-states-of-design-5bfe9b3d6d85) the app can be in: no data, some data, too much data, logged in, logged out, etc. This makes it very easy to quickly check the impact of CSS changes on the widget in all its different faces.

The storybook is made possible by the aptly-named project [Storybook](https://storybook.js.org).

## Visual regression tests

Still, having to manually inspect the widget in all its various states on every change is rather time-consuming. Luckily this, too, can be automated. Whenever a pipeline is run, the storybook is opened in a browser, and screenshots of the widget in every state are compared to checked-in reference screenshots. When the two diverge, the pipeline fails and a report is generated highlighting the differences between the deployed widget and the reference image.

![Screenshot of a report on a visual regression test, displaying one failed test case showing three images: the reference image, the taken screenshot, and the two overlapped with their differences highlighted in pink.](/img/fearless-deployments/visual-regression-test.png)

This will make sure that I receive a notification whenever I alter the layout of the widget, which is very helpful when those changes were unintentional. For example, I recently found out that Chrome does not support `<button>` elements acting as [flex containers](https://developer.mozilla.org/en-US/docs/Glossary/Flex_Container), due to which a supposedly minor accessibility change ended up messing up the layout in a specific state.

The regression tests are executed by [Gemini](https://gemini-testing.github.io).

## Wrapping up

Combined, all these tools provide me with a lot of reassurance whenever I am ready to push my code to production. They have prevented repeated occurrences of actual problems I have run to in the past, and seeing a row of green checkmarks before merging into `master` provides a lot of reassurance.

![Screenshot of a merge request report, showing successful security, performance, code quality and regular tests.](/img/fearless-deployments/code-checks.png)

These tools were added incrementally over time, in response to actual encountered problems. I hope that showing what's possible using off-the-shelf open source tools will encourage you to investigate what could be possible and useful in your own projects as well. The exact setup will vary per project, but you're welcome to look at [ours](https://gitlab.com/Flockademic/plaudit/blob/master/.gitlab-ci.yml) as a starting point!

[![License](//i.creativecommons.org/l/by/4.0/80x15.png)](//creativecommons.org/licenses/by/4.0/)

_This work by [Vincent Tunru](https://vincenttunru.com/) is licensed under a [Creative Commons Attribution 4.0 International License](//creativecommons.org/licenses/by/4.0/)._


I describe a number of measures that help me to regularly deploy to production without fear of my code breaking.

Fearless deployments


When it comes to unit testing, there are three schools of thought that I'm aware of:

1. We don't do unit testing, since they are overrated/too hard/take too much time/whatever.
2. We only test critical/error-prone parts of our code, and bugs that have been fixed.
3. We want a significant part of our code to be covered by tests.

I'm usually in the latter camp: I think that automated tests are one of the most effective ways to ensure code quality on a long-running project.

This strategy, however, is often accompanied with a policy of maintaing at least a certain level of code coverage. And since there are usually some parts of the codebase where the effort required to bring them under test is considered not to outweigh the benefits, unit test policies usually stipulate a minimum amount of code coverage of 80%, 90% or 95%.

I'd like to argue that best practice is to require code coverage analysis to report a coverage of 100%.

## Why not <100%

> When a measure becomes a target, it ceases to be a good measure.

—[Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law)

The risk of _requiring_ a certain level of coverage is that achieving that level becomes the goal, instead of the goal being to write proper tests. And what is the easiest way to achieve 80% code coverage? Simple: write tests for the 80% of your code that is easiest to test.

However, you do not want to test the code that is _easiest_ to test: you want to test the code that is _most important_ to test. In other words: whether a certain line of code is worth the effort of writing a test should be consciously decided, rather than certain parts not being tested simply because the other parts were enough to achieve the required coverage metrics.

## Reaching 100% without losing your mind

Code coverage reports should offer guidance, not a goal to meet. More specifically, they can help you find out which parts of your code you actually forgot to test. It regularly happens that there are some unhappy paths that I did not think of when writing tests, but of which it was obvious that they would need to be dealt with when writing code. When anything below 100% coverage is configured to be insufficient, my testing tools will then point out to me that those parts still need a test.

Of course, it is still likely that there are parts of your code for which the benefits of testing do not outweigh the costs. However, rather than lowering my coverage target by some arbitrary number, I [mark those parts as irrelevant for the coverage report](https://github.com/gotwarlost/istanbul/blob/master/ignoring-code-for-coverage.md), _with a comment explaining the reasoning behind not testing it_.

In other words: I don't want 100% my code to be covered by unit tests, but I at least want 100% of my code _to have been considered_ for unit tests. That way, what to test is still left to the programmer's best judgment, and code coverage analysis becomes helpful rather than yet another tool to satisfy.

Now, I haven't seen the above mentioned as best practice before, so I'm very much interested in what people have to say about it. That said, regardless of the testing strategy you use, I think developer buy-in is always a good starting point. The tooling we use should help us achieve our goals, rather than set them for us.

[![License](//i.creativecommons.org/l/by/4.0/80x15.png)](//creativecommons.org/licenses/by/4.0/)

_This work by [Vincent Tunru](https://vincenttunru.com/) is licensed under a [Creative Commons Attribution 4.0 International License](//creativecommons.org/licenses/by/4.0/)._


Testing