Implement contract testing with Pact to get rid of bugs due to front-back and back-back communication
jeanlucc10 min read
Image from the Pact documentation
Exec summary
Modern software development organizations scale their development efforts by spreading the development of a system across different teams. Such projects can have team work issues because interconnected applications that change often, create a lot of errors breaking other teams’ application. A typical problem due to a large organization is when the backend team deploys a change that is not compatible with the mobile app and breaks it.
Contract testing is a way to make sure that several applications developed by collaborating teams can understand each other. With no need to take extra manual care when modifying code, the teams avoid human errors, get rid of bugs, and on the way, develop faster.
With contract testing, a team can easily create a test checking that the application they are developing receives viable responses from the provider application that another team is developing. On their side, the provider application team only has to run a single automatically generated test. This allows both teams to know if something is broken before they deploy their applications: they can talk quickly about this precise problem, with little rework cost and no bug for the users.
Pact is the main framework to implement contract tests, and it works with many languages.
For Whom?
In order to avoid creating bugs coming from miscommunication between two parts of your applications you can set up different tactics, contract testing is one of them. Here are some alternatives with an evaluation of how they perform. You can find some comments on these alternatives in the appendix at the end of the article.
| Alternative | Maintenance cost | Setup cost | New interaction cost | Learning curve | Efficient |
|---|---|---|---|---|---|
| End-to-end testing | $$$ | $$$ | $$ | $$ | ++ |
| Shared types | $ | $$ | $ | $ | ++ |
| Versioned API without breaking change | $$$ | $$ | $ | $$ | +++ |
| Massive documentation and conception | $$$ | $ | $$$ | $$$ | +++ |
| Contract testing | $ | $$ | $ | $$$ | +++ |
All these tactics can be used together where it is possible.
In fact contract testing needs some conception to create the contract or update it. But contract testing allows to have an accurate documentation when you want to do this conception. Once you discover a breaking change you still have to take a decision on how to handle it and versioned API are a good way to do it but the contracts will show you what you do not need to maintain because no one uses it anymore.
Conclusion: if you develop a distributed system communicating via HTTP APIs with heterogeneous technologies you should find someone to help your team set up contract testing.
Benefits
Benefit #1: Defects are stopped before wasting others time
Usually to achieve testing of several services interacting, we use some kind of end-to-end testing. This means you can test only after deploying your service to put it on the same environment as other services (e.g. staging, preprod, CI). What is called end-to-end testing here could be called broad stack tests, it matches the definition found here. If there are several teams, it becomes difficult to have someone responsible for end-to-end tests.
With contract testing you can test directly in your continuous integration (CI) workflow which means no one else than the developer invested time in checking the work done, this allows problems to be caught earlier. (cf. process just below)
doing > manual local test (few minutes) > push (few seconds) > CI (few minutes) > code review (some minutes) > merge (few seconds) > deploy (few minutes) > manual test on staging > validation by the PO
Benefit #2: Developers know their mistakes and correct them quickly
As you can see on the process above having the result in CI instead of after deployment creates a faster feedback loop for the developer and he can correct it before the merge which means corrections are done faster also.
Let me introduce the pyramid of tests briefly, there are several kinds of tests with different drawbacks, the idea is that even if useful, the more drawback you have, the less tests of this kind you do.
Image from the Martin Fowler article Test Pyramid
The main advantage is that contract tests are lower on the pyramid of tests (between unit for the cost and service for the speed) than end-to-end tests (UI). It means they are faster and cheaper, so we can test more cases which gives quickly more confidence for less money.
Benefit #3: Contract tests are easy to create and maintain
With the pyramid we saw that contract testing is cheap compare to the end-to-end alternative. Our experience is that once set up the cost is close to unit tests (only slower) which are widely used on most projects.
Each test, tests exactly one case, this means they do not change often (compared to end-to-end tests where any change can break the test) and that a change is easy to do in the test because the complexity is low (like a unit test).
Benefit #4: Contract tests give auto-documented API usage
Thanks to Pact that registers all your contract tests, you know precisely what are the interactions tested in your project. This is up to date even before going to production and you can easily visualize it from Pact.
Services interactions:
Image from the Pact documentation
Details of an interaction:
Image from the Pact documentation
Trade-offs
Trade-off #1: It is useful only when several teams work together
Tests are used to perform check that are hard to remember and take a lot of time to perform manually. For contract tests this means they are less useful when you only have one or two teams as you usually know exactly what the other team uses.
They remain relevant but the cost for an inexperienced team on how to use them could create more problems than it solves at first. For an experienced team the cost is not an issue making them a practical solution even to test the interactions between one frontend and its API developed by the same team.
Trade-off #2: The setup is quite complex at first
To make contract tests work you need several days of work.
- All teams need to understand the core concepts and how to write good tests with the good level of testing.
- You need to add a pact broker in you environment (meaning you have to deal with security issues and keeping it running) this is mitigated by the fact the provided docker container works fine.
- You need to make sure your providers applications run in your CI. By default there is no need to have a running instance in your CI but this is quite easy to do with docker.
- You need to find a way to handle state in your provider apps to simulate all kinds of responses for instance you need to simulate your own dependencies to make the test reliable. Depending on the complexity of your dependencies (number of dependencies and number of possible outputs) this can be more or less difficult.
My Opinion
You can benefit of contract testing in many situations. Once you have overcome the initial cost to set up and understand how you should work with it for a big project it can even be a cost effective solution for smaller projects in your organization. So join the Pact users’ community to improve your test stack and help your teams to ship faster. You are not the first, many companies like Atlassian use this tool already as they explain it in this video.
Go Further
To know if you can get benefit from Pact you should read this page of the documentation, in case of doubt read this. Then to learn how to use it you should begin here and here. To know where these tests should be used you can read the excellent article The Practical Test Pyramid which describes most kinds of test you can use in a project.
Appendix
If you want more details, here are some elements I used to create the evaluation table of the alternatives in the “For Whom?” part:
End-to-end testing
Pros:
- When the tests is green you are sure everything works together since the tests checks everything at once
Cons:
- The cost of maintaining an environment to test everything.
- Test tend to be very flaky due to external causes.
- You need to run all tests (and they are slow) before the real deployment of any part, which creates a congestion in your teams workflow.
- The coverage of error cases costs even more because you need to set up your system to fail in expected ways and because each case is quite long to execute you tend to have few.
- When you discover a real bug, you still need to work to find the root cause (slow to fix)
Use: When you want to check your most critic happy path and you have a way to have enough control on your system in an environment before prod (set up mocks for external services, set up well known data…)
Shared types
Pros:
- Easy to set up
Cons:
- You do not have the vision of what versions work together, this force to deploy the different pieces together which is a smell for down time.
Use: When you have a small enough system that you can deploy all at once and a technical way to share code between consumers and provider.
Versioned API without breaking change
Pros:
- The problem cannot exist
Cons:
- Maintenance cost is high, you either have to live with the burden of all your previous choices or find a way to know what is not used anymore (one way is a deprecation plan of older versions).
- Risk of human error and still create a bug
Use: For public APIs this is a good solution as there is no way to contact each consumer directly.
Massive documentation and conception
this kind of massive
Pros:
- Reduces the amount of bugs of any type.
Cons:
- Forces you to make the perfect conception, keep track of every decision, to have comprehensive checklists and to train your whole team to follow all items.
- Your process needs to be at the highest level, like the NASA for space shuttles.
Use: When lives depends on your software working perfectly
Contract testing
Pros:
- Catch bugs between teams before deployment
- Easy to maintain
- Teams can deploy at their own pace
- Documents expectations of all consumers and ensure they are met with no human intervention
- Keeps track of what versions are compatible
Cons:
- You need to carefully set up your workflow to integrate these tests.
- You need to create tools to handle complex cases.
- There is yet another tool to teach to your team.
- You still need other kinds of tests to check the business rules of each application work as expected
Use: When you want to quickly develop a system with more than two interconnected applications (if you do not control the update of some applications like for mobile, each used versions count). And each application is developed by independent teams which communicate frequently. It can be better to have someone to train your team to reduce the setup cost.
