Pact.io claims that it can fully replace End-to-End testing. I’m not so sure about that, but it’s an intersting tool!
End-to-End (or E2E) testing usually involves mimicking a user working through a flow in the UI using some kind of specialized tool like Cypress or Playwright. The frameworks can click buttons, navigate pages, fill form, and the like.
Pact claims it can replace end-to-end tests with it’s contract-based testing. In one talk, they specifically refer to consumer-driven contracts. APIs sometimes expose schemas specifying what endpoints and data structures they support. Consumer-driven contracts flip this model. The consumers of an API maintain a specification of what they need from the endpoints they call. Then, when the called API wants to update, it can validate changes against the consumer needs. This allows the API to ensure it won’t break any consumers when it publishes and improves planning for trimming API elements.
This certainly is much faster to run than an end-to-end test suite. The contracts can be compared statically without running the software at all, let alone trying to run the UI.
One issue with Consumer-driven contracts is the effort required to maintain schemas for every consumer of the API endpoint. It’s not possible to use ensure such a schema if the endpoint is public and can be consumed by any arbitrary 3rd-party with an http client.
Checking for compatible structure also isn’t a complete test. It ensure that the shape of the data is correct, but not that the behavior is correct. It doesn’t guarantee that the contents of the response are correct for a given input, just that it has all the right fields.
Pact sounds like it has partial solutions to the contract maintenance and behavioral testing issues. It appears (1,2) they build the API contracts by intercepting http traffic. They then verify services in isolation by replaying the captured requests (or responses) to the service being tested. This would make Pact a kind of Operational Profile Testing.
However, there are some issues here too. Pact doesn’t work unless it can mock HTTP traffic. It doesn’t work if you want to run some of your services in the same process, and a well architected system is oblivious to how you deploy components. More fundamentally, pact may not observe all kinds of traffic. We can’t tell if some caller depends on a previously supported part of our API, but just doesn’t call the service very often.
Further, Pact isn’t testing everything we would expect an E2E test to cover. Pact mocks the HTTP layer, which can test an API without further modification, but can’t test the UI state without additional support from a UI automation tool like we’d use for E2E tests. Even when testing the back-end API in isolation, this automated testing isn’t a substitute for the critical thought and interdisciplinary collaboration a good set of behavioral tests encourage (i.e. Specflow or Cucumber). If interdisciplinary collaboration on behavioral tests is too much investment, then well-written behaviorally-focused developer tests are usually enough of a substitute. Behaviorally-named tests can offer light non-developer verification of tests and the behaviorally-focused developer tests can be reused as unit, integration, or system tests.
I’ve been poking at what Pact can’t do, but I do actually think Pact is very cool. It’s the most modern and well-documented tool I’ve seen for contract and (maybe) operational profile testing. This kind of testing can add wide coverage for less investment than E2E tests. It also offers distinct value for testing the system with a sampling of real workloads in any environment. Pact is definitely a tool I hope to explore further.