Testing Pyramid & its importance in Agile Test Automation

With focus shifting towards quick time-to-market, we have to look for ways to deliver the software faster without sacrificing its quality. Continuous Integration & Continuous Delivery (CI-CD) can help here, a build pipeline to automatically test the software and deploy it to testing and production environments. Automating everything — from build to tests, deployment and infrastructure — is the only way forward. That’s where Testing Pyramid helps to formulate an effective Test automation strategy.

Mike Cohn came up with the concept of “Testing Pyramid” in his book “Succeeding with Agile”. It’s a great visual metaphor telling you to think about different layers of testing. It also tells you how much testing to do on each layer.

What is Testing Pyramid?

The “Test Pyramid” is a metaphor that tells us to group software tests into buckets of different granularity. It also gives an idea of how many tests we should have in each of these groups.

A framework that can help teams to create high-quality software by reducing the time required to identify if a change breaks the code. It can also be helpful in building a more reliable test suite. The whole point is to offer immediate feedback to ensure that code changes do not disrupt existing features.

Testing pyramid

The assumption here is that automated unit tests are cheap, easy, fast to run and isolated, compared to those slow, brittle, hard to write end-to-end tests that require a full working system and a web browser or mobile device. Cohn proposed the foundation of a test effort should be unit tests, with fewer service tests and very few end-to-end tests, creating a bit of a pyramid.

Level-1 : Unit Tests

Unit tests form the base of the testing pyramid. They test individual components or functionalities to validate that it works as expected in isolated conditions. They interact directly with product code, meaning they are “white box.” Typically, they exercise functions, methods, and classes. Unit tests should be short and focused on one thing/variation. They should not have any external dependencies – mocks/monkey-patching should be used instead.

Since this is the largest subset, the unit test suite must be written to run as quickly as possible. This test suite needs to be run every time a new feature is added. Consequently, developers receive immediate feedback on whether individual features are working as they are meant to.

Level-2 : Integration Tests

Integration tests are in the middle. Essentially, these are tests that validate the interaction of a piece of code with external components. These components can range from databases, external services (APIs) and the like. It should not be run as frequently as unit tests. Service call tests (REST, SOAP, etc.) are examples of integration tests.

Level-3 : End to End Tests

At the top of the pyramid are the end-to-end tests. End-to-end tests do exactly what the name suggests: test that the application is working flawlessly from start to finish. Typically, they interact with the product like a real user. When running these tests, it is important to imagine the user’s perspective. How can tests be written to replicate that interaction? Web UI tests are examples of End-to-End tests.

End-to-end tests are at the top of the testing pyramid because they usually take the longest to run. They can also be fragile since they have to test a large variety of user scenarios. Tests that run end-to-end through the UI are: brittle, expensive to write, and time consuming to run. So, the pyramid argues that you should do much more automated testing through unit tests than you should through traditional GUI based testing.

The Testing Pyramid is triangular for a reason

There should be more tests at the bottom and fewer tests at the top. Why?

  • Distance from code: Ideally, tests should catch bugs as close to the root cause as possible. Unit tests are the first line of defence. Simple issues like formatting errors, calculation blunders, and null pointers are easy to identify with unit tests but much harder to identify with integration and end-to-end tests.
  • Execution time: Unit tests are very quick, but end-to-end tests are very slow. If test suites have hundreds to thousands of tests at the upper layers of the Testing Pyramid, then they could take hours to run.
  • Development cost: Tests near the top of the Testing Pyramid are more challenging to write. They’re longer. They need more tools and packages (like Selenium WebDriver). They have more dependencies.
  • Reliability: Black box tests are susceptible to race conditions and environmental failures, making them inherently more fragile.

The total cost of ownership increases when climbing the Testing Pyramid. When deciding the level at which to automate a test (and if to automate it at all), taking a risk-based strategy to push tests down the Pyramid is better than writing all tests at the top.

Benefits of implementing – The Testing Pyramid

  • Speed and efficacy: Streamline the testing process with a clear progression and logic introduced to the testing pipeline – work gets done faster.
  • Right priorities: Since turnaround time of UI tests is high, it leads to lower test coverage overall. By implementing the pyramid, such situations are completely avoided.
  • Shortened feedback loop: Fueled by effective test automation approach, teams can move fast and with confidence. It goes hand in hand with agile development practices, continuous delivery and DevOps culture.

The Testing Pyramid – a guideline, not a hard rule!

The Testing Pyramid should be a guideline, not a hard rule. A team might skip valuable end-to-end tests or write needless unit tests just to hit numbers. Instead, use loose proportions to foster better retrospectives. Are we covering too many input combos through the Web UI when they could be checked via service tests? Are there unit test coverage gaps? Each layer’s test count should be roughly an order of magnitude smaller than the layer beneath it.

Testing pyramid

Beware: Watch out that you don’t end up with a test ice-cream cone that will be a nightmare to maintain and takes way too long to run.

Due to its simplicity the essence of the test pyramid serves as a good guideline when it comes to establishing your own test suite. The more high-level you get the fewer tests you should have. Stick to the pyramid shape to come up with a healthy, fast and maintainable test suite: Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end.


Leave a Reply

Your email address will not be published.