Comprehensive Automated test suite

A comprehensive set of automated tests will improve development productivity by allowing for continuous testing, which will allow bugs to be found and fixed while the code is still fresh in developer's mind.

Additionally, a comprehensive test suite will ensure that the codebase stays stable at all times. This isolates the potential causes of a defect to the current changes, making defects much easier to debug.

Some of the advantages to the approach include:

• Whole-team automation
• Distributed maintenance
• Lower cost
• Run-time speed (allowing for more coverage on a finely grained check-in level)
• Consistent coverage and ongoing feedback

Structure tests using Automated Test Pyramid

A well constructed test suite will have tests for each level. Organizing tests around this model can avoid the common problems of having too many slow and fragile end-to-end integration tests.

Unit tests:

Unit tests are written to test the cooperation of several  classes or even major subsystems. This will make the unit tests slower, and diminish the frequency that the tests are run. To be fully useful, the unit tests need to run continuously, ideally every time that a file is saved in the integrated development environment (IDE) and these tests are the most in number and take several minutes and even hours to run.

The solution is to have a full set of fast unit tests that run extremely quickly. This can be done by having each test only test one method on one class and extensively use test doubles¹ to replace collaborative classes in a test.

Component tests:

These tests are designed to ensure that modules or components function correctly and that the various classes in the service work together correctly.

Tests should be fast and portable enough so that they can be run by developers before each code change is committed to source code repository.

Service/API tests:

The tests are an expression of the contract that the software implements, any change to the service that breaks an existing test is also likely to break production software that the software supports.

Tests should use service virtualization² to isolate the service under test. API tests should be very robust, even when the software is changing.

Acceptance tests:

These tests are tied to a specific functionality to verify that overall software functions correctly. They mat be slower but they must be portable enough to run in the development environment.

Tests should use service virtualization² to eliminate dependencies on external slow or difficult to deploy services.

Integration tests: 

These tests are designed to test the software in a environment that closely matches production. These end-to-end tests are used to test the interaction of modules in specific environment.

Tests should be run continuously or nightly on dedicated test hardware or using cloud-based testing resources as tests are tend to be slow and environment-specific.

 Specialized tests:

These tests are designed to test the performance, scalability and security of software when running in an environment as close to the production system as possible and require production-like machines with large datasets and take longer to run.

Use cloud testing resources and automated performance test tools, and increase the frequency that these tests are run whenever possible.

¹Test Double: Stubs and Mock objects are forms of test doubles. These are replaced with actual objects which are slow and difficult to run without a complete test environment.
²Service Virtualization: Rather than virtualizing entire systems, virtualizes only specific slices of dependent behavior critical to the execution of development and testing tasks

Source: Parasoft (http://alm.parasoft.com/api-testing-gartner)