(Written in collaboration with Marco Bonanni – Capgemini Italian Testing Practice Leader)

Test Process vs. Development Process
While Agile is more and more coming of age inside development teams, it is important to understand what are the specific characteristics of Testing in an Agile environment. The first thing to notice is that Testing in modern engagement is no more a separate phase, but – as one can likely expect – test activities are fully integrated in the development process. According to the Agile Manifesto, Testing must comply with the principle “deliver value to the business as soon as possible”: therefore testing is a continual activity, starting from early identification of requirements to the final deployment.
The focus is of course to deliver (and thereby test) high-value area first so as to reduce the risk from the early stages of the project. To such purpose testers play a fundamental role in order to clearly define and agree the acceptance criteria with the Customer.

Test stages
With respect to traditional approaches, we notice a lower emphasis on “test levels”. Whilst in the traditional life-cycles, we have seen the rise a plethora of test stages (system test, user acceptance test, end-to-end test, production acceptance test and more), this was mainly due to different parties/ stakeholders with different responsibilities being involved in the validation phase
Since in an Agile environment testing is continuously performed throughout the whole lifecycle in strict collaboration with the Customer’s stakeholders (possibly final users), it is less relevant to talk about test stages such as system test or acceptance test. Of course we can’t assume that test stages are supposed to be eliminated, but the need for different stages will be depending on project size/type
If a need to provide specific test levels still holds, we might say that all the product backlog items with the quality characteristic of “functionality” are good candidates to represent the system test, while the validation of non-functional requirements on the target environment might concur to build the acceptance test suite.

The role of Testers: definition of the Testing Strategy
Of course test specialists will be playing a fundamental role also inside Agile projects, to guarantee proper expertise to the team. The Test Specialist will be supposed to define the test strategy, perform the risk analysis, evaluate specific (difficult) cases and coach the rest of the team as to testing activities to broaden knowledge and skills
The strong message here is anyway that testers are fully integrated in the development team i.e. all the team members must be prepared to perform test activities. In particular all the members in the team may (and they will actually) be asked to support the specification and execution of test cases.
In an Agile team, the tester will support:

  • The Product Owner in the formulation of the product (and prioritization) of backlog and acceptance criteria
  • The Business Analyst in the elicitation of software requirements (Definition of Ready)
  • The Developers, possibly pairing them during programming and evaluating unit test

Most of all the Test Specialist will be required to define a convenient testing strategy for the project to drive the validation activities. A test strategy in an Agile environment should provide two different levels

  • Project Test Strategy
  • Sprint Test Strategy

The most relevant aspect regarding a testing strategy in Agile is the consideration of Risk: basically testing in Agile should more and more leverage on a Risk-driven approach

Defining the Test Strategy: Project Test Strategy
The Project Test Strategy takes place during the Scrum “planning” event (typically holding during the so-called Sprint 0) and is incorporated in the product backlog: this is the moment to determine the high-level product risk, the test process and the so-called Definition of Done (DoD)
The DoD is one of the differentiating aspects of Agile delivery and of course the test strategy plays a fundamental role in its formulation. From a test perspective the DoD must contain:

  • The agreements concerning the test process (how the test will be performed)
  • The kind of tests to be included for each Sprint
  • The criteria to be met with regard to the defects identification and classification (what can be considered a defect and what will be its Severity and Priority)
  • The agreements concerning the test results (when the test can be considered passed) (Quality Gate)

Defining the Test Strategy: Sprint Test Strategy
The Sprint Test Strategy is defined during the Sprint planning event and aims to include testing activities in the Sprint backlog. During the Sprint planning stage, the team estimates (e.g. via Planning Poker) the amount of time required for each task in the Sprint backlog. For testing activities such effort is influenced by a consideration of whether the validation should be thorough or light: such consideration is therefore strictly related to the product risk. It is therefore advisable to include the risk classification for each backlog item in addition to its priority, with particular respect to user stories before the Planning Poker is initiated
A practical way to evaluate risk by the Scrum team may be the following:

  1. Gather the team members together (a 1-hour meeting usually suffices)
  2. List all the backlog items for the current sprint on a whiteboard
  3. Ask the team members to identify the characteristics of each backlog item to be validated Test characteristics could identified according to FURPS+
  4. Determine the possible damage and chance of failure for each item and quality characteristic
  5. The product risk can be then calculated as: damage x chance of failure

The test strategy will be determined in the next step, specifying the intensity (i.e. depth of testing) with which a combination of backlog items and quality characteristic is to be tested. To make the test applicable in practice, a couple of columns can simply be added to the previous table to:

  1. Determine the test intensity
  2. Determine the test design technique on the basis of the test intensity and quality characteristic

Theory vs. Practice: Deviating from Agile
According to Agile approaches, a working product is supposed to be delivered at the end of the Sprint, which means that it is tested and accepted by the Product Owner: from a testing perspective this means that all tests (unit test, system test, acceptance test) take place during the Sprint.
Nevertheless in real-life practice deviation from the Agile theory is likely to occur quite often. And yet we would be wary to encourage the proliferation of anti-patterns.
In particular it should be deprecated to develop and perform unit tests in one Sprint, while postponing system test and UAT to another: such approach creates a sort of “technical debt”, causing defects to be fixed later, which may cause the quality of the software to considerably decline in case of long projects with many Sprints