A primer on acceptance testing

Foreword

Last week, I took the exam for the ISTQB® Acceptance Testing certification. This is a Foundation Level certification, specializing on acceptance testing, including all related activities.

In this post now, I would like to share the, in my opinion, essential information included in the official learning material, plus one or two things I learned over the last years. While the certification, as well as acceptance testing itself, targets quite a broad range of roles, the learning material focuses on the two roles Tester and Business Analyst (BA). Now, at the moment, we don’t have either of these roles inside Human Made, and I don’t know how many projects (or rather clients) actively involve testers and/or BAs in the project work we are involved in. However, following Agile methodologies, engineers (just like project managers) ideally should be involved in some of the activities that either testers and or BAs would be doing. In Scrum, this is also true for the Product Owner (PO) role.

This post is no extensive document on all the various aspects of acceptance testing, for example, roles, perspectives, activities, and objectives. It should rather be seen as an introduction, going deeper in some of the aspects, while only scratching the surface for others.

Acceptance Testing

Acceptance testing is the name of a test level. It is performed to asses a system’s readiness for deployment and its use by the end user.

Acceptance testing also is the shared responsibility of the end users, business users, testers, product owners and administrative staff, and any other stakeholder.

Where is Acceptance Testing

Using the “common” V-model, as illustrated in the following diagram, acceptance testing is situated at the end of the development process (i.e., the top right side).

However, this does not mean that acceptance testing is the last thing you would do. Even less so in Agile development. While acceptance testing, in general, targets the complete system, it is both possible and common to also perform acceptance testing for a specific feature, or user interface (UI) element, or business process. Acceptance testing is the last test level simply because testing the (user) acceptance of something that is not yet complete is almost useless.

The reason I like the above model is that it provides an easy-to-understand and yet complete overview of all the different (typical) test levels, the different development phases and activities (which exist in both traditional and Agile development, in some form and shape), and, most importantly, the relationship between those. Acceptance testing assesses whether or not the implementation meets the Customer Level requirements—in Agile development, requirements are often represented by user stories, or features.

Objectives of Acceptance Testing

Acceptance testing focuses on the behavior and capabilities of a whole system or product. Its main objectives are:

• establish confidence in the quality of the system (feature);
• validate that the system (feature) is complete and works as expected;
• verify that both functional and non-functional behaviors are as specified.

Forms of Acceptance Testing

The most common (and for us relevant) forms of acceptance testing are:

• User Acceptance Testing (UAT)
• Operational Acceptance Testing (OAT)
• Beta Testing

User acceptance testing aims at building confidence that end users can use the system, fulfill requirements and perform business processes with minimum difficulty, cost, and risk. Operational acceptance testing, on the other hand, focuses on building confidence that administration staff can keep the system working properly for end users, even under exceptional or difficult conditions. OAT includes things like upgrading, user management, data load and migration, and performance testing.

Beta testing is conducted to obtain feedback from the market after development and in-house testing. Since it is performed in various realistic configurations by potential or existing users at their own location, beta testing may discover defects that escaped during development and previous test levels.

Depending on the project context, there might be more, for example, regulatory acceptance testing, which is testing that the system or product adheres to specific government, legal or safety regulations.

Acceptance Criteria

When working on a product or other project, you usually have a set of requirements. These might cover both the business and the product side, and, typically, at least BAs, testers and developers are involved in the requirements engineering process. If you intend to perform acceptance testing, this is the ideal time to also develop acceptance criteria, and, based on these, acceptance tests. Doing this as a joint effort of this multi-disciplinary team ensures a mutual understanding of what acceptable means from the business, development, and testing perspectives, right from the start.

Requirements and Acceptance Criteria

Each acceptance criterion relates directly to a specific requirement or user story, and it is either part of the detailed description or an attribute of the related requirement. When using user stories, acceptance criteria are part of the user story’s definition.

If requirements or user stories are vague or ambiguous, people will likely make assumptions—assumptions which may be incorrect. As a consequence, resulting acceptance tests might be flawed or invalid, and thus create unnecessary costs, as well as risks and uncertainty about the overall quality. Therefore, it is critical that developers, testers and BAs work closely to make sure that requirements are clear and well understood by all stakeholders.

Acceptance Criteria Best Practices

Acceptance criteria refine requirements or user stories, and they provide the basis for acceptance tests. They are formulated as one or more statements, which can all either be true or false. Acceptance criteria also represent the test conditions used to check whether or not a requirement or user story has been implement as expected.

In order to create acceptance criteria, one needs to think about both functional and non-functional requirements, from a stakeholder and user perspective. This provides a high chance of detecting inconsistencies, contradictions, or missing information, and it supports early verification and validation of the related requirement or user story.

Well-written acceptance criteria are precise, measurable and concise. For each criterion, testers need to know if a given test complies with it, or not. Acceptance criteria do not contain technical information such as technologies used, or implementation details. Acceptance criteria do not refer to UI elements.

Sample Acceptance Criteria

Good examples of acceptance criteria:

• A registration form is displayed on the screen.
• I cannot register if the username I choose already exists.
• After sucessful registration, I am informed about it via an email.

Bad examples would be:

• The registration form is realized with Gravity Forms.
• The cursor is initially set on the login field.
• Pressing the TAB key, the cursors switches to the password, repeat password, email and repeat email input fields.

Reviewing Acceptance Criteria

As with requirements and user stories, acceptance criteria should be reviewed regularly to ensure they are clear, consistent and comprehensive, and that they also cover non-functional characteristics. These quality assurance activities can be performed during sprint planning or refinement meetings, or formal technical reviews.

Acceptance Tests

Acceptance tests are derived from acceptance criteria, or other requirements documents. While the latter determine what to test, acceptance tests specify how to test, including detailed test procedures.

Designing Acceptance Tests

Acceptance tests represent scenarios of usage of the system or product. When performing requirements-based acceptance testing, there are several work products that can be used as a test basis. The obvious ones are user or business requirements, or user stories. But also system requirements, documentation, and regulations, contracts and standards oftentimes form a good basis for certain acceptance test cases.

Acceptance tests are typically Black-box Tests. This means that the test does not care about the implementation details, but treats the subject under test as a black box, only interacting with its public interface. Acceptance testing is about behavior, functionality, or usage, not about implementation. Relevant acceptance testing techniques include Equivalence Partitioning and Boundary-value Analysis.

Other test techniques or approaches often used for acceptance testing are:

• business process-based testing, validating business processes and rules;
• experience-based testing, leveraging the tester’s experience, knowledge and intuition;
• risk-based testing, using previously identified product or business risks;
• model-based testing, using graphical (or textual) models to obtain test cases.

Using Acceptance Tests to Drive the Development

There are several approaches to acceptance testing where both test analysis and test design are formally part of the requirements engineering process. The most prominent ones are Acceptance Test-driven Development (ATDD), and Behavior-driven Development (BDD). The names already indicate that the whole development process is driven by either acceptance testing aspects, or the system behavior (which is to be validated and verified).

ATDD relies on different forms of textual or graphical acceptance test design, for example, representations of application workflows. BDD, on the other hand, uses a domain-specific scripting language, Gherkin, that is based on structured, natural statements. Requirements are defined in a “Given—When—Then” format, and they are also used as acceptance test cases, as well as for test automation. This means that acceptance tests become living documentation, easy to understand by all stakeholders, important for the complete development process.

Using the Gherkin language, all requirements or acceptance test cases follow a standardized pattern:

• Given [a situation]
• When [an action]
• Then [the expected result]

The “Given” block specifies the initial state of the test object, the “When” block indicates what actions to peform, and the “Then” block includes the expected consequences. This Given—When—Then triple is no different than the Arrange—Act—Assert triple, which you might know from unit or integration testing (i.e., white-box testing techniques).

As with acceptance criteria, acceptance tests using the Gherkin language do not refer to user interface elements, but rather to user actions on the system.

Here are two sample requirements/tests:

Given I have specified a post title
When I navigate to the Social tab in the Multi-Titles block
Then I see the the social headline reflect the post title

Given I am creating/editing a post
And the content has triggered legal warnings
When I publish the post
Then I see a failing Legal Warnings item in the publication checklist

Non-functional Acceptance Tests

Meeting the expectations for non-functional quality charateristics strongly influences user acceptance. In terms of acceptance testing, the most relevant product quality properties are Performance, Usability, and Security. That is not to say that the other ones are not important. However, the three mentioned characteristics directly affect both the business and the end-user perspective.

In addition to that, performance testing, usability testing and security testing oftentimes require specific approaches to obtain a desired level of coverage, especially if there are (legal) regulations to meet.

What Now?

I think there is some low-hanging fruit. And, to be fair, there’s some rather tedious initial setup to do, too. But let’s focus on the easy wins first, and start writing clear and standardized requirements. Requirements that can be used for manual testing. Which means they can also be used for automated testing!

Let’s not do that for an entire codebase all at once, but gradually. And when the time is right, we can build up more. I’m positive that sooner or later, one or the other person on your project team—maybe you—will benefit from that.

I also think that we would benefit greatly by creating and maintaining business process/rule models for select workflows or processes. And this is true for requirements engineering, for implementation, for testing, and also for communication with stakeholders and/or third parties. Using a standardized approach allows these models to be understood by a variety of roles, and to be used for a variety of activities. If there’s interest, I might write a follow-up post specifically on this subject…

Functional acceptance testing always is specific to a project, and it requires a lot of manual work, initially. For non-functional acceptance testing, however, there are existing tools, services, plans to use. Verifying that a certain workflow or a whole product complies with select security regulations, or accessibility criteria, or performance metrics, this is easy, and at the same time rather important.

What do you think about all this? 🙂