What is acceptance testing and do you really need to do it?

“It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them.” - Steve Jobs

Are you Steve Jobs?

Whether you’re insulted by this question or in awe of the possibility that someone might compare the two of you, you’re not Steve Jobs. You don’t project your own reality distortion field. You’ve never owned Pixar, not even for a little while (unless you’re George Lucas, and if you are, hi George). Ashton Kutcher never played you in a bad movie.

Steve Jobs and Apple built things that they themselves wanted to use. Turns out that a lot of other people wanted to use those things too.

Are you building things that people want to use? Hopefully. Probably. (You may not be Steve Jobs, but I still believe in you.)

Will they use those things the way you expect them to? Some of them will. Some of them won’t.

Will they discover your software has a bunch of bugs, despite the fact that your team has tested everything ten million times? They will.

The good news is that there is a way to be more confident that your product will solve your user’s problems instead of saddling them with more: acceptance testing.

What Is Acceptance Testing?

Acceptance testing can be poorly understood, partially because it goes by so many names. User acceptance testing, user testing, alpha testing, beta testing, phase 3 moon testing — all are terms that refer to the same basic principle (and only one of them is made up).

While you may find some variation in meaning among these terms, the purpose is always the same: you’re getting close to shipping the final version of your product, and some real users or potential users are gonna kick the tires to make sure that 1) they can even find the tires and 2) everything about the tires make sense.

This testing must involve the end user. This is the part that distinguishes acceptance testing from every other kind of testing in the software life cycle. Other test phases, such as unit testing and system testing, focus on the technical, development side of things. Depending on the methodologies you use, the steps and processes involved may vary, but the goal of acceptance testing is always the same: to learn how the end user (consumer, client, or even an internal team) will interact with your product.

Why Should Product Teams Conduct Acceptance Testing?

Without acceptance testing, you might wind up with a product that engineers think is awesome and end users hate. Though, to be fair, if your product is specifically for engineers that’s pretty good.

Isn’t this what UX designers are for? No, UX designers are for designing the UX. You still have to make sure users like that experience. Much like you, UX designers are not Steve Jobs. They’re good at what they do, but they’re not clairvoyant. They’re probably not even wearing a turtleneck sweater.

I mean, software developers are for writing code and look how that works out? No matter how skilled a developer is, they still ship bugs. That doesn’t mean their code isn’t great, it just means that no one is perfect and it's important to test in order to find issues that anyone might overlook. Everyone writes bugs and “bugs” can exist in the way that users interact with the product just as much as they can exist within the code itself.

So, just as you wouldn’t launch your product without rigorously testing your code (and I'm sure you've never shipped anything that wasn't rigorously tested, lol), you shouldn’t launch your product without rigorously testing your user experience.

Common Kinds of Acceptance Testing

There are a number of different ways to run acceptance testing. You might ultimately mix and match methods to create the right test for your product. You might use both alpha and beta testing, or you might find you need to layer compliance acceptance testing over your beta test, with early alpha testing also thrown in for good measure.

Black-Box Testing means that the tester goes in without any knowledge of the code or structure of the application. This is the most common form of acceptance testing, simply because it’s a major component of every other method.

Alpha Testing runs in a dedicated test environment and usually involves people within the company (although plenty of companies run limited public alphas). Alphas are often used as a precursor to a beta test phase, which you probably already know from experience and from their names.

Beta Testing involves external users as your testers, in either a public or a private environment. Public betas are open to anyone who is interested, while private betas involve outreach and recruitment by the team doing the testing. Public or private, beta tests are always run as a black-box test. Everyone from industry giants like Microsoft to indie game developers run beta tests to allow a select group of users to try things out ahead of a wider rollout. This kind of testing allows you to find problems and fix them, while also providing you an opportunity to build buzz ahead of a wider release.

Contract Acceptance Testing is a special type of acceptance testing that ensures an end product fulfills the requirements of a contract between an internal or external client. In this case, the person testing the software is often the client or someone from their team.

Compliance Acceptance Testing (also called Regulation Acceptance Testing) is run specifically to ensure that the software lives up to all relevant regulations. This testing is critical for software built for industries with a lot of those, like healthcare, government, finance, or legal.

While there are other kinds of acceptance testing, most software developers will use some combination of alpha and beta testing, which involves gathering a team, preparing a test (and sometimes a dedicated test environment), and running a test period of anywhere from a few days to a few months.

8 Ways to Make Acceptance Testing More Efficient and Effective

Nothing is 100% effective 100% of the time for 100% of cases. If it was possible for things to be 100% perfect, it’d probably remove the need for ever testing anything.

But these eight things will help set you up for success.

1. Don’t leave user experience until the end.

If acceptance testing is the first time you’re thinking about the user experience, the process will be far less efficient and will likely force you to make a lot of fixes. Remember those UX designers from earlier? Incorporate UX design thinking into your development from start to finish to make your acceptance testing—and your software—as effective as possible.

2. Think through your test scenarios.

It’s tempting to throw your users into the test with a little documentation and a “have fun!” message. After all, real users will be signing up for your product and may not have much guidance, so why not test that experience. While you might get some deep insight into users this way, your test process will be messy and inefficient. Think through your test scenarios ahead of time. Design them to imitate common real-life use cases (such as managing a backlog). The goal is to strike a balance between seeing how users interact with your product and conducting an efficient test process. Not having rules will give you very little in the way of usable data.

3. Gain input from all involved teams when planning your acceptance test.

Let’s be honest—many hands don’t always make light work. But they definitely make far better software. Your product might have been primarily in developers’ hands up to this point, but when it comes to preparing your product for release, it’s good to have multiple perspectives. Marketing, UX, support, and various development teams will all have a range of potentially useful insights when it comes to planning test scenarios.

4. Recruit the right testers.

Unless you decide to do a public beta, you’re going to need to recruit users for your acceptance test. Who you recruit will make a big difference as to quality. For the most effective acceptance test possible, look for testers who fit your target market.

5. Choose the right tools for the job.

Just like there is no single right approach to acceptance testing, there is no single tool. Everything from a survey to a recorded Zoom call can work for getting the feedback you need. There are also a variety of dedicated software systems available for this. Tools like Rainforest are centered on generating useful reports from in-depth surveys, while tools like Usersnap focus more on getting users to provide screenshots. Then there are platforms like Centercode that provide a more comprehensive testing environment. The tool that is most useful to you will depend on your product, your approach to your tests, and your specific goals for the test.

6. Document everything.

The easiest way to make acceptance testing a frustrating waste of time is to not properly document the results. Make sure the tool you choose allows you to capture the right kind of data for your test. If you want to observe users interacting with key parts of your software, for example, make sure you’re recording calls.

Make sure your documentation is useable; it’s not efficient for everyone to be sifting through recordings or survey results. Make reports based on user data, and note issues that came up as either common or outliers (it isn't necessarily a problem just because a few people have an issue with something). Good documentation is key to moving forward with the information you gather.

7. Don’t rush the acceptance testing phase.

It might be tempting to reach out to a handful of user candidates in a private beta that consists of a single Zoom call or to recruit a few internal people to try out your product in exchange for food. That may work for some products—for example, a simple marketing tool could theoretically be tested in an afternoon by your marketing team. But resist the urge to do the bare minimum. It’s worth the time it takes to plan your test scenarios, recruit the best users, and thoroughly cover use cases. Otherwise, you may get bad data and find yourself panic-fixing a live product with a lot of frustrated, confused users.

8. Ask questions.

Last but not least, ask questions. Ask questions of yourself and your team when you begin planning your acceptance tests. Ask questions of your colleagues as you prepare to run the test. Ask questions of your users before the test to understand their assumptions and expectations. Ask questions before and during the acceptance test phase. (I think you get the picture.) The goal is to understand how end users interact with your product, not just whether or not their experience is smooth. Understanding their process—and their assumptions—will give you far better data than simply looking at this as another bug hunt.

Verdict: acceptance testing is 100% worth the effort

Maybe you know your team has done their best, and you believe the end user will love your product as much as you do (and they just might). Maybe you’re absolutely sure that you did all the research and design necessary to ensure people like what you and your team have made. Maybe you’re just tired and want to go home. Why do you need to go through another phase of it?

Acceptance testing is not just a cherry on top; it’s an essential part of making sure the product you deploy lives up to all your hard work. If you focus on UX from the beginning and make sure you’re well prepared leading into it, acceptance testing doesn’t have to upend your world.

If you do it right, it might make it easier to design products that upend other people’s worlds, just like that Steve Jobs guy we mentioned earlier.