A Page From the Tech Industry
We are going to talk later about a derivative of Beta testing which is a bit different from those applied by the tech industry (more specifically, the software industry). Nonetheless, I would like to offer a primer on the "true" Beta testing.
"Pure" Beta, as applied in tech, tends to be more heavily weighted toward finding bugs and flaws in the product more than garnering live, revealed preference data on how the market receives the offering. Because this desire toward usability and performance testing tends to be the emphasis in the tech industry, their Betas tend to have little to no emphasis on understanding purchase behaviors, instead recruiting participants through either "closed" or "open" Betas.
Closed or "by invitation" Beta tests are as they sound: They are closed to the average Joe, and instead rely on either an outbound invitation from the organization, or for you to apply for consideration for a Beta slot. You will sometimes see this structure implemented when the offering being tested is of a sensitive or confidential nature, when a company wants to hand-pick a group of Beta testers based on past purchases or behaviors, or when they simply want to limit the number of participants.
Open Beta tests allow virtually anyone to participate, perhaps with minimal barriers to ensure the software/product will be used in the proper environment. This testing may allow free and open download of a Beta software version and simply follow up with all users as to their experiences, or in the case of limited physical products, may allow the first 500 or 1000 participants into the Beta before closing it.
As we will see, depending on where we are in our offering development process, we can apply the Beta logic in a wide variety of ways to meet our learning needs at that specific time. For example, if we were early in the offering development and had a prototype we needed early feedback for, we would likely lean toward a closed Beta with customers or organizations with which we are familiar to get some of the prototypes into the field and see how they perform. This would provide us fast feedback without having to recruit new and unproven participants, etc.
Example of a Closed Hardware Beta
As an aside, the below is from one of the most well-executed closed betas I have seen in quite some time. It was for the Steam Machine, a new gaming console from an established content provider.
The specific byproduct I would like to point out here is how Betas, especially closed Betas, can be a fantastic engagement tool for customers and prospects alike.
In the case of the Steam Machine, there were hundreds of thousands, if not millions, of people not among the 300, who were vicariously participating through hourly updates and postings as these mysterious crates began arriving at homes. While I am personally not a gamer, I followed this story in 2013, as it was a fascinating example of what a beautifully deployed Beta can do in a high-engagement group.
If it is any evidence, the video below of unboxing the Steam Machine Beta has almost 500k views. (Feel free to scrub through the following 7:23 video to see how the Beta was presented.)
Video: Steam Machine Unboxing (7:23)
Six Steps of Beta
If we are considering Pure Beta as applied by myriad software companies, the following offers a simplified view of the six steps of Beta.
From the Centercode Beta Testing Process:
Step 1: Project Planning
Before beginning a beta test, the objectives of the project must be defined. It's common for the number of unique goals in a beta test to range from just a few to upwards of 20. Defining these goals in advance ensures that the appropriate number and composition of participants are selected, an adequate amount of time is available, and everyone involved understands what needs to be accomplished.
Step 2: Participant Recruitment
Beta testing begins with the selection of test candidates. The ideal candidates are those who match the product's target market and whose opinions won't be swayed by a prior relationship with the company. Most private beta tests include anywhere from 10 to 250 participants. However, this number is highly dependent on the complexity of the product, the audience involved, the time available for testing, and the individual goals you'd like to achieve.
Step 3: Product Distribution
Next, products are distributed to beta participants. The focus of a beta test is to understand the customer experience as though they purchased the product themselves. With this in mind, beta is most effective when a complete package including all appropriate materials (software, hardware, manuals, etc.) are sent to participants.
Step 4: Collecting Feedback
Once your participants begin to use the beta product, feedback needs to be gathered quickly. This feedback comes in many valuable forms including bug reports, general comments, quotes, suggestions, surveys, and testimonials. With good beta management and communication tools, you can get a lot of feedback from test participants.
Step 5: Evaluating Feedback
A beta test provides a wealth of data about your product and how your customers view it. However, that information is useless unless it's effectively evaluated and organized to be manageable. All feedback should be systematically reviewed based on its impact on the product and relevant teams.
While bugs are often the core focus of a beta, other valuable data can also be derived from the test. Marketing and public relations material, customer support data, strategic sales information, and other information can all be collected from an effective beta test.
Step 6: Beta Conclusion
When a beta test comes to a conclusion, it's important to provide closure to both the project and the beta participants. This means providing feedback to the participants about their issue submissions, updating them on the status of the product, and taking the time to thank and reward them for their effort.
Weakness of Pure Beta
As you can perhaps imagine, having a closed pure Beta with 250 participants as described would provide some extremely useful feedback from users on the software, usability, instructions, and the entire use experience.
What it would NOT help us understand is any revealed preference data on our offering, and if we were to use Beta testing to try to understand market preference, we would have quite a few issues with this methodology:
- Betas of this type are essentially always free, or, in some cases, participants may be compensated with product or vouchers
- Issue #1: No Dollars for us to understand real preferences and behaviors.
- Betas of this type usually have specific guidance and instructions on how to use the product, and what feedback is desired at each step.
- Issue #2: Minutes of participant attention/use are therefore skewed because we are setting behavior, and not letting users find their own way.
- In the case of closed Beta, the organization will be responsible for the composition of those testing the product.
- Issue #3: Bias may be introduced into participant selection in a variety of ways, from heavy early adopter loading to those who would likely have a favorable opinion of the organization and its products. Furthermore, it would be difficult for us to choose users in the target market if we have not yet proven what that target market is.
For those reasons, in our next topic, we will explore a philosophy which has foundations in Beta, but is better suited to our needs in testing the proposition and offering in the market quickly and inexpensively. We may call this philosophy "microtesting."