Blog

Monday, May 10, 2021

Sample Size for Win / Loss Analysis

Note: This blog was written by Landon Fried, a Research Analyst at Anova, about how he helps clients decide on one of the most fundamental aspects of a win / loss program: the sample.

For companies new to implementing win / loss, figuring out the right sample size can feel like a shot in the dark. While quantitative research has widely accepted standards for determining sample size, qualitative research, such as win / loss analysis, has no rigid rules for defining a data set. Experts, such as the ones in this study by the National Centre for Research Methods, have responded “it depends” when asked how many qualitative interviews are enough.

“So, what are the factors to consider in order to determine the best win / loss sample?” The list of factors is seemingly endless, but many of them can fall into one of three categories: characteristics of the deals you want to interview, the degree of representativeness you want to achieve, and the timeframe you are trying to complete your program in. Altogether, these elements will determine the best sample size for your program.

Deal characteristics: How similar / different are your deal types?

As a rule of thumb, if you have many different deal types (e.g. different product types, different sales channels, different competitors, etc.) the more interviews you will need in order to confidently distinguish aggregate trends. A deal population that is more similar can yield actionable findings with fewer interviews.

Here are some deal-related factors to consider before settling on your target sample size:

How far did you get in the sales process? An interview for a deal that reached the finalist stage often has many more and many different insights from one that fell apart in the RFP stage. While Anova typically recommends choosing the former type of situation to do interviews on because there is much more to be learned, sometimes it is important to find out why you are getting eliminated earlier on in your sales cycle. Consider where in the sales cycle you think your team is struggling the most and target deals that fell apart in that stage to do interviews on.

How different are the sales processes across your deals? If your company utilizes different sales processes, perhaps for different product lines or geographic regions, you will likely need to adjust your win / loss program’s sample size or scope accordingly to capture feedback from the different deal types.

In Anova’s experience, there are a few different ways to account for the differences in deal types. One is by expanding the number of interviews conducted and then segmenting the aggregate data by the different characteristics as part of one large program.

Rather than increasing the number of interviews for one program, another way to account for these differences is to conduct multiple, smaller programs that target specific market segments. This concept is frequently used by one of Anova’s clients in the enterprise software space. This client prefers to utilize multiple, smaller programs for different market segments and conducts them separately from each other. By having multiple programs with more focused scopes, they can focus the feedback and learnings on specific segments of their business.

Representativeness of the sample

What deal flow volume does your company have? Does your company see hundreds or thousands of sales situations in a year? Or is your deal flow lower volume? The more deals your company is involved in, the more interviews you will need in order to have a sample that accurately represents your business. On the other hand, a lower volume pipeline will require fewer interviews to achieve a similar level of representativeness.

How important is each deal at the individual level? While lower deal flow may require fewer interviews to achieve an acceptable level of representativeness, it is also important to consider the importance of each individual deal.  If your company has relatively fewer deals, each deal is more significant and so it makes sense to include as great of a percentage of them as possible. Also consider doing multiple interviews for each situation. Since each deal is of greater significance, interviewing two or three people who were involved in making the decision can provide you with different perspectives on individual deals and increase your overall sample size.

Balancing Time Against the Number of Data Points

One of the primary tradeoffs in any win / loss program is between the size of the data set and the timeframe in which an organization is looking for aggregate results. While programs with a smaller sample size can get your company emerging trends relatively quickly, a more robust program with a larger sample will yield deeper insights.

More robust programs generally take a longer time to complete but provide greater confidence in the overall findings when compared to smaller programs. Additionally, a larger program can allow for segmentation and more precision when calculating numbers, which can be particularly useful in discerning trends over time.

However, some companies prefer to view results more frequently. Smaller programs have the advantage of a faster completion time. For some organizations, this is preferred given the nature of their business (e.g. companies in cyclical industries). Although smaller programs produce fewer data points, these programs can uncover insights quicker and can be particularly useful in rapidly changing industries where it is imperative to get actionable feedback as fast as possible.

A smaller program can also serve to demonstrate the value of doing win / loss analysis. In Anova’s experience, companies new to win / loss often opt for a smaller sample size in their first program in order to receive aggregate results which help in securing internal buy-in and showing the value of win / loss. Once the organization sees the value, these companies often move to a larger, longer-term program.

Although there is no “one size fits all” method for determining the sample size your company will need, considering the factors above can help you to determine the ideal sample size for your company’s win / loss program. By taking the time up front to identify the best opportunities and decide on the right sample size, you can lay the groundwork for a successful win / loss program before even completing a single interview.