Five real life examples:

1. A business is implementing a new IT system that will need to integrate with hundreds of other business systems due to the B2C nature of the organization.  
2. An organization wants to run a design thinking workshop to reimagine the future of their product.
3. A nonprofit is designing a new strategic plan for the next ten years that is focused on serving their community.
4. A think tank wants to define and understand the landscape of adult learning organizations.
5. A social impact agency needs to redesign their business model to better serve their constituents and employees.  

Most of us get it. We can’t just design a product, service, or solution alone in a windowless room. We have to go out and speak to end users to determine whether what we have in mind actually responds to their needs or preferences. Without consulting them, our efforts may fall flat. That’s why sampling is key in any design process.

But the way we go about sampling is equally, if not more, important than the sampling itself. This is something that fewer of us may know or understand.

Sampling often looks a little something like this: we are in the process of designing a product or solution and want to get a sense of how the end user will respond to it. So we find a couple of people we know or like, or people who have a desk near us, or talk to a committee of users who opted in, or else rely on a group of people who are easily accessible and are often involved with our organization.

This might sound like sampling done right. But the problem with doing it in this way is that it creates an immense bias in our design work — and puts the initiative at risk of failure. We think we’re designing for all users by sampling some users. After all, the whole point of sampling is inferring results from a subset of people to apply to the whole. But this approach to sampling leaves us vulnerable to bias.

Now, the bias we’re talking about isn’t a reflection of who you are or your values — it’s a reflection of circumstances, context, and environment. Speaking to people in our known universe, such as those who are easily accessible or who opt in, can create blindspots. These blindspots, in turn, can create a set of data that may accurately reflect other people who look, act, or think like those we sampled, but won’t reflect the whole universe of users.

So we need to ask ourselves, is the group I handpicked substantially different from the population? Usually, the answer is yes. When Google tested Glass, the people who opted in were enthusiasts. That meant that Google’s data was based on folks who were likely to be excited about the new technology, which skewed their findings. That excitement was felt by a subset of the population but didn’t reflect the general whole.

In the scientific process, we call this generalization. To test a hypothesis, a scientist would ideally want to sample everyone — but that’s not always feasible. So instead she will rely on statistics to take a random sample to ensure that her findings are generalizable. That way the phenomenon that she observes in the sampled subset reflects the same proportion of the same phenomenon occurring in the whole population. We believe this approach is equally critical and informative in business as well.

Yet if you are hoping to go beyond observing or describing a population and towards designing new products, systems, structures, or ideas, it is not enough to simply rely on a generalizable sample as is common in academia. The generalizable representative tends to preserve the known universe as it is. In the process, it can inadvertently promote legacy structures, serving to perpetuate a system whereby marginalized voices are not heard or accounted for.

So how do we go about doing this?

It’s not as hard as you might fear. Sure, it might take a little bit more effort than poking your head over your cubicle or opening up slack and asking your colleague what they think, but it doesn’t have to be impossible.

Here’s the approach we advocate with our clients: first, we need to figure out the distribution of experiences. Second, we strategically target the voices of edge cases or marginalized voices to ensure the parameters of the world of the population are clearly defined. Only once we have done both of those things can we design solutions that generalize back to the whole — while promoting equity and elevating voices that are often systemically or institutionally marginalized. Samples for design work must be both generalizable and strategic.

There are a few simple steps you can take today to improve your sampling.

1. Think about your sample size. While 20% of the population is a good rule of thumb, after a sample of about 500 people you get diminishing returns.
2. Put all of the possible users in a spreadsheet. Then, using a random number generator (which you can find online), put in the number range, such as 1-900 users. Generate a row to start at, and then get the generator to pick a number between 1 and 10. Then sample every whatever number it says until you finish (i.e., you start at 459. Then you take every 5th row on the list, so you would do 459, 464, 469, etc.).
3. Think about who all the extreme or non-mainstream users are in your system. This can include big or small, frequent or infrequent users, etc. Make a list and then identify users that fit into each of those “mainstream and extreme” categories and include them in the sample as well.
4. Finally, do an equity check: whose voices are still missing? Ensure those are added in the sample.

Next time you need a sample, remember to take a random sample large enough to be reasonable, and then add in voices of extreme members of the population or members whose voices are typically marginalized or underrepresented. This will ensure that you extract findings from the most representative sample possible.