Let’s engage, let’s do a survey

/in , , /by

by Danielle Armengaud, Specialist, Research and Communications

How often have you heard engagement and a survey be joined as if they were one and the same?

While there are many online and in-person engagement techniques to choose from, online surveys have become increasingly popular, especially during this COVID-19 time where in-person events are simply not an option.

The choice of using online surveys should be as carefully considered as with any other engagement technique. As with all technique selection, a survey should only be chosen if and when it fits with the objectives and the planned engagement outcomes. When chosen, sound survey design principles are important, as is understanding the pros and cons, and most importantly, the limitations inherent in data analysis and interpretation of the survey findings. While this occurs at the “back end” of an engagement, this latter point is important to understand and clearly communicate to everyone involved, including decision makers and the broader project team, from the get-go.

Many clients are familiar with seeing survey results from public opinion research (also known as market research) studies. They typically describe a sample size and an audience or universe from which that sample was drawn (“1,000 Canadians were surveyed in May 2020”). They also frequently talk about a level of statistical significance of such a study (“The margin of error of this survey is 3.1%, 19 times out of 20”). These studies are representative surveys, meaning that the respondents reflect the population from which they were drawn.

For a national, Canadian study, for example, respondents will typically be selected proportionately to the regional distribution of the country (i.e. the number of adults in each province and territory), as well as gender and age distribution. The key is that not every Canadian was invited to participate in the survey, but only a representative group.

Engagement surveys, however, are usually a very different kind of animal. They enable many stakeholders, including those who may not be able to or inclined to come to an in-person session, to relatively quickly share their views, from wherever they may be. They could be targeted to an organization’s membership, or to concerned citizens with a passion for a particular local issue. The key is that these surveys are for “everyone” which means by that almost by default they cannot be representative.

In order to promote participation, clients often post and widely share an open link to their online survey. It shows true openness to being inclusive, to hearing from and listening to as many stakeholders as possible. It is not uncommon to receive thousands of responses, resulting is a large, robust dataset and a lot of valuable insights. We have a lot to work with as we sift through what was heard.

Then, when we present the findings to our clients and discuss what we heard, inadvertently, someone will ask “What does this say about our membership’s /stakeholders’ views in general?” or “What is the margin of error of this survey?” And here’s where things can get challenging if the limitations of the data from an engagement survey were not clearly discussed from the start. The thing is, that unless we have put in place strict design protocols or know a lot about the stakeholders or population of our engagement, we cannot project these findings in a statistically sound way to the views of an entire group.

It is easy to imagine that a particular interest group or passionate sub-section of an organization’s stakeholder landscape may be more inclined to participate in the survey and encourage others with like views to do so as well. Or that a certain segment, for example those less technologically savvy, will opt for a different way of sharing their views. We should discuss this in the survey design stage and always consider asking some pointed questions in the survey to figure out whether people may have a special interest. The key take-away: an findings of an engagement survey with unknowns about the stakeholder population, no recruitment or controls cannot be quantified as representative. Rather, it results in a robust dataset that gives us some strong evidence of how participants feel about a particular topic or project.

This is why we talk about what we heard from “participants” or “stakeholders who provided feedback in the survey,” rather than what we heard from “members” or “Canadians” or any other larger group. This is why we typically do not talk about margins of error or statistically significant sample sizes.

While you may feel that this takes away from the validity or impact of engagement survey results, there are many reasons why this is not the case and why surveys can be a very appropriate and useful engagement technique. Consider, for example:

  • It is often a relatively easy, cost effective and quick way to reach out to a large number of stakeholders. It may be the only way to involve certain people or a subset of the community.
  • Engagements support a change or a decision to be made. The decision-maker considers many inputs from different sources of information. A large-scale survey could be one of these, but it is not “the decision” and should not be seen as a vote or a ballot. Even if a majority of survey respondents express that the change should be X and that they want the decision to go one way over another, other inputs may show that this is not the best decision.
  • While for public opinion research, the results are what it is all about, in engagement, the process is also very important (some would argue, even more important.) Authentic engagement considers the openness of the decision-maker to listen to its stakeholders and to create opportunities to be, at some level, a part of the decision. Online surveys often allow for those who otherwise would not have their voices heard, to be included.
  • We need to go where our stakeholders are. Increasingly, that is online. Increasingly, that is also in short bursts of time, at some point in a very busy day with many different priorities. Engagement builds relationships with stakeholders, which can be done effectively by offering short, well-designed and even fun online surveys.

Understanding, discussing and clearly communicating with your project team and decision makers the objectives of the engagement and purpose of the survey early on in the process is key. If everyone understands the value of survey and how results can be and will be used in a decision-making process, we can avoid misunderstanding later on.