Malicious and Multiple Responses to Surveys: How to Manage the Risks

We are sometimes asked “What are the risks of people completing multiple surveys, or unintended people taking part, in an employee survey or customer survey?”. The concern behind this question is usually that some people might take part in a survey with malicious intent to influence the survey results. Managers sometimes fear that employees or customers may try to exaggerate negative opinions by completing multiple surveys with negative feedback. And employees and unions sometimes fear that management may stack survey responses with excessively positive feedback to exaggerate their own performance or meet KPIs.

There are two main approaches to running online surveys, an “invitational” approach and an “anonymous” approach. Each approach manages the risk of unintended participation in different ways.

Invitational Survey Approach

The invitational approach involves respondents receiving a personalised email with their own unique survey link. This approach directly controls who can fill in the survey - only invited participants receive an email, each email contains a different survey link, and each unique survey link can only be used to complete one survey. Another advantage of the invitational approach is that reminders to complete the survey can be targeted at only those participants who have not yet completed the survey, with early responders saved from receiving unnecessary communications about the survey.

Challenges in an Invitational Survey

At first glance the invitational approach seems like a great solution to managing the risk of unintended participation in a survey. But this approach comes with some challenges.

• First, it requires an accurate email list for all staff or customers, something that not all organisations have or are willing or able to share outside of their organisation.
• Second, this approach is more time consuming because of the need to monitor and respond to email bounce-backs and incorrect email addresses. Hence the invitational approach incurs additional time and cost managing the email list.
• Third, the invitational approach raises fears for some staff or customers that their responses are not confidential because their email address is linked with their responses – this fear is greater when there is lower trust between participants and those administering the survey. Although confidentiality is protected by Xref Engage not providing raw data back to clients, the use of a personal email address may result in some people not participating in the survey or providing more guarded responses to the survey.

If you don’t have an accurate email list for all your intended respondents, or if the costs or confidentiality concerns discourage you from using an invitational approach, then the alternative is an anonymous approach.

Anonymous Survey Approach

With anonymous surveys, a single link is used for all respondents to complete an online survey. Anyone who has access to the survey link can complete the survey, and hence this approach occasionally raises concerns that some respondents might complete the survey multiple times, or the link may be forwarded to people who aren’t intended participants for the survey.

However, the risk of unintended participation is extremely low. Over Xref Engage's history, since starting commercial work in 2002, we have delivered surveys to over 2 million people across more than 3,000 organisations, across private, not-for-profit and public sectors. Across all this time there have been very few occasions where we have clear evidence of unintended or malicious participation in a survey.

Protective factors in an Anonymous Survey

There are several protections from unintended participation in an anonymous survey.

• First, for large-scale surveys involving thousands of participants, any attempt to influence overall results would require a time-consuming effort to fill in a very large number of surveys. A small number of unintended survey completions will have negligible impact upon overall results.
• A second form of protection is through our tracking of response rates for smaller organisations, smaller customer samples, or work areas within larger organisations. If someone attempted to influence the results for a small group by filling in multiple surveys we would likely see an unusually high response rate. If a small organisation or work area had an unusually high response rate, or perhaps even a response rate above 100%, that would be a red flag for caution and further investigation of results. Here is another reason to strive for a high response rate to your survey – the larger the number of responses the smaller the impact of malicious responses, and an unusually high response rate (perhaps over 100%) will stand out more when the clear majority of staff in all work areas have completed the survey.
• Finally, longer surveys including at least one open-ended question offer greater protection from malicious responses because it will take more time and thought for someone to submit fake surveys. This isn’t a reason to artificially make your surveys longer or to put in open-ended questions if the extra questions won’t deliver valuable feedback. The length of the survey and type of questions should be determined by the range of topics you want to assess. Shorter surveys are better if you can achieve the same quality of assessment, or if your survey audience is unlikely to complete longer surveys (this is typically less of a concern for employee surveys, but is often a significant concern for customer surveys). But if you are running short surveys (eg, a single question NPS survey, or a “pulse” employee survey), and perhaps only focusing on numerical or rating-scale feedback, you need to be that much more vigilant in discouraging and monitoring for malicious responses.

Identifying Multiple Responses

If there are reasons to suspect unintended participation, we can drill into meta-data for survey responses, looking at information such as when surveys were completed, how quickly they were completed, and through what type of device and browser they were completed. We can also look at variability in responses to rating scale questions and the content of text responses. Unintended responses show patterns such as being completed back-to-back, being completed much more quickly than legitimate surveys, being completed all through the same type of device and browser, and show little variability in rating scale responses (eg, all being “Strongly Disagree”) and text responses (eg, no text responses, or little or no change in wording of text responses).

All of the 360 leadership surveys we run use an invitational approach. But because of the above means of managing risk, and the lower cost and constraints, the majority of employee and customer surveys we run use an anonymous approach. Nevertheless, if you’re confident you have an accurate email list, are OK with the additional time and cost of administering an invitational survey, and you believe there is a good level of trust among survey participants, then an invitational approach can provide more control over who can respond to your survey and has the additional benefit of being able to target survey reminders only to participants who have not yet completed their survey.