By Ray Seghers

Over the years there has always been concern about sampling bias when conducting surveys or polls. Although this concern is greater for consumer polls and market research studies, it applies to employee surveys as well. The discussion used to focus on the telephone (those that had one versus those that didn't) now it focuses on land lines versus cell phones and, of course, on the internet (those that have access versus those that don't.)

But in employee surveys there is a bigger concern - non-response bias. That is, what about the people that did not respond? Are they similar to those who did or are they different? Well, it is very difficult to know what those people would have said had they participated. Sure we could try to track them down (if we even knew who they were), but would we believe their responses now?

Several years ago I conducted an employee survey for a manufacturing plant in the Midwest that was part of a multinational organization. They were going through many changes at the time and needed to know how to motivate and retain their current workforce. They decided to use an employee commitment survey to determine which organizational factors were driving the commitment/engagement of their employees.

Well, we got survey results that indicated that their employees had a very low level of commitment to the organization. This really did not surprise the plant manager, but since the participation rate was abysmal (31%), many on his staff were trying to downplay the significance of these low results. They suggested that only disgruntled workers must have taken the survey and that the "silent majority" did not see the need to speak up.

After all, the results made them look bad and, frankly, some of them didn't want to devote any energy to this endeavor. (Side Note: We often remember to get employee buy-in before conducting an employee survey, but we often forget to get mid-level manager buy-in.)

The basic question facing the organization was "Are these results accurate?" That is, can we build and execute action plans based on these results and not be heading off into the wrong direction. As you can see, my concern was a practical one not a statistical one.

There are some things that we can do when faced with this situation:

1. We can try to get an idea if our sample is biased in terms of demographics - gender, age, department, employee status, etc. This is nice to know but probably doesn't answer our basic question since we probably don't know how these subgroups differ in their perceptions of the organization. (Note: if you have conducted previous surveys/interviews, you may have this information. The plant that I was working with didn't.)

2. We can ask whether it matters if only disgruntled workers took the survey. I asked the management group if, in fact, the other 69% were happy but these 31% were disgruntled would that be OK with them. Of course, they admitted that having nearly one-third of their workforce being dissatisfied would still represent a real problem that needed to be addressed. So, even if the sample were biased, the survey results could still provide guidance for needed improvement.

3. We can determine if these results really do represent only dissatisfied employees. We looked at the survey results and determined that that was simply not the case. Yes, the results were lower than those seen at other companies but not all of the responses were negative. The pattern of their results was similar to what we typically saw in other organizations, just lower. The management team agreed that the survey results painted a valid picture of their work environment.

So, while we can never know for sure what employees who didn't participate might have said if they had, we can determine if the survey results provide us with an accurate road map for organizational improvement.

To read more employee survey topics, please go to Ray's blog: http://rayseghers.wordpress.com

Also, please see Ray's LinkedIn.com profile: http://www.linkedin.com/in/rayseghers

Article Source: Non-Response Bias in Employee Surveys