Our second doctoral research group round table for the year focused on survey design. One of the PhD students shared her tips for designing and validating surveys. The tips are grouped into survey design, piloting and implementation.
- Background – Follow a theoretical as well as an empirical approach to designing the questionnaire. Review the literature related to your topic of interest and consider how data was collected. Make use of existing questionnaires where possible (that have been already validated). If you want to use an existing questionnaire, you may have to ask permission. More likely you will use items from different questionnaires and design your own items too. If possible, conduct interviews with experts in the field to gather their feedback and identify gaps in your data collection approach.
- Support: Make use of any institutional training or support (internally or externally) from experts in questionnaire design and validation. Participate in any relevant research methodology seminars and courses. Talk with your fellow researchers.
- Tools: Use an appropriate tool. Take the time to get to know the tool properly and the features and limitations. Google Forms is a popular choice and is free. Other online tools like SurveyMonkey have free basic designs, with fee structures for more detailed features. There are also licensed tool options such as Netquest.
- Question types: Remember to consider the type of question used, what information you want to obtain and the analysis that will result from that type of question (open questions, likert scale questions, multiple choice questions, multiple response questions). Limit the use of open questions where possible, as this will require extensive analysis and coding. The more questions that are mandatory, the better.
- Style: Ensure that each question/statement deals only with one issue and not multiple issues. Ensure questions are written consistently. Demographical questions are important at the beginning of the survey.
- Length: It is difficult to define an ideal number of questions as it depends on the research topic and design. However the shorter the survey the better and more likely it is to be completed. Questionnaires with too many questions (60+) can lead respondents to abandon them.
- Process: Clearly document each of the the steps taken in the survey design and implementation, as well as the justification for the decisions taken. For example, why a certain sample size was selected, or why the sample was limited to a certain institution. It is important for research publication and for future studies that your data collection methods are replicable. When you publish your research, you may need to include the instruments and instructions to the respondents.
- Purpose: Pilot the questionnaire to obtain comments, opinions and suggestions. This allows you to validate the questionnaire and test the effectiveness before sending it out to your population sample. Piloting the survey will provide information like how long respondents take to complete the survey, whether there are any wording changes required, etc.
- Process: Aim for 15-30 respondents to give you data to analyse. The questionnaire should be as “real” as possible in comparison with the final questionnaire.
- Validity: To check validity, ask pilot respondents:Are you able to clearly understand the questions and the answer options?Do you have any comments or suggestions to improve the survey? For reliability, conduct a reliability test (e.g. Cronbach’s alpha). Review the data collected in the survey to ensure this adequately addresses your research questions and that you are able to analyse the data.
- Permissions: Be sure to contact relevant people (administrators or lecturers) in good time before the survey to obtain permission and cooperation. Ensure you are following any institutional ethical and research processes. Make a list of all stakeholders that you need to contact beforehand and who you can contact if you problems arise.
- Backup: Always create a backup plan in case you are unable to conduct the survey with your initial target group. Or if your response rate is much lower than expected. A response rate of 10-15% can be considered to average in online surveys.
- Reminders: Send a maximum of 2 reminders to respondents to complete the survey.