Automated Question Generation for Delphi Studies

Abstract

Surveys are a relatively easy and low-cost method for gathering a group opinion on a given subject. Despite there being several survey platforms the adaptation of these platforms for complex Delphi studies still leaves much to be desired. ‘Qualtrics Delphi Toolkit’ (QDT) is a tool that is designed to make the Qualtrics survey platform better adapted for Delphi studies.The Delphi method has been devised to aid forecasting and decision-making. It is a method that gauges the wisdom of crowds to estimate a solution or to come up with an accepted definition. A Delphi study consists of a number of rounds. In the first round, a panel of experts is gathered, each expert shares their opinion on the selected topic. In the second round, the opinions are compiled into a single unified summary that is then presented back to the experts. Each expert is then given an option to alter their original answer for the changes to then be propagated into the summary answer. This procedure is repeated until a consensus is reached. As the use of the Internet has become widespread it has allowed an increase in the magnitude of such studies, through facilitating access to a greater range of experts, as well as enabling the rapid gathering of more information. The tool support, however is often not able to efficiently manage this scale of data collection. Certain functionality, such as the ability to randomly sample a set of questions is not efficiently achievable at the scale required for large Delphi studies. The need for better tool support has become evident after facing the challenge of presenting a list of over 300 hacking techniques to a group of experts. This required stochastically presenting a subset of these techniques maximising expert engagement whilst minimising fatigue and performance decline.QDT is a tool that is written to extend the functionality of a well-known survey platform, Qualtrics, to automatically generate questions relating to a Delphi study. Currently, Qualtrics offers a feature called ‘Piped text’ that passes information from one question to the next. However this feature alone is not enough when there is a requirement to generate large volumes of questions based on previous questions. QDT is designed to automatically create questions from a list of strings. Together with Qualtrics's conditional formatting it allows efficient creation of large volumes of questions where the question text depends on user's previous choice. This will enable the efficient creation of large-scale survey-assisted Delphi studies where the list of topics is very large.