What is Real Time Delphi?

In the late 1950s and early 60s, researchers at the think-tank RAND explored the use of expert panels to address forecasting issues, usually about weapons and conflict. Their reasoning went something like this: experts, particularly when they agree, are more likely than non-experts to be correct about questions in their field. However, they found that bringing experts together in a conference room introduces factors that may have little to do with the issue at hand. For example, the loudest voice rather than the soundest argument may carry the day; or, a person may be reluctant to abandon a previously stated opinion in front of his peers. As with almost all conferences, the give-and-take of face-to-face confrontations often gets in the way of a true debate.

The Delphi method emerged as a means for avoiding these conference room problems, and to encourage a true debate, independent of personalities. Anonymity was required in the sense that no one knew who else was participating. Further, to eliminate the force of oratory and pedagogy, the reasons given for extreme opinions were synthesized by the researchers to give them all equal “weight” and then fed back to the group as a whole for further analysis in a series of
sequential questionnaires or rounds. These aspects, anonymity and feedback, represent the two irreducible elements of the Delphi method. This general approach has been used thousands of times since the first published Delphi study in 1964. Personal computers were a decade or more away.

Yet, despite its wide use, the original Delphi design is time-consuming and costly.

Use of the Internet to conduct a Delphi-like study speeds the process and reduces costs. On line applications were explored as early as the 1970s when Turoff experimented with computer communications to link experts in networks. (Turoff, 1972). Turoff and colleagues also presented a social decision-support system that could accommodate large numbers of people who interacted and voted dynamically as they might in a conventional Delphi. (Turoff et al, 2002). Researchers in Finland developed a system now called eDelfoi that has been used extensively in that country.

The United States (US) Defense Advanced Research Projects Agency in September 2004 awarded a Small Business Innovation Research (SBIR) grant to Articulate Software, Inc. to develop a Delphi-based method designed to improve Delphi speed and efficiency. The idea behind the grant was to provide a means for rapidly synthesizing judgments involved in tactical decisions under conditions of high uncertainty. (Gordon, T., and Pease, A., 2006). The Real Time Delphi code that came out of this work was first presented as an open source program that was later developed by the Millennium Project for use in numerous studies.

The primary operational advancements of a RTD study include the absence of repeated rounds and 24-hour, simultaneous computation and delivery of participant responses resulting in an uninterrupted, single round process; greatly reducing the overall time frame normally required to conduct complex studies. In addition, experts are not restricted by the number of rounds to make subsequent judgments based on additional data. The number and location of experts that can participate in a RTD is limited only by Internet access and can be administratively terminated at any time upon satisfaction with existing responses. Finally, RTD’s anonymous environment allows for increased safe opportunity for experts’ interactions, as well as the inferred push for greater intellectual scrutiny, thus, amplifying the global validity of a study as seen through the qualitative comments.

But does Real Time Delphi produce reliable results that are at least equivalent to the original process? Gnatzy et al say:

(In our paper) we demonstrate how the real-time (computer-based) method increases the efficiency of the process, accommodates expert availability, and reduces drop-out-rates. Modifications in the Delphi procedure … not only increase efficiency but also change the nature and process of the survey technique itself. …Empirical data obtained from both conventional as well as real-time Delphi studies is analyzed based on multiple statistical analyses. The research findings indicate that significant differences between the two Delphi survey formats do not exist and final survey results are not affected by changes in the survey procedure. (Gnatzy et al, 2011)

How does RTD work? Imagine a Delphi study involving a set of questions. When each respondent joins the on-going, on-line study, he or she is presented an on- screen form that contains the questions, and, for each question:

  • The average (or median) response of the group so far (and possibly the distribution of responses)
  • The number of responses made so far
  • A button that opens a window showing reasons that others have given for their responses.
  • A window that allows the respondent to type in justifications for their own answer.
  • And finally, a space for the new respondent’s numerical estimate, answering the question.

Considering this information, the respondent types an input and instructs the computer to save the answer. The group average or median is updated immediately for numerical questions and presented back to the respondent and anyone else who has signed on. If the respondent’s answer to any question is beyond a pre-specified distance from the average or the median, an attention-getting indicator may be used to flag the question for the respondent. When the flag is “up” the respondent is asked to give reasons for their response which, when saved, become an entry in the reasons window, seen later when anyone opens that window. There is no explicit second round. When the respondent comes back to the study in a minute or a day, the original input form is presented to him or her. Of course, by then others may have contributed judgments, so the averages or medians may have changed and other questions may be flagged since the group response may have changed sufficiently to move the respondent’s previous answers outside of the pre-
specified distance from the average or the median since the last time the input page was viewed. In this way the Delphi requirements of anonymity and feedback are met and the process, once underway yields the distribution of the group’s responses and reasons for the extreme positions. The process can be synchronous
or asynchronous, and if implemented on an Internet site, can involve a worldwide panel.

References

Gnatzy, T, Warth, J., von der Gracht, Darkow, I., “Validating an innovative real-time Delphi approach – A methodological comparison between real-time and conventional Delphi studies” Technological Forecasting and Social Change, Volume 78, Issue 9, November 2011, Pages 1681–1694

Gordon, T. “The Delphi Method,” a chapter in Gordon TJ, Glenn JC. Futures Research Methodology- v3.0. Washington, DC: The Millennium Project, 2009.

Gordon TJ, Pease A. RT Delphi: An efficient, “round-less” almost real time Delphi method. Technological Forecasting and Social Change, 2006; 73(4):321-333.

Turoff, M. Delphi Conferencing: Computer-Based Conferencing with Anonymity, Technological Forecasting and Social Change, 1972; 3: 159-204.

Turoff M, Starr RH, Hee-Kyung C, Zheng L, Yuanqiong W. Social Decision Support Systems (SDSS), Proceedings of the 35th Hawaii International Conference on System Sciences, 2002.