If you do not already have an eDelphi.org account. You can create one by clicking
Four phases of the eDelphi Process, Part 4: Analyzing, Presenting And Reporting the Delphi Results
January 24, 2020
Four phases of the eDelphi Process part 4
During its twenty years of development, eDelphi has been built as a network utility for qualitative delphi research. It emphasizes the interactive learning process of an all-round and multidisciplinary expert panel, instead of just data collecting and recycling. This kind of argumentative eDelphi process consists of four phases that are:
Creating a panel and a query
Building an expert panel
Executing the delphi rounds
Analyzing, presenting and reporting the delphi results
In the fourth part, we will introduce some tools eDelphi offers for analyzing, reporting, and presenting the data of a Delphi query. We will start from the reports, since they are available and ready to share immediately when the data has been collected and the panel closed. eDelphi will produce graphs from the panel results, which is helpful not only in reporting, but also in making the presentations from the project. In analysis, it is important to recognize and structure differences between the evaluation and argumentation of different futures. eDelphi facilitates this task by arranging the arguments according to the answers of scale -or time series- questions. Another important analyzing tool is the possibility to divide the research material into subgroups, whose answers can then be compared.
In eDelphi, there is a separate analysis function for illustrative (ILÖSTRÖTIV) comparison of the panel’s subgroups, where the report comparison will happen in real time with two simultaneous windows .A more complex analysis will take place outside of eDelphi, in content and statistical analysis. For that purpose, you can export material from eDelphi in different text and table formats for further analysis. You can find different alternatives for further analysis in the monthly workshops and from the online environment. See the comment section for the link to the online environment.
As an example of the Delphi process, we will be using the same Technological Singularity -panel as in the previous instruction videos: https://www.edelphi.org/technological-singularity . This panel is referred to as the Case Panel later in the instructions.
Part 4: Analyzing, Presenting And Reporting the Delphi Results
Reports and Graphics
It is possible to export a Delphi query as a file in two ways, in addition to opening the report in eDelphi.
Open the ADMINISTRATION tab of your query. Below the PROCESS heading, see the QUERY RESULTS section. Click on it. Now you can see the QUERY RESULTS -view in front of you. On the left side of the page you will see all the queries of your panel. Click on the name of the query you want to examine.
On the right side you can see a set of filter options. With those, the manager can edit the report. At the top of the page you fill find the expertise matrix of the panel. It reveals whether the reported material has a lack of expertise. This is important information when targeting the communication. With the help of the matrix and grouping, it is also possible to divide the panel into subgroups. Next we will show you how this is done.
After filter options you can see a heading EXPORT REPORT. Here you can choose in which format you want to examine the report. The options for the whole content are a PDF -file and a google document. For a Google document you need to have a Google ID and an email address. Even in this case, the report is identical to the report in eDelphi, with the exception of formatting. The essential thing is to have the report in word processing mode, so you can delete, add and edit all the material. The strength of the Google document is -in addition of it being free of charge- is that being a cloud service, it makes cooperation possible in processing the report. eDelphi will send the report to the manager’s email. This will take up to ten minutes, depending on the size of the file.
The comments of the report will automatically be arranged in descending order according to the criterion variable. This will help the analysis of especially = one-dimentional ,two-dimensional, time series and trend questions. The first comments are those with the most positive value according to the first criterion variable. In the Case Panel, the probability estimation is marked with 3 pluses +++, as well as desirability. In the end of the list are the estimations with both the probability and desirability marked with 3 minuses - - -.
The Comment Order
In scale questions, as well as in the time series questions, the material will be arranged according to the choices the panelists have made. Panelists’ choices on the scale predict their choices when it comes to futures, and the arguments can be found from the experts’ comments.
During the Delphi round, the most interesting picks are the questions that strongly divide the panel. It signals that the question is in a controversy state, which means that versatile argumentation and discussion are especially necessary. Based on the dispersion of the answers, the status of the future thesis can be divided into three: Controversy, dialogue, and solution state. In solution state, the answers concentrate strongly in one point on the scale, and do not disperse much. In the dialogue state, the dispersion is relatively even throughout the scale, and in the controversy state, the layout is polarized, and the answers pile up on the edges of the scale, forming a so called bactrian camel.
When the thesis is in controversy state, opinions and arguments of the future development tend to polarize into emotional conflict. In the dialogue state, the understanding of future development divides into many different point of views and future paths, that can and should be taken into common discussion. In the solution state, the panelists are relatively unanimous. In that case the main question is, when and how the change will happen.
The three different future states can be defined with the traffic light metaphor. When the light is green, the future is ready to happen. With the yellow light on, the choice of the route is still under discussion, and with the red light, it is necessary to have a detailed debate about the route and schedule.
The controversy questions are often the ones, that will deepen on the second round of a multiround Delphi. Respectively, the questions with solution status can usually be considered as results after the first round.
After arranging the answers in order, the most simple way of analyzing the panel material is to divide it according to both criteria. A more workintensive way is to arrange the material as a fourfold table according to criterion variables.
A fourfold table that can be a starting point for creating scenarios
If the two-dimensional bubble graph is divided into four equal parts, the arguments with positive stance to both probability and desirability (++) are placed in the upper right corner. The opposite arguments - when the future is considered unlikely and unsolicited (- -), are in the lower left corner. The arguments with the future being desired but unlikely to happen (- +) are in the upper left corner. Respectively, in the lower right corner are the arguments considering the future probable but unsolicited (+ -). If natives, the scales should be made even, so that there is no possibility to answer in the middle. The default setting of eDelphi’s likert-scale is seven (7), therefore it has to be edited.
Arranging the material is one step away from scenariating the material. For this purpose, there are several possible methodical (METHOTICOL) solutions from Futures Tables to Cluster Analysis. You can get more information and support for these alternatives from the Delphi community’s workshops and online services.
Partitions (PARTISONS) an Comparisons
An essential part of the result analysis is to compare the answers of different groups. For that purpose, it is possible to partition the material of the whole panel in many different ways.
You can affect the appearance of the report by filtering the results to concern only some of the panelists. From the expertise matrix, you can select any cell or group of cells for analysis. Respectively, if you have created User Groups for the panel, you can limit the results according to them. If the query has any background variables, the panel can be partitioned into subgroups according to them .
There is also a possibility in eDelphi to compare reports. Open the ADMINISTRATION tab of your panel. Under the PROCESS heading, you will see the title REPORT COMPARISON. Click on it.
At first, you will see two empty fields in front of you. You can compare a filtered report either to the panel’s whole data, or in a another filtered report. Select a query and the filtering options from the icons above the fields. You can download or export the report in the usual way. You can go back to the ADMINISTRATION tab through the browser’s Back-button.
If the panel in question is a long-term barometer-type Delphi process, the manager has to be able to make comparisons between different time periods. For this purpose, eDelphi has a possibility to “time stamp” the panel. In the lower part of the QUERY RESULTS page you can see a title PANELS STAMPS. Click on the stamp in order to see the results of that date. If you do not choose a stamp, eDelphi will show you the results of the current situation. Sometimes time stamps are used for differentiating the results of different stages of the Delphi round. You can stamp the panel by clicking the title PANELS STAMPS from the ADMINISTRATION tab.
The Graphs and Presenting the Results
You can export the results of any panelist group into graphics in PNG-format. The graphics will be downloaded as a zip-file to your computer. When you open the file, the graphs will be unpacked as separate files in one folder.
You can export or copypaste the images into a PowerPoint presentation program or its corresponding free Google-version, Google Slides. We have created a slide series in Google Slides as a sideline for this video instruction. The graphs are exported -and in some cases, drawings added- in a way that is analogical to a situation, when the manager wants to create a presentation for their own Delphi study.
The Data and The Comments
More thorough (THAROU) analysis will be done outside of eDelphi, either with quantitative or qualitative methods. Be it either, the research data has to be exported in table or text format from eDelphi. In statistical operations, commonly used software is SPSS (http://spss.fi/), where the 2D-type questions can be run as scenarios by using cluster analysis. Table-based data also fits for qualitative analysis, for example with the Atlas.TI-software ( https://atlasti.com/). It is the most commonly used qualitative tool of eDelphi, and the Delphi developer community has good experience on it.
In eDelphi, the data will be exported in table format into Google Sheets. The format is compatible with Excel and SPSS. Usually you have to decode the table a bit before you can continue the analysis. The case-table is created for further analysis of a real-life Delphi panel. It has been refined from the original table by changing the plus’s and the minuses into a numeric scale of one to seven. For doing that, you can use the “Find and Replace”-function of Google Sheets. In the case-table, the manager has also entered the average figures (the average and the standard deviation), and created several diagrams from the query results.
If you want to create argument fourfold tables from the query answers, a good alternative is to export the data in text format. CSV is a suitable format, since it does not include any formatting, but the comments are easy to export for further analysis or presentation. It is helpful to know, that also in this format, the data will be arranged in descending order according to answers in scale or time series -questions.