Login

You can login by clicking

Register

If you do not already have an eDelphi.org account. You can create one by clicking here

Four phases of the eDelphi Process, Part 4: Analyzing, Presenting And Reporting the Delphi Results

January 7, 2019

Four phases of the eDelphi Process 4/4

VIDEO WILL COME LATER

During its twenty years of development, eDelphi has been built as a network utility for qualitative delphi research. It emphasizes the interactive learning process of an all-round and multidisciplinary expert panel, instead of just data collecting and recycling. This kind of argumentative eDelphi process consists of four phases that are:

  1. Creating a panel and a query
  2. Building an expert panel
  3. Executing the delphi rounds
  4. Analyzing, presenting and reporting the delphi results

In the fourth part, we will introduce some tools eDelphi offers for analyzing, reporting, and presenting the data of a Delphi query. We will start from the reports, since they are available and ready to share immediately when the data has been collected and the panel closed. eDelphi will produce graphs from the panel results, which is helpful not only in reporting, but also in making the presentations from the project. In analysis, it is important to recognize and structure differences between the evaluation and argumentation of different futures. eDelphi facilitates this task by arranging the arguments according to the answers of scale- or time series -questions. Another important analyzing tool is the possibility to divide the research material into subgroups, whose answers can then be compared.

In eDelphi, there is a separate analysis function for illustrative comparison of the panel’s subgroups, where the report comparison will happen in real time with two simultanious windows. More demanding analysis will take place outside of eDelphi, in content and statistical analysis. For that purpose, you can export material from eDelphi in different text and table formats for further analysis. You can find different alternatives for further analysis in the monthly workshops and online environment http://edelfoi.ning.com .

As an example of the Delphi process, we will be using the same Technological Singularity -panel as in the previous instruction videos: https://www.edelphi.org/technological-singularity . This panel is referred to as the Case Panel later in the instructions.

Part 4: Analyzing, Presenting And Reporting the Delphi Results

Reports and Graphics

Reports

It is possible to export a Delphi query as a file in two ways, in addition to downloading it straight on the screen.

Open the ADMINISTRATION tab of your query. Below the PROCESS heading, see the QUERY RESULTS section. Click on it. Now you can see the QUERY RESULTS -view in front of you. On the left side of the page you will see all the queries of your panel. Click on the name of the query you want to examine. If the material is very large, downloading might take a while. You can change the query or thesis by clicking the Show Report-icon next to its title. If you want to examine the results on paper, click on the Export report -icon, and export the report as PDF.

Before the actual report, you can see a set of filter options. With those, the manager can edit the report. At the top of the page you can the expertise matrix of the panel. It reveals weather the reported material has some expertise holes. This is important information when targeting the communication. With the help of the matrix and grouping, it is also possible to divide the panel into subgroups. We will show how it is done in the next phase.

The comments of the report will automatically be arranged in descending order according to the criterion variable. This will help the analysis of especially 1D, 2D and time series or trend questions. The first comments are those with the most positive value according to the first criterion variable. In the Case Panel, the probability estimation is +++, as well as desirability. In the end of the list are the estimations with both the probability and desirability as - - -. 

The other reporting option is to export the material as Google document. For that, you need to have a Google ID and email address. Even in this case, the report is identical to the report in eDelphi, with the exception of formatting. The essential thing is to have the report in word processing mode, so you can delete, add and edit all the material. The users of Word can export the file into more familiar environment. The strength of Google document is -in addition it being free of charge-  that being a cloud service, it makes cooperation possible in processing the report.

The Comment Order

In scale questions, as well as in time series questions, the material can be arranged according to the choices the panelists have made. Panelists’ choices on the scale predict their choices also when it comes to futures, and the arguments can be found from the experts’ comments.

During the Delphi round, the most interesting picks are the questions that strongly divide the panel. It signals that the question is in a controversy state, which means that versatile argumentation and discussion are especially necessary. Based on the dispersion of the answers, the status of the future thesis can be divided into three: Controversy, dialogue, and solution state. In solution state, the answers concentrate strongly in one point on the scale, and don’t disperse much. In dialogue state, the dispersion is relatively even throughout the scale, and in controversy state, the layout is polarized, and the answers pile up on the edges of the scale, forming a so called bactrian camel. 

When the thesis is in controversy state, opinions and arguments of the future development tend to polarize into emotional conflict. In dialogue state, the understanding of future development divides into many different point of views and future paths, that can and should be taken into common discussion. In solution state, the panelists are relatively unanimous. In that case the main question is, when and how the change will happen.

PICTURE (WATCH THE VIDEO)

The three different future states can be defined with a traffic light metaphora. When the light is green, the future is ready to happen. With the yellow light, the choice of the road is still under discussion, and with the red light, it is necessary to have a detailed debate about the road and the schedule.

The controversy questions are often the ones, that will deepen on the second round of a multiround Delphi. Respectively, the questions with solution status can usually be considered as results after the first round.

After arranging the answers in order, the most simple way of analyzing the panel material is to divide it according to both criterions. A bit more work intensive way is to arrange the material as fourfold table according to criterion variables.

PICTURE (WATCH THE VIDEO)

A fourfold table that can be a starting point for creating scenarios

If the 2D bubble graph is divided into four equal parts, the arguments with positive stance to both probability and desirability (++) are placed in the upper right corner. The opposite arguments when the future is considered unlikely and unsolicited (- -), are in the lower left corner. The arguments with the future being desired but unlikely to happen (- +) are in the upper left corner. Respectively, in the lower right corner are the arguments considering the future probable but unsolicited (+ -). If the manager wants to place all the arguments explicitly in one of the four alternatives, the scales should be made even, so that  there is no possibility to answer in the middle. The default setting of eDelphi’s likert-scale is seven (7), therefore it has to be edited.

Arranging the material is one step away from scenariating the material. For this purpose, there are several possible methodic solutions from Futures Tables to Cluster Analysis. You can get more information about and support for these alternatives from Delphi community’s workshops and online services.

Partitions an Comparisons

An essential part of the result analysis is to compare the answers of different groups. For that purpose, it is possible to partition the material of the whole panel in many different ways.

You can affect to the appearance of the report by filtering the results to concern only some of the panelists. From the expert matrix, you can select any cell or group of cells for analysis. Respectively, if you have created User Groups for the panel, you can limit the results according to them. And if the query has any background variables, can the panel be partitioned into subgroups according to them also.

An important detail when it comes to 2D questions, is the possibility to replace 2D bubble graph with two bar graphs. This alternative is often more illustrative way to estimate the dispersion of the panel’s views according to probability and desirability -if they are the criterion variables in use. You can do the change by choosing the SHOW 2D GRAPHS AS 1D GRAPHS, and then clicking the APPLY FILTER button.

There is also a possibility in eDelphi to compare reports. Open the ADMINISTRATION tab of your panel. Under the PROCESS heading, you will see the title REPORT COMPARISON. Click on it.

At first, you will see two empty fields in front of you. You can compare a filtered report either to the panel’s whole data, or to another filtered report. Select a query and the filtering options from the icons above the fields. You can download or export the report in the usual way. You can go back to the ADMINISTRATION tab through the browser’s Back-button.

If the panel in question is a long-term barometer-type Delphi process, the manager has to be able to make comparisons between different time periods. For this purpose, eDelphi has a possibility to “time stamp” the panel. In the lower part of the QUERY RESULTS page you can see a title PANELS STAMPS. Click on the stamp in order to see the results of that date. If you don’t choose a stamp, eDelphi will show you the results of a current situation. Sometimes time stamps are used for differentiating the results of different stages of a Delphi round. You can stamp the panel by clicking the title PANELS STAMPS from the ADMINISTRATION tab.

The Graphs and Presenting the Results

You can export the results of any panelist group into graphics in three different formats: Download as PNG, download as SVG, and export as Google images. When it comes to PNG and SVG formats, eDelphi works similarly, ie. the graphics will be downloaded as a zip-file into your computer. When you open the file, the graphs will be unpacked as separate files in one folder.

You can export or copy paste the images into PowerPoint presentation program or its free Google-version, Google Slides. We have created a slide series in Google Slides as a sideline for this video instruction. The graphs are exported -and in some cases, drawings added- in a way that is analogical to a situation, when the manager wants to create a presentation for their own Delphi study.

The Data and The Comments

More thorough analysis will be done outside of eDelphi, either with quantitative or qualitative methods. Be it either, the research data has to be exported in table or text format from eDelphi. In statistical operations, commonly used software is SPSS (http://spss.fi/), where 2D-type questions can be run as scenarios by using cluster analysis. Here is more sophisticated Disaggregative Policy Delphi-technique (https://helda.helsinki.fi/handle/1975/274), developed by Petri Tapio. Table-based data fits also for qualitative analysis, f.ex. with Atlas.ti-software ( https://atlasti.com/). It is the most commonly used qualitative tool of eDelphi, and the Delphi developer community has good experience on it.

In eDelphi, the data will be exported in table format into Google Sheets. The format is compatible with Excel and SPSS. Usually you have to decode the table a bit before you can continue the analysis. The case-table is created for further analysis of one real-life Delphi panel. It has been refined from the original table by changing the plus’ and the minuses into a numeric scale of one to seven. For doing that, you can use the “Find and Replace”-function of Google Sheets. In the case-table, the manager has also entered the average figures (the average and the standard deviation), and created several diagrams from the query results.

If you want to create argument fourfold tables from the query answers, a good alternative is to export the data in text format (download as CSV). CSV is a suitable format, since it doesn’t include any formatting, but the comments are easy to export for further analysis or presentation. It is helpful to know, that also in this format, the data will be arranged in descending order according to answers in scale or time series -questions.

 

Open Panels