Login

You can login by clicking

Register

If you do not already have an eDelphi.org account. You can create one by clicking here

Four phases of the eDelphi Process, Part 3: Executing the​ Delphi Rounds

January 7, 2019

Four phases of the eDelphi Process 3/4

VIDEO WILL COME LATER

During its twenty years of development, eDelphi has been built as a network utility for qualitative delphi research. It emphasizes the interactive learning process of an all-round and multidisciplinary expert panel, instead of just data collecting and recycling. This kind of argumentative eDelphi process consists of four phases that are:

  1. Creating a panel and a query
  2. Building an expert panel
  3. Executing the Delphi rounds
  4. Analyzing, presenting and reporting the Delphi results

In the third part, we are in the middle of a Delphi round, collecting the data produced by the panelists. In the final part, this data will be described, analyzed, and reported. During the Delphi round, the panelists will be activated, but It is downright proactive time also for the manager.

In this video instruction, we will first repeat the instructions from the previous video on how to invite and manage panelists.

Secondly, we will show how to keep track of the query activities. The manager should also observe, if the Expertise Matrix will fill up evenly during the round. Conversely to survey-type queries, the Delphi-management prefers active communication. The goal is not only to gather already existing, scattered information, but also to form it through the panelists’ interaction. This interaction becomes possible when the panelist, in addition to answering, familiarizes themselves with other panelists’ comments, and possibly reconsiders their own original answer. In this video instruction, we will demonstrate two different techniques on how to involve and inform the panelists. Finally, we will describe one possible technical solution to move from the first round to the second.

As an example of the Delphi process, we will be using the four possible routes to end up to technological singularity, sketched by Vernor Vinge. By technological singularity he means a new evolutionary phase when the significance of the machines will outweigh the human abilities. The phase will be a result of a changing relationship between the human and the machine. The use of the theme is not a statement for singularity, but a tactical decision to choose a theme that does not grow old immediately. For this video instruction there is an open delphi panel with the title Technological Singularity: https://www.edelphi.org/technological-singularity . This panel is referred to as the Case panel later in the instructions.

Part 3: Executing the Delphi Rounds

Inviting the Panelists

Open the ADMINISTRATION tab of your panel. At the bottom right corner of the page, you will see the USERS heading and below it INVITE/ADD USERS. Click on the title.

Now you will see the INVITE USERS TO THE PANEL -tab view. On the right side, you will see the panelists already participating to the panel. At the top left you can see the INVITE/ADD USERS section. This allows you to invite individual users to the panel.
Enter the e-mail-addresses of the invitees to the field below the title one at a time.

When the software offers the correct address, click on it. The addresses of the invitees will appear below the USERS TO BE INVITED heading.

In the INVITATION section, you can edit the invitation mail to be sent to all new panelists. You can customize the invitation according to your the panel. The data in square brackets will be automatically updated. Do not modify or delete the text ACCEPT URL. It contains the invitation link to join the panel. However, in the end of the letter there is a text part that offers a link to turn down the invitation. It is used only in the rare case that the manager hasn’t agreed with the panelist about their participation to the panel beforehand. This text part should be deleted, since the panelist can also accidentally   click on the link, which will eliminate the panelist for good. After the refusal, the manager won’t be able to invite the same panelist again without the chargeable help of a coder.

Below the label ACCEPTING THE INVITATION REDIRECTS TO you can define into which query the acceptance link will lead. The default setting is the main page of the panel.

If the panel consists of several queries and you would like the panelists to answer only one of them, choose the query in question.

By clicking on ADD USERS DIRECTLY, chosen panelists will not receive an invitation, but they will be added directly to the panel. In this case, eDelphi will announce the password of the panelist or panelist group in the uppermost part of the page. Save the password for the occasion you want to instruct the panelists on how to change their password.

When you are ready, click on the SEND INVITATION button.

After a while, the panelist will receive an invitation to their email. By clicking on the invitation link, the panelist can access the panel page specified by the Manager. When using the invitation link the panelist does not need to  log in to eDelphi.

If you want to add a large number of panelists at once, you can do it effortlessly by making a CSV or TXT file of the panelists email addresses. You can do it, for example, with Google Sheets, by typing each e-mail on its own cell without punctuation, and downloading the file as a CSV. You can create a TXT file with google docs, using the same procedure. In google docs type each e-mail address on a separate row and download the file as a TXT. Corresponding measures will work with the same definitions in Microsoft Excel and Word.

When you have created a CSV or TXT-file, open the ADMINISTRATION tab of your query. At the bottom of the page, there is the USERS section, and below it, INVITE/ADD USERS. Click on the title.

On the left side of the page, you will see the title INVITE/ADD USERS and below it, CSV FILE. Click on ADD FILE. Select the CSV or TXT file you created from your own files and then click on DOWNLOAD CSV. Now you will see all the email addresses of the file below USERS TO BE INVITED. To invite panelists, click on SEND INVITATION at the bottom of the page. Before sending the invitations you can edit the invitation text.

Query Activities

To see the number and quality of panelists who have answered a single query, you can review it from QUERY ACTIVITIES. Open the ADMINISTRATION -tab. Under the PROCESS header, there is a section QUERY ACTIVITIES. Click on it.

Now you can see the QUERY ACTIVITIES view in front of you. You will see the QUERIES heading on the left, and below it, all the queries of your panel. Select the query you want to examine. On the right side, you will now see the information of the panelists who have answered to the query. The panelists are listed in alphabetical order. After the name, you can see the last login date of the panelist, and the date when they last time answered to the query. By clicking on LAST LOGIN, the panelists are organized by their last login date, either from the latest to the earliest or from the earliest to the latest.

By clicking ANSWERED, the panelists are organized by the answering date.

In addition to query activities, you can follow the logins of the panel according to panelists’ login status. eDelphi will divide panelists into different groups according to the way they have been added to the panel, and weather they have reacted to the invitation.

Open the ADMINISTRATION tab of your panel. At the bottom right corner of the page, you will see the USERS heading and below it INVITE/ADD USERS. Click on the title.

On the right side, you will see all your panel’s invited panelists, grouped according to the way they have been added to the panel, and whether they have reacted to the invitation. If the manager has added the panelist without a separate invitation via email, they belong to the group under USERS ADDED BY MANAGER. Again, if they have logged in to the panel by themselves -which is possible in an open panel- they belong to the group under REGISTERED USERS. When the manager has invited the panelists via eDelphi’s invitation mail, they will be listed under PENDING INVITATIONS. After they have accepted the invitation ie. clicked themselves to the panel, the status will chance into ACCEPTED INVITATIONS.

You can send email to each group by clicking the link RESEND INVITATIONS TO ALL USERS next to the heading. You can also send email to an individual panelist. Let’s choose the most common situation, when the communication is directed to the panelists who haven’t responded to the invitation. Find the status heading PENDING INVITATIONS, and click on the link RESEND INVITATIONS TO ALL USERS. Now you can customize the invitation to fit the situation. You can document the invitation by sending it also to yourself. Then choose the query you want to invite the panelists to below the heading ACCEPTING THE INVITATION REDIRECTS TO. Finally, click on the SEND INVITATIONS.

It is helpful to use a chart when keeping track on the panelists. Here is an example of the bookkeeping of one executed Delphi study.

PICTURE (WATCH THE VIDEO)

In the case chart, all the panelists and their expertise groups are written down. During the Delphi round, the participation of the panelists will be monitored, and it is also possible to follow the group-specific situation in percentage. From the other tabs of the google sheet,  you can find graphs describing the query activities.

Active Communication

The panel manager has two clear information policies to guide the communication with the panel. The first one is based on the structure of the panel and the query activities. This has been demonstrated earlier in the video. The other policy is based on the distributions and comments accumulated in the queries and questions. From those the manager has a possibility to highlight interesting “news” to motivate the panelists to participate.

Based on the monitoring, the manager estimates the need to communicate with different subgroups of the panel. The basic monitoring includes naturally the division into answered and unanswered panelists. The communication to these groups should always be differentiated. The manager should thank the answered panelists by informing them of the development of the query and its results. It is of course especially important to inform the unanswered panelists. The manager can appeal to for example the uniqueness of their expertise. For both groups, regular and planned mailing is important also because every time a panelist receives an email, they will also get a direct link to the panel.

If and when some subgroup is underrepresented, the manager should communicate with the group separately, and appeal to the argumentation that conforms their interests. Individual mailing is most efficient when the manager has some content to personify the message. It is important to acknowledge, that getting the first answer is only a temporary goal. In the argumentative panel, the final goal is to get the panelists to participate and comment also other panelists’ comments, and possibly change their own answers. Both features are signs that the expert groups are in learning mode, and the result of the Delphi round will be more than just a collection of individual point of views.

You can look for panel “news” the same way you examine and analyze query results after the Delphi round. Open the ADMINISTRATION tab of your query.

Below the PROCESS heading, see the QUERY RESULTS section. Click on it. On the left side of the page you will see all the queries of your panel. Click on the name of the query you want to examine.

Now you can see the QUERY RESULTS -view in front of you. On the left side, you can see all the thesis' of the chosen query, and the rest of the queries. Before the actual report, you can see a set of filter options. With those, the manager can edit the report. At the top of the page you can the expertise matrix of the panel. It reveals weather the reported material has some expertise holes. This is an important information when targeting the communication. With the help of the matrix and grouping, it is also possible to divide the group under examination into subgroups. We will show how it is done in the next phase.

the manager can hunt the “case news” by browsing the report question by question. The most promising questions are the ones that disperse according to one or both criterion variable. This usually means, that the panelists’ views of the future development differ from each others, and it is necessary to deepen the conversation and the argumentation for and against different futures. For most people it is easier to examine the differences, if 2D graphs are switched into 1D diagrams. You can do it by choosing the option SHOW 2D GRAPHS AS 1D GRAPHS, and then clicking the APPLY FILTER button.

Let’s scroll down the panel until the question about Biological evolution. It is divided according to probability, but it is almost unanimous when it comes to desirability. The question is a dystopia thesis, and half of the panel thinks the alleged future will actualize. It  is a snapshot, which should be pointed out to the panel, and ask them to continue and deepen the conversation.

For a change, let’s send the mail from SEND EMAIL-page. Open the ADMINISTRATION tab, and click on the SEND EMAIL under the PROCESS heading. The downside of this function is that the message doesn’t include the link to the panel. The other weakness is that the mailing list doesn’t support subgrouping. 

Reviewing the results

Let’s go back to the Query Results. At the top of the page, you can see the title FILTER. From the fields below you can choose how the results will be filtered. From the INTERESTS AND EXPERTISE GROUPS section you can filter the results according to the expertise matrix, if you have created one. Activate the cell or cells of the matrix by tapping them. From the USER GROUPS section you will see all the panel groups you have created. Activate the group or groups which results you want to examine.

From the OTHER OPTIONS you can choose whether the result graphics will be shown in one dimensional or two dimensional form. When you have chosen all the filters, click on the APPLY FILTER -bar.

Now you can view the filtered results in eDelphi. Below each of the theses, you can see the results of the answers in graphic form. Below the graph you will see all the panelists' replies combined with the comments, as we already showed in previous phase.

You can also export the report as a PDF-document or a google-docs document. Below the filters you will see the EXPORT REPORT section. From REPORT CONTENT you choose whether you want the report as a PDF or as a Google document. From REPORT CHARTS, select the type of file  you want the report graphics in. From REPORT DATA you choose whether you want the report’s data as a CSV -file or as a Google spreadsheet. Exporting or downloading the results might take a while depending, how large the response data is.

eDelphi has also a possibility to compare reports. Open the ADMINISTRATION tab of your panel. Under the PROCESS heading, you will see the title REPORT COMPARISON. Click on it.

At first, you will see two empty fields in front of you. You can compare a filtered report either to the panel’s whole data, or to another filtered report. Select a query and the filtering options from the icons above the fields. You can download or export the report in the usual way. You can go back to the ADMINISTRATION tab through the browser’s Back-button.

During the Delphi round, especially interesting picks are the questions that divide the panel strongly. It signals that the question has a controversy status, which means that versatile argumentation and discussion are especially necessary. Often the controversy is a good basis for communication during the Delphi round. In addition to communication, the manager has a few other “dramaturgical means” on hand.

Facilitating the panel

The most common manuscript for a Delphi round is to start the query so that the panelists won’t see each others’ answers, but answer independently. Then on the second week of the round - if not earlier - the manager opens the queries. It is also a good way to motivate the panel to reanswer.

It is also possible to open and close queries. Sometimes an opening or a narrow query is used, which means that the manager adds new queries only when the panel has already answered the existing queries. You can hide a ready made query from the panelists by clicking the EYE-icon next to the query’s name. You can do the same with individual questions, which can be open or closed. Each question has four options: VISIBLE, COMMENTABLE, DISCUSSIONS VISIBLE, REPLIES VISIBLE.

It is also possible to arrange the questions in new order in any phase of the round. For example, if the manager wants the discussion to concentrate around a certain question, it can be dragged as the first question in the query.

Let’s also remember, that the manager can direct the invitation straight to a certain query instead of the panel’s main page. In eDelphi, we have had some good experience in arranging a “rush hour” at some point during the Delphi round -usually towards the end. It means 1-2 hour span, when the panelists are especially advised to answer.

Follow-Up Rounds

The Delphi method has originally been based on iteration, ie. cumulative and gradual accumulation of data. With modern technology, it is not always necessary to go through multiple Delphi rounds, but the iteration can be executed as so called one-round solution. For this method variation, we have an established concept called Real-Time Delphi, which has been increasingly popular during the last years. Real-Time Delphi is fast to execute and it doesn’t burden the panelists as much as a Delphi process that is formed during several Delphi rounds. Even in roundless Delphi, it is possible to produce more questions  towards the end of the round from the comment data the panelists have produced.

However, the standard Delphi form is still to execute the panel in 2-4 rounds. For each round, the manager creates a new query or edited version from the previous query or queries. There are several techniques at hand to do that. Probably the most commonly used technique is to copy the first round with its data as a basis for the second round. The manager then deletes the questions with clear results already from the first round, for example in the form of the panel’s relative consensus.

Let’s copy the first-round query “Technological Singularity” by clicking the icon next to its title. This will open a dialogue window. Write the name of the copy in the title field, and then copy the query with data. Then delete from the copied query those questions, that won’t continue to the second round. In order to do that, you have to first empty their data through QUERY RESULTS-page, by clicking the TRASH CAN-icon next to a question. When the data has been removed, open the query in edit mode. Now you can delete the empty questions and create new ones to replace them.

For the second round, the manager should choose the parts with the process clearly still in progress for voting. New questions are created from the panelists’ comments. Let’s take the comment from the “Biological Evolution” -question: This will further increase the inequality in the world. It is a potential new question. Let’s form it a bit differently to create a new thesis for the panel to vote in the next round: “Biological evolution will increase the inequality”.

Often during the second round the research will widen and orientate, whereas it will deepen and concentrate around the most essential hot topics during the third round. The possible fourth round will often be so called feedback-round, where the manager asks the panelists to assess the essential results and conclusions of the panel.

In the end of the process the panel will be closed, unless it is used as a media page for the results, which is often the case. Barometer-type panels are a case of their own, when the data collecting and creation will renew in regular intervals.