Four phases of the eDelphi Process, Part 2: Building an Expert Panel

Four phases of the eDelphi Process 2/4

 

During its twenty years of development, eDelphi has been built as a network utility for qualitative delphi research. It emphasizes the interactive learning process of an all-round and multidisciplinary expert panel, instead of just data collecting and recycling. This kind of argumentative eDelphi process consists of four phases that are:

  1. Creating a panel and a query
  2. Building an expert panel
  3. Executing the delphi rounds
  4. Analyzing, presenting and reporting the delphi results

In the second part, we will build an expert panel. The Delphi method is fundamentally based on different cognitive and social abilities as well as skills of the experts. These abilities should be diverse enough to cover the whole phenomenon under discussion. That’s why recruiting the experts is a fundamental part of managing the method, and a requirement for reliable research.

In this video instruction, we will first go through the criteria for recruiting panelists, and how to create a so called Expertise Matrix. Secondly, we will introduce how to invite panelists to the panel, and how to manage their profiles as a panel manager. Classifying the panelists will be important for the reporting phase, when the manager examines and analyzes the results.

As an example of the Delphi process, we will be using the four possible routes to end up to technological singularity, sketched by Vernor Vinge. By technological singularity he means a new evolutionary phase when the significance of the machines will outweigh the human abilities. The phase will be a result of a changing relationship between the human and the machine. The use of the theme is not a statement for singularity, but a tactical decision to choose a theme that does not grow old immediately. For this video instruction there is an open delphi panel with the title Technological Singularity: https://www.edelphi.org/technological-singularity .This panel is referred to as the Case panel later in the instructions.

Part 2: Building an Expert Panel

Expertise Matrix

Linstone & Turoff formally defined Delphi method “as a method of structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem”

This expertise group formed by individuals is chosen by a two phase recruiting process.

In the Delphi-method, it is crucial to recruit the panel in a way that covers the expertise and parties that are relevant to the phenomenon in question and its future perspective.

This ensures that the argumentation of the thesis and themes will be diverse and comprehensive. In eDelphi, it is possible to create a two dimensional matrix for this purpose. The matrix will help both in building the panel and in analysis, when the answers and results of any part of the matrix can be examined separately. Dividing the panel also helps in management and communication during and between the delphi rounds.

It is possible to place the panelists into the matrix in any phase during the process. That means the manager does not have to make it beforehand, although they should at least sketch the matrix on paper in the beginning, since it will be helpful when choosing the panelists.

Open the ADMINISTRATION tab of your query. At the top of the page you will see the PROCESS header, and below it, PANEL’S EXPERTS. Click on the title.

Now you will see the MANAGE PANELS EXPERTS view in front of you. When you are creating a new panel, the EXPERTISE MATRIX is empty. The matrix will be built with the interests on the vertical axis and the expertises on the horizontal axis. The matrix should not be too large, because it makes it difficult to control. Five interests and expertises are usually the maximum amount. If one cell has only one or two panelists, the division does not  serve the analysis. Even in the case of too few panelists per cell the division might be useful for ensuring the diversity of the panel.

The matrix is created under ADD INTERESTS AND EXPERTISE GROUPS OF A PANEL. Type the interests of the vertical axis in the ADD NEW INTEREST field, and click on the ADD button. Type the expertise of the horizontal axis in the ADD NEW EXPERTISE field, and click on the ADD button.

For the case panel, we will create five cognitive expertise categories according to the  PESTE model: Economy, Technology, Ecology, Social Expertise and Politcal Expertise. For the Interest groups, we will use three operative sectors of the society.

You can edit the titles of the matrix by clicking on the ”paper and pen” -icon next to them. You can delete a title by clicking on the ”trash can” -icon next to it. Editing can also be done after you have placed panelists in the matrix cells.

When you have created an expertise matrix, you can place the panelists into the matrix by dragging the name of the panelist to the cell of your choice. One panelist can be placed in multiple cells. You can delete panelists from the matrix by clicking the cross next to a name. There is a more natural way of classifying the panelists, especially if the matrix is very big. This will be introduced later. It is not unusual for a matrix to have some empty or half-empty cells. The subjects differ in this regard. There might be combinations of expertises and interests that do not mix logically . For example, an expertise for experience can not by definition be simultaneously an academic researcher.

Finally, we will introduce one more feature that prevents flexible editing of the matrix. It is possible to create a question for panelists based on the matrix. This feature is mostly used when the panelists are exceptionally collected openly, through a so called snowball method. In this case, the manager does not know the panelists beforehand. If the manager creates an Expertise Matrix Question, it is clear that the matrix can not be changed afterwards. The self-classification of the panelists will not affect the report, so it is not possible to make comparable material divisions.

User Administration

The manager can examine and manage the placings of an individual panelist in the Expertise Matrix from the ADMINISTER USERS page. Open the ADMINISTRATION tab. On the bottom right corner you will find the USERS heading, and below it, ADMINISTER USERS. Click on the title!

Under PANEL’S USERS you will see all the panel experts. Clicking on the panelist’s name will display the panelist's profile on the right side of the page. First you will see the basic information of the panelist. You can specify a person's role in the panel under ROLE.

The options are a Panel Manager, a Panelist, and a Panel Guest.

Here the panel manager can add manager partners for themselves.

Below the USERS EXPERTISE heading, you can see the expertise matrix and the panelist's placements in it. You can add and remove the expertises and interests of individual panelist here too.

The Expertise Matrix has significance also during the reporting phase of the panel. You can filter the query results through the expertises and interests of your choice. Thus the matrix offers a lot of different possibilities to analyse the results.

You can also remove panelists in User Administration, for example when the open panel has resulted in unwanted guests. You can do it simply by clicking the profile-icon next to the panelist’s name. Removing the panelist will require a confirmation!

Administering Usergroups

Through user grouping, you can divide your panel more, in addition to what the Expertise Matrix allows.  This makes it possible, for example, to filter the female panelists in one group, and the male panelists in another. Respectively, in the reporting phase you can examine the results of one individual group, or compare the results of different user groups. You can create a user group in any phase of the process, but the panel’s stamps will not allow you to do it retroactively.

In the lower right corner there is the heading USERS and below it, ADMINISTER USERGROUPS. Click on the title.

Now you can see the MANAGE PANEL’S USERGROUPS view in front of you.

On the left side of the page, you will see the title PANEL’S USERGROUPS. You can create a new user group by clicking on the icon next to it.

Enter the name of the group in the upper text field. You can see all the users of the panel in alphabetical order on the left side. On the right side, you can see an empty column below the title USER GROUP MEMBERS.

To move members to a user group, activate the name of the panelist from the left side and click on the arrow key pointing towards the new user group. You can remove a panelist from the group in the same way. If you want to find a certain panelist, you can enter the name you are looking for in the SEARCH FROM PANELISTS/ SEARCH FROM USER GROUP USERS. When you are ready, click on the SAVE-button.

Now you can  see the new user group on the left side of the page. You can edit the existing user groups by selecting the group title, and delete a group by clicking on the cross next to the title.

Inviting Panelists

Open the ADMINISTRATION tab of your panel. At the bottom right corner of the page, you will see the USERS heading and below it INVITE/ADD USERS. Click on the title.

Now you will see the INVITE USERS TO THE PANEL -tab view. On the right side, you will see the panelists already participating to the panel. At the top left you can see the INVITE/ADD USERS section. This allows you to invite individual users to the panel.
Enter the e-mail-addresses of the invitees to the field below the title one at a time.

When the software offers the correct address, click on it. The addresses of the invitees will appear below the USERS TO BE INVITED heading.

In the INVITATION section, you can edit the invitation mail to be sent to all new panelists. You can customize the invitation according to your the panel. The data in square brackets will be automatically updated. Do not modify or delete the text ACCEPT URL. It contains the invitation link to join the panel. However, in the end of the letter there is a text part that offers a link to turn down the invitation. It is used only in the rare case that the manager hasn’t agreed with the panelist about their participation to the panel beforehand. This text part should be deleted, since the panelist can also accidentally   click on the link, which will eliminate the panelist for good. After the refusal, the manager won’t be able to invite the same panelist again without the chargeable help of a coder.

Below the label ACCEPTING THE INVITATION REDIRECTS TO you can define into which query the acceptance link will lead. The default setting is the main page of the panel.

If the panel consists of several queries and you would like the panelists to answer only one of them, choose the query in question.

By clicking on ADD USERS DIRECTLY, chosen panelists will not receive an invitation, but they will be added directly to the panel. In this case, eDelphi will announce the password of the panelist or panelist group in the uppermost part of the page. Save the password for the occasion you want to instruct the panelists on how to change their password.

When you are ready, click on the SEND INVITATION button.

After a while, the panelist will receive an invitation to their email. By clicking on the invitation link, the panelist can access the panel page specified by the Manager. When using the invitation link the panelist does not need to  log in to eDelphi.

CSV-File

If you want to add a large number of panelists at once, you can do it effortlessly by making a CSV or TXT file of the panelists email addresses. You can do it, for example, with Google Sheets, by typing each e-mail on its own cell without punctuation, and downloading the file as a CSV. You can create a TXT file with google docs, using the same procedure. In google docs type each e-mail address on a separate row and download the file as a TXT. Corresponding measures will work with the same definitions in Microsoft Excel and Word.

When you have created a CSV or TXT-file, open the ADMINISTRATION tab of your query. At the bottom of the page, there is the USERS section, and below it, INVITE/ADD USERS. Click on the title.

On the left side of the page, you will see the title INVITE/ADD USERS and below it, CSV FILE. Click on ADD FILE. Select the CSV or TXT file you created from your own files and then click on DOWNLOAD CSV. Now you will see all the email addresses of the file below USERS TO BE INVITED. To invite panelists, click on SEND INVITATION at the bottom of the page. Before sending the invitations you can edit the invitation text.

Finally a few words about the qualities of the panelists in addition to what is required for the expertises and interests.

Useful qualities for panelists are the ability to discuss and interact, and openness for alternative perspectives. When it comes to learning abilities of a panel, it is useful to have individual panelists with forecasting abilities, imagination and creativity. Hopefully some of the panelists have the ability to see patterns when the others will see only random elements.

The energization of a panel also depends on the panelists’ readiness to seize the unexpected future possibilities, which the eDelphi process offers. But now we are already talking about the qualities of a manager, which might in many ways differ from the profile of a traditional researcher. This is an important side theme of the next instruction video. The main theme is the actions during the Delphi round.