Back to Main Decision Making BluePrint ​http://trac.sahanapy.org/wiki/BluePrintDecisionMaking

VERSION 1.0

SUMMARY AND DISCUSSION -- paper is attached too

14.1 Introduction

The objective of this chapter is to provide a summary of this research effort. A software system was designed and developed to support large groups of crisis experts making decisions in time critical extreme events. The overarching question that drove this research effort was: Is it possible to create a web-based system that will enable dispersed groups of experts or knowledgeable individuals to share and evaluate information and opinions, expose disagreements, and reach decisions more quickly than they could have without such a system?

Built as a module to add to the existing Sahana Disaster Management System, this addition later named The Delphi Decision Maker, was created and implemented. A series of tests were run on the prototype including: individuals as system testers, a smaller group used as a pilot study, and a larger set of participants in a field study. Analysis was conducted by triangulation where proof of concept, quantitative (simple statistics) from survey analysis and qualitative (descriptive) statistics of content coding were used from the discussions in the forums. Reliability measures were used to test the survey questions. A prototype of the system was built and implemented, thus satisfying Proof of Concept. Results from the participants taking the Post Study questionnaire also provided positive support for the overarching question. Further, comments from the participants provided more positive feedback that it was possible to create such a system. Hence, there is support for the main research question.

In order to help this system to be successful in that crisis experts would want to use it, other research questions asked, What features or functions would be most useful for this objective? And, what would be required to make the design of the system understandable and useable? A basic set of four tools {Voting, List Creation, Feedback Scale, Discussion Forum} was created to help guide a group through the decision making process. The design of the system mimicked that of another software program, Web2Py whose design was considered easy to use. There was positive support for these research questions. Participants were asked if they found the tools useful, both individually as well as together. People surveyed confirmed that the tools were useful. Also, research findings supported that the design was easy to use. However, in both areas, there was room for improvement.

In emergency management, especially given time critical situations, it’s best if an emergency management information system is used for day to day tasks, or at least on regular occasions. If people frequently use a system, they will be more apt to use the system during an emergency where duress and other factors may place undue pressure on a user and their ability to make effective decisions (Turoff, et al, 2004). Participants were tested to see if they perceived that the system and the function it served, generating and creating a ranked list, could be useful in general. The following question was posed: Can the system be used for planning and other tasks that would make it a regularly used system, eliminating need for special training for use in emergencies? Survey questions were created to test this; reliability measures confirmed that the set did in fact test the same thing. Participants were surveyed providing positive feedback and statistical results to support that this system could be used for other, non emergency tasks. Further, through content coding, various comments supported these questions with agreement. Thurstone’s Law of Comparative Judgment influenced the design of the system. A scale based on this method provides further information versus a ranked list where each item is proportioned one against the next equally. Thurstone’s method shows how much closer an item A is to another item B, versus that same item A to another item C. This provides more information to the user as it shows how much more one item is desired over another. Participants were tested to see if they perceived this to be additional information to help them make better decisions. The research question was posed, Will using a visual scale based on Thurstone’s Law of Comparative Judgment help the expert make a more informed or confident judgment? Participants were surveyed using a set of questions designed to measure this question. Reliability measures confirmed that all questions were asking the same thing. The results to these survey questions found that participants did believe that they were making more informed or confident decisions. Thurstone’s Law of Comparative Judgment also influenced the design of how people would rank a list of items, by paired comparisons. Participants were used to determine the next research question, Will experts use voting to reflect their opinion over time? A set of questions was designed to answer this question. First, it needed to be known if the user used the voting tool at all. Voting by paired comparisons is not a natural or normal way people create ranked lists, so it was important to determine if users would accept this form of ranking. Most participants voted and further, they used voting to reflect changes they may have had. However, as described at length in prior chapters, this feature was not properly tested as there was a high level of agreement to start with, in the main trial. Further, it was determined that the task type and stakeholder involvement would influence the usefulness of such tools. If a user has a vested interest in the outcome of the rank where there are repercussions somehow in the results of that rank, the user will need to view the information provided more because of a vested interested and the ramifications of the end results. (Move what follows to the future section) This is the basis behind the model that is represented next. This shows how the tools are possibly influenced by the task type, group proximity and size.

14.2 Limitations A limitation of the study was that a group of students was used to test the system. Second only to the hope of winning $100, extra credit was the motivator for many as they were in one of the researcher’s classes. The Delphi Decision Maker was designed for a particular group of people, experts. Not being able to use crisis experts was the limitation where the problem task didn’t meet the demands that are required to use a system such as the one developed. This system was developed also for time-critical situations and this is an element that cannot be tested given the guidelines for research on humans. So, basically, the toolkit presented was tested at a basic level which served other purposes, but did not test the system for what it has been developed to tackle.

In addition, the number of subjects was low, which made it impossible to test hypotheses about relationships among variables. Also, the problem chosen for the main trial did not generate much disagreement, which limited the usefulness of the voting and discussions. All of these problems point to the need for further research, discussed below.

14.3 Contributions Summary The primary contribution of this work is that an innovative and useful free and open source module has been developed and implemented to meet the unique characteristics of extreme events and to support the decision making needs of the crisis experts trying to manage them. This system was found to be useful for day-to-day tasks of the same nature which would keep the users of the system more familiar with it and thus, more successful at using it during time critical, emergent situations. In addition, the system can be used as a learning exercise where other situations are the focus. Another major contribution to this work is that a method was developed where a modified version of Thurstone’s Law of Comparative Judgment was created to calculate uncertainty with missing data. This will give the users further information from which their insight and intuition may be further utilized, decreasing gaps where no information or knowledge exist. This will be implemented in the next version which is described below.

14.4 Future Research Plans This research effort is guided by the seven steps of Design Science according to (Hevner, March, Park and Ram, 2004). The sixth requirement guiding design science is that the software should be tested to determine if it is satisfying the needs of the organization and to modify the software system after testing it to make it a better fit to the characteristics of the group for which it is being designed. Presented next is a model that was developed to guide the next iteration of studies. Modifications to the system are listed and the functional requirements for the Delphi Decision Maker, Version 2.0 are described.

14.4.1 Hypothesized Model of Relationships amongst Constructs

Figure 14.1 Model of Constructs

The constructs that underlie the evaluation effort are: Usefulness, Ease of Use, Confidence in the results, satisfaction, usage, and the major functions of the discussion forum, the linear Thurstone’s scale, and the voting process. The constructs were created to measure the system against the research questions. The tools were measured as constructs and were found to be useful. However, there is a direct relation between a basic set of group support tools and the context in which they are used that influence the Usefulness of this system. It is influenced by the task the group is working on, the group’s proximity and size. These characteristics will be investigated further as testing continues for their positive or negative influences on the usage of the system. Another influence on the construct Usefulness is the accuracy the experts find in the information provided by the Scale and Voting tools provided. This was very important as the expert needs to be confident that those tools provide an accurate reflection of what they are thinking and what the group is thinking. Participants found both that using the paired comparisons for voting was an accurate way to reflect what they were thinking, and that the scale provided an accurate reflection of the result. That the users had confidence in these results increased the chances of this system being used. A system that is easy to use will more likely be used. That this system was found easy to use which users needs to be able to accomplish tasks with minimal effort. This also made it such that the system would be likely to be used. This model will be tested in the next round of studies after the next version (2.0) is developed and implemented. 14.4.2 Major Problem and Design Areas Identified The system was found to work well when the number of items in the Option List was low. There weren’t any complications and the system worked well for up to 6 items. As the number of items on a list grew, the problems surfaced and were linked to other problems. We found that there was a direct correlation between the number of items and voting as a problem. This is because there is a logic check in the paired comparisons ensuring there are no cyclic triads. This is where some option A is selected over B, then B is selected over C, but then C is selected over A. Although we conducted a logic check, this is not anywhere in the former literature and is something we are doing to help experts in their consistency in judgment during the trying times of crisis management. In our studies, users found it most difficult to be consistent. This was not a stressful problem but was interpreted as simply having too many items under consideration to remember the sequence. This is going to be addressed two ways: one, users will be able to select a subgroup of items and then, the items that are not correct in logic will be brought before the user and the user will have the option to clarify further or enter as initially selected. In version 1.0 of the Delphi Decision Maker, users were forced to compare all items. Given that the number of comparisons is based on the formula where N(N-1)/2, the number of comparisons gets unreasonable when reaching even smaller numbers of 5+. So, since people could not get through the logic check, their votes could not go through. Since uncertainty was based on members who had not voted yet, this problem snowballed. Since people’s votes wouldn’t go through, the uncertainty grew to a point where results were equal to those who had already voted. This rendered the uncertainty useless; it was barely useful initially and will need major analysis. This is discussed in the conclusion of the chapter on the Field Study.

Transaction Log For further studies, it was determined that a list of a variety of transactions needed to be gathered and maintained. This will help researchers determine which tools or actions trigger other tool usage or actions. A transaction log will help support the ability to study how the system is used, by which users and why. This can help determine if new items are created when disagreement is high and other relationships between the available toolset. Each ‘Active Problem’ will need its own transaction log that only records the actions that are between the members of that particular group. Particular information needs to be recorded:

• When someone votes - for each item - (not a change in vote but just if it's voted on initially)
• When someone changes a vote (for Each item available)
• When someone posts or replies to a discussion
• When someone adds an item in the Options List
• When someone views the Scale
• When someone views the Option List
• And time-date for each user,

o Initial vote on an item o Change in vote o Post/Reply o Add item o View item list and range or item number All integer fields need to have sorting ability so that analysis is easier and more structured for a more accurate analysis. It should also be available in a format such that it can easily be imported into statistical software and spreadsheet programs, i.e. a simple table. To maintain anonymity of personal data, each instance can be identified by a primary key. Monitoring functions such as group formulation, edits of material, the addition of new members, etc., needs to be part of the transaction log as well as other roles that can make some contribution.

Model A model of the system is presented. This model shows some preliminary views on how these tools may be used together. One of the underlying features of The Delphi Decision Maker is that the scale will be used as feedback to help groups focus on areas of disagreement, thus saving them time. Through saving data through a transaction log, other information like this will be available to discover such patterns of use.

Figure 14.1: Model of Possible Tool Use This model describes one possible way in which quantification (the scale) could be used to help focus groups on areas of disagreement. However, until a transaction log is created, such patterns of use cannot be confirmed.

Miscellaneous Usability Changes These are changes that are small but make a big difference. This will be implemented in Version 2.0.

1. The bottom of discussion screen needs a ‘go to top’ AND options to vote, scale, etc at bottom (Same as on top as menu, but located at the bottom to make availability easier and quicker).
2. After a User votes and clicks Submit, a confirmation that the vote went through needs to be made to the user along with a temporary visual scale of their results.

Item Selection for Option List Creation This tool requires modification so that larger lists of Options can be managed effectively and efficiently. If a user selects a subset of a larger list, this will expedite the time it takes for the crisis manager to create a ranked list using voting as a tool to reflect what the expert is thinking. This was initially in the Version 1.0 proposal, but due to the lack of time, could not be implemented. We still hold that the following design would be a good design to implement for this task. This sort of item selection is already used in other software systems like Moodle and SAS, Statistical Analysis Software. Figure 14.2 Item Selection of Sub-List Screenshot This Muddling Through is supported in the research to make decision making more efficient and effective by allowing experts the flexibility to select only what they want to consider. Version 1.0 had a forced vote where all pairs of items had to be voted on. Otherwise, an error would occur and the vote would not go through. This was not good as there were too many paired comparisons which caused other problems when getting through the logic check. That will be described later with solutions posed.

Items will be more dynamic and informative. The modification for this will be:

• When a User adds an item to the Option List, they can give a description to define or explain it better. This is to lessen ambiguity between individuals and the options they pose.
• When a User adds an item to the Option List, a corresponding Discussion Forum will be created automatically. This Forum will automatically be given the same name as the Option item created.
• The version 1.0 Discussion Forum in place will be removed.

Voting This tool gave this research effort the most trouble and needs modification if this system is to be successfully used in the real world. Having so many voting pairs caused great problems given there was a logic check in place checking for cyclic triads. We have worked to modify the system to reduce the number of paired comparisons in two ways. One, allowing smaller sub-lists to be selected to rank will reduce the number of items to compare. Two, by having each item paired up with another item such that the least amount of required comparisons is made. On a user’s initial vote, for example, if there are four items A, B, C and D, the following comparisons will be made: (A,B), (B,C), (C,D), (D,A). From this information, transitivity will be used and the remaining cells will be filled with the information that is present. Next, if an item E is added to the list, a binary search algorithm will be used to place the new item E, into the existing ranked list, if and only if the user wants to accept the item as a consideration. Otherwise, the user can ignore the new item suggested altogether.

Logic Check Modifications When a User votes and does not pass the Logic Check – A screen will return to the user telling him/her of this error. The screen will have the Users vote on selections returned and the items that are not passing the Logic check will be pointed out to the User. The items could be bolded and in red with an asterisk by them so that Users can quickly see the problem. At this point, the User can either:

1. Make other selections and click Re-Submit.
2. Not make any changes and click Submit Anyway.

Discussion Forum The Discussion Forum will be improved by adding more flexibility to the user and more alternatives on what functionalities are available for them. Discussion Forum Improvements

• A User will be able to Edit posts – anywhere in the thread.
• A User will be able to Delete own posts if no thread exists
• A User will be able to Insert pictures
• A User will be able to Upload Attachments
• A User will be able to make an Anonymous post
• A User will be able to Post or Reply using their Pen Name
• When a User reads a post, it will be marked as ‘read.’ This will be accomplished by having new posts in bold letters and ‘Read’ posts NOT in bold.
• A User can click a box next to any message/comment/reply item and Mark as ‘Read’
• A User can have the option to ‘Mark All Read’ where they did NOT have to check the box next to the post.
• We would like to have hypertext ability where, given you are in Option A’s Discussion Forum and mention Option X, Option X will automatically be a keyword which will link the Option X keyword to Option X’s Discussion Forum. This would hold true for any Option’s Forum. If any Option on the list is referenced, when the keyword is typed in, a link will automatically be created linking this option’s keyword to the option’s discussion forum.

Moodle is a Free and Open Source Software system for course management. Although it is written in PHP, the developer is trying to change it over to Python and implement its discussion forum structure for our use. Forums will be available for every single item in the Option List and can also be started for new topics.

User Permission User permissions need to be implemented such that groups can have more control over membership. This is important for a variety of reasons. There could be a need for a closed group of members who are representatives and have a vote for or against some policy. This will also allow more structure in large groups such that some people have the ability to vote, and others can contribute ideas but not vote or whatever combination of needs that may be required for the given problem. User permissions can be seen in the User’s Profile. New options will be:

• Pen Names – Pen Names will be added so that users can be anonymous but still have an identity.
• Profile Picture – pictures can be added so that the user has some visual representation along with a name. This is important in online environments so that when these people are face to face, they recognize one another. Avatars also make you feel like you know the person better because it’s another form of representation/identification and richer than a text based name.
• Profile Email – a primary email should be given so that experts can be mailed by the moderator.
• Profile Update Information – the user will be able to update their information under their profile
• Description – A user will be able to add information on their domain expertise, listing the areas in which they are considered to have expertise.

Group Permissions and Classifications The studies were conducted and many problems were created, but there was a problem in that every problem reflected the total number of users of the system and not the group targeted for that specific problem. A solution is posed for these related problems where group permissions will expand. There will be different permission levels desired for the system.

1. Guest –guests will be able to view the Active Problems List, Scale, and Item List.
2. Guest - will NOT be able to vote.
3. Guest - can NOT view or contribute to the Discussion Forum.
4. Contributors – Contributors inherit the permissions of the Guest.
5. Contributors – can see and post/reply in the Discussion Forum.
6. Contributors – can add items to the Item List.
7. Contributors – can NOT vote.
8. Participant – inherits both the Guest and Contributor privileges. However, the Participant can Vote.
9. Moderator – A moderator inherits all available permissions.
10. Moderator – can create a new problem
11. Moderator – can edit or delete any item as an option.
12. Moderator - can set up groups by allowing or inviting individuals to the group.
13. Moderator – can email all participants.
14. Moderator – can designate members in a group able to review and accept another person who wishes to join the group.
15. Any member of the system in any role can be an observer in a given group and request a participant or contributor role.
16. Groups need to be able to select the group members and/or accept membership requests. It needs to be such that a can accept or reject individuals for participation if a participant requests to join, but each group needs to have a designated set of individuals that are working on a single problem area. This will have to be tested after it’s designed to see if it fits the needs of the organizations.

14.5 Uncertainty Calculations in Problems As has been stated throughout these observations of the studies, support for types of uncertainty needs to be explored. For this Version 2.0, the Voting Uncertainty feature will be implemented. Given that groups will be closed and membership privileges will define which members are allowed to vote, it is hoped that the Version 2.0 information from the calculation will be of more benefit to the user as it was intended. Uncertainty comes in many forms and there is no one overall calculation that is applicable to any or all problems. The uncertainty cannot be identified until the problem is identified. Many variations of uncertainty representation should be explored in hopes that a set of choices can be selected by the group members in a given group. Some examples are:

• Votes and voter uncertainty can come into play when each member gets to add weight to a decision using a Vote.
• Uncertainty can be calculated by forecasting events that are likely to occur.
• Uncertainty can be in resource availability.
• Uncertainty can be calculated from a subset of ranked items.
• Uncertainty can be created from Risk Assessment.

Until the problem is identified, the uncertainty cannot be identified and therefore cannot be calculated. So, each problem should be explored for the types of uncertainty that will be of some benefit in the decision making process of the expert. Eventually, a set of uncertainty calculations should be available to the expert where the group can choose to use the ones that are beneficial for a designated problem type.

Voting and Uncertainty Voting was used as a calculation for uncertainty in Version 1.0. Uncertainty was calculated use of by those participants in the trial who had not voted on an item. Although this did not work well for the Study, it could easily work for the right group under the right problem type. For example, recently the House passed the Health Care Reform Bill. Votes meant everything. The number who had not voted yet was used to calculate a measure of uncertainty by assuming all those not voting could all vote one preference or the opposite one to give high and low values for the final result. This did not work well and resulted from trying to a quick estimate as opposed to the original specified uncertainty measure. This type of uncertainty calculation, although it was in need of interpretation and fine tuning, could have been used for such a situation as a Bill being brought before either the House of Representatives or Congress. A Screenshot 14.1 is offered as an example of a real vote occurring from the Health Care Reform Act.

Screenshot 14.1 Voting Used as Uncertainty Problem Type From the study conducted, many problems were created on The Delphi Decision Maker. It’s from the analysis of these individual problems showed particular areas where uncertainty could occur, but each was different and no one uncertainty calculation satisfied all. Until the problem is identified, the uncertainty cannot be identified. Each of the problem areas are explored as such next.

1. Resource Allocation Problem. This problem was created during the development of the system to use between the designer, developer, and testers. It was based loosely on the aftermath of Hurricane Katrina and had been used in a paper published by the researcher and some committee members. Although this problem was not part of the study, this problem seemed to draw interest from testers and participants alike. In a real world problem, if this was under a time critical situation, this would test the system and perhaps have the tools utilized and leveraged as intended. There would be the basis for arguments. Stakeholders have a vested interest in the outcome and they could be life or death decisions.
2. Quadrennial Homeland Security Review. This is a problem based on past uses of Thurstone’s Law of Comparative Judgment that we ran prior pilots on; see the research proposal. This is where the Option List is already created and the participants vote on those items only. This problem used a group of experts who were knowledgeable about the information before them. This was a bit different in that no ideas were to be added to the Option list, but rather, a ranking of the most important ongoing topics was the goal. Good debates occurred and agreement between participants was used to judge the information and ongoing debate. The groups had a bit of bias in one category over another as they had submitted group projects presenting the most compelling arguments behind their ‘own self selected’ topic areas. However, there was nothing implemented, no consequence of the results.
3. Campus Threat Assessment. This is the problem that was used for the study. Although this was a good problem, it held no consequences. There was no budget tied to implementation, there was no outcome to be had. A problem requires that there be some compelling reason to want outcomes based on some need or requirement.
4. Dessert Problem. Although this was created as a Sandbox for participants to play in and use their newly found tools, the problem also brought up an interesting issue – some problems are based on things like taste and although people can be persuaded for a variety of reasons, if something tastes good, or bad, there’s no persuading someone to taste something differently. However, other considerations may make a participant willing to give something up for the greater good of the group. For example, diabetics should be considered, not to mention there’s an obesity problem so, low fat desserts should matter, although that is doubtful. So, all of these sorts of issues should be considered as some things, belief systems or taste cannot be argued although maybe negotiated. For example, if there were a problem saying ‘list and rank the best religions,’ it is doubtful that people could be persuaded to support anything but their own belief.
5. Sahana Framework Problem. This is a problem for the Sahana development community. For a while now, various frameworks have been proposed for the system to use. However, these discussions are unorganized and discussion is supported by an email list. There’s no way to know what the group thinks so, the site mentor of this research effort asked that this problem be added to The Delphi Decision Maker. The system will support something like this well as there are only a few alternatives being considered. This is a group of experts who know how to interact with one another in a post/reply format. One framework will be selected over another and so, argumentation could be used. However, this problem brings up another problem that actually, the Resource Allocation also brings up – that accommodations need to be made that help the decision makers by providing a list of factors that directly relate to the options being explored. For example, in the Resource Allocation problem, the number of generators, how much gas they use and how long they need gas should be available for group members to see. These sorts of pieces of information directly relate to the decision making and make a more informed decision for the participant.

The system worked when there was a small number of items on the Option List from which to choose. This meant that voting wasn’t confusing and also, the forum would be able to support so few topics. However, as the number of items increases, the system doesn’t manage the complexity well.

So, questions should be asked: 1) Will a solution be implemented? 2) Does the end result affect the stakeholder? 3) What is the end result going to be used for? What is the goal? 4) What is the uncertainty to be considered for the problems? 5) What type of problem is this? Forecasting? Ranking? Clustering? Building Categories from scale of results like in the Unified Scale paper?

14.6 Future Features for The Delphi Decision Maker 3.0 There are other features that should be developed to further support the efforts of the Delphi Decision Maker.

14.6.1 Expert Domain Group Identification When a New Problem is entered into The Delphi Decision Maker, there needs to be a way for the groups of experts to be formed. Groups could be formed using a number of alternative methods.

1. Database of Experts. First, a database of experts should be built. Information on level of expertise, domains of expertise, location and availability could all be attributes. This would give users the ability to search and retrieve experts and allow programs to be written to automatically identify and mine experts.
2. Moderator Determined. This is where the moderator or person in charge of ‘forming a subgroup’ of experts selects participants and reaches out to the experts for their contributions.
3. Expert’s Surfing. Experts could have methods bringing awareness to situations where they may be able to provide expertise. An expert could do this by having RSS feeds from the Sahana community or other sources. This way the expert reaches out to the problem. When the moderator is in charge, they are reaching out to the expert. Using a variety of methods for group formation may prove more beneficial. However, there may be times when a closed group method is preferred due to issues such as decision making power of authority.
4. Automatic Expert Mining. Other programs could be written automating lists of possible experts who could contribute to a particular problem. Experts could be registered to particular domains and upon selecting a series of related domains, a program could automatically contact and confirm membership participation.

These systems could use multiple methods of communications in order to quickly contact experts and confirm if they are willing, ready and available. Expert lists could be updated real time so that the most efficient use of experts could be deployed.

14.6.2 Dynamic Discussion Forum Although there are standard forums that can be used to satisfy the Discussion aspect of the system, a much more versatile format would be much better for the complexities and information needs of experts for this system. If forums could act more as wikis, this would be a beneficial technology to implement where terms could be linked and interlinked and new information better managed. However, there is a lot of research that needs to be conducted in this area for solutions. Other information from the Sahana system needs to be available and integrated into the discussion area such that real time information is existing and information exchange can be seamless.

14.6.3 LIVE Information Aggregation Sites Information needs to be collected in a way that is most useful to the expert’s needs, not overloading the user but supporting the user’s needs. Although search engines can retrieve bits and pieces of information, tagging information and other sorts of ways of aggregating data to a collective space should be a consideration. Information on a particular subject could be spread out between forums and other relevant places. This information is chronological by nature due to the timeline of actions and decision making. However, information should also be directed, either by humans or automatically or both such that there is a direct link that experts can go to for a comprehensive but organized representation of the information on some given option, problem or situation.. An example of a live site is provided by Information Systems for Crisis Response and Management, www.iscram.org/live. ISCRAM has it such that tagging information collectively by group members and having a site receive and organize such information – real time – makes for a useful place to find information during an event. Real time information may be coming in from various people using a variety of methods, information is streamed to the site, viewable by anyone, and others have space to contribute documentation. This is a media rich environment that supports both outgoing and incoming information and activity. It’s this sort of ‘portal’ that could be effective in managing the particular needs of the experts and decision makers.

Figure 14.2 Information Aggregated on ISCRAM Site 14.6.4 Integration of Sahana Modules

Many other upgrades will be made in Version 3.0 such that information from other modules in the Sahana system can be integrated with the Delphi Decision Maker. If experts and crisis managers can get the real-time information they need directly from the same system, it would make decision making even more efficient and effective. This would save time and since the information would be real time, decisions could be made that wouldn’t be outdated by the time the actions are reached by those on the ground.

14.7 Future Research Questions A set of research questions that were left unanswered from the Version 1.0 system will drive the study efforts where the Version 2.0 system will be implemented and tested. They are:

1. Will emergency management experts use such a system? What do they see as the advantages and disadvantages?
2. Will the discussion from disagreements lead to more new options being proposed?
3. How can voting be used to aid experts in ‘muddling through’ an initial set of items to create a subset which the experts determine to be the most important items for the group to work with?

Ways of answering these questions are under development. These core questions will help identify if the system is getting closer to its intended use.

14.8 Future Studies Future studies are planned where groups of crisis experts will be used. A request was posted to the International Association for Emergency Managers mailing list. From this post both groups of individuals volunteered as well as two groups of emergency affiliated responded and are available. It is from these studies where the true design efforts of the system will be tested. To help the researcher better understand the needs of the users, studies are in the works where the researcher would be part of a RELIEF effort that will hold the following series of evaluations:

• Sahana Testing at Camp Roberts (in the works)

 6 months Tabletop  9 months Functional  12 months Full Scale

It is very difficult to test the system using problems or scenarios. Testing the system in its environment will provide researchers with a more realistic picture of the situations in which the system can be best used and where other modifications can be realized to better fit the needs of the users. Once the system is deployed with the Sahana Disaster Management System, real cases can be evaluated. To be able to study the system from a behavioral perspective may aid the researcher’s effort to further support the needs of the users. It is important to study how people use the system. New information can be derived from these studies where the users could use the system for situations not considered by the researchers previously.

14.9 Conclusion Overall, this research effort went well. The system proved useful at the core, but many improvements need to be implemented so that The Delphi Decision Maker can be used by the emergency domain community to support them in the most critical times. A new model has been developed which will guide the next iteration of studies. Research questions have been developed to help measure the success of the design. Future studies are planned where the new version of the system will be tested again and further modified according to the needs identified through analysis. The last guideline to Design Science Research is that the work from this effort should be published. Many publications are planned from this work addressing the variety of issues that surfaced. Hopefully, the system will be studied as part of a larger effort where its use would be implemented in the field. This would shift the project from Design Science Research to Action Research. It is only through the real use of this system that the real contributions can be identified; if extreme events are better managed, the death toll is minimized, and a more efficient recovery is actualized then the system can be declared fully realized.