Changes between Version 6 and Version 7 of Deployments/SAMBRO/Evaluation

06/18/16 09:27:39 (6 years ago)
Nuwan Waidyanatha



  • Deployments/SAMBRO/Evaluation

    v6 v7  
    2626== Methodology ==
    28 We have broken the SAMBRO Early Warning System (EMS) dissemination subsystem activities into two components; namely Publishers and Subscribers. Publishers are essentially authorized alerting organizations who would originate or relay a warning/alert message. Both Publishers and Subscribers will undergo a series of tests to determine their aptitude to:
    29 * publish effective CAP-enabled messages
    30 * Subscribe to actionable (or inactionable) CAP-enabled alert/warning messages
    31 * operationalize a cross-agency situational-awareness platform
     28The evaluation hinges on the simple question: '''“did the technology and the people perform as planned on the day of the exercise?”'''
    33 We are applying two Human Computer Interaction (HCI) evaluation methodologies:
    34 Technology Acceptance Model (TAM)
    35 * Goal Oriented HCI
    36 * Technology Acceptance Model (TAM)
    37 TAM provides insights on the ease of use and usability of the technology presented to the users. This applies to both Publishers and Subscribers. This is a subjective analysis of the intervention.
     30'''NOTE''''''' that the technology and procedures should not be a surprise to the users. They should have undergone training and a series of silent-tests to be familiar with the system and be competent in operating and reacting to the system for all known scenarios.
     32=== Actors (System Users) ===
     33We have separated the SAMBRO users into three roles:
     341. Implementers - would ready the system with predefined data and processes necessary for the efficient use of the system by the Publishers and Subscribers
     351. Publishers - are essentially authorized alerting organizations who would originate or relay CAP warning/alert messages
     361. Subscribers - are authenticated users who would receive restricted or private CAP messages
     38With respect to the alerting cycle, there is a backward dependency among the actors. The performance of the Subscribers in achieving their intended SOPs depend the accuracy and timeliness of the messages produced by the Publishers and delivered by the SAMBRO system. The performance of the Publishers depends on the completeness of the implementation with predefined data and training regime carried out by the Implementers.
     40=== Research Questions ===
     41The evaluation process will attempt to answer the questions:
     42* Have the Implementers operationalized the National cross-agency situational-awareness platform to meet the user expectations and ease-of-use?
     43* Are the National Alerting Authorities competent in publishing effective CAP-enabled messages for a warming cycle?
     44* Are the Subscribers, namely First-Responders, able to decipher the CAP-enabled messages, received over the various channels, to determine the necessary actions or inactions SOPs?
     46=== Evaluation Process ===
     47These questions would be answered through a series of exploitation-evaluation activities, namely a series of controlled-exercises following a format of Simulations and Tabletop Exercise (TTX). The controlled-exercise activities constituted three steps:
     481. discussion of the operating procedures,
     491. actual execution of those procedures, and
     501. evaluating the outcomes
     51These activities are designed as verification exercises to determine the system’s usability (or complexity), usability, and utility. The table discusses the indicators, methods, instruments, and process of the verification process. [[br]]
     53The two types of users will be subject to scenario driven simulation and tabletop exercises.
     541. Publishers - will be presented with various scenarios and they will act on those scenarios by executing an appropriate alerting cycle (i.e. alert, update, clear).
     551, Subscribers (specifically First-Responders) - will be presented with alert messages, taking them through a complete cycle, to determine their response.
     57These exercises '''DO NOT INVOLVE THE PUBLIC'''.
     59==== Objective Evaluation
     60The objective evaluation is part of the controlled-exercises. The Publishers and Subscribers will exercise successive iterations of the steps forming:  a goal, intentions, and actions. This is a Human Computer Interaction (HCI) technique. The Publishers would execute the actions related to issuing alerts. The Subscribers would discuss the actions they would execute for specific alerts received. In each case the participants would perceive the state of the world, interpret the state of the world, and evaluate the outcome.
     62We will install the CAM Studio screen capture software to record the user Interaction Techniques. The screen capture software will capture the behaviour of the user and the sequence of executing each process. This will also provide the Evaluators with insights of the Usability and the Operational Efficiency of the system. The alert messages saved in SAMBRO will be used to determine the data quality as well as Email and SMS received by the Subscribers will determine the Data Quality.
    3963=== Goal Oriented HCI ===
    4064There are two elements to the Goal Oriented HCI: