wiki:Deployments/SAMBRO/Evaluation

Version 13 (modified by Nuwan Waidyanatha, 9 years ago) ( diff )

--

SAMBRO Evaluation Guidelines

Table of Contents

  1. Introduction
  2. Technology
    1. Issuing Alerts (Publish)
    2. Receiving Alerts (Subscribe)
  3. Methodology
    1. Actors (System Users)
    2. Research Questions
    3. Evaluation Process
      1. Objective Evaluation
      2. Subjective Evaluation
  4. Evaluation Instruments
    1. Cam Studio
  5. References

We provide a methodology and a set of evaluation instruments to be used in evaluating the implementation.

Introduction

Sahana Alerting and Messaging Broker (SAMBRO) is designed to foster Cross-Agency Situational-Awareness in support of improving institutional responsiveness to All-Hazards: http://eden.sahanafoundation.org/wiki/Deployments/SAMBRO . Alerting Authorities (Publishers) and warning recipients like the public and first-responders (Subscribers) can interconnect through the SAMBRO. It offers an all-hazard all-media approach whereby consistent, concise, and controlled alerts and warnings are channelled through multiple technologies to geo-targeted recipients.

To evaluate the performance of SAMBRO, along with the underlying Common Alerting Protocol (CAP) content standard and procedures, we are introducing an objective and subjective evaluation approach. The evaluation is specific to early warning and alerting. Through structured scenario-based simulation exercises, we will evaluate the performance of the users and the usability and ease of use of the system.

Technology

Issuing Alerts (Publish)

Primarily there are two types of devices that can be used for issuing messages:

  • Personal Computer (PC) - Desktop, Laptop, Notebook
  • SmartPhones (including Tablet PCs)

SAMBRO offers two types of applications:

  • Browser-based application that is accessible through a PC or SmartPhone
  • Mobile-based application (Android & iOS) that is accessible using a SmartPhone only

Receiving Alerts (Subscribe)

There are multiple channels for receiving alerts: SMS, Email, RSS, Twitter, Facebook. Only “public” alerts are received over Twitter and Facebook. SAMBRO offers two RSS feeds: one that carries public warnings and the other that carries restricted and private warnings. SMS and Email are purely for those Administrator and Self controlled Subscribers.

The user may chose to receive alerts on a PC, SmarPhone, or Standard Mobile Phone. Each of the devices have their limitations.

Methodology

The evaluation hinges on the simple question: “did the technology and the people perform as planned on the day of the exercise?”

NOTE that the technology and procedures should not be a surprise to the users. They should have undergone training and a series of silent-tests to be familiar with the system and be competent in operating and reacting to the system for all known scenarios.

Actors (System Users)

We have separated the SAMBRO users into three roles:

  1. Implementers - would ready the system with predefined data and processes necessary for the efficient use of the system by the Publishers and Subscribers
  2. Publishers - are essentially authorized alerting organizations who would originate or relay CAP warning/alert messages
  3. Subscribers - are authenticated users who would receive restricted or private CAP messages

With respect to the alerting cycle, there is a backward dependency among the actors. The performance of the Subscribers in achieving their intended SOPs depend the accuracy and timeliness of the messages produced by the Publishers and delivered by the SAMBRO system. The performance of the Publishers depends on the completeness of the implementation with predefined data and training regime carried out by the Implementers.

Research Questions

The evaluation process will attempt to answer the questions:

  • Have the Implementers operationalized the National cross-agency situational-awareness platform to meet the user expectations and ease-of-use?
  • Are the National Alerting Authorities competent in publishing effective CAP-enabled messages for a warming cycle?
  • Are the Subscribers, namely First-Responders, able to decipher the CAP-enabled messages, received over the various channels, to determine the necessary actions or inactions SOPs?

Evaluation Process

These questions would be answered through a series of exploitation-evaluation activities, namely a series of controlled-exercises following a format of Simulations and Tabletop Exercise (TTX). The controlled-exercise activities constituted three steps:

  1. discussion of the operating procedures,
  2. actual execution of those procedures, and
  3. evaluating the outcomes

These activities are designed as verification exercises to determine the system’s usability (or complexity), usability, and utility. The table discusses the indicators, methods, instruments, and process of the verification process.

The two types of users will be subject to scenario driven simulation and tabletop exercises.

  1. Publishers - will be presented with various scenarios and they will act on those scenarios by executing an appropriate alerting cycle (i.e. alert, update, clear).
  2. Subscribers (specifically First-Responders) - will be presented with alert messages, taking them through a complete cycle, to determine their response.

These exercises DO NOT INVOLVE THE PUBLIC.

Objective Evaluation

The objective evaluation is part of the controlled-exercises. The Publishers and Subscribers will exercise successive iterations of the steps forming: a goal, intentions, and actions. This is a Human Computer Interaction (HCI) technique. The Publishers would execute the actions related to issuing alerts. The Subscribers would discuss the actions they would execute for specific alerts received. In each case the participants would perceive the state of the world, interpret the state of the world, and evaluate the outcome.
We will install the CAM Studio screen capture software to record the user Interaction Techniques. The screen capture software will capture the behaviour of the user and the sequence of executing each process. This will also provide the Evaluators with insights of the Usability and the Operational Efficiency of the system. The alert messages saved in SAMBRO will be used to determine the data quality as well as Email and SMS received by the Subscribers will determine the Data Quality.
Observers, namely members of the project team, will be provided assessment guidelines to evaluate the Publishers and Subscribers during the controlled-exercises. The Observer will report details on the operational usability, timeliness, and any other noteworthy observations.

Subjective Evaluation

The subjective evaluation will be carried out at the end of the controlled-exercises to give the users the opportunity to express their opinion on the adoption of the system. It is follows the Technology Acceptance Model (TAM) and the HCI-based Gulf of Execution / Evaluation methodologies. These insights will evaluate the utility of the system.
For the TAM evaluation, the users will be given a structured questionnaire which they will provide answers to. The questionnaire is anonymized to allow for the user to provide an unbiased opinion. The users are provided with a section in the TTX guideline to provide feedback on the Gulf of Execution / Evaluation, which would indicate their ability to achieve the goal, intents, and actions using the system.

Evaluation Instruments

We will use the following evaluation instruments: questionnaires, tools, and guides in the controlled-exercises

  1. CAM Studio for capturing usability
  2. Technology Acceptance Model Questionnaire
  3. Human Action Cycle Guideline
    1. for Publishers
    2. for Subscribers

Cam Studio

References

Note: See TracWiki for help on using the wiki.