SAMBRO Evaluation Guidelines

We provide a methodology and a set of evaluation instruments to be used in evaluating the implementation.

Introduction

Sahana Alerting and Messaging Broker (SAMBRO) is designed to foster Cross-Agency Situational-Awareness in support of improving institutional responsiveness to All-Hazards: http://eden.sahanafoundation.org/wiki/Deployments/SAMBRO . Alerting Authorities (Publishers) and warning recipients like the public and first-responders (Subscribers) can interconnect through the SAMBRO. It offers an all-hazard all-media approach whereby consistent, concise, and controlled alerts and warnings are channelled through multiple technologies to geo-targeted recipients.

To evaluate the performance of SAMBRO, along with the underlying Common Alerting Protocol (CAP) content standard and procedures, we are introducing an objective and subjective evaluation approach. The evaluation is specific to early warning and alerting. Through structured scenario-based simulation exercises, we will evaluate the performance of the users and the usability and ease of use of the system.

Technology

Issuing Alerts (Publish)

Primarily there are two types of devices that can be used for issuing messages:

• Personal Computer (PC) - Desktop, Laptop, Notebook
• SmartPhones (including Tablet PCs)

SAMBRO offers two types of applications:

• Browser-based application that is accessible through a PC or SmartPhone
• Mobile-based application (Android & iOS) that is accessible using a SmartPhone only

Receiving Alerts (Subscribe) There are multiple channels for receiving alerts: SMS, Email, RSS, Twitter, Facebook. Only “public” alerts are received over Twitter and Facebook. SAMBRO offers two RSS feeds: one that carries public warnings and the other that carries restricted and private warnings. SMS and Email are purely for those Administrator and Self controlled Subscribers.

The user may chose to receive alerts on a PC, SmarPhone, or Standard Mobile Phone. Each of the devices have their limitations. For example, unless you have configure a PC to receive SMS, you may only receive SMS alerts over SmartPhones and Standard Mobile Phones. While Email can be received on any device.

Methodology

We have broken the SAMBRO Early Warning System (EMS) dissemination subsystem activities into two components; namely Publishers and Subscribers. Publishers are essentially authorized alerting organizations who would originate or relay a warning/alert message. Both Publishers and Subscribers will undergo a series of tests to determine their aptitude to:

• publish effective CAP-enabled messages
• Subscribe to actionable (or inactionable) CAP-enabled alert/warning messages
• operationalize a cross-agency situational-awareness platform

We are applying two Human Computer Interaction (HCI) evaluation methodologies: Technology Acceptance Model (TAM)

• Goal Oriented HCI
• Technology Acceptance Model (TAM)

TAM provides insights on the ease of use and usability of the technology presented to the users. This applies to both Publishers and Subscribers. This is a subjective analysis of the intervention.

Goal Oriented HCI

There are two elements to the Goal Oriented HCI:

• Objectively analyze the functional transition
• Subjectively analyze the functional transition

Screen Capture

We will install a screen capture software. The screen capture software will capture the behaviour of the user and the sequence of executing each process. This will provide the evaluators with insights of the ease of use and usability of the system. It may relate to the level or training and expectation of the user’s aptitude. Focus Group Discussion

Tools

Several survey instruments and evaluation guidelines

For Observers

1. [Observe Alerting Authorities publishing CAP messages

For Publishers

for Subscribers

1. ​Perception of SAMBRO using TAM
2. [

References

• ​Kirk Patrick's Four Level Training Evaluation