Solution Quality Evaluation Guideline

General Quality Requirements

The general quality requirements are defined here as attributes of the solution that is to be evaluated ("The solution must be ...").

These attributes are grouped in four quality "levels" - a higher level representing a higher quality standard.

Projects can refer to these levels to define minimum quality goals, to prioritize certain quality aspects vs. other requirements, or to define an iterative approach of continuous quality improvement.

In order to pass a level, all criteria of this level and all previous levels must be met.

Level	Priority (MoSCoW)	Key Attributes	Description
0 - Experimental	-	-	any solution that does not (yet) meet all criteria of level 1
1 - Minimal	Must	valid, correct, safe	minimum standard for any solution
2 - Appropriate	Should	usable, maintainable, secure	minimum standard for production use
3 - Good	Can	consistent, robust, configurable	long-term development goals
4 - Optimal	Want	efficient, scalable, interoperable	optimization and enhancement goals

Level 1: Minimal

Valid

"Writing the right code"

• The specified solution solves the targeted problem and meets the minimum functional requirements

• The required input data are available in the target scenario

• The user actions required to apply and/or use the solution or its responses are possible in the target scenario

• All dependencies of the feature set are resolvable, and hardware and infrastructure requirements can be met in the target scenario

Correct

"Writing the code right"

• Processing does not fail for any valid input data or user action.

• All responses are correct for the respective input data or user actions according to the specification

Safe

• The solution does not break other functionality of the application (except for functionality it replaces or supersedes)

• All processing always ends with a valid database status regardless whether it succeeds or fails

• All failed processes are effectively removed from the system and their transactions properly rolled back

• The solution effectively prevents uncontrolled indefinite object or process proliferation

• The solution always returns to a defined system status after a failure so that the application remains accessible

Level 2: Appropriate

Usable

• The solution can be used by the intended user group in the target scenario, i.e. the users can:
  • discover and access the features
  • understand and perform the required user actions
  • understand and control the consequences of actions
  • understand and provide the required type, quality and format of input data
  • discover and access the results, and apply them to achieve their goals

• Usability criteria apply analogously for back-end libraries and tools with developers in the role of the user

Maintainable

• All critical dependencies are obvious at their starting point

• All non-critical dependencies have meaningful fallbacks

• Processing failures produce useful and accessible feedback about the reasons

• The design of the solution does not obstruct diagnostics of defects

• The code is structured in a way that the processing flow can easily be determined from it

Secure

• The solution applies all relevant system configurations for authentication, authorization and accounting (AAA) appropriately and correctly.

• The solution does not allow the user to perform actions or initiate processes beyond the intended functionality.

Level 3: Good

Consistent

"Maintaining Conceptual Integrity across the Application and beyond"

• The solution applies the application framework appropriately and correctly

• The data structures defined in the solution are consistent with the global application data model

• The solution interoperates appropriately and correctly with other system components

• The solution applies the application's global design and style guidelines (both for code and interfaces)

• Where appropriate or required, the solution complies with specified standards (see Standards)

Robust

• The solution delivers consistent behavior, results and performance for all valid configurations

• The solution handles invalid input data, unexpected user actions and irregular system conditions appropriately and consistently for all valid configurations

• The solution detects invalid configurations and prevents any inadvertent effects that could result from them

Configurable

• The solution can be easily and safely turned on and off without inadvertent side-effects (where appropriate, not for framework solutions)

• The solution applies global configuration settings correctly and consistently

• Processing parameters, behavior and/or appearance of the solution can be adapted to requirements variations using configuration patterns separate from its core code

Level 4: Optimal

Efficient

• The solution provides acceptable performance with the minimum available hardware/infrastructure resources in the target scenario

• The solution can be applied with minimal configuration effort

• The solution can be applied, developed and maintained with minimal learning effort

Scalable

• The solution can handle/adapt to varying problem sizes (e.g. amount of data, frequency of events) without failure, inconsistent behavior or significant loss of performance

• The solution can utilize hardware/infrastructure enhancements and/or concurrency to provide better performance

• The solution can be applied in any number of concurrent instances (given the appropriate hardware and infrastructure) without failure, inconsistent behavior or significant loss of performance

Interoperable

• Different instances of the solution can, where appropriate or required, interchange data and/or process information

• The solution can, where appropriate or required, interchange data and/or process information with other relevant solutions in the target scenario

• The solution provides external APIs (application programming interfaces)

User Experience (UX) Requirements

Wikipedia: ISO 9241-210[1] defines user experience as "a person's perceptions and responses that result from the use or anticipated use of a product, system or service". According to the ISO definition, user experience includes all the users' emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviors and accomplishments that occur before, during and after use.

User experience, as a result of human-computer-interaction, involves various aspects of a software solution - both in view of its design, its functionality as well as its quality, but also personal aspects of the individual user (e.g. skills, previous experiences, cultural background), and of the situations and environments in which the software is to be used.

"Good UX" is though not a requirement in itself - but the generic name for any additional design, functional and quality requirements in view of human-computer-interaction that may be defined for a solution, and which must be met in order to achieve certain quality levels.

Even though UX as such appears to be a largely subjective matter, there are usually a number of generalizable UX requirements for any user interface.

Example: User interfaces which are typically used in the darkness should not blind the user, but use a reduced palette of primarily reddish colors on dark background in order to cause less or no iris contraction. If the interface is though to be used under varying lighting conditions it should be adaptable (or even auto-adapt).

In order to be valid and accomplishable, UX requirements must be specific and clearly defined (like the above example) for the quality level for which they shall apply - even when their reasoning may be more complex sometimes.

Some general quality requirements also influence the user experience (e.g. correctness, usability, efficiency), but are not specific for the use-case and therefore usually not separately discussed as UX requirements. They are however relevant for UX evaluations and should thus be taken into account in such analyses.

Advanced Quality Requirements

Advanced quality criteria can be used to compare different solutions, or to define quality roadmaps for particular feature sets beyond the standard criteria.

Advanced quality criteria do not apply equally for all use-cases - in certain cases they may even constitute anti-patterns.

Advanced quality criteria do not enhance the general quality level of a solution: a level-3 solution is still of better quality than a level-2 solution - regardless how many advanced criteria the level-2 solution meets.

Stability

A solution is stable when its black-box model rarely changes, and it maintains backwards-compatibility and/or easy migration paths. Stable solutions are required for all critical dependencies, so that they can be built upon without risk for frequent regression issues or need for constant migration. Stability is especially critical in shared data models (i.e. where data are shared between different modules within the application or even across applications), and in authorization frameworks - and can thus be defined as quality requirement for a solution.

Stability is typically perceived as a sign of maturity, but can actually be taken into account already during the initial solution design. Pragmatic approaches (e.g. integrated migration, or enforced backwards-compatibility) can create stability early on without limiting or obstructing further development.

Generalization

Generalizable solutions are designed and/or implemented in a way that they can be generalized. Typically, they separate their core patterns/algorithms from the use-case specifics through parametrization and/or configuration patterns. This can be defined as an advanced quality requirement for a solution.

Generalization means to modify and/or extend the core patterns/algorithms of the solution in a way that they can be re-used in other solutions which involve the same or similar functional requirements, thereby reducing future development efforts. There is a wide range of quality attributes - "flexible", "customizable", "re-usable", "extensible", just to name a few - all of which involve different degrees of and approaches to generalization.

Standards

Solutions can both define or implement standards (e.g. data standards, communication protocols, taxonomies). Standard compliance can be defined as quality requirement for a solution.

Standards can define additional requirements beyond the immediate requirements of the targeted problem, which must however be met in order to comply with the standard. Such additional standard requirements would usually be seen as quality requirements under Consistency in level 3.