3.2. quality of a work system in use: The extent to which specified goals can be achieved with effectiveness, efficiency and satisfaction in a specified work system.

3.4. efficiency: The resources expended in relation to the accuracy and completeness with which users achieve goals.

3.5. satisfaction: The comfort and acceptability of use.

3.6. context of use: The users, goals, tasks, equipment (hardware, software and materials), and the physical and social environments in which a product is used.

3.7. work system: A system, consisting of users, equipment, tasks and a physical and social environment, for the purpose of achieving particular goals.

3.8. user: The person who interacts with the product.

3.9. goal: An intended objective.

3.10. task: The activities undertaken to achieve a goal.

3.10. product: The part of the equipment (hardware, software and materials) for which usability is to be specified or evaluated.

The framework describes the components of usability and the relationship between them.

In order to specify or measure usability it is necessary to decompose effectiveness, efficiency and satisfaction and the components of the context of use into sub-components with measurable and verifiable attributes. The components and the relationships between them are illustrated in Figure 1.

When specifying or measuring usability, the following information is needed:

• A description of the components of the context of use including users, equipment, environments, and tasks. This may be a description of an existing context, or a specification of intended contexts. The relevant aspects of the context and the level of detail required will depend on the scope of the issues being addressed. The description of the context needs to be sufficiently detailed so that those aspects of the context which may have a significant influence on usability could be reproduced.
• Usability measures consisting of target or actual values of effectiveness, efficiency, and satisfaction for the intended contexts.

Figure 1 - Usability framework

Relevant characteristics of the users need to be described. These may include knowledge, skill, experience, education, training, physical attributes, and motor and sensory capabilities. It may be necessary to define the characteristics of different types of user, for example with different levels of experience or performing different roles.

Relevant characteristics of the equipment need to be described. The description of the hardware, software and materials may be in terms of a set of products, one or more of which may be the focus of usability specification or evaluation, or it may be in terms of a set of attributes or performance characteristics of the hardware, software and other materials.

Relevant characteristics of the physical and social environment need to be described. Aspects which may need to be described include attributes of the wider technical environment (e.g. the local area network), the physical environment (e.g. workplace, furniture), the ambient environment (e.g. temperature, humidity) and the social and cultural environment (e.g. work practices, organisational structure and attitudes).

The goals of use of a product should be described. Goals may be decomposed into sub-goals which specify components of an overall goal and the criteria which would satisfy that goal. For example, a telephone sales clerk may have the goal to "Maintain customer orders". This overall goal might then be decomposed into sub-goals such as:

• "Make accurate record of all orders placed by customers"
• "Provide information rapidly in response to customer inquiries about orders placed".

The level at which the overall goal is set is a function of the boundary of the work system which is under consideration and which provides the context of use. In the example above the work system under consideration consists of clerks taking telephone orders.

The specification or measurement of the usability of a particular product should identify the overall goal, specific sub-goals, the relevant context of use, including the tasks and resources involved, and the measures of effectiveness, efficiency and satisfaction which are chosen as being relevant to the goals which have been identified.

Tasks are the activities undertaken to achieve a goal. Characteristics of tasks which may influence usability should be described as part of the context of use, e.g. the frequency and the duration of performance.

Detailed descriptions of the activities and processes may be required if the description of the context is to be used as a basis for the design or evaluation of details of interaction with the product. This may include description of the allocation of activities and steps between the human and technological resources. Tasks should not be described solely in terms of the functions or features provided by a product or system. Any description of the activities and steps involved in performing the task should be related to the goals which are to be achieved.

For the purposes of evaluating usability, a set of key tasks will typically be selected to represent the significant aspects of the overall task.

Annex A gives examples of how the components of the context of use can be described in terms of characteristics which may be relevant to usability.

A description of usability measures consists of target or actual values of effectiveness, efficiency, and satisfaction for the required contexts. It is normally necessary to provide at least one measure for each of effectiveness, efficiency and satisfaction. Because the relative importance of components of usability depends on the context of use and the purposes for which usability is being described, there is no general rule for how measures should be chosen or combined.

The choice of measures and the level of detail of each measure, is dependent on the objectives of the parties involved in the measurement. The importance each measure has relative to the goals should be considered. For example where usage is infrequent, high importance may be given to measures of learning and relearning.

If it is not possible to obtain objective measures of effectiveness and efficiency, subjective measures based on the user's perception can provide an indication of effectiveness and efficiency.

Measures of effectiveness relate the goals or sub-goals of the user to the accuracy and completeness with which these goals can be achieved.

For example if the desired goal is to accurately reproduce a 2-page document in a specified format, then accuracy could be specified or measured by the number of spelling mistakes and the number of deviations from the specified format, and completeness by the number of words of the document transcribed divided by the number of words in the source document.

Measures of efficiency relate the level of effectiveness achieved to the expenditure of resources. Relevant resources may include mental or physical effort, time, materials or financial cost. For example human efficiency could be measured as effectiveness divided by human effort, temporal efficiency as effectiveness divided by time, and economic efficiency as effectiveness divided by cost.

If the desired goal is to print copies of a report, then efficiency could be specified or measured by the number of usable copies of the report printed, divided by the resources spent on the task such as labour hours, process expense and materials consumed.

Measures of satisfaction describe the comfort and acceptability of the use.

Satisfaction can be specified and measured by attitude rating scales or measures such as the ratio of positive to negative comments during use. Additional information may be obtained from longer term measures such as rate of absenteeism, health problem reports, or the frequency with which users request transfer to another job.

Measures of satisfaction may assess attitudes to use of the product, or assess the user's perception of aspects such as efficiency, helpfulness or learnability.

Further examples of measures that can be used for assessing usability are included in Annexes B and C.

Care should be taken in generalising the results of any measurement of usability to another context which may have significantly different types of users, tasks or environments. If measures of usability are obtained over short periods of time the values may not take account of infrequent events which could have a significant impact on usability, for example intermittent system errors.

For a general-purpose product it will generally be necessary to specify or measure usability in several different representative contexts, which will be a subset of the possible contexts and of the tasks which can be performed. There may be differences between usability in these contexts.

Information about the characteristics of users, their goals and tasks and the environments in which the tasks are carried out provides important information for use in the specification of overall product requirements, prior to development of specific usability requirements.

Prior to development of a custom system, a purchasing organisation can use the information in this part of ISO 9241 as a framework for specifying the usability requirements which the system must meet and against which acceptance testing may be carried out. Specific contexts in which usability is to be measured should be identified, measures of effectiveness, efficiency and satisfaction selected, and acceptance criteria based on these measures established (an example is given in Annex C).

The definition and framework for usability can be used by product development teams to establish a common understanding of the concept of usability, and can help product development teams address the breadth of issues associated with product usability.

A developer can use the guidance in this part of ISO 9241 to help specify usability targets for the product (see Annex C). At various stages during the development process the developer can measure the usability achieved against these targets. This information enables objective decisions to be taken about the need for design changes to enhance usability, and about trade-offs which may be appropriate between usability and other requirements.

The guidance on context of use can be used to identify the users, tasks and environments so that more accurate judgements can be made about the needs for particular product attributes.

This part of ISO 9241 provides information to support measurement of usability. For example, description of the characteristics of users can assist with the selection of users to participate in evaluation. Identification of user's goals can assist with the selection of appropriate tasks for usability testing or reviews. The characteristics of the environment in which a product is likely to be used need to be described if that environment has to be simulated to ensure the validity of test results.

This part of ISO 9241 also provides a basis from which measures of usability can be generated. Product developers can develop appropriate measures of efficiency, effectiveness, and/or satisfaction (see Annex B).

The activities listed in 6.1 to 6.5 can provide a basis for defining, documenting and verifying usability as a part of a quality system. For example, Figure 2 illustrates how usability can be incorporated into a quality system which conforms to ISO 9001.

Figure 2 Quality Plan

This part of ISO 9241 can be used to help structure comparative evaluations of usability. Evaluators can use this part of ISO 9241 to identify which parts of the context of use (the users, their tasks, and the environment of use) need to be controlled when conducting comparative evaluations.

The comparative user performance test methods in the Annexes of parts 3 and 4 of ISO 9241 are examples of usability evaluation for the purpose of equivalence testing of a product (display screen or keyboard). These test methods, which are intended to provide alternative means of testing conformance, enable a manufacturer to demonstrate the usability of a new product by showing that, in a specific context of use, the measures of usability with the product are at least as good as the measures of usability with a product which has physical attributes which are known to meet the requirements of the standard.

A purchasing organisation can use the guidance in this part of ISO 9241 to help choose between products already available. Having specified the usability requirements in terms of the intended context of use and which measures of effectiveness, efficiency and satisfaction are to be used, the guidance may then be used to specify test conditions and evaluation criteria. The test conditions should be representative of important aspects of the overall context of use.

Other parts of ISO 9241, such as Part 14, contain recommendations which are applicable in particular contexts of use. The guidance in this part of the standard can be used to provide a framework for identifying the context of use which is relevant to the design decision to be made. Annex D contains more information on the relationship with other parts of ISO 9241 and other standards.

Measures of usability apply to products used within the overall work system. Measures of effectiveness, efficiency and satisfaction can be used to evaluate how any component of a work system affects the quality of the whole work system in use.

If the aim is to improve the quality of the overall work system in use, any part of the work system may be the subject of design or evaluation. For example it may be appropriate to consider the amount of user training to be provided, changes in lighting, or re-organisation of the task. In this case the element which is the object of design or evaluation is considered to be subject to potential variation, while the other elements of the work system are treated as fixed.

When designing a complete work system its quality in use may be optimised by changing components of the context of use of a product, such as the operating system, lighting or amount of user training. In this case measures of effectiveness, efficiency and satisfaction can be used to specify or evaluate the effect of different operating systems, different types of lighting, or different amounts of user training respectively.

When the quality of a work system in use is recognised to be unsatisfactory it is often necessary to carry out systematic analyses of the contribution of different components of the context of use and the interactions between them, in order to determine what the principle causes of the problems are. This process may also be used to identify which components are amenable to change, in order to bring about improvements in the quality of the work system in use. Diagnostic activity relating to the context of use is often a necessary step in the specification and evaluation of product usability, since it provides the basis for judging whether problems can be resolved by the re-design of the product.

Annex A (informative)

To specify or measure usability, measures of effectiveness, efficiency and satisfaction are required for goals. Usability measures may be specified for overall goals (e.g. produce a letter) or for narrower goals (e.g. perform search and replace). Selecting usability measures for the most important user goals may mean ignoring many functions, but is likely to be the most practical approach. Examples of appropriate measures are given in Table B.1.

Additional measures may be required for particular desired properties of the product which contribute to usability. Examples of some of these properties and additional specialised measures are given in Table B.2. In addition, where appropriate the measures given in Table B.1 can also be used for the usability objectives given in Table B.2.

Table B.2 Examples of measures for desired properties of the product

The choice of criterion values of measures of usability depends on the requirements for the product and the needs of the organisation setting the criteria. Usability objectives may relate to a primary goal (e.g. produce a letter) or a sub-goal (e.g. search and replace) or secondary goals (e.g. learnability or adaptability). Focusing usability objectives on the most important user goals may mean ignoring many functions, but is likely to be the most practical approach. Setting usability objectives for specific sub-goals may permit evaluation earlier in the development process.

It may be necessary to specify criteria both for the minimum acceptable level of usability and for the target level of usability.

When setting criterion values for a group of users, the criteria may be set as an average (e.g. average time for completion of a task to be no more than 10 minutes), for individuals (e.g. all users can complete the task within 10 minutes), or for a percentage of users (e.g. 90% of users are able to complete the task in 10 minutes).

Measures of usability should be based on data which reflect the results of users interacting with the product or work system. It is possible to gather data by objective means, such as the measurement of output, of speed of working or of the occurrence of particular events. Alternatively data may be gathered from the subjective responses of the users expressing feelings, beliefs, attitudes or preferences. Objective measures provide direct indications of effectiveness and efficiency while subjective measures can be linked directly with satisfaction.

It should be noted that it is possible to obtain data relating to each component of usability from objective or from subjective measures. For example, satisfaction can also be inferred from objective measures of the behaviour of the users, and estimates of effectiveness and efficiency can be derived from subjective opinions which the users express about their work and its outputs.

The validity of the data gathered to predict the level of usability achieved when a product is actually used will depend upon the extent to which the users, tasks and context of use are representative of the real situation and the nature of the measures chosen. At one extreme one may make measurements in the "field" using a real work situation as the basis for the evaluation of the usability of a product. At the other end of the continuum one may evaluate a particular aspect of the product in a "laboratory" setting in which those aspects of the context of use which are relevant are re-created in a representative and controlled way. The advantage of using the laboratory based approach is that it offers the opportunity to exercise greater control over the variables which are expected to have critical effects on the level of usability achieved, and more precise measurements can be made. The disadvantage is that the artificial nature of a laboratory environment can produce unrealistic results.

Evaluations may be conducted at different points along the continuum between the field and laboratory settings depending upon the issues which need to be investigated and the completeness of the product which is available for test. The choice of test environment and measures will depend upon the goals of the measurement activity and their relationship with the design cycle.

Effectiveness is defined as the accuracy and completeness with which users achieve specified goals.

To measure accuracy and completeness it is necessary to produce an operational specification of the criteria for successful goal achievement. This can be expressed in terms of the quality and quantity of output, for example, the specification of a required format for output documents together with the number and length of documents to be processed.

Accuracy can be measured by the extent to which the quality of the output corresponds to the specified criteria, and completeness can be measured as the proportion of the target quantity which has been achieved.

If a single measure of effectiveness is required, it is possible to combine measures of accuracy and completeness. For example, completeness and accuracy may be calculated as percentages and multiplied together to give a percentage value for effectiveness [2,12]. In cases where it is not appropriate to trade accuracy off against completeness, the two measures should be considered independently.

Efficiency is measured by relating the level of effectiveness achieved to the resources used. Temporal efficiency can be defined as the ratio between the measure of effectiveness in achieving a specified goal, and the time it takes to achieve that goal. This provides an absolute measure of temporal efficiency in a particular context. Similar calculations can be made with respect to efficiency in the use of mental or physical energy, materials or financial cost.

Satisfaction (defined as comfort and acceptability of use) is a subjective response of users to interaction with the product. Satisfaction can be assessed by subjective or objective measures. Objective measures may be based on observation of the behaviour of the user (e.g. body posture, body movement, frequency of absences) or can be based on monitoring the physiological responses of the user.

Subjective measures of satisfaction are produced by quantifying the strength of a user's subjectively expressed reactions, attitudes, or opinions. This process of quantification can be done in a number of ways, for example, by asking the user to give a number corresponding to the strength of their feeling at any particular moment, or by asking users to rank products in order of preference, or by using an attitude scale based on a questionnaire.

Attitude scales, when properly developed, have the advantage that they can be quick to use, have known reliabilities, and do not require special skills to apply. Attitude questionnaires which are developed using psychometric techniques will have known and quantifiable estimates of reliability and validity, and can be resistant to factors such as faking, positive or negative response bias, and social desirability. They also enable results to be compared with established norms for responses obtained in the past. QUIS [3,16] and SUMI [10,11] are examples of attitude questionnaires.

Workload entails both physical and mental aspects of tasks. Physical workload is the load resulting from movements, walking, carrying, etc. and also arising from constrained postural positions. The activities that are of a mental nature (perception, information processing, thinking, etc.) contribute to 'cognitive workload'. The design of computer hardware should take into account physical demands caused by high rates of input and sustained periods of activity. Interactive dialogues result in cognitive demands which in some circumstances can have a significant effect on usability of software products.

It is possible to measure the cognitive workload on the user. Cognitive effort is one resource expended in relation to the accuracy and completeness with which users achieve goals and can therefore contribute to the measurement of efficiency. However, cognitive workload has certain special characteristics in that both under- and over-loading may result in lowered efficiency. A task demanding too little mental effort may result in a lowered efficiency because it leads to boredom and lack of vigilance, which directly lowers effectiveness. In such a case overall efficiency would be enhanced by increasing demand. Excessive cognitive workload may also result in lowered effectiveness, if it causes information to be missed and results in errors. This is a particularly important issue in situations where safety is critical, e.g. air traffic control and process control. Measures of cognitive workload can be used to predict these types of problems.

Two examples of questionnaires which have been validated for measuring mental effort and perceived work demands are the Subjective Mental Effort Questionnaire (SMEQ) [19] and the Task Load Index (TLX) [13]. The SMEQ is a unidimensional, whereas the TLX distinguishes between six aspects of work demands.

The term usability is often used to refer to the capability of a product to be used easily. This corresponds with the definition of usability as a software quality in ISO/IEC 9126:

However the attributes which a product requires for usability depend on the nature of the user, task and environment. A product has no intrinsic usability, only a capability to be used in a particular context. Usability cannot be assessed by studying a product in isolation.

There are therefore three potential ways in which the usability of a product could be measured.

1. By analysis of the product and the context of use.

Usability could be measured by assessing the product features required for usability in a particular context. Appropriate features are specified in other parts of ISO 9241. However ISO 9241 only gives partial guidance. Of the many potential design solutions compatible with ISO 9241, some will be more usable than others.

2. By analysis of the process of interaction.

Usability could be measured by modelling the interaction between a user carrying out a task with a product. However, current analytic approaches do not give very precise estimates of usability. As the interaction is a dynamic process in the human brain, it cannot be studied directly. However, measures of mental effort and acceptability are important indirect measures of factors contributing to usability.

3. By analysing the effectiveness and efficiency which results from use of the product in a particular context and measuring the satisfaction of the users of the product. As these are direct measures of usability, they are the ultimate test of usability. If a product is more usable in a particular context, usability measures will be better.

It should be noted that usability as defined in this part of ISO 9241 also depends on software qualities which are distinct from usability as defined in ISO 9126, such as functionality, reliability and computer efficiency. These software qualities all contribute to quality of the work system in use.

Usability defined in terms of the quality of a work system in use necessarily depends on all factors which may influence use of a product in the real world, including organisational factors such as working practices and the location or appearance of a product, and individual differences between users including those due to cultural factors and prejudice. This broad approach has the advantage that it concentrates on the real purpose of design of a product - that it meets the needs of real users carrying out real tasks in a real technical, physical and organisational environment. This is consistent with the objectives of ISO 9241 described in ISO 9241-1.

Annex E (informative)