STEP 8: SET METRICS 3-73
• Define and check metrics
• Produce GQM plan and measurement plan
Please note that this step requires expertise of building measurement programmes.
Define measurement goals Activity
8-1
• Consult project team members
• Specify measurement goals
• Prioritize measurement goals
• Select measurement goals
The first step in the GQM definition phase is to define formal measure ment goals. To do this, the PROFES team must consult all the people partici-pating in the measurement programme. In principle, these people are probably software engineers, although some improvement goals might involve other people. All people collecting measurement data also need to be involved in measurement goal setting. Any possible measurement goals stated by these people should be made explicit, after which the goals can be prioritized and a final decision made on the measurement goals that are selected.
Measurement goals in the context of PROFES are based on the product quality goals, process conformance goals, or product-process dependency goals.
Measurement goals must be defined and described on such a level of abstraction that goal attainment can be evaluated by analysing data collec-ted for the purpose. Therefore, measurement goals must be defined in a structural and logical way. For this purpose, templates are available that support the definition of measurement goals by specifying the purpose (what object and why?), perspective (what aspect and who?), and the environmental characteristics. The PROFES template for measure ment goals is included in the PROFES GQM plan template in the appendices to this manual.
STEP 8: SET METRICS 3-75
Conduct GQM interviews Activity
8-2
• Select project team members for interview
• Prepare interview
• Hold interview
• Report interview
The project team must be closely involved in the development of the measurement programme. With respect to the defined measurement goals, structured interviews with individual members of the project team must be held to extract the knowledge from the project team
In Step 4, interviews were also conducted. The object of these interviews was to determine the current capability of the process, while the GQM interviews clarify the GQM questions and metrics. However during the PROFES project, in some cases it was possible to combine these interviews, as described in Chapter 4 of this manual.
So-called ‘abstraction sheets’ are used during the interviews. An abstrac-tion sheet summarizes the main issues and goal dependence as described in the GQM plan and presents this information divided into four sections on a page:
Quality Focus: What are the measured properties of the object, according to the project members?
Baseline Hypothesis: What is the project member’s current knowledge, with respect to these measured properties?
Variation Factors: Which (environmental) factors do the project members expect to have an effect on the measured properties?
Impact on Baseline Hypothesis: How do these variation factors influence the measured properties?
An example of an abstraction sheet is shown in Figure 3.4. The four sections can be checked for consistency and completeness. The four sections are mutually related. For example, for every quality focus,
hypothesis should be stated; or for every variation factor, its effect on the hypothesis should also be made explicit.
Object Purpose Quality Viewpoint
Focus
Delivered Understanding Reliability SW Development
Product and its causes Team
Quality Focus Variation Factors
Number of Failures Level of Reviewing
lBy Severity
lBy Detection group
lNumber of Faults
lBy Module
Baseline Hypothesis (estimates) Impact on Baseline Hypothesis Distribution of Failures The higher the level of reviewing, By Severity the less failures, and the less faults
lMinor 60% slip through the implementation
lMajor 30% phase
lFatal 10%
Figure 3.4. Example abstraction sheet Interviews should be recorded for future use and as a feedback mechanism to the interviewee. Such an interview report contains the results of the interview and can include an abstraction sheet of the issues raised in the interview.
Define questions and hypotheses Activity
8-3
• Analyse interview reports
• Specify GQM questions
• Specify hypotheses
With respect to the measurement goals, questions must be defined to support data interpretation. As the goals are defined on an abstract level, the questions are refinements of the goal that lead to a more operational
STEP 8: SET METRICS 3-77
level suitable for interpretation. By answering the questions, it should be should be possible to conclude whether the goal is reached.
For each question, expected answers are formulated as ‘hypotheses’.
Formulating hypotheses encourages the project team to think about the current situation and therefore stimulates a better understanding of the process and/or product. Furthermore, during data interpretation, these hypotheses on measurement results are compared to the actual measure-ment results. The purpose of this is not to evaluate the possible correctness of the hypotheses, but rather to encourage the project team to identify and analyse the underlying reasons that caused results to conform to or deviate from their expectations.
To ensure that the right questions and hypotheses have been recorded and correctly formulated, a review must be made. The questions are the basic translation from goals to metrics. When the actual data is collected and presented to the project team, it should help in answering the questions of the project team. In this way, the questions take a central role, not only during definition but also during interpretation. Therefore it is important to make sure that the questions are correct. The questions were also reformulated from the project team’s input during the interviews. It is possible that mistakes were made during transcription, or that the PROFES team misinterpreted the questions.
The hypotheses must be reviewed as well, as the hypotheses are used together with the questions to define the metrics to be established for data collection.
Define and check metrics Activity
8-4
• Analyse GQM questions
• Specify the metrics required to answer those questions
• Compare metrics with process and product models
• Incorporate necessary changes
Once the goals are refined into a list of questions, the metrics must be defined to provide all the quantitative information for answering the questions in a satisfactory manner. Therefore, metrics are a refinement of questions into a quantitative level of abstraction that identifies process and/or product measurements. Once all these measurements are
collec-ted with respect to the defined metrics, sufficient information should be available to answer the questions completely.
Furthermore, factors that influence the outcome of the metrics must also be defined: factors that directly influence metrics also influence the answers to the questions that the metrics are related to. If these influen-cing factors are not considered when defining the measurement prog-ramme, incorrect conclusions on interpretations of the collected data may be drawn. These influencing factors are also usually defined as metrics.
The defined goals, questions, and metrics must be consistent and com-plete in relation to the process and product models of the respective project (see Figure 3.5). To ensure this, consistency and completeness checks have to be performed throughout the entire definition phase. If definitions appear to be missing, incomplete, or inconsistent, either the definitions have to be adjusted to comply with the process/product models, or the process/product models have to be adjusted to comply with the goal, question, and metrics definitions.
Metric Metric Metric
Question Question GOAL
Process Phase
Metric Metric Metric
GQM Definition
Process Modelling Check on
Consistency and Completeness
Figure 3.5. Checking for consistency and completeness
STEP 8: SET METRICS 3-79
Produce GQM plan and measurement plan Activity
8-5
• Assemble GQM goal, questions, hypotheses, and metrics
• Document them in a GQM plan
• Operationalize each metric
• Document in a measurement plan
A GQM plan is a document that contains the goals, questions, metrics, and hypotheses for the measurement programme, as defined in the previous steps. The GQM plan serves as a guideline for data interpre-tation, and provides a basis for the measurement plan and analysis plan.
The GQM plan describes the refinement from measurement goals into questions and from questions into metrics. As some of these metrics may be calculated (indirect metrics) using other metrics, all direct measure-ments that will also be collected, are listed in the GQM plan.
A measurement plan is a document for each identified measurement in the GQM plan that describes who should collect it, when it must be collected, and how it must be collected. Furthermore, all automated data collection tools and manual data collection forms are defined in the measurement plan.
The measurement plan describes the following aspects for each indirect measurement identified in the GQM plan:
• It provides a formal description of the direct measurement
• It provides a textual description of the direct measurement
• It defines all possible outcomes (values) of the direct measurement
• It identifies the person that collects the direct measurement, i.e. a programmer, engineer, project manager, tester etc.
• It defines at which specific moment in time this person must collect the direct measurement
• It defines by which medium (tool or form) this person must collect the direct measurement.
As the GQM plan and measurement plan represent the formal definition of the measurement programme and describe all related data collection procedures, they must be reviewed and approved by all project members before data collection can actually begin.
The review session should focus on:
• Do the project members agree upon the defined goals, questions, and metrics?
• Can the project members identify any missing or unnecessary definitions?
• Do the project members agree with the proposed data collection procedures and tools?
Average Duration and Effort
The duration for this step is about one month with an effort of about 8 person weeks.
Tools and Templates
• GQMAspect, to be used for the definition of a measurement programme
• MetriFlame, to be used to perform data analysis of the measurement data, and to provide feedback to the software development team.
• SW engineering tools, such as compilers or static analysers, since these tools can provide plenty of useful data and can therefore be a practical tool for collecting measurements.
• Spreadsheets, to be used to perform some basic data analysis in trial situations.
• PROFES provides templates for a GQM plan and for a measurement plan in the appendix of this manual.
STEP 8: SET METRICS 3-81
Work Products
Input work products Output work products
• Prescriptive process model (including selected process changes)
• Product quality needs and target profile (result from Step 2)
• Current status of product quality (result from Step 3)
• Product improvement goals (result from Step 5)
• Process assessment reports and profiles (result from Step 4)
• PPD models (result from Step 6)
• GQM abstraction sheets
• GQM plan
• Measurement plan
Resource Allocation
Roles, responsibilities and requested skills Expert roles
PROFES expert
The PROFES experts are responsible for managing and per-forming all GQM activities. The PROFES team involves the project development team as much as possible, but however makes sure that the effort spent by the project team remains acceptable.
Support roles Project team
The project team provides all necessary information for proper GQM planning. This means participation in goal definition, interviews, hypothesis formulation, and document reviews.
However, the project team does not carry out operational GQM tasks.
Expected effort/role PROFES experts
Depending on the experience of the PROFES team, the effort needed for GQM definition is 4 weeks for an experienced PROFES team, and 3 months for an inexperienced PROFES team. In case the project team is less than 10 engineers, this effort can be lower.
Project team
The project team spends about one day per team member on GQM definition.
Methods and Techniques
Goal/Question/Metrics (GQM) method:
• For short introduction into the principles see Basili, V., Caldiera G.
& Rombach, H. 1994. Goal Question Metric Paradigm. In Marciniak, J.J, (ed.), Encyclopaedia of Software Engineering. John Wiley &
Sons, Vol. 1, pp. 528-532.
• For practical guidelines and examples see Solingen R., Berghout E.
1999. The goal/ question/ metric method, a practical method for quality improvement of software development. McGraw-Hill ISBN 007-709553-7.
1. VERIFY COMMITMENT
4. DETERMINE CURRENT PROCESS CAPABILITY 3. DETERMINE CURRENT PRODUCT QUALITY 2. IDENTIFY PRODUCT QUALITY NEEDS
10. IMPLEMENT AND MONITOR IMPROVEMENTS 7 . DESCRIBE PROCESS CHANGES
8. SET METRICS FOR THE PROCESSES AND PRODUCT
11. EVALUATE RESULTS
12 . UPDATE EXPERIENCE BASE
5. SET PRODUCT IMPROVEMENT GOALS
6. DETERMINE NECESSARY PROCESS CHANGES
9. PREPARE IMPROVEMENT IMPLEMENTATION CHARACTERIZE
PLAN
PACKAGE ANALYSE EXECUTE SET GOALS
PROFES PHASES PROFES STEPS
STEP 9: PREPARE IMPROVEMENT IMPLEMENTATION 3-85