ADVANCED ASSESSMENT AND MEASUREMENT TECHNIQUES 4-1
Why Use Integrated Software Process Assessment and Measurement?
Software process assessment and goal-oriented measurement require a good understanding of the organization and projects that they are to be applied to. Techniques for collecting, using, and reporting information from organization and projects have common features. The need to integrate software process assessment and goal-oriented measurement is based on this approach. The objectives for an organization applying integrated process assessment and goal-oriented measurement are as follows:
• To complement the views of assessors and Goal/Question/Metric (GQM) experts and make their work more effective
• To make best use of the engineers’ time
• To avoid redundant activities that increase improvement costs
The purpose of this chapter is to demonstrate how software process assessment and goal-oriented measurement can be performed in an integrated way. In the PROFES project, assessments were made using the ISO 15504-compliant BOOTSTRAP assessment methodology, and measurements were made according to the GQM method. This section deals with these two methods, although integration can generally also be conducted with other assessment and measurement methods.
Integration Principles
Preliminary product improvement goals assist and focus the forthcoming organizational and project level assessments. These are carried out with the ISO15504 process assessment method and measurement definition according to the GQM method respectively.
The processes to be selected are assessed by applying specific product-process dependency (PPD) models that indicate which product-processes are most likely to have an influence on the required product quality. By using these PPD models, it is possible to focus solely on the most critical processes. Focused assessment offers a good starting point for a preliminary measurement programme to be run in conjunction with assess-ment. More information on PPDs can be found in this user manual.
When discussing the integrated use of measurement and software process assessment, two main principles can be defined (see Figure 4.1):
ADVANCED ASSESSMENT AND MEASUREMENT TECHNIQUES 4-3
1. By integrating the preparation phase, interviews, and feedback, assessment and measurement activities in the improvement project can be combined.
2. Measurement data collected by the software development project for assessing the process capability can be used both during and after the software development project.
Figure 4.1. Principles of the integrated use of software process assessments and measurements
The integration of activities is described in Chapter 4.1. The use of measurement data for assessing processes is discussed in Chapter 4.2.
Integration and the PROFES Improvement Methodology
Step 4 of the PROFES improvement methodology steps contains software process assessment activities, and Step 8 contains measurement activities. Process changes are implemented during Step 10 and measurement data is collected.
Integration of assessment and measurement activities is closely related to these steps. The purpose of integration is not to replace the step-wise approach of PROFES, but instead to combine and conduct certain activities of software process assessment and measurement earlier than the PROFES step-wise approach would otherwise recommend.
Integration is suitable for organizations and projects familiar with software process assessment and measurement, and does not require any sub-stantial effort to begin these activities.
Focused BOOTSTRAP assessment
GQM
measurement
Metrics in GQM plan Process
capability
Measurement data Integrated interviews
Produces Defines
Applies to
Collects and analyses
SW development
project Is used to assess
1
2 Is used to refine
Integrated feedback Integrated preparation
Integrated Activities
The following software process assessment and measurement activities can be integrated:
• Preparation
• Interviews with software producing unit (SPU) project personnel
• Feedback
In this section, we will describe these activities and explain how they can be integrated. There are other activities that can also be carried out during software process assessment and measurement, but as they cannot be conducted in an integrated manner, they will not be discussed here.
Integrated Preparation
Integrated preparation combines assessment preparation and GQM preparation. GQM preparation includes environment characterization and measurement goal definition, both of which are activities carried out during GQM planning.
Table 4.1 describes the integrated preparation activities with more detail:
Table 4.1. Activities carried out during integrated preparation
Assessment1 GQM method
Integrated preparation
§ Select sponsor
§ Define assessment purpose
§ Identify and characterize the organization and projects
§ Set assessment scope
§ Select processes to be assessed
§ Identify sponsor and stakeholders
§ Characterize environment
§ Set measurement goals
§ Select assessment team
§ Select key persons to interview
§ Select measurement team
§ Select key persons to interview
§ Collect supporting material § Collect supporting material
§ Schedule interviews
§ Define assessment schedule
§ Schedule interviews
§ Define measurement schedule
The sponsor, purpose, improvement goals, measurement goals, and software process assessment scope should all be defined. A sponsor within the organization is required, who is typically a person responsible
1 Activities in this and other activity tables are based on BOOTSTRAP assessment method. Activities in other ISO15504 compliant assessment methods may vary.
ADVANCED ASSESSMENT AND MEASUREMENT TECHNIQUES 4-5
for the SPU and provides the resources and budgeting for product and process improvement.
In general, the purpose of assessment is to aid process improvement, but the specific purpose of the assessment is based on discussions with top management. If measurements are used together with assessment, they should be co-ordinated. Measurements can be used for measuring process improvements or process capability.
A preliminary proposal for SPU-level product improvement goals and measurement goals should be defined. This requires that the organization has been previously characterized and some important issues for analysis have already been identified. This proposal can be used as a basis for defining measurement goals in SPU-level interviews.
Measurement goals can directly reflect business goals or project goals.
Measurement goals must be carefully selected, based on selection criteria such as project or organization priority, risk, time available to achieve goal, or potential for understanding, control, or improvement. The better the measurement goals match the improvement goals and assessment scope, the less time and effort is needed to define the measurements.
Assessment scope includes identification of the SPU, projects to assess, and processes to focus on. In the case of a focused assessment, we recommend that processes be selected that will have the greatest impact on the organization's business, product, and process improvement goals.
PPDs offer valuable support in helping to select processes. Previous assessment recommendations and ongoing improvement initiatives also provide pointers for identifying prospective processes for assessment.
The PROFES team is responsible for carrying out the complete PROFES cycle. Roles in the team include:
• Lead assessor, who is responsible for the assessment and competent and qualified for this assignment. He or she may either be working in the organization or someone from outside, such as a consultant.
• Assessor, assists the lead assessor and is either an internal or external person.
• GQM expert, responsible for the GQM process.
• Measurement facilitator, an internal person responsible for providing the measurement infrastructure and conducting the measurements.
• PROFES facilitator, an internal person who is responsible for arranging the assessments, interviews, required material, infrastructure, etc.
In most cases, some of these roles could be combined when a complete PROFES software process improvement is to be carried out in a cost-efficient way. For example, a qualified PROFES expert might be able to take on the roles of lead assessor and GQM expert. However, we recommend that there are at least two people in the PROFES team, so that both assessment and measurement aspects are covered as thoroughly as possible. Also for the reliability of the assessment this is usually advisable.
Supporting material is collected to familiarize the PROFES team in advance with the organization and the application domain. It is based on a study of all the relevant documentation, and includes for example:
• Quality manuals
• Process descriptions
• Possible previous assessments
• Possible results of previous measurement programmes
• Project documentation examples
• Product documentation examples
It is important to use existing GQM plans and measurement plans for including additional information on measurements defined during the integrated interviews. Integrated interviews provide a more comprehensive view of measurement than stand-alone GQM interviews, and so existing measurements can be used to complement the GQM and measurement plan.
The PROFES team negotiates an interview schedule with selected interviewees allowing for their availability. A schedule of both assessment and measurement activities is defined together with the sponsor. The resulting plan should contain at least the following information:
• Activity responsibilities
• Interview schedule
• PROFES team roles
• Schedule for possible complementary interviews. A draft GQM plan is created during integrated interviews. Depending on organization/-project’s measurement experience, measurement goals and GQM plans may be created based on integrated interviews. Additional interviews may also be required.
ADVANCED ASSESSMENT AND MEASUREMENT TECHNIQUES 4-7
• Schedule for review and acceptance of GQM and measurement plans
• Feedback sessions
The selection of interviewees should be included in the plan, and invitations sent to the persons chosen.
Integrated Interviews
Integrated interviews combine assessment interviews held during assessment and GQM interviews held during GQM planning.
The SPU and one or more application projects are assessed. These activities form a major part of the characterization phase in the PROFES cycle. During assessment, interviews are held at both SPU and project level. However, SPU interviews are not obligatory for focused assessment.
GQM interviews are held as part of the assessment interviews, and are usually conducted during the PROFES planning phase. By using integrated interviews, it is possible to combine planning phase activities with characterization phase activities, thus saving time and effort.
Table 4.2 describes the activities of integrated interviews:
Table 4.2. Activities conducted during integrated interviews
Assessment GQM method
Integrated interviews
§ Opening briefing § Opening briefing
§ Carry out SPU level interviews § Review and refine GQM goals if necessary
§ Carry out project level interviews § Carry out GQM interviews
§ Create draft version of assessment report
§ Review assessment report by the organization
§ Update and finalize assessment report
§ Create draft version of GQM plan
§ Review GQM plan
§ Identify additional measurements if necessary
§ Create measurement plan
Integrated interviews begin with an opening briefing. This is to minimize any potential psychological barriers and explain the plans for the assess-ment and measureassess-ment programme, so that subsequent activities can be carried out effectively and efficiently. All assessment and measurement participants should have a clear understanding of the task’s purpose, schedule, and what is expected from them. At least all those directly contributing to the assessment and measurement activities should be
invited. We suggest that all of those who will benefit and/or will be affected by the assessment/measurement results should be involved.
Interviews are conducted at both SPU and project level in that order.
Several viewpoints may be put forward for the SPU interviews, and each viewpoint requires a separate interview. At least one person should be appointed for each viewpoint. Adequate time should also be allocated to each viewpoint as appropriate for the plan. No interview should take more than two hours. Please note that PROFES team roles (assessors and GQM experts) may also be combined thus requiring fewer people.
• The purpose of the assessment viewpoint is to understand important organizational issues such as organizational structure and culture, strengths and weaknesses, business objectives and improvement goals.
• The main purpose of the GQM measurement viewpoint in the SPU interviews is to review the measurement goals and refine them if necessary. The measurement goals are reviewed primarily to ensure their importance for the SPU. They should be clearly related to the important issues in software development for the SPU.
The SPU-level interviews have two main aspects, an assessment aspect and a GQM measurement aspect. The assessment aspect dominates the SPU interviews and the GQM measurement viewpoint dominates the project interviews. The focus of each interview should be defined in advance and explained to the interviewee.
The preliminary measurement goals should be verified and fine-tuned, if necessary. The results can be reviewed at the SPU level before the project level interviews begin. The main purpose is to present the results to SPU interviewees, collect feedback, verify findings, and make corrections, if and when required.
The project level interviews have the same assessment and measurement aspects as the SPU level interviews. However, the measurement aspect dominates at the project level. A draft GQM plan is created according to the measurement goals defined previously. During the interview, the knowledge of people interviewed is recorded to provide information for a GQM plan.
Ideally, the needs of the assessment and GQM planning should be covered in a single interview with each interviewee. Due to limited time, this often means that assessments have to be focused and/or specific details of GQM plan and measurement plan are completed later on.
A preliminary version of the GQM and measurement plans should be developed before project level assessment, if measurements are already
ADVANCED ASSESSMENT AND MEASUREMENT TECHNIQUES 4-9
collected already at an early stage in the project. Such measurements might exist in situations where projects are constantly measured with certain sets of metrics. In this case, a second version of the GQM plan is created during the integrated interviews to ensure that the final improve-ment actions are followed either by measureimprove-ments or measureimprove-ments providing data for continuous assessment.
In stand-alone GQM interviews, quality focus, variation factors, and related hypothesis for both the quality focus and the variation factors, are collected for each measurement goal. Due to the broader scope of assessment interviews, it is not possible to collect data in as much detail in integrated interviews. Therefore, there are two possibilities for completing any missing details in the GQM plan:
a) Additional complementary GQM interviews are organized that validate and add data to the GQM plan.
b) Some of the data in the GQM plan is based on already existing GQM plans carried out previously. The project team should review them in both cases.
After the integrated interviews, the results are documented. The assess-ment report includes assessassess-ment profiles, detailed findings and improve-ment recommendations. The GQM plan and measureimprove-ment plan contain information on the measurement programme. The interviewees and inter-viewers together review the results, in order to collect feedback, verify findings, and make corrections if necessary.
Integrated Feedback
Integrated feedback combines the presentation of assessment results with GQM feedback sessions.
Feedback sessions are organized meetings involving members of the project team and the PROFES team. They provide an essential mecha-nism for supporting analysis and interpretation of the measurement results. The objective of the integrated feedback is a) to discuss the preliminary findings and results of the measurement programme, b) for the project team to interpret the data collected so far, and c) to discuss the status of improvement actions with complementary measurements. It is also possible to use the feedback sessions to evaluate PPDs using data from product and process measurements.
Feedback sessions are an integral part of goal-oriented measurement. In GQM measurement, regular feedback sessions are arranged with the application project. Integrated feedback sessions combine GQM feedback
sessions and assessment presentations, which contain a presentation of assessment results and a follow-up of improvement actions. It is important that the final assessment presentation is performed as soon as possible after completion of the interviews, so that assessment results can be presented and accepted, and improvement recommendations discussed.
Case experiments during the PROFES project indicated that application projects are interested in participating in integrated sessions, since this method of implementing the PROFES improvement methodology was considered both efficient and useful for the target project and the PROFES team. Table 4.3 describes the integrated feedback session activities:
Table 4.3. Activities conducted during integrated feedback
Assessment GQM method
Integrated feedback
§ Implement improvement actions
§ Follow-up of the status of software process improvements
§ Collect measurement data
§ Feedback from assessment-related measurements
§ Refine improvement actions
§ Prepare and conduct feedback session
§ Write feedback session report
§ Update measurement programme
Two kinds of integrated feedback sessions can be arranged:
1) An initial feedback session in which assessment results, improvement recommendations, and project responses are discussed.
2) Further feedback sessions, during which collected measurements are analysed. This also includes a status presentation of improvement actions.
If measurements are linked to assessment indicators, improvements in process capability can also be presented.
We recommend that the initial feedback session be held soon after the assessment results are available. The main emphasis at this session is on assessment results, findings, and recommendations, as measurement data is rarely available. GQM plans and measurement plans can be presented to the project members at once, in order to hear their immediate comments. If the company has already collected measurements that are part of the GQM plan, then those measurements can also be presented.
The subsequent feedback sessions arranged are closer to the feedback sessions recommended by the GQM approach. The main focus of the feedback sessions is to present the collected measurement data and analyse it together with the project team. An integrated feedback session differs from the standard GQM feedback session in that it also contains assessment feedback.
There are two kinds of assessment feedback that can be given:
ADVANCED ASSESSMENT AND MEASUREMENT TECHNIQUES 4-11
a) Feedback can be given on status of improvement recommendations b) Feedback can be given on process capability measurements
Usually, several improvement recommendations were made in the assess-ment, but it is not possible to measure all of them. Improvement recom-mendations not followed up with measurements should at least have their current status checked. Measurements collected to continuously monitor the influence of improvement recommendations and fine-tuning naturally offer more detailed information than a simple status check.
Issues related to process capability measurements are discussed in the next section, “Continuous assessment”.
How to Use Continuous Assessment to Support Product-Driven Process
Improvement?
In this section, we will focus on continuous assessment and discuss both its benefits and limitations. The main benefits are increased visibility of the actual software process and the ability to detect process deviations earlier than before. However, successful application of continuous assessment requires a focused improvement area, experience in goal-oriented measurement, and an adequate data collection infrastructure.