Situational Method Engineering (SME) in agile contexts have been reported by Karlsson (2013) and Karlsson and Ågerfalk (2009b). Agile ISD methods promote stakeholder communication and collaboration by fostering practices that rely
heavily on socialization to access and share tacit knowledge within a project organization (Chau, Maurer & Melnik, 2003). Iterative activities and continuous integration of WorkProducts, coupled with a strong sense of collective owner-ship have a positive effect on stakeholder satisfaction and project success (Fer-reira & Cohen, 2008). Such principles quite aptly address the important Stake-holder and Context dimensions of evaluation, promoting the consideration of the situational factors that are essential in the SME activities. Building on the widely adapted agile ISD method Scrum (Schwaber & Sutherland, 2014), pre-sented in the following, the agile principles are applied in a conceptual instanti-ation of Artifact II into an agile usage situinstanti-ation. The high-level conceptualiza-tion is illustrated in FIGURE 28. It represents the dynamic part of Artifact II, whereas the method elements form the static part.
The situational evaluation effort is divided into three logical phases: pre-evaluation, pre-evaluation, and post-evaluation. The evaluation effort is run in iter-ations called sprints, each including the selected method elements of the meth-od base of Artifact II. Each sprint is initiated with a meeting in which all the stakeholders should be present. The length of the sprint can be any amount of days that is agreed upon, within the limit of a few weeks. Daily meetings should also be implemented among the core stakeholders to address the con-crete issues faced in the everyday operations. Depending on whether the evalu-ation effort is distributed or not, an online tool and/or a physical board for pro-ject management should be utilized to monitor the progress and to promote process transparency and ownership as well as communication.
In the pre-evaluation phase an evaluation framework is iteratively devel-oped by structuring and prioritizing requirements into the organizational framework. In the case of DSM tool requirements, they are linked into technical framework as criteria, using the Artifact I presented in Chapter 3 as guidance.
The technical framework also includes the metrics, scales, standards, and meas-urement techniques for the criteria.
In the evaluation phase a set of requirements, in the order of the set priori-ties, are taken under consideration and goals are derived for the evaluation sprint. The goals are then transformed into concrete tasks to be taken and add-ed into the sprint backlog list, steering the activities taken in the evaluation sprint. The evaluation phase can include any of the method elements of Artifact II. In the case of Evaluation WorkUnits, the tool candidates are measured by the criteria and guidance provided by the technical framework. As the outcome of the evaluation phase, a data increment is produced. In the case of Evaluation WorkUnits, the increment should always provide a complete piece of evalua-tion data of the tool candidates, in regards to the sprint goals.
In the post-evaluation phase, the data increments produced in the sprints are integrated into corresponding WorkProducts, e.g. an evaluation report or an update to the evaluation framework. As the outcome of the final sprint, tool selection activities produce a recommendation for a tool. At the end of the eval-uation effort, a selection workshop is organized, in which the recommendation is validated, and the experiences from the effort are analyzed and documented.
The evaluation framework refined during the evaluation effort will be an asset in future evaluations.
FIGURE 28 Conceptual Instantiation of Artifact II into an Agile Usage Situation, Adapted from Abrahamsson et al (2002, p. 28)
5.7 Summary
The chapter discussed the components of the baseline method for the engineer-ing of situational evaluation methods for DSM tools, called Artifact II. First, the SME foundation, along with the related process engineering metamodels and CAME tools, were discussed. Second, a conceptual method base and its three types of method elements, WorkUnit, WorkProduct, and Producer, were dis-cussed. The WorkUnits were derived from the synthesis of the activities of the existing evaluation methods. The WorkUnits were further divided into the high-level concepts of Preparation, Structuring, Evaluation, and Selection. Fur-thermore, the WorkProduct and Producer elements were constructed and pre-sented. Third, situational factors were discussed on a general level, and aligned with the previously discussed socio-technical dimensions of evaluation. Fourth, the various approaches to method construction and tailoring, i.e. the construc-tion guidelines, were presented. Fifth, the resulting high-level architecture of Artifact II was presented, based on the previous sections. Finally, Artifact II was conceptually instantiated in an agile usage situation.
6 EVALUATION OF META-METHOD
This chapter presents the empirical and conceptual evaluation of Artifacts I and II, discussed in the previous sections. The main objective is to seek validation for the artifacts by structuring them into a single design theory called Meta-Method and by comparing it against the similarly structured design theory of ISO 14102. In this evaluation process we use the evaluation criteria of progress for IS design theories.
First, the evaluation methodology is presented. The methodology is divid-ed into four sections: Grounding Approach, Empirical Approach, Conceptual Approach, and Evaluation Criteria. Here, the current state of the discourse on the evaluation of DSR artifacts as well as the rationale behind the evaluation approaches chosen are first discussed. Then, the internal, external, and empiri-cal grounding of the artifacts are delineated. Next, the empiriempiri-cal approach to evaluation, i.e. case study, is discussed, followed by the presentation of the con-ceptual approach, in which the relationships between design theories, instantia-tions, and humans are discussed and reflected to the phenomena of the case study. Additionally, design theory componentization and the evaluation criteria of progress for IS design theories: utility, internal consistency, external con-sistency, broad purpose and scope, simplicity, and fruitfulness of new research, are introduced.
Second, the design theories of Meta-Method and ISO 14102 are structured as design theory components, which is conducted to prepare them for the com-parative evaluation with the criteria of progress.
Third, a case study in which Meta-Method was initially designed and em-pirically evaluated is discussed. The empirical evaluation is based on the analy-sis of the data collected during the case study, which included a company oper-ating in the industry of professional mobile radio networks and devices. The empirical evaluation addresses the utility criterion.
Fourth, the conceptual evaluation is provided on the basis of the other cri-teria of progress. Finally, a summary of the chapter is presented.