The initial framework described in chapter 8.3. and ANNEX 4 (detailed slides about the framework) together with ANNEX 3 - the cover letter along with ANNEX 5 (the questionnaire) were sent to the companies in April 7th year 2019.
The purpose was to demonstrate the capability of the framework to solve the comprehensiveness problem in red teaming. This is the fourth phase of the DSRM (Peffers et al., 2007). Additional material was also sent along with results from the initial survey to present the feedback that is required in Delphi process (Dalkey, 1967). Shortened version from chapters 1-5 to elaborate details of the framework (25 pages) was also sent. The overall package was quite heavy consisting of approximately 50 pages of text. Answering deadline was set to 26th of April, but last answers were received by 7th of May. Questions which were asked in the first Delphi-round were;
1. How are you acquinted with the background material? 1-5. Open comments if any?
2. Is the CART framework conceivable? Can you understand and differentiate the purpose of continuous activities, phases, steps and products? 1-5. Please submit issues for development.
3. Give grade for the continuous activities from (1) to (5), with the help of reference grading below. Should something be deleted, combined etc?
4. Give grade for the phases from (1) to (5), with the help of reference grading below. Should something be deleted, combined etc?
5. Give grade for steps from (1) to (5), with the help of reference grading below.
Should something be deleted, combined etc?
Responses were submitted with a 1-5 Likert scale with predefined meanings for 1,3 and 5 leaving 2 and 4 blanks. Open comments were also asked for each response. Questionnaire sheet is in the ANNEX 5.
7.4.1 Evaluation of Delphi 1 – answers
Delphi-survey round 1 produced some variation on responses due to different background of companies. Divergence of answers is not a problem. It demonstrates that there has not been a negative “committee effect” among the sample group. (Dalkey, 1967)
The initial answer was still, yes - The model solves the comprehensiveness problem. Maybe too well, being a bit overwhelming and covering almost all aspects of red teaming as one response phrased the issue. The numerical assessment of the framework is presented in the table below.
Question one measured the background work which the recipients committed to and that shows an average of 3,6 which means that the heavy background package was read by all.
Question two showed that the framework is understandable, and one company (4) could have implemented it as such whilst two companies (1,5) thought of it a bit obscure and crowded. The result is that the framework as such is conceivable with minor changes.
Question three affirmed that continuous activities are viable even though their substance is not fully understood. Activities were considered useful to connect the phases by company 4.
Question four proved that phases are valid, but one response was criticizing their proportionality (5) whilst other complemented the variability (4). One (1) added that phases are set up like in security consulting.
Question five assured that steps are detailed and actionable although respondents did not always understand the substance inside every step. This was not even the meaning. One response (4) noted that the line between steps is very narrow and several steps could be completed simultaneously inside a phase which is entirely correct and adaptive use of the planning process. One response (1) suggested removing the “internal development”-step from the active framework although it should be kept as a continuous activity.
TABLE 12 Numerical results from Delphi 1 – questions.
Subject/ comprehension has not always followed as one response (5) noted. This is most likely due to vast amount of background material and people’s unfamiliarity with military or agile methodology. To raise a few misconceptions; One response (3) claimed that the attacks in engage phase have no campaign planned and are based just on gathered intelligence. In this case the responder did not understand that the campaign is planned in the previous phase and presented as the concept of operation (CONOPS). Same responder also confused the activities and phases in the replies and analysis was proposed as additional phase, when it is one element of intelligence activity.
One response (5) claimed that there are no feedback loops, or the model is not operating iteratively and would have like to have an IPO (input-process-output) loop presented. The feedback loop is in every step’s retrospective and every step is its own IPO-loop which has an input from the previous step and runs through five activities as a process and gives output to the next step. Both misconceptions are not the fault of the responder, but of bad communication from the researchers. This led to the first discovery that the model should be
communicated in a clear and precise way to the audience taking their background into consideration more thoroughly.
The most positive respondent (4) claimed that the framework is a great background plan but requires a mature organization to handle all the steps. It takes effort from a team to go through every step, but when done correctly the result is more consistent. The responder also noticed that the phases are the baseline of the framework and steps are the details inside the phase which complete the process. This led to the second discovery that two models should be created; 1. for the client to present what is done and 2. to the red team to show how it’s done. Naturally the model 2 is the more detailed one and needs more training. This presents an old Finnish folk wisdom; “The trick and how it’s done are two different things” meaning that what may seem simple, actually needs a lot of work in the background but it’s needless to show it for the audience because then the trick is ruined.
One response (2) noted that it would be beneficial in some cases before the engage phase to provide pre-training for the client if the maturity is not sound enough. This was noted in the previous phase of the study in initial survey, but the authors simply forgot to add this element to the model. Second respondent (1) noted that the word “agile” is in the headline, but it’s not mentioned in the model and could be presented more clearly. The main notion was that most of the issues brought forth by the respondents are already included in the framework, but they are hard to see, hence better communication is needed for comprehension of the framework.
7.4.2 Processing of Delphi 1 – answers
The answers were classified into themes and three main issues were discovered that were constant. These issues are depicted in the table below.
First notion is about the overwhelming nature of the framework. The model is intentionally comprehensive, or heavy. However, when conducting the joint planning process, fulfilment of all possible steps with related products is not mandatory. User can pick the things she needs from the framework if she follows the basic idea of phases and continuous activities. The product backlogs are created in the planning phase and can be adapted in the engagement phase if needed. Conclusion is that in the refined model this needs to be communicated more clearly and visualized better as several respondents (1,2,5) claimed. Also, a simplified cyclical model is needed for marketing purposes and customer relations.
The second and third issue were about understanding the substance of military way of thinking, planning and executing operations. The terminology and products like intelligence collection plan (ICP), Joint intelligence preparation of environment (JIPOE), concept of operation (CONOPS) were not very familiar to all respondents but since they form the basis of the framework they cannot be banned. Terminology can be changed, but the understanding comes through learning which requires training. For the researchers these issues are very clear
due to military background, but most of the people in ICT-business do not have a military background. Therefore, better communication of the model and its terminology are pursued.
TABLE 13 Issues for remediation from Delphi 1 - questions
ISSUE Rationale
Framework is heavy
(5 respondants) Picture is crowded
Requires good maturity from the red teaming company
Difficult to sell for clients
Challeging to implement
Contains too many elements Activities are
obscure (4 respondants)
Idea of continous activity is not obvious
Targeting not understood, could align with intel.
Activities and products of different phases and are not understood
Presenting of stakeholders might help to understand the model better i.e. Chief intel, red team leader, etc.
Military terminology and agile events not understood (4 respondants
Intelligence cycle or products are not familiar to recipients
Retrospective role as feedback/iterative mechanism not understood
Planning products (COA, CONOPS) not understood to be the campaign plan.
All the presented issues are real, and they lead to two questions:
Is the initial CART-model too complicated?
Was the initial CART-model communicated properly for the companies?
The framework was complimented by comprehensiveness noting that it requires maturity from the red team as well. This led to the following conclusions that are implemented in the next Delphi round:
1. Create a simplified cyclical model for presentation.
2. Implement minor changes suggested by the respondents.
3. Highlight that completion of every single detail is not needed to complete the framework and it’s just a framework for flexible use.
4. Create a training program for the model for the red teams if someone tries to implement the framework in practice.