Empirical study on the adoption, use and effects of cloud-based testing

(1)

Leah Riungu-Kalliosaari

EMPIRICAL STUDY ON THE ADOPTION,

USE AND EFFECTS OF CLOUD-BASED TESTING

Acta Universitatis Lappeenrantaensis 572

Doctoral dissertation for the degree of Doctor of Science (Technology) to be presented with due permission for public examination and criticism in the Auditorium 1382 at Lappeenranta University of Technology, Lappeenranta, Finland, on 12th of June 2014, at noon.

(2)

Supervisors Professor Kari Smolander

Department of Software Engineering and Information Management School of Industrial Engineering and Management

Lappeenranta University of Technology Finland

Associate Professor Ossi Taipale

Department of Software Engineering and Information Management School of Industrial Engineering and Management

Lappeenranta University of Technology Finland

Reviewers Professor Ilkka Tervonen

Department of Information Processing Science University of Oulu

Finland

Professor Scott Tilley

Department of Computer Sciences Florida Institute of Technology USA

Opponent Professor Tommi Mikkonen

Department of Pervasive Computing Tampere University of Technology Finland

ISBN 978-952-265-575-2 ISBN 978-952-265-576-9 (PDF)

ISSN-L 1456-4491 ISSN 1456-4491

Lappeenrannan teknillinen yliopisto Yliopistopaino 2014

(3)

Abstract

Leah Riungu-Kalliosaari

Empirical study on the adoption, use and effects of cloud-based testing Lappeenranta, 2014

105 p.

Acta Universitatis Lappeenrantaensis 572 Diss. Lappeenranta University of Technology

ISBN 978-952-265-575-2, ISBN 978-952-265-576-9 (PDF) ISSN-L 1456-4491, ISSN 1456-4491

Cloud computing is a practically relevant paradigm in computing today. Testing is one of the distinct areas where cloud computing can be applied. This study addressed the applicability of cloud computing for testing within organizational and strategic contexts. The study focused on issues related to the adoption, use and effects of cloud- based testing.

The study applied empirical research methods. The data was collected through interviews with practitioners from 30 organizations and was analysed using the grounded theory method. The research process consisted of four phases. The first phase studied the definitions and perceptions related to cloud-based testing. The second phase observed cloud-based testing in real-life practice. The third phase analysed quality in the context of cloud application development. The fourth phase studied the applicability of cloud computing in the gaming industry.

The results showed that cloud computing is relevant and applicable for testing and application development, as well as other areas, e.g., game development. The research identified the benefits, challenges, requirements and effects of cloud-based testing;

and formulated a roadmap and strategy for adopting cloud-based testing. The study also explored quality issues in cloud application development. As a special case, the research included a study on applicability of cloud computing in game development.

The results can be used by companies to enhance the processes for managing cloud- based testing, evaluating practical cloud-based testing work and assessing the appropriateness of cloud-based testing for specific testing needs.

Keywords: Cloud computing, cloud-based testing, software testing, quality, cloud gaming, grounded theory

UDC 004.7:004.41:004.415.53

(4)

(5)

To the African girl child

Your right and access to education is a cause worth fighting for.

“The future must not belong to those who bully women. It must be shaped by girls who go to school and those who stand for a world where our daughters can live their dreams just like our sons.” – Barack Obama, at the United Nations General Assembly

2012

(6)

(7)

Acknowledgements

I consider it a blessing to have had the opportunity to carry out this research work. It would not have been possible without the support of many wonderful people. I am not able to mention everyone here, but I acknowledge and deeply appreciate all your invaluable assistance and support.

I would like to thank my supervisors, Professor Kari Smolander and Associate Professor Ossi Taipale, for their guidance, encouragement and contribution throughout this research. Thank you for all your efforts and for providing a great research environment.

I wish to thank the reviewers of this dissertation, Professor Ilkka Tervonen and Professor Scott Tilley for your valuable comments and feedback that helped me to finalize the dissertation.

I would like to express my gratitude to Dr. Ita Richardson for your warm reception and guidance during my research visit at the Irish Software Engineering Research Centre (Lero). Thank you also for your efforts as a co-author. Additional thanks to another co-author, Dr. Jussi Kasurinen, for your contributions.

For financial support, I would like to acknowledge the following sources: The Finnish Funding Agency for Technology and Innovation (TEKES), the Graduate School on Software Systems and Engineering (SoSE), European Union Regional Development Fund project administered by the Council of South Karelia, the companies participating in ESPA, STX and SOCES research projects, LUT Foundation and the Irish Software Engineering Research Centre (Lero) grant 10/CE/I1855.

I appreciate the support and assistance of my colleagues at the Department of Software Engineering and Information Management. Special thanks to Tarja Nikkinen, Ilmari Laakkonen and Petri Hautaniemi, for providing administrative and technical support.

My many friends, both near and around the globe, thank you for sharing my joys and frustrations. This dissertation is probably a little bit late, but better late than never.

Thank you - Joan Mallai and family, Seija and Sakari Kiiskinen, and Pirkko and Martti Rouhiainen – for being like family away from home.

Special gratitude to my family, mother Isabella, siblings Kagendo, Mugambi, Murage, Martin and Edwin, parents-in-law Eila and Jorma Kalliosaari, brother-in-law Sami, and the extended family for unconditional support and encouragement. I fondly

(8)

remember my late father Eustace Riungu Muchiri, who highly upheld the value of education and encouraged all my academic endevours. Death did not allow him the opportunity to celebrate this milestone; I know he would have been very pleased.

My dear husband Jyrki, I am truly grateful for your love, patience, understanding and support.

Lappeenranta, March, 2014 Leah Riungu-Kalliosaari

(9)

List of publications

I. Riungu, L.M., Taipale, O., and Smolander, K., 2010. “Research Issues for Software Testing in the Cloud”, In Proceedings of the 2^nd International Conference on Cloud Computing Technology and Science (CloudCom), pp.

557-564.

II. Riungu-Kalliosaari L., Taipale, O., and Smolander, K., 2013. “Software Testing as a Service: Perceptions from Practice”, Book chapter appearing in the book, Software Testing in the Cloud: Perspectives on an Emerging Discipline, IGI Global, pp. 196-215.

III. Riungu-Kalliosaari, L., Taipale, O., and Smolander, K., 2012. ”Testing in the Cloud: Exploring the Practice”, Special Issue on Cloud Computing for Software Engineering, IEEE Software, Volume 29, Issue 2, pages 46-51.

IV. Riungu-Kalliosaari, L., Taipale, O., Smolander, K. and Richardson, I.

“Adoption and Use of Cloud-Based Testing in Practice”, Submitted to the Software Quality Journal.

V. Riungu-Kalliosaari, L., Taipale, O., and Smolander, K., 2013. ”Desired Quality Characteristics in Cloud Application Development”, Proceedings of the 8^th International Joint Conference on Software Technologies (ICSOFT), pp. 303- 312.

VI. Riungu-Kalliosaari, L., Kasurinen, J., and Smolander, K., 2013. ”Cloud Services and Cloud Gaming in Game Development”, Proceedings of Game and Entertainment Technologies (GET), 22-24.7.2013, Prague, Czech Republic, pages 197-206.

In this dissertation, these publications are referred to as Publication I, Publication II, Publication III, Publication IV, Publication V, and Publication VI.

After Publication I, my surname changed from Riungu to Riungu-Kalliosaari – due to getting married.

(10)

(11)

Symbols and abbreviations

API Application Programming Interface

BPEL4WS Business Process Execution Language for Web Services COTS Commercial-off-the-shelf

CRM Customer Relationship Management

EU European Union

GSD Global Software Development

Haas Human as a Service / Human -as-a-Service

HD High-definition

IaaS Infrastructure as a Service / Infrastructure -as-a-Service ISO International Organization for Standardization

IT Information Technology

MDD Model-Driven development

NIST National Institute of Standards and Technology OSS Open Source Software

PaaS Platform as a Service / Platform-as-a-Service QoE Quality of Experience

QoS Quality of Service

SaaS Software as a Service / Software-as-a-Service SDLC Software Development Life Cycle

SLA Service Level Agreement

SME Small and Medium sized Enterprise

(12)

SOA Service-Oriented Architecture SOAP Simple Object Access Protocol SQL Structured Query Language

STaaS Software Testing as a Service / Software-Testing-as-a-Service

SUT System Under Test

TaaS Testing as a Service / Testing-as-a-service

ToS Terms of Service

VM Virtual Machine

WSDL Web Services Description Language

XP eXtreme Programming

(13)

1 Introduction

In the information technology (IT) industry, software products and services are constantly evolving due to changing technologies, trends and market requirements.

Innovations towards service-oriented architecture (SOA) and software-as-a-service (SaaS) models have greatly affected the nature of software systems and organizations (Collard, 2009; Goth, 2008). As software products and services continue to evolve, it implies that the methods, tools and concepts to test them must also change. Over the last five years, a new paradigm called cloud computing has attracted the attention of many IT practitioners and researchers. There have been different descriptions of cloud computing (Armbrust et al., 2009; Buyya et al., 2009; Geelan, 2009; Mell & Grance, 2011). For example, Armbrust et al. (2009) look at cloud computing from the point of view of software applications delivered as services, while Buyya et al. (2009) pay attention to the technical concepts that cloud computing is based on, such as, parallel and distributed computing.

Mell and Grance (2011) define cloud computing in terms of its essential characteristics (e.g. measured service and resource pooling), service models (e.g. platform as a service), and deployment models (e.g. public cloud and hybrid cloud). Even though the definition of cloud computing has been subjected to many debates, cloud computing is viewed as a “promising paradigm that could enable businesses to face market volatility in an agile and cost-efficient manner” (Hassan, 2011, p.16). The definition by Mell and Grance (2011) is used in this dissertation.

Cloud computing follows the concept of utility computing to provide users with on- demand access to computing resources that are billed on pay-per-use basis (Buyya et al., 2009). There are many opinions about the purpose of cloud computing, how to adopt it and use it in practice. Many have wondered whether cloud computing is an old idea dressed in a new name, or whether they should consider it at all (Voas &

Zhang, 2009). Some see it as way to cut down on expenditure, while others see it as a

(18)

means to attain flexibility and focus on business goals. However, many people are unsure of what they could gain from cloud computing because it introduces more challenges, e.g., security. Cloud computing can be applied to support many business operations to achieve different goals. Therefore, it is important to understand cloud computing in a specific context within which it is applied.

The goal of this dissertation is to increase empirical knowledge of the adoption, use and effects of cloud-based testing. In this regard, we use the term cloud-based testing to refer to testing of software applications using infrastructure hosted in cloud computing environments. We address this goal by providing empirical results that help to define and understand cloud-based testing, as well as how it is applied in practice. Because testing is connected to development and quality, the dissertation also looks at cloud gaming and quality in the context of cloud application development.

This dissertation contains a series of empirical studies that focus on cloud-based testing. To provide more evidence on applying cloud computing in different contexts, the dissertation includes a study on cloud gaming. The cloud gaming example demonstrates the importance of considering the specific context within which cloud computing (and consequently, cloud-based testing) is applied. The results in the publications were obtained using qualitative research methods with data collected through interviews from different practitioners. The research method selected for this dissertation was grounded theory, which is an iterative research process for gathering and analysing data with the purpose of denoting theoretical concepts from the data (Corbin & Strauss, 2008).

The contributions of this dissertation are four-fold: (1) We define and understand cloud-based testing by describing the facets of testing in the cloud, and investigating the benefits, challenges, requirements and effects of cloud-based testing. (2) We provide insights on the applicability of cloud-based testing in practice and develop a roadmap and strategy for adopting cloud-based testing. (3) We explore quality issues in cloud application development. (4) We provide insights on the applicability of cloud computing within the gaming industry.

This dissertation consists of two parts, an introduction and an appendix containing the scientific publications. The introduction presents the background of the research area along with the research goal, research question, research method, overview of the publications, and a summary of the research contributions. The appendix is composed of the publications, which contain the research results in detail. Five of them have been through a scientific referee process and Publication IV is submitted to a journal for the evaluation process.

The introduction contains six chapters. Chapter 2 presents the literature review discussing the topics covered by the dissertation i.e. cloud computing, cloud-based

(19)

testing, software quality assurance in the context of cloud computing, and cloud gaming. Chapter 3 describes the research process including the research questions and the research method. Chapter 4 summarizes the results contained in the publications.

Chapter 5 discusses the contributions and their implications for research and practice.

It also evaluates the research and mentions the limitations of the research. Chapter 6 summarises the contributions of dissertation and proposes future research topics.

(20)

(21)

2 Cloud computing and cloud-based testing

The purpose of this chapter is to set the scope of the study by describing the definitions and concepts related to this dissertation. The definitions and concepts are obtained from the related research and literature. Before going into the descriptions of cloud computing and cloud-based testing, a brief description of the software technology trends preceding cloud computing is presented.

2.1 Software technology trends during the 21

^st

century

The software industry has changed significantly during the past decades. In an analysis of the future trends and implications for software engineering, Boehm (2006) identified eight trends that would befall the software industry. These trends are:

 increased emphasis on agility, usability and end value.

 increased emphasis on software criticality and dependability.

 increasing needs for commercial-off-the-shelf (COTS) systems and components, open source and legacy systems.

 model-driven development (MDD).

 the increasing interaction of software engineering and systems engineering.

 increasing global connectivity and need for systems to interoperate.

 increase of software-intensive systems of software.

 increasing computing capabilities across different platforms and applications.

By the start of the 21^st century, the software industry had transformed from the use of formal and waterfall development procedures, to an escalated inclination towards agile methods such as Adaptive Software Development, eXtreme Programming (XP), Feature Driven Development, and Scrum (Boehm, 2006). Agile development methods

(22)

primarily focus on building functional and usable software within short iterations.

The agile manifesto was published in 2001, highlighting four main principles for agile software development: (1) individuals and interactions over processes and tools; (2) working software over comprehensive documentation; (3) customer collaboration over contract negotiation; and (4) responding to change over following a plan (Agilemanifesto.org, 2001).

Alongside agile methods, there was also an increase in open source software (OSS) and commercial-off-the-shelf (COTS) systems and components. There were some shortcomings with OSS and COTS such as difficulties with updating the software and the lack of “usability, dependability, interoperability and localizability in different countries and cultures” (Boehm, 2006). To address some of these problems, model- driven development (MDD) was applied. MDD focuses on developing software for a specific domain such as banks or automobiles, and it leads to the development of interoperable applications (Boehm, 2006). However, the main challenge with MDD was its inability to adapt to the changing software infrastructure caused by massive distribution, mobile computing and evolving Web objects (Boehm, 2006). To support the needs of the evolving software infrastructure requirements, there was a rise in the use of object-oriented models and service-oriented architectures (SOAs), which led to greater interaction between software systems.

Service-oriented architecture (SOA) became popular and was widely explored (Ebert, 2008; Gold et al., 2004; Goth, 2008; Sahoo, 2009). SOA was seen as “logical way of designing a software system to provide services either to end-user applications or other services distributed in a network, via published and discoverable interfaces”

(Papazoglou et al., 2007, p.38). SOA is based on open Internet-based standards, such as Simple Object Access Protocol (SOAP), Web Services Description Language (WSDL) and Business Process Execution Language for Web Services (BPEL4WS) and uses the Internet as the communication channel. The application of open standards to implement SOA helped to produce loosely coupled and interoperable software applications, referred to as services.

A service is “an act or performance offered by one party to another. Although the process may be tied to a physical product, the performance is essentially intangible and does not normally result in ownership of any of the factors of production”

(Lovelock, 2000, p.3). Extending this definition to software engineering, service-based software, better known as software as a service (SaaS), is defined as software that is configured and executed to meet a specific set of requirements at a point in time and then discarded after use (Bennett et al., 2000). When delivering software as a service, a SaaS vendor handles the IT infrastructure, hosting, maintenance and support services (Sahoo, 2009; Sun et al., 2007), and provides fully functioning software that can be configured and used at delivery time (Turner et al., 2003). By using SaaS, end users are

(23)

able to reduce the Total Cost of Ownership (TCO) while using software that is tailored to suit their unique needs and working environments (Olsen, 2006; Sun et al., 2007).

The use of SaaS influenced a service-oriented mindset across the IT industry.

Organizations wanted to quickly deliver value to the users by providing software as a service (SaaS)-based applications (Sahoo, 2009; Turner et al., 2003). SaaS supported the evolution of software from being delivered as a product, to being delivered as a service that was provided to users on a demand basis. Over time, it became common for people to access online content with limited knowledge about the data centers hosting the content – a paradigm that evolved into cloud computing (Buyya et al., 2009).

2.2 Cloud computing

The advent of cloud computing generated many attempts to give it a descriptive, clear and understandable definition. Geelan solicited for cloud computing definitions from twenty-one experts (Geelan, 2009). For example, one of the experts defined cloud computing as Internet centric software which shifts the traditional single tenant software to scalable, multi-tenant, multi-platform, multi-network, and global software (Geelan, 2009). Another expert described cloud computing as being the capability to access resources and services needed to perform functions with dynamically changing needs, such that an application or service developer can request access from the cloud rather than a specific endpoint or named resource (Geelan, 2009).

Buyya and his team define “a cloud as a type of parallel and distributed system consisting of a collection of inter-connected and virtualized computers that are dynamically provisioned and presented as one or more unified computing resource(s) based on service-level agreements established through negotiation between the service providers and consumers” (Buyya et al., 2009, p.601). Parallel computing refers to multiple processors communicating with each other while located in the same machine (Sulistio et al., 2004). On the other hand, distributed computing has to do with different machines located in different physical locations and communicating with each other through networks such as the Internet (Sulistio et al., 2004). This definition emphasises two unique features of cloud computing:

 Virtualization: this is the capability that computing resources have to enable a single physical machine to function and present the illusion of many smaller virtual machines (VMs) running different operating system instances (Barham et al., 2003; Buyya et al., 2009). The VMs can host isolated operating system environments and make use of different

(24)

portions of the resources – on the same physical machine (Buyya et al., 2009).

 Dynamic provisioning: a technique whereby the computing resources are started and stopped to meet the changing demand by spreading the load among the available virtual machines under the specified conditions (Buyya et al., 2009; Lu et al., 2013).

Armbrust and his team referred to cloud computing as “the applications delivered as services over the internet and the hardware and systems software in the data centers that provide those services” (Armbrust et al., 2009, p.4). They described the services delivered over the internet as the earlier known software as a service (SaaS); and the data center hardware and software as a cloud; which could either be a public cloud (available to everyone) or a private cloud (intended to a specific organization).

According to the National Institute of Standards and Technology (NIST), cloud computing is “a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.” (Mell & Grance, 2011, p.2). The NIST definition descomposes the cloud model into five essential characteristics, three service models and four deployment models (Figure 1).

The following text summarizes the essential characteristics, service models and deployment models of cloud computing as described by Mell and Grance (2011), and adapted from Publication I. Cloud computing has the following essential characteristics:

 On-demand self-service: Computing services such as servers, storage and virtual machines can be acquired automatically if and when needed by a user without human interaction with the cloud service providers.

 Broad network access: Computing services can be accessed over a network using different devices e.g. laptops and mobile phones.

 Resource pooling: Computing services are pooled for use by many users through a multi-tenant approach that enables the services to be allocated as per the demand of the users.

 Rapid elasticity: Computing services are unlimited and can quickly be scaled in and out as required by the customers.

 Measured service: Appropriate metrics like storage, active user accounts, and bandwidth are used to measure the usage of the computing services. This provides transparency of the utilized service to the cloud service provider and customer.

(25)

The cloud contains three main layers referred to as service models (Mell & Grance, 2011). These include:

(1) Software as a Service (SaaS): Customers are able to make use of applications that are running in cloud environments, usually by means of a web browser. The customer does not have the rights to control or manage the cloud infrastructure that is running the software.

(2) Platform as a Service (PaaS): In this case, customers are provided with programming and execution environments through which they can run and access applications of their own choice. Similar to the SaaS model, customers cannot control the underlying cloud infrastructure but have control over the applications they create and to a certain degree, configuration settings of the hosting environment.

(3) Infrastructure as a Service (IaaS): This is where computing services such as storage, processing and networks are provided by the IaaS provider for the customers to deploy and run their applications. IaaS gives a customer

Figure 1: A visual overview of the NIST definition of cloud computing

Measured service

Resource pooling Rapid elasticity

On-demand self-service Broad network access Essential Characteristics

Service Models

Software as a Service (SaaS)

Platform as a Service (PaaS)

Infrastructure as a Service (IaaS)

Deployment Models

Private Cloud

Public

Cloud Community Cloud

Hybrid Cloud

(26)

the flexibility to control and run software over the computing environment.

In addition to the three service models mentioned in the NIST definition of cloud computing, Lenk et al. (2009) proposed two additional service layers, namely Human as a Service (HaaS) and supporting services. These layers can also be viewed as service models. Human as a Service (HaaS) incorporates crowdsourcing so that a crowd of people can use the cloud technology from different geographical places and work together to complete a task requiring effort from a large group of people. A good example of crowdsourcing is uTest, a company that provides software testing solutions to its customers through on-demand access to a community of professional testers (uTest, 2013). Another form of HaaS is Information Aggregation Services (IAS) that deals with generating a unified figure that represents a popular opinion of the crowd (Lenk et al., 2009).

Cloud computing can be implemented through different clouds, referred to as deployment models. These deployment models are:

• Private cloud: The cloud infrastructure operates mainly to serve one organization only and may be managed by the organization itself or an external cloud provider.

• Community cloud: Several organizations share the cloud infrastructure and provide services to a specific community that has similar needs.

• Public cloud: The cloud infrastructure is available for use by anyone and is usually owned by a large organization.

• Hybrid cloud: This is a combination of two or all of the above-mentioned clouds.

The above definitions describe cloud computing in terms of its composition (computing resources and services), distribution channel (internet/networks), acquisition approach (on-demand, pay-per-use) and stakeholders (providers, users).

Cloud computing is enabling the delivery of computing as a utility (Armbrust et al., 2009), which Buyya, et al. (2009) envision to be a 5^th utility in the future (in addition to water, electricity, gas and telephony).

The NIST definition of cloud computing has gained wide acceptance and is used in this study. The research also adopts the view about Human as a Service (HaaS) being another deployment model of cloud computing (Lenk et al., 2009), because it supports the delivery of services through crowdsourcing.

According to the literature, some benefits of cloud computing include:

(27)

 Reduced costs: Cloud computing provides quick access to computing resources without upfront capital investments for the users, which helps to reduce the total expenditure for an organization (Marston et al., 2011). Cloud computing also frees the users from the expense and effort of buying, installing and maintaining their own computing resources (Leavitt, 2009).

 Availability: Large cloud service providers that have abundant resources and redundant equipment may offer more availability than if the resources are hosted within the premises of smaller organizations (Leavitt, 2009).

 Scalability: Cloud computing enables the users to acquire and free the resources according to the changing demands (Marston et al., 2011).

 Flexibility: Leavitt (2009) claims that most cloud computing vendors don’t require contracts and let users work with their services as needed, which makes the cloud a convenient way of getting additional resources for activities such as testing new applications and services.

 Possibilities for new applications: Cloud computing enables the availability and delivery of applications and services that might not have been possible earlier, such as data and resource intensive business analytics and parallel batch processing requiring huge amounts of processing power (Marston et al., 2011).

 Other benefits include lower costs of entry for smaller organizations, which boosts innovation for start-up companies (Marston et al., 2011); and the ability to integrate many applications and services into powerful composite systems, for example, Salesforce.com (Leavitt, 2009).

Many concerns have prevented the adoption of cloud computing. Some of these concerns are:

 Security: For many IT organizations, security and privacy have been primary concerns. For example, the privacy regulations in some European countries prohibit some types of personal data from being distributed outside the European Union (EU). This made organizations such as Microsoft and Amazon to locate some data centres in the EU and allow the users to select the data centre using geographical preferences (Sultan, 2011).

 Loss of control: Many IT organizations have been afraid of losing control of their operations to a third party who could change the underlying technology without the knowledge and permission of the customers (Marston, et al. 2011;

Sultan, 2011).

 Lack of cloud computing standards: There have been concerns regarding the lack of interoperability between services from different cloud providers. The efforts to address this have come from cloud service providers such as Google and Microsoft, industry experts, independent organizations (e.g. the cloud computing interoperability forum) as well as the International Organization

(28)

for Standardization (ISO) (Marston et al., 2011). For example, the ISO has a study group that is working on cloud computing standards (ISO, 2013).

 Vendor lock-in: Due to the lack of standards, cloud providers offer their services through provider-specific interfaces (Sultan, 2011). This further limits the interoperability between the cloud services, resulting in a situation whereby users are ‘stuck’ with one cloud service provider, even when the users might wish to move to another provider.

 Reliability: Some of the cloud service providers have experienced service outages in the past, the most current being in August 2013 by Amazon (Amazon’s online retail operation was off for about an hour) and Google (a four-minute blackout across all applications). Service outages can be a serious problem for the users and the cloud provider, especially in the case of prolonged outages.

Despite the concerns surrounding cloud computing, an increasing number of organizations have migrated some of their business operations to the cloud. Some of the areas in which cloud computing is being applied are (Sheehan, 2013):

 File storage, synchronization and sharing, e.g., dropbox.

 Financial services/eCommerce such as Amazon and eBay.

 Email, e.g., Gmail.

 CPU intensive applications across various domains such as healthcare, social media and advertising.

 Going global, i.e., utilizing different data centres across the world.

 Test and development, e.g., load testing.

 Short-term projects, for instance, setting up a microsite for a few months.

 Handling seasonal capacity, for example, when there is a surge in the number of users in need of your web applications.

 Business intelligence/big data for businesses to find out more about their prospects and customers.

 Website hosting.

 Conducting proof of concepts.

 Web and mobile applications for social media, online communities and blogging.

 Advertising, for example, by hosting advertising servers in the cloud.

(29)

2.3 Software testing

This section starts with a brief description of software testing, followed by cloud- based testing

2.3.1 What is software testing?

There are many definitions of software testing in literature. According to Myers (2004), testing is the process of executing a program with the intent of finding errors. By finding and removing errors, one ascertains that the program does what it is supposed to do and does not do what it is not supposed to do; hence improving the reliability and quality of the program (Myers, 2004).

Kit (1995) defines testing as verification and validation.

Verification is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase (IEEE/ANSI, 1990). The aim of verification is to answer the question: are we building the product right? Basic verification methods are inspections, walkthroughs and technical reviews. Checklists are verification tools that can be used to verify, for example, requirements, technical design and code.

Validation is the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements (IEEE/ANSI, 1990). Validation ensures that the built product satisfies the user requirements (as described in the requirements specification) and that the product’s behaviour is according to the expected behaviour (as stated in functional specification). Validation includes activities such as unit testing, integration testing, usability testing, function testing, system testing and acceptance testing.

Testing can be divided into two groups of testing methods: black-box testing and white-box testing. Black-box testing deals with the functional specifications (i.e. what the software can do) and white-box testing is concerned with the technical specifications (i.e. the internal structure of the software).

Black-box testing includes testing methods such as syntax testing (systematic method to generate valid and invalid inputs to a program), state transition testing (using finite-state machines to design the tests), equivalence partitioning (identifying equivalence classes and test cases), error guessing (using intuition and experience to guess), cause-effect graphing (transform a natural-language specification to a formal- language specification) and graph matrices (using graphs to represent and organize data) (Kit, 1995).

(30)

White-box testing methods, include, for example, statement coverage (executing each statement at least once), decision coverage (each decision or branch takes on all possible outcomes at least once), condition coverage (each condition in a decision takes on all possible outcomes at least once) and path coverage (all possible combinations of condition outcomes in each decision occur at least once) (Kit, 1995).

Testing techniques can also be divided into static and dynamic testing (Heiser, 1997).

Static testing may also be referred to as static analysis, and it includes testing techniques that do not involve executing the software, e.g., reviews, walk-throughs, inspections, audits, program proving, symbolic evaluation and anomaly analysis.

Dynamic testing refers to any testing technique that involves executing the software, for example boundary-based techniques, decision table-based methods, statistical techniques, path testing, and data flow testing (Heiser, 1997).

Kit’s (1995) definition of software testing i.e. verification and validation, does not link software testing to any specific software development method or life cycle model and it is widely applied in software testing and quality practices (Taipale, 2007). For these reasons, this definition was used in this dissertation.

2.3.2 Cloud-based testing

By the year 2009, the IT industry was experiencing a shift towards service-oriented architectures (SOA) and software-as-a-service (SaaS) (see Section 2.1). Convinced about this shift, organizations such as Google started reconsidering their development and testing practices by taking into account the SOA and SaaS dynamics (Goth, 2008).

With regard to testing, the terms “software testing as a service (STaaS)”, “testing as a service (TaaS)”, “testing in the cloud”, “cloud testing” and “cloud-based testing” are now widely used as synonyms. As can be seen in Publication II, at the beginning of this study, the existing literature about cloud-based testing was in the form of industrial white papers and reports. However, as the field has evolved and gained popularity, scientific research has also been growing (Priyanka et al., 2012).

Cloud-based testing is essentially an intersection of two areas: software testing and cloud computing. Linthicum (2010) defines testing-as-a-service as the ability to test local or cloud-delivered systems using testing software and services that are remotely hosted. He emphasizes that while a cloud service requires testing unto itself, testing- as-a-service systems have the ability to test other cloud applications, web sites, and internal enterprise systems, and they do not require a hardware or software footprint within the enterprise.

In an industrial white paper, van der Aalst defines software testing as a service (STaaS) as “a model of software testing used to test an application as a service provided to customers across the Internet” (van der Aalst, 2009, p.1). STaaS enables daily

(31)

operation, maintenance and testing support through web-based browsers, testing frameworks and servers.

A typical STaaS process involves interaction between a customer and STaaS provider in a demand and supply fashion (Figure 2). The customer sends a test demand through the Internet to the STaaS provider who handles the testing task and sends the customer the test results (van der Aalst, 2009). The provider needs to manage the test infrastructure and availability of the test service, as well as take into account the overall communication with the customer.

In conceptualizing testing in the cloud, we defined facets of testing in the cloud in Publication I. These are (1) the system or application under test is accessible online. The system under test (SUT) might be SaaS software or non-SaaS software. In addition, this includes testing at different test levels e.g. performance testing; (2) testing infrastructure and platforms are hosted across different deployment models of the cloud i.e. public, community, private or hybrid clouds; (3) testing of the cloud itself (Figure 3). To add on the human as a service (HaaS) layer onto testing in the cloud, crowdsourced testing is also a form of testing in the cloud whereby crowdsourced pools of testers from around the globe provide testing services to customers. A practical example of crowdsourced testing is uTest (uTest, 2013).

uTest (2013) uses a community of professional, crowdsourced testers to test mobile, web and desktop applications for multiple testing types (e.g. functional, security, load, localization, and usability). A project manager is assigned to work closely with the customer to understand the customer’s testing needs, select and manage testers, assess the results and minimize overheads to the customer’s in-house team.

Cloud-based testing introduces new requirements and features for the testing process (Gao et al., 2011). These include: (a) Cloud-based testing environments containing different computing resources, system infrastructures and tools that are scalable and can be provisioned for use as needed. (b) Service level agreements (SLAs): Just like in other cloud services, service level agreements are used to describe the testing and quality assurance requirements e.g. reliability, availability, security and performance

Figure 2: STaaS process Customer

STaaS provider

Test infrastructure

24/7 availability Test demand

Test supply

Web interface

(32)

agreements. (c) Price models and service billing: The computing resources, infrastructures, and tools are charged along with the testing services based on pre- defined cost models. (d) Large-scale cloud-based data and traffic simulation:

Applying and simulating online user accesses and traffic data is necessary in cloud- based testing, for example, during performance testing.

A cloud-based testing framework should be based on a reliable and scalable infrastructure that can handle changing resource demands with service transparency and data confidentiality (Wu et al., 2011). The capabilities of a cloud-based testing service can be supported by an architecture containing several layers. For example, Yu et al. (2010) developed a testing as a service (TaaS) architecture with five layers.

(1) Service and contributor layer. This layer allows interaction between the users and the TaaS platform, in order for the user to access the testing services.

(2) The task management layer acts as a testing service bus that handles all the activities associated with a particular test task e.g. checking the testing capabilities, scheduling and dispatching the tasks and publishing the services. This layer is also responsible for dispatching the scheduled tasks to the VMs and collecting the results.

(3) The resource management layer monitors the resources such as the physical and virtual machines, and assigns them to perform testing as required.

(4) The test layer generates the test cases, handles test execution and gets the results.

(5) The database layer stores the test tasks, service images, and bug tracking results.

Figure 3: Facets of testing in the cloud

(33)

According to Robinson & Ragusa (2011), there are various scenarios for utilizing cloud-based testing infrastructure. These are:

i. Testing in the Cloud: A simple model of using a single cloud infrastructure provider for hosting the software under test and test suite.

ii. System under test in the Cloud: The tester places the system under test on a cloud provider’s infrastructure but hosts the test suite locally. The remote system under test receives the test payloads over the Internet. The software under test could be owned either by the tester or by a software- as-a-service (SaaS) solution under test.

iii. Test suite in the Cloud: The testing service is provided such that the cloud testing service provider hosts and controls the test suite outside the domain of the system under test.

iv. Multi-site testing in the Cloud: More than one provider hosts both the system under test and the test suite in different domains.

v. Brokered multi-site testing in the Cloud: The same as scenario IV but includes an intermediary for brokering test and infrastructure management requests and operations. The broker also provides support for resources selection, reservation and deployment.

Cloud-based testing supports testing across various testing levels including unit, integration, system and acceptance testing (Incki et al., 2012). It is also possible to perform different types of testing in the cloud e.g. functional, performance, security, compatibility and interoperability testing (Incki et al., 2012; Wu et al., 2011), model- based testing, symbolic testing, fault injection testing, random testing and privacy aware testing (Priyanka et al., 2012).

There has been an increase in the literature about testing in the cloud over the last few years (Incki et al., 2012; Priyanka et al., 2012). This demonstrates the importance of the topic and need for further investigation. The success of testing in the cloud will rely on how researchers and industry practitioners address all aspects related to its adoption, implementation, delivery and management.

2.4 Software quality and cloud application development

The key objectives of software engineering are reducing costs and improving the quality of the products (Osterweil, 1997). The quality of a software product is achieved as a result of actions performed throughout the software development life cycle (SDLC), including planning, design, implementation and testing (Seth et al., 2012). In particular, testing has a vital influence on the quality of a software product (Kasurinen et al., 2011; Tassey, 2002) and it needs to be taken into account to produce quality

(34)

software. Other factors that affect quality are for example, outsourcing and communication (Kasurinen et al., 2011) as well as the work culture and individual attitudes (Wilson & Hall, 1998).

2.4.1 What is software quality?

Software quality is a complex concept that is difficult to define (Blaine & Cleland- Huang, 2008; Kitchenham & Pfleeger, 1996; Sommerville, 2001). When dealing with software systems, it is especially difficult to find a common and agreeable definition due to problems such as (Sommerville, 2001, p. 536):

i. The requirement specification should be oriented towards the characteristics of the product that customers wants. However, the development organization may also have requirements e.g. maintainability requirements which are not included in the specification.

ii. We do not know how to specify certain quality characteristics in an unambiguous manner, e.g., maintainability.

iii. It is very difficult to write complete software specifications. Therefore, even though a software product may conform to its specifications, users may not consider it to be a high-quality product.

The third problem mentioned above indeed demonstrates that software quality is a multi-dimensional concept that means different things for different people (Kusters et al., 1997). The ISO 9000 (2000) quality standard defines quality as the degree to which a set of inherent characteristics fulfil the requirements. In the ISO/IEC 25010 standard (ISO/IEC, 2010), software product quality is defined as the degree to which the software product satisfies stated and implied needs when used under specified conditions.

The ISO/IEC 25010 quality standard contains two parts for software systems - product quality model and quality in use model. The product quality model applies to computer systems and software products. It is made up of eight characteristics;

functional suitability, reliability, performance efficiency, usability, security, compatibility, maintainability, and portability (ISO/IEC, 2010). Each of these characteristics contains related subcharacterictis e.g. availability as one of the subcharacteristics for reliability and interoperability as a subcharacteristic for compatibility.

The system quality in use model relates to the impact that a computer system or software product has on the users after it is used in a given context. It contains five main characteristics - effectiveness, efficiency, satisfaction, freedom from risk and context coverage together with their associated eleven subcharacteristics, such as, trust and flexibility (ISO/IEC, 2010). Factors that influence the quality in use of a system or

(35)

product are: the quality of the software, hardware and operating environment as well as the users, tasks and social environment. This dissertation uses the ISO 25010 definition of software product quality and quality in use to describe the quality concept.

Garvin (1984) deduced different perspectives on product quality, and even though his work did not focus on software products, the different perspectives also apply in software engineering. He identified five different perspectives that define quality (Garvin, 1984):

 Transcendent perspective: This is a product’s ‘innate excellence’ comprising of high standards that cannot be defined precisely but can be recognized through experience.

 The product-based perspective says that the product contains attributes that can be quantified and measured. Therefore, the quality is assessed objectively based on measurable attributes, instead of using preferences.

 The user-based perspective focuses on satisfying the individual preferences.

However, users have different preferences, and it is not possible to accommodate all of them in one product. Therefore, a high-quality product is assumed to be one that meets the needs of a majority of the users.

 The manufacturing-based perspective aims at fulfilling the product requirements and specifications, and any deviation leads to a compromise in the quality.

 The value-based perspective evaluates the quality of a product in terms of costs and prices such that a quality product is one that delivers customer satisfaction at an acceptable price.

2.4.2 Cloud application development

Cloud computing is increasingly gaining appreciation for enabling easier, cost- effective and faster development of many applications, for example, e-business, collaborative research and development, enterprise computing infrastructures, healthcare, and military applications (Yau & An, 2011). Cloud computing strengthens the web as an application development environment leading to the delivery of dynamic web applications (Mikkonen & Taivalsaari, 2013). Cloud computing does not only enable application development and testing, but it also supports delivery of the developed and tested applications through (Yau & An, 2011):

 Service virtualization, hence providing easy application deployment and maintenance for service providers.

 Interfaces that provide users with access to the applications.

(36)

 Dynamic resource virtualization and allocation, enabling service providers to conduct Quality of Service (QoS) management.

The cloud provides a “complete technology stack covering the ground from database and security to workflow and user interface—so you can focus on assembling, building, and instantly deploying solutions” (Patidar et al., 2011, p.1010). A team from IBM observed that utilizing the cloud for application development can significantly reduce the time required to set up the development environment (Zhou et al., 2011).

They developed a cloud platform that organizations can use for developing applications. The development environment incorporates standardized methods and tools that can help to improve the efficiency of application development processes and applications. According to Lu et al. (2010), Microsoft’s Windows Azure platform can support the development of computation and data intensive scientific applications.

In order to succeed in utilizing the cloud for application development, there needs to be appropriate development frameworks, environments and tools. For example, Hosono et al. (2011) suggested a framework that streamlines and shortens the flow of development and production (or deployment) activities. Their framework is based on the concept of lifecycle management with supporting tools and roles for each phase of the development lifecycle. The service designer processes the requirements to lay out the design; the resource planner handles the virtual machine (VM) structure; the application implementer develops the application; the service administrator manages the running applications and the cloud administrator manages the virtual machines (VMs) on the physical servers.

When adopting cloud computing for application development, Sodhi & Prabhakar (2011) advocate that the organizations should first evaluate the various platforms and select the one that supports an application’s most important quality characteristics.

They name three types of development platforms.

i. Traditional non-cloud platforms: These are essentially the development platforms where data is stored in regular structured query language (SQL) relational databases, and the designer does not have to take into account any virtualization, elastic and distributed properties.

ii. Virtualized platforms: These are are similar to traditional non-cloud platforms, but use virtual machines (VMs) for developing and deploying the applications. The designer does not take into account properties such as elasticity of the platform.

iii. Cloud-aware platforms: These platforms hold virtual, elastic and distributed properties supported by robust data centres. Subsequently, the applications developed and deployed on these platforms can also be virtualized, elastic and distributed in nature.

(37)

Cloud-based application development requires that various aspects are considered in a different manner from developing applications in conventional software development. When developing applications in the cloud, the developers need to interact with the cloud providers, which is not the case with conventional software development. The developers need to follow formal and secure guidelines in order to align their code with the cloud provider’s platform. Different service and deployment models have different technical, legal, security and trust boundaries that determine the kind of interaction that will exist between the developers and the cloud providers (Patidar et al., 2011). For example, using a private cloud provides more control and security to the developer as opposed to when using a public cloud.

2.4.3 Quality in the context of cloud application development

Developing and deploying applications in cloud computing environments introduces features that have significant impact on the quality of the applications. Some of these features are (Hobfeld et al., 2012):

 Increased network distance between the user and the service, e.g., if a user’s geographical location is far from the data centre where the service is running from, he/she might experience unacceptable latency which will limit the level of interaction with the service.

 Service delivery implementation, e.g., how to deal with latency and bandwidth bottlenecks.

 Real-time resource management as users access and use the cloud services.

 Having multiple parties involved in providing the service where before there were one or two.

 Geographical distribution of the user base.

 Multi-party sharing and communication, which necessitates real-time requirements and measures for evaluating the quality of service.

 SLAs and pricing.

Lee et al. (2009) came up with a model for evaluating the quality of SaaS applications.

They first looked at SaaS features (reusability, customizability, pay-per-use, data managed by providers, scalability and availability) and then derived a set of SaaS- specific quality characteristics (reusability, efficiency, reliability, scalability and availability). Then they defined metrics that could be used to measure each quality characteristic using specific formulae, value ranges, and relevant interpretations. For example, ‘reliability’ could be measured using the metrics ‘service stability’ and

‘service accuracy’. Their quality model is relevant for both providers and customers

(38)

because it focuses on SaaS features and therefore, provides guidelines for evaluating and assessing SaaS quality.

The quality of cloud applications can also be assessed by evaluating the quality of the components that make up an application (Zheng et al., 2010). The components are ranked based on their past performance while in use by other users. The results enable the user to select the components that would generate optimal performance for the relevant cloud application. In addition, the application designers can use the quality ranking to assess the performance of the components (Zheng et al., 2010).

Chauhan & Babar (2012) evaluated the quality characteristics of a cloud infrastructure for providing Global Software Development (GSD) teams with a service to support the tools in use. They identified the quality characteristics from studying existing GSD and cloud computing literature. For a cloud infrastructure to efficiently support the delivery of tools as services, it should contain these quality characteristics: multi- tenancy, versioning capabilities, compatibility with commercially available tools, ability to work on decentralized artifacts, support multiple databases, support multiple devices and comply with the SLAs. Chauhan and Babar (2012) also designed architectural solutions that could help to achieve the identified quality characteristics.

For example, the architectural solution for handling ‘compatibility with commercially available tools’ is to ensure that the hosting infrastructure uses platform neutral Application Programming Interfaces (APIs).

Cloud computing services are considered to be relatively cheap, such that providing services at low prices ceases to be a competitive advantage (Durkee, 2010; Hobfeld et al., 2012). This means that cloud computing providers have to find other ways to differentiate themselves from their competitors. It is for this reason that many cloud service providers are looking to enhance the quality of service (QoS), and the quality of experience (QoE).

Quality of service is the ability of a service or an application to “satisfy the stated or implied needs of the user” (ITU-T, 2008a, p.3). QoE refers to “the overall acceptability of an application or service, as perceived subjectively by the end-user” (ITU-T, 2008b, p.2). QoS and QoE concepts complement each other, in that QoS deals on the technological aspects of an application or service, and QoE concentrates on the user satisfaction with the technological aspects of the particular application or service (Hobfeld et al., 2012).

Hobfeld et al. (2012) claim that quality of experience (QoE) has the potential to become the guiding paradigm for managing quality in the cloud. True to this claim, research on QoE and QoS in the cloud is growing. The following are some studies that have been conducted.

(39)

Hobfeld et al. (2012) discuss the challenges of QoE management for cloud applications. They evaluate the challenges based on the features of the cloud applications which are determined by the degree of multimedia intensity, interactivity, primary usage domain and service complexity. Cloud gaming, high-definition (HD) teleconferencing, and customer relationship management (CRM) services (e.g.

Salesforce.com) require varying degrees of interactivity, which demand QoE requirements, for example, high degree of responsiveness.

Kafetzakis et al. (2012) developed a QoE-driven framework for effectively managing the quality of cloud computing environments. Their framework considers four aspects of quality. (1) System/hardware-related QoS aspects (IaaS management); (2) Network QoS aspects (PaaS QoS optimization); (3) QoE aspects (SaaS QoE optimization); and (4) Business-related aspects (Cloud provider operational excellence). The framework emphasizes on the QoE aspects in order to ensure that the cloud provider complies with the end user QoS expectations. By doing so, the cloud service provider guarantees that the end user’s QoE is optimal to keep the end user happy with the service.

The Cloud2Bubble framework takes into account the content generated by users within a specific environment and then uses this information to evaluate the user’s QoE (Costa et al., 2012). The aggregated information is used to improve the performance of the application and to improve the overall QoE when delivering personalized services to the users.

Other studies include, for example, a framework for measuring the quality of cloud services aimed at encouraging cloud services providers to better fulfill users’ QoS requirements (Garg et al., 2013); a model that enables users to evaluate cloud service providers based on the QoE that service providers are able to provide (Qian et al., 2011); and an adaptive QoS framework that allows video-on-demand (VoD) service providers to target user-specific QoS requirements under different computing capacities and resources (Wang et al., 2010).

2.5 Cloud gaming

In support of the idea of ‘adopting cloud computing for various needs within an organization’, the research also includes a study on cloud gaming. The cloud gaming study serves to demonstrate the applicability of cloud computing in different organizational contexts to improve efficiency.

In recent years, the video game industry has grown into a large industry that is almost thrice the size of the music industry (PricewaterhouseCoopers, 2010). It was predicted

Empirical study on the adoption, use and effects of cloud-based testing