The main objective of this thesis was to research and evaluate options on how to perform automated testing for microservice applications in a cost-effective and time-effective manner. Theoretical background about software architectures and technologies used in the Internet and automated testing in general were discussed on the way. All the knowledge gathered was in the end combined to create a test automation framework for Intel Insight, a microservice system.
The design process of the automation framework was discussed from the perspective of high-level requirements. These, in turn, were used to try out and evaluate a number of different tools that provided useful features for test automation. The tools that were se-lected for the framework were examined in-depth, and the rest were briefly discussed from the functionality point of view and the reason they were not selected.
The use of the framework in action was examined by looking at how a core use case was automated to facilitate testing the end-user workflow at multiple levels of the appli-cation. Test results can be determined from what state the system is left in after the automation has finished its execution for the end-to-end use cases and from server re-sponses on the API level.
The resulting framework was evaluated from the perspective of developing automated tests with the tools. The framework was regarded as suitable for its purpose, but with some evident downsides to it. All the tools in the framework do their core jobs well, but as use cases get more complicated, some functionalities have plenty of room for im-provement.
Future work on improving the framework is possible, as all tools have their source code publicly available. The most pressing improvements would be improving SikuliX by im-plementing better error reporting and some kind of tool for static code analysis to catch faulty syntax and poor logic in the code before executing scripts. Improving the SikuliX IDE would also be high on the list of improvements. Postman improvements include making the collection execution more functional, with more versatile ways than environ-ment variables of injecting parameters for test runs. Selenium works well on the things it is capable of and as such, no immediate improvement needs came up.
This thesis focused on using free test automation tools, and commercially distributed automation tools were intentionally left out of scope. The free tier of Postman offers a wide set of features for API testing, similar tools are available on the commercial side
with more functionality and license prices starting from approximately $500 per year.
Using them would make sense for an organization working primarily in the SaaS busi-ness domain where they could be utilized to their full potential. In the busibusi-ness environ-ment the work of this thesis was done in, it did not make sense to explore further those options.
REFERENCES
[1] Gartner, Inc, Gartner Forecasts Worldwide Public Cloud Revenue to Grow 17.3 Percent in 2019. [Online]. Available (accessed 9.5.2019): https://www.gart- ner.com/en/newsroom/press-releases/2018-09-12-gartner-forecasts-worldwide-public-cloud-revenue-to-grow-17-percent-in-2019
[2] Mozilla, Evolution of HTTP, MDN web docs. [Online]. Available (accessed 9.5.2019): https://developer.mozilla.org/en-US/docs/Web/HTTP/Ba-sics_of_HTTP/Evolution_of_HTTP
[3] Tim Berners-Lee, The Original HTTP as defined in 1991. [Online]. Available (ac-cessed 9.5.2019): https://www.w3.org/Protocols/HTTP/AsImplemented.html [4] T. Berners-Lee, R. Fielding, H. Frystyk, RFC 1945: Hypertext Transfer Protocol
-- HTTP/1.0, Internet Engineering Task Force, 1996. [Online]. Available (ac-cessed 9.5.2019): https://tools.ietf.org/html/rfc1945
[5] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, T. Berners-Lee, RFC 2068: Hyper-text Transfer Protocol -- HTTP/1.1, Internet Engineering Task Force, 1997.
[Online]. Available (accessed 9.5.2019): https://tools.ietf.org/html/rfc2068 [6] R. Fielding, J. Reschke, RFC 7230: Hypertext Transfer Protocol (HTTP/1.1):
Message Syntax and Routing, Internet Engineering Task Force, 2014. [Online].
Available (accessed 9.5.2019): https://tools.ietf.org/html/rfc7230
[7] M. Belshe, R. Peon, M. Thomson, RFC 7540: Hypertext Transfer Protocol Ver-sion 2 (HTTP/2), Internet Engineering Task Force, 2015. [Online]. Available (ac-cessed 9.5.2019): https://tools.ietf.org/html/rfc7540
[8] International Organization for Standardization, ISO-7498: Basic Reference Model: The Basic Model. Available (accessed 9.5.2019): https://stand- ards.iso.org/ittf/PubliclyAvailableStandards/s020269_ISO_IEC_7498-1_1994(E).zip
[9] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, T. Berners-Lee, RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1, Internet Engineering Task Force, 1999. [Online]. Available (accessed 9.5.2019):
https://tools.ietf.org/html/rfc2616
[10] T. Berners-Lee, R. Fielding, L. Masinter, RFC 3986: Uniform Resource Identifier (URI): Generic Syntax, Internet Engineering Task Force, 2005. [Online]. Availa-ble (accessed 9.5.2019): https://tools.ietf.org/html/rfc3986
[11] Mozilla, HTTP messages, MDN web docs. [Online]. Available (accessed 9.5.2019): https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages [12] R. Fielding, J. Reschke, RFC 7231: Hypertext Transfer Protocol (HTTP/1.1):
Se-mantics and Content, Internet Engineering Task Force, 2014. [Online]. Available (accessed 9.5.2019): https://tools.ietf.org/html/rfc7231
[13] Roy Thomas Fielding, Architectural Styles and the Design of Network-based Software Architectures. Doctoral dissertation, University of California, Irvine,
2000. Available (accessed 9.5.2019): https://www.ics.uci.edu/~fielding/pubs/dis-sertation/top.htm
[14] Roy Thomas Fielding, REST APIs must be Hypertext-driven, 2008. [Online].
Available (accessed 9.5.2019): http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
[15] Open API Iniative, A Short History of the Open API Initiative and the OpenAPI Specification. [Online]. Available (accessed 9.5.2019): https://www.opena-pis.org/faq#OAIFAQ-History
[16] ReDoc documentation. [Online]. Available (accessed 9.5.2019):
https://github.com/Rebilly/ReDoc
[17] Docker Engine API documentation. [Online]. Available (accessed 9.5.2019):
https://docs.docker.com/engine/api/v1.39/#operation/ContainerTop
[18] Zalando, Zalando RESTful API and Event Scheme Guidelines. [Online]. Availa-ble (accessed 9.5.2019): https://opensource.zalando.com/restful-api-guide-lines/#api-first
[19] Nadareishvili I, Mitra R, McLarty M, Amundsen M. Microservice Architecture. 1st ed.: O'Reilly Media, Inc; 2016. p 4.
[20] Nadareishvili I, Mitra R, McLarty M, Amundsen M. Microservice Architecture. 1st ed.: O'Reilly Media, Inc; 2016. p 6.
[21] Martin Fowler, James Lewis, Microservices. [Online]. Available (accessed 9.5.2019): https://martinfowler.com/articles/microservices.html
[22] M. D. McIlroy, E. N. Pinson, B. A. Tague, Foreword, The Bell System Technical Journal, Volume 57, Number 6, Part 2, 1978. [Online]. Available (accessed 9.5.2019): http://emulator.pdp-11.org.ru/misc/1978.07_-_Bell_System_Tech-nical_Journal.pdf
[23] M. J. Scheepers, Virtualization and Containerization of Application Infrastruc-ture: A Comparison, Proceedings of the 21st Twente Student Conference on IT June 23rd 2014, 2014. Available (accessed 9.5.2019):
https://thijs.ai/pa-pers/scheepers-virtualization-containerization.pdf
[24] Docker documentation, Dockerfile reference. [Online]. Available (accessed 9.5.2019): https://docs.docker.com/engine/reference/builder/
[25] Amazon, What is DevOps?. [Online]. Available (accessed 9.5.2019):
https://aws.amazon.com/devops/what-is-devops/
[26] Martin Fowler, Continuous Integration. [Online]. Available (accessed 9.5.2019):
https://martinfowler.com/bliki/ContinuousDelivery.html
[27] Peter Murray, Traditional Development/Integration/Staging/Production Practice for Software Development. [Online]. Available (accessed 9.5.2019):
https://dltj.org/article/software-development-practice/
[28] Tuukka Virtanen, Literature Review of Test Automation Models in Agile Testing, Master of Science Thesis Information and Knowledge Management, Tampere University of Technology, May 2018, p. 14. Available (accessed 9.5.2019):
https://dspace.cc.tut.fi/dpub/bitstream/handle/123456789/25869/Vir-tanen.pdf?sequence=3&isAllowed=y
[29] Mike Cohn, The Forgotten Layer of the Test Automation Pyramid, Mountain Goat Software. [Online]. Available (accessed 9.5.2019): https://www.moun-taingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid [30] Crispin, L. & Gregory, J. 2009, Agile testing: a practical guide for testers and
ag-ile teams, Addison-Wesley, Upper Saddle River, NJ.
[31] Myers, G.J., Sandler, C., Badgett, T. & ebrary, I. 2012, The art of software test-ing, 3rd edn, John Wiley & Sons, Hoboken, N.J.
[32] Wikipedia, List of unit testing frameworks. [Online]. Available (accessed 9.5.2019): https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks [33] International Software Testing Qualifications Board, Standard Glossary of
Terms used in Software Testing. [Online]. Available (accessed 9.5.2019):
https://www.istqb.org/downloads/send/20-istqb-glossary/186-glossary-all-terms.html
[34] Atif M. Memon, Martha E. Pollack, Mary Lou Soffa, Using a goal-driven ap-proach to generate test cases for GUIs, Proceedings of the 21st international conference on Software engineering, Los Angeles, California, USA, May 16-22, 1999. pp 257-266. Available (accessed 9.5.2019): https://dl.acm.org/cita-tion.cfm?id=302632
[35] Intel Corporation, Intel Insight platform – Drone Data Management. [Online].
Available (accessed 9.5.2019): https://www.intel.com/con-tent/www/us/en/drones/solutions/intel-insight-platform.html
[36] Alan Walford, What is photogrammetry?. [Online]. Available (accessed 9.5.2019): http://www.photogrammetry.com/
[37] Connexion documentation. [Online]. Available (accessed 9.5.2019):
https://github.com/zalando/connexion
[38] Meqa documentation. [Online]. Available (accessed 9.5.2019):
https://github.com/meqaio/swagger_meqa
[39] SoapUI main page, SmartBear. [Online]. Available (accessed 9.5.2019):
https://www.soapui.org/
[40] Katalon studio main page, Katalon. [Online]. Available (accessed 9.5.2019):
https://www.katalon.com/
[41] Oatts documentation. [Online]. Available (accessed 9.5.2019):
https://github.com/google/oatts
[42] Floating Keyboard Software Inc, Insomnia main page. [Online]. Available (ac-cessed 9.5.2019): https://insomnia.rest/
[43] Postman, Postman main page. [Online]. Available (accessed 9.5.2019):
https://www.getpostman.com/
[44] Postman Learning Center, Environments and globals. [Online]. Available (ac-cessed 9.5.2019): https://learning.getpostman.com/docs/postman/environ-ments_and_globals/variables
[45] Postman Learning Center, Intro to scripts. [Online]. Available (accessed 9.5.2019): https://learning.getpostman.com/docs/postman/scripts/in-tro_to_scripts
[46] Postman Learning Center, Command line integration with Newman. [Online].
Available (accessed 9.5.2019): https://learning.getpostman.com/docs/post-man/collection_runs/command_line_integration_with_newman
[47] Postman Learning Center, Viewing monitor results. [Online]. Available (ac-cessed 9.5.2019): https://learning.getpostman.com/docs/postman/moni-tors/viewing_monitor_results
[48] Selenium HQ, Selenium history. [Online]. Available (accessed 9.5.2019):
https://www.seleniumhq.org/about/history.jsp
[49] Selenium HQ, Selenium IDE. [Online]. Available (accessed 12.5.2019):
https://docs.seleniumhq.org/selenium-ide/
[50] Tom Yeh, Tsung-Hsiang Chang, Robert C. Miller, Sikuli: Using GUI Screenshots for Search and Automation, Proceedings of the 22nd annual ACM symposium on User interface software and technology, Victoria, British Colombia, Canada October 4-7, 2009. pp 183-192. Available (accessed 9.5.2019):
http://up.csail.mit.edu/projects/sikuli/sikuli-uist2009.pdf
[51] Raimond Hocke, SikuliX main page. [Online]. Available (accessed 9.5.2019):
http://sikulix.com/
[52] SW Test Academy, Quick Start to SikuliX (Sikuli Script). [Online]. Available (ac-cessed 9.5.2019): https://www.swtestacademy.com/quick-start-to-sikulix/
[53] PyAutoGUI documentation. [Online]. Available (accessed 9.5.2019):
https://pyautogui.readthedocs.io/en/latest/
A. FILE UPLOAD BY USING SELENIUM
# Start the browser and go to Insight staging server DRIVER = webdriver.Chrome("../chromedriver.exe", options=START_OPTIONS) DRIVER.get("https://stagingserver.net")
# Locate textboxes for login credentials and type to them username = DRIVER.find_element_by_id("username")
password = DRIVER.find_element_by_id("password") username.send_keys("test_username")
password.send_keys("test_password") password.send_keys(Keys.ENTER)
# Wait for the main view to appear, then find and click the upload button
time.sleep(2)
DRIVER.find_element_by_xpath("/html/body/app-root/ng-compo- nent/div/ng-component/div[1]/app-projects-panel/app-projects-panel-header/div/div[2]").click()
# Wait for the JavaScript element for uploading appear
# Launch OS file dialog by clicking the 'Browse' button on screen time.sleep(3)
pyautogui.click(x=966, y=575)
# Once file dialog is visible, use keyboard to navigate to test data location
# Wait for the frontend code to read image files
# Then press 'Next' and 'Upload' buttons to finish the upload