Performance

Figure 5.2Availability before the launch of the renewed www.talouselama.fi website. Total of 2 outages that in total resulted in outage of about 2 minutes. Data was gathered using pingdom service and data points are from 18.7.201 6 to 2.10.2017.

code that might account for these outages. Mainly these are problems considering fetching data from different APIs.

Figure 5.3 Availability after the launch of the renewed www.talouselama.fi website. Total of 42 outages that in total resulted in outage of about an hour. Data was gathered using pingdom service and data points are from 3.10.2017 to 21.1.2018.

5.2 Performance

Performance measurements were done using Apache JMeter tool. Apache JMeter is a tool which can be used to measure performance of a webserver by sending multiple sequential and parallel requests to the server. After Apache Jmeter has sent the requests it measures the time it took for the server to respond to a particular request and draws a graph from the results. It has to be noted that Apache JMeter does not measure websites performance as a whole as it focues only on the time it took for the server to respond with a payload.

5.2. Performance 33 Subsequent requests were sent with Apache JMeter to new www.talouselama.fi ap-plication running in AWS Lambda. To test latency under load two load tests were created: medium load test and heavy load test.

Response time under medium load was tested by creating a test in which 16 threads ran in parallel each sending 1000 sequential requests to the server. All of the threads were started at the same time.

Response time under heavy load was tested by creating a test in which 32 threads ran in parallel each sending 500 sequential requests to the server. All of the threads were started at the same time. Total amount of requests 16000 stayed the same in both test cases.

To compare results gathered from www.talouselama.fi running on AWS Lambda to a site with similar functionality but with different technology stack, m.kl.fi running on AWS Elastic Beanstalk was chosen. Comparison to Elastic Beanstalk solution gives some rough estimates of Lambdas performance even thought the application tested was not the same.

While testing latency it became evident that latency almost doubled when load increased to double. Hypothesis was formed that increase in latency might be due to underlying API not scaling fast enough. To make sure hypothesis of frontends scalability not being a problem, it was necessary to test www.talouselama.fi with it’s APIs mocked.

Underlying API was replaced by placing API responses to S3 bucked and then queried from there instead of the normal API. This way problems relating to scaling of the underlying API were removed. This was done only to the www.talouselama.fi application.

With correct test setup all request were able to complete with an HTTP status of 200 meaning successful computation of the request even without mocking of the underlying API.

In medium load test the AWS Lambda implementation of www.talouselama.fi with 3 GB of memory had an average latency of 749 ms and median latency of 681 ms. 3 GB is the maximum amount of memory AWS Lambda function can have at the time of writing. This also meant that the function was having as much CPU performance as possible from the AWS Lambda system. Charts of medium and heavy tests can

5.2. Performance 34

be seen in figures 5.4 and 5.5.

In heavy load test without API mocking the numbers rose to almost double of the medium load test. Avarage latency was 1354 ms and median latency was 1397 ms.

As Lambda should be able to scale quickly increase in latency hinted that some parts of the program were not able to scale fast enough.

As m.kl.fi had completely different code base but similar functionality it was a meaningful target to compare www.talouselama.fi’s performance. In a medium load test m.kl.fi had an average latency of 825 ms and median latency of 773 ms. As with www.talouselama.fi as the amount of clients doubled so did the latency: average latency was 1621 ms and median was 1437 ms.

To make sure the problem was not in scalability of AWS Lambda the code of www.talouselam.fi was changed to mock used APIs by querying them from S3 bucket. Same doubling in latency did not happen with mocked API. Avarage latency stayed almost the same in medium and heavy load tests used. Median in medium test was 474 ms and 517 ms in heavy test.

Figure 5.4 Chart showing latency of medium load test on AWS Lambda with 3 GB of memory.

In medium load test the AWS Lambda implementation of www.talouselama.fi with 3GB of memory had an average latency of 864 ms and median latency of 667 ms.

In medium load test the AWS Lambda implementation of www.talouselama.fi with

5.2. Performance 35 3GB of memory had an average latency of 944 ms and median latency of 781 ms.

Figure 5.5 Chart showing heavy load test results on AWS Lambda with 3 GB of memory.

As seen in figures 5.4 and 5.5 and in the rest of the test result set, few long responses causes the average latency to not be the best metric. Instead median was used as the main metric for comparison. Test results are summarized in table 5.1

Load test www.talouselama.fi m.kl.fi www.talouselama.fi with mocking

medium 681 773 474

heavy 1397 1437 517

Table 5.1 Results of medium and heavy load tests. Rusults are medians of latency in milliseconds

5.2.1 Difficulties in producing reliable test results in load tests

There were problems getting consistent results from the latency tests when working with AWS Lambda setup. At times over half of the sent request came back with other status codes than 200, meaning something went wrong. Main ones were 502 Gateway unavailable and 404 Not Found.

Small percentage of responses were 502 Gateway unavailable errors. AWS Lambda application has a timeout time of 10 seconds after which the execution is terminated.

5.3. Developer experience 36