Forecasting Emergency Department arrivals with Facebook Prophet library

(1)

FORECASTING EMERGENCY DEPARTMENT ARRIVALS WITH FACEBOOK PROPHET LIBRARY

Bachelor of Science Thesis Faculty of Information Technology and Communication Sciences Examiners: MSc Francesco Lomio Prof. Joni Kämäräinen May 2021

(2)

ABSTRACT

Antti-Jussi Mäkipää: Forecasting emergency department arrivals with Facebook Prophet library Bachelor of Science Thesis

Tampere University Electrical Engineering May 2021

Emergency departments are prone to overcrowding due to mismatch between service demand and available resources. Forecasting the number of future visitors would enable more intelligent resource allocation and ensure timely care for each individual patient. This thesis aims to predict the arrivals for the next day in the Tampere University Emergency Department Acuta using a machine learning library called Facebook Prophet. The dataset used to train and test the model contains hourly Emergency Department data over a three year period between year 2015 and 2019.

Time series forecasting is a subfield machine learning where the predictive model is trained with past data in order to predict the future. Time series model can be composed of different components and for instance, Facebook Prophet is formed with the following components: trend, seasonality, holidays and the error term. There are three different metrics used in this thesis for evaluating the model, one of them being mean absolute percentage error (MAPE).

70 % of the dataset is used to train the model and optimize the hyperparameters, while 30 % is retained as a test set. Out of sample validation is performed on the test set using rolling origin cross validation. Alongside to fitting the model, hyperparameter tuning and variable selection are implemented. Hyperparameters are built-in parameters that define the model structure and in this thesis, the variables are exogenous variables, such as weather or calendar variables. Both hyperparameter tuning and variable selection are part of the optimization of the model.

The results show that Prophet managed to forecast the future quite accurately. Even without hyperparameter tuning and variable selection the model yielded 7.24 % MAPE. Hyperparameter tuning and variable selection managed to decrease the error rate to 6.57 %.

Keywords: machine learning, time series forecasting, emergency department arrivals, Facebook Prophet

The originality of this thesis has been checked using the Turnitin OriginalityCheck service.

(3)

TIIVISTELMÄ

Antti-Jussi Mäkipää: Sairaalan yhteispäivystyksen kävijämäärien ennustaminen Facebook Prop- het -kirjastolla

Kandidaatintyö Tampereen yliopisto Sähkötekniikka Toukokuu 2021

Ensiapuklinikat ovat alttiita ruuhkautumiselle johtuen ajallisesta epäsuhdasta päivystyspalve- luiden kysynnän ja tarjonnan välillä. Kävijämäärän ennustaminen riittävällä tarkkuudella voisi aut- taa resurssien ajallisessa kohdentamisessa, jonka avulla voidaan turvata oikea-aikainen ja laadu- kas hoito. Tämän työn tarkoituksena on ennustaa Tampereen yliopistollisen keskussairaalan yh- teispäivystyksen Acutan kävijämäärää seuraavalle päivälle Facebook Prophet -koneoppimiskirjastolla.

Koneoppismallin kouluttamiseen käytetty tietoaineisto sisältää dataa ensiapuklinikasta yli kolmen vuoden ajalta.

Koneoppimisen yksi alueista on aikasarjaennustaminen, jossa koneoppimismalli koulutetaan menneen ajan datalla, jonka jälkeen malli pyrkii ennustamaan tulevaisuuteen. Aikasarjamalli voidaan jakaa erilaisiin komponentteihin, ja esimerkiksi Facebook Prophet muodostuu seuraavista komponenteista: trendikäyrä, kausiluonteisuus, juhlapyhät sekä virhetermi. Tässä työssä hyödyn- netään kolmea erilaista metriikkaa mallin evaluointiin, yhtenä niistä toimii prosentuaalinen keski- poikkeama (mean absolute percentage error).

Tietoaineisto on jaettu koulutusaineistoksi ja testiaineistoksi niin, että noin 70 % on koulutusai- nestoa, ja jäljelle jäävät 30 % testiaineistoa. Mallin validointi on toteutettu tässä työssä vieriväl- lä origolla (rolling origin). Pelkän mallin sovittamisen lisäksi työssä toteutetaan hyperparametrien virittäminen sekä muuttujien valinta. Hyperparametrit ovat sisäänrakennettuja parametrejä, jot- ka määrittävät mallin rakenteen. Muuttujat ovat tässä työssä eksogeenista dataa tietoaineistosta, kuten esimerkiksi sää ja kalenterimuuttujat. Sekä hyperparametrien virittäminen että muuttujien valinta ovat osa mallin optimointia.

Tulokset osoittavat, että Prophet-malli kykenee ennustamaan tulevaisuuteen melko tarkasti.

Ilman hyperparametrien virittämistä sekä muuttujien valintaa malli onnistui saamaan 7.24 % pro- sentuaalisen keskipoikkeaman. Hyperparametrien virittäminen ja muuttujien valitseminen laskivat virhetason 6.57 %.

Avainsanat: koneoppiminen, aikasarjaennustaminen, sairaalan yhteispäivystyksen kävijämäärä, Facebook Prophet

Tämän julkaisun alkuperäisyys on tarkastettu Turnitin OriginalityCheck -ohjelmalla.

(4)

PREFACE

This thesis was written as part of an interdisciplinary research which included researchers from Tampere University and Tampere University Hospital. I would like to thank my su- pervisors Francesco Lomio and Joni Kämäräinen for their guidance. Special thanks to MD Jalmari Tuominen for his helpfulness and eagerness towards this project and PhD Antti Roine for coming up with the research topic. Additionally, I am grateful for Prof. Niku Oksala, PhD Satu-Liisa Pauniaho and Unitary Healthcare Oy for producing and acquiring the data for this thesis.

Tampereella, 23rd May 2021

Antti-Jussi Mäkipää

(5)

1. INTRODUCTION

Emergency Department (ED) is a key component in any healthcare service network pro- viding care for people arriving without scheduled appointments. The initial health status of these patients can vary significantly and some of them may be life-threatening. Due to the unpredictable nature of the arrivals, Emergency Departments worldwide suffer from sudden overcrowding and peaks which can be sporadic or recurring [1]. The goal of the thesis is to predict the number of arrivals using data which is provided from Tampere University Hospital ED Acuta.

Time series data can be predicted with a variety of different methods, one of them being Autoregressive Integrated Moving Average (ARIMA) which is suited for short-term forecasting [2]. In this thesis, the prediction is made with a machine learning library called Facebook Prophet which was published in 2017 [3].

The research questions can be separated into three parts:

1. How accurately Prophet is able to predict future arrivals?

2. Which hyperparameters and variables does the model consider important?

3. How well Prophet performs compared to other machine learning models for the same dataset?

The thesis is structured as follows. Chapter 2 covers the background of the study and reviews related work in the field. The Facebook Prophet model and evaluation methods for the model performance are wrapped up in Chapter 3. Chapter 4 contains detailed information about the dataset and the experiments conducted. The fifth chapter discusses results and compares them with previous studies from the same dataset. The final chapter concludes the previous chapters.

(7)

2. BACKGROUND

Machine learning is a subdomain of artificial intelligence, which lately has been one of the hottest trends in the field of technology. Every machine learning system can be split into three components:

• Data

• Model

• Task

The data is fed to the model as an input, the model can be e.g a function or a program that is selected specifically for the task. Machine learning is achieved by the process in which adjusting the model to the data, the model becomes better at the desired task. [4]

2.1 Time series forecasting

Time series data is one of the most common data types available nowadays. It can be anything that changes with time e.g how stock market values change over time or electricity consumption in certain periods. However, every data with a timestamp cannot be considered as time series data, rather the data should consist of observations over cyclic or continuous intervals. Time series can be either univariate or multivariate depending on how many variables are being observed, univariate having only a single variable and multivariate having multiple variables respectively. [5]

Certain types of patterns can be usually found from time series data. These patterns can be for instance:

• Trend: "Trend can be a linear or nonlinear component and its value may either decrease or increase with respect to time by changing its directions.".

• Seasonal: "A linear or nonlinear pattern that repeats at particular intervals. Sea- sonality is a very common feature or characteristics of Sales data.".

• Cyclic: "It is a wave-like pattern that persists over a longer period. Cycles are often irregular and mostly appears in combination with other patterns.".

• Noise: "A random component that does not have or follow a specific pattern.". [5]

(8)

Additionally, time series data regularly has a level component, which is described as "The baseline value for the series if it were a straight line." [6]. Time series data is guaranteed to have a level, most of them have noise but the trend and seasonality may not occur.

The previously mentioned patterns and components can be formed into a model which provides the observed time series. An example of the model could be:

y=level+trend+seasonality+noise, (2.1) ybeing the output of the model.

Figure 2.1. Multiple data sources for time series forecasting [7].

Usually, time series data tends to be multivariate, hence many different data sources are needed. These sources can be observed inputs from the past (e.g electricity consumption on a certain day), known future inputs (e.g upcoming national events), and static metadata (e.g location) which are demonstrated in Figure 2.1. Lim et al. [7] state that due to the heterogeneity of the data sources, multi-horizon time series forecasting becomes quite challenging.

2.2 Related work

Time series forecasting has proven itself to be a valuable tool in many different fields. It has been used in predicting solar irradiance [8][9], finance [10][11], petroleum production [12], and likewise in this thesis, predicting emergency department arrivals and occupancy levels [13].

(9)

Over the years there has been significant academic effort aiming to increase the accuracy of time series forecasting tools in ED context. However, most of these studies have focused on neural networks or some variations of ARIMA’s. A limited amount of research has been done with the Facebook Prophet model yet due to it being available from 2017.

It has been used e.g in predicting air temperature and based on Toharudin et al. [14]

paper, it did a fairly decent job in it. In a very recent paper (2021) Park et al. [15] tested Prophet for data collected from urban community-based hospital in New York Metropoli- tan area, which yielded reasonable results, however, according to the paper the limitation was that there were only three years of data from one hospital.

Two studies that have the same dataset as this thesis, have also tried different methods for forecasting time series. The former which is currently under review utilized simple and seasonal naives, Seasonal Autoregressive Integrated Moving Average (Exogenous) (SARIMAX), General Linear Model (GLM), both simple and seasonal naive models and finally Prophet [16]. The latter used Temporal Fusion Transformers (TFT) and Long Short- Term Memory methods [17]. In Chapter 5, the results of this thesis are compared with these previous papers’ results.

(10)

3. METHODOLOGY

This chapter covers detailed information on Facebook Prophet library and the evaluation metrics which are used to determine how the model performs. Whilst the library has a very intuitive application programming interface (API) for Python, which follows the Scikit- learn model API [18], it is necessary to dig deep into it in order to understand how Prophet works under the hood.

3.1 Prophet

Facebook’s Core Data Science team released open source time series forecasting soft- ware called Prophet which has an API available for R and Python [19]. With a fairly low amount of effort, a simple prediction can be made with Prophet, however, if one desires to optimize the model, it is necessary to tune the hyperparameters. The next subsection covers the structure of Prophet model thoroughly.

3.1.1 Structure

As mentioned in Chapter 2, a time series forecasting model can be constructed as stated in equation 2.1. In the original paper, Taylor & Letham [20] proposed a model for Prophet which can be represented in the following equation:

y(t) = g(t) +s(t) +h(t) +ϵ_t, (3.1) where g(t)is the trend function which models a non-periodic change in the time series, s(t)describes periodic changes e.g monthly and yearly seasonality, h(t)describes the effect of holidays and events which may occur in irregular schedules and finally ϵ_tis the error term which represents any quirky changes which are not adapted by the model.

Two different types of trend functions are proposed for the Prophet, nonlinear trend with saturating growth and linear trend with changepoints [20]. The first one can be formulated as:

g(t) = C

1 +exp(−k(t−m)), (3.2)

where C is the carrying capacity, k growth rate, and m an offset parameter. Typically

(11)

the C and m are not constant, which is not taken into account in equation 3.2. Taylor

& Letham proposed [20] a model where these are considered, but in order to utilize the model, extra information is needed for the parameters which are not acquired for this thesis. Therefore this thesis will focus on the linear trend which is formulated as follows:

g(t) = (k+a(t)^Tδ)t+ (m+a(t)^Tγ), (3.3) where k is the growth rate, δ has the rate adjustments, m being still the offset parameter and γ is set as−s_jδ_j in order to make the function continuous. The parameter s_j describes changepoints S at certain timesj = 1, ..., S.

The seasonality model is constructed with the standard Fourier series and thus it can model arbitrary smooth seasonal effects. Hence seasonality can be modeled as:

s(t) =

N

∑︂

n=1

(a_ncos

(︃2πnt P

)︃

+b_nsin

(︃2πnt P

)︃

). (3.4)

Here P is the regular period expected to have for the time series, e.g P = 365,25for a year and P = 7for a week. The parameters a_n and b_n need to be estimated in order to fit the seasonality, the estimation is in format β = [a₁, b₁, ..., a_N, b_N]^T. This can be implemented by making a matrix from seasonality vectors for every value of t from the past and the future data. Therefore, the seasonal component is s(t) = X(t)β, when X(t)can be formulated for example:

X(T) = [cos

(︃2π(1)t 7

)︃

, ...sin

(︃2π(8)t 7

)︃

], (3.5)

with weekly seasonality and N = 8. For approximating β in a general model, Taylor &

Letham [20] assumed thatβ∽N(0, σ²), which means thatβis normally distributed and centralized around zero.

Holidays are tricky to forecast because they rarely follow a periodic pattern. However, national holidays are known to occur on certain days, so they must be also considered.

The Prophet model allows the user to manually add columns for the desired holiday or event. Assuming holidays are independent, combining the model with the holidays is quite direct. For every holidayi,D_iis the list of dates for the holiday in the past and in the future. Next, an indicator function is used whether time t is during a holiday. After that, each holiday is assigned with a parameterκ_i which stands for the change in the forecast.

This is done in a similar manner as in seasonality

Z(t) = [1(t ∈D₁), ...,1(t ∈D_L)] (3.6)

(12)

and eventually taking

h(t) = Z(t)κ. (3.7)

Same assumption is made forκbeing normally distributed, soκ∽N(0, ν²). [20]

Codewise the Prophet model is based on Stan’s backend. Stan is a cutting-edge platform utilized for statistical modeling and it has high performance for statistical computation [21].

The Facebook’s Core Data Science team released Prophet as open source. The source code can be found from the reference link guiding to GitHub [22].

3.2 Evaluation

When implementing machine learning models, it is important to receive feedback on how the model performed and ultimately compare the results with other implementations. Er- ror functions (loss functions) are a handy tool for this and there are a few common loss functions used in machine learning such as mean squared error (MSE) and mean absolute error (MAE) [23]. Root mean square error, MAE and mean absolute percentage error (MAPE) will be used in this thesis to evaluate Prophet’s performance.

MAPE is a common metric used to measure the outcome of time series forecasting models [24]. MAPE is also often used in real world applications if the quantity to predict is guaranteed to stay above zero, like in this thesis’ data [25]. The loss function itself is calculated in a following way:

M AP E = 1 n

n

∑︂

t=1

⃓

y_t−yˆ_t yt

⃓

∗100%, (3.8)

wherey_tis the actual value,yˆ_tis the predicted value andnis the total number of samples [23]. While MAPE represents the percentage error, MAE expresses absolute value of the prediction error. MAE can be formulated as:

M AE = 1 n

n

∑︂

t=1

|y_t−yˆ_t|, (3.9)

where y_t, yˆ_tandn are the same as in MAPE [26]. RMSE is basically the same as MSE but square rooted. MSE resembles MAE with only difference being error term squared.

This causes larger errors to have more weight. RMSE is described in a following manner [26]

RM SE =

⌜

⃓

⎷ 1 n

n

∑︂

t=1

(y_t−yˆ_t)². (3.10)

The parameters of RMSE represent the same variables as in MAPE and MAE. The results these metrics provide are the better the lower the results are, which means they are

(13)

negatively oriented.

(14)

4. EXPERIMENTS

The primary experiment in this thesis is to take a portion of history from the data and use the data to train the Prophet model to predict arrivals one day ahead. If implemented in production setting, this predictive horizon would enable ED management to process the forecasts and act upon the information, both of which are required in order to draw any benefit from the provided predictions.

The dataset is split into a training set (the first ~1200 days) and a test set (the last ~400 days). As their name describes, the training set is used to train the model and the test set is to evaluate the model. In machine learning, it is very important to separate the training and test sets, because otherwise, the model would overfit. This means that the model fits a certain dataset "excessively well", and when a new previously unknown dataset is introduced to the model, it performs poorly [27].

Other experiments are hyperparameter tuning and finding the variables which the model thinks are relevant regarding getting better results. These variables are for instance weather, Tampere University Hospital website visits and holidays. These topics will be discussed more extensively in Sections 4.2 and 4.3.

4.1 Data

The dataset contained hourly ED occupancy data from Tampere University Hospital ED Acuta and there are over 35 000 hours of data available from May 2015 to September 2019. In addition to the hourly arrivals, the dataset also contained several independent variables such as national holidays, several weather variables and availability of hospital beds in several local hospitals.

Initially, the dataset needed some preprocessing because it was in hourly and a fairly raw format. For example, NaNs (Not a Number) needed to be replaced with zeros and Boolean values with zeros and ones respectively. In terms of required computing power it is much lighter to train the models when the data is in daily format rather than in an hourly format. For the Prophet model itself, the API requires exactly two columns with specific column names for the data. These columns aredsandy which represent the timestamp and the desired forecasting unit. The dataset didn’t have a timestamp column ready, but

(15)

this was quite easy to create using Pandasdataframe.index.

Training the model with the dataset (without any additional variables) and predicting the future with rolling origin (see Section 4.4) takes around 7 minutes with Intel Core i5-7200U CPU. Nonetheless, when additional variables are considered, the computing time grows proportionally. For instance, when all of the variables available were used, the computing time rose to ~2 hours. The source code for this Prophet model implementation can be found from GitHub [28].

4.2 Hyperparameter tuning

In machine learning, hyperparameters are higher-level parameters that are set before training the model and these parameters are tuned based on the features of the dataset [29]. The Prophet model is packed with 16 different hyperparameters. However, only 4 of them are recommended to tune according to the documentation [30]. Prophet’s hyperparameters recommended to tune are listed below.

1. Changepoint prior scale. This parameter determines the trend flexibility and how the trend changes at the trend changepoints.

2. Seasonality prior scale. As well asChangepoint prior scale this determines the flexibility of the seasonality. Bigger values allow the model to fit to larger fluctuation points.

3. Holidays prior scale. This specifies the holiday effects flexibility and it is similar to Seasonality prior scale.

4. Seasonality mode. This parameter defines if the seasonality is either additiveor multiplicative. For instance, if the seasonality grows along the trend, then multi- plicativemode may be considered.

Table 4.1 shows all of the considered hyperparameters for the tuning.

Table 4.1.Different hyperparameter values

Changepoint prior scale

Seasonality prior scale

Holidays prior scale

Seasonality mode

0.001 0.01 0.01 additive

0.05 1 1 multiplicative

0.5 10 10

There are many ways to optimize these hyperparameters, such as grid search, random search and Bayesian optimization [31]. Prophet has an inbuilt method for the hyperparameter tuning calledparallel cross validationand this method is used in this thesis.

(16)

4.3 Variable selection

In time series forecasting, finding the variables which the model considers important regarding the results, is an interesting task. The following variable groups which are considered are listed in Table 4.3

Table 4.2. Variables and their definitions.

Variable name Definition

Hospital beds available The amount of free hospital beds in health centers around Pirkanmaa.

Weekday E.g Monday.

Month E.g January.

Days around holiday Describes whether a day is a holiday, after a holiday or before a holiday

Holiday name Is working day Weather

Website visits Number of website visits to tays.fi and tays.fi/acuta.

Ekströms visits Number of website visits to tays.fi between 6 PM to midnight divided by rest of the visits.

Google trends Acuta per month normalized

Number of Google searches for "Acuta" normalized for each month.

Number of public events per day

E.g festivals held in Pirkanmaa.

Daily peak occupancy

Overall, there are 12 variable groups and these can be combined in 4096 different ways and this is calculated with the following formula:

combinations=

12

∑︂

n=0

(︃12 n

)︃

. (4.1)

The result is as it is because all combinations are considered (for instance, one combination is using no variables at all), and the order of the variables is irrelevant.

Trying out all of these combinations would be too exhaustive, so the process of choosing the variables is done in a simplistic way. First, the model is trained with all the variables considered, this gives a reference error value to compare the future results to. After that the model is fitted with 11 variables 12 times in such a way, that one variable is dropped off. Doing this procedure reveals which variables affect the prediction error in either increasing or decreasing manner.

(17)

4.4 Model validation

As discussed before, machine learning datasets are usually split into training and test sets. The model is fit to the training set and it is validated on the test set. However, in time series forecasting testing the model once on "fixed origin" can give misleading information on the performance. This is because if the data has outliers, a poor model performs better if a fixed origin is used. Hence one should use "rolling origin" for fitting time series data.

[32]

The idea of rolling origin is very simple and Figure 4.1 does a good job visualizing it.

Initially, the model is fit to the training set which is 15 days as the image shows. Then a prediction is made for the next three days ahead. After that, the model is fit again, but this time the training set is 16 days. This process is repeated until the whole dataset is covered. [32]

Figure 4.1.The basic principle of rolling origin with constant holdout size [32].

While the image represents the procedure efficiently, rolling window usage in this thesis is slightly different. The dataset’s training set size is approximately 1200 days and the test set 400 days. Also, the prediction is made one day ahead, not three as the image shows.

Hence the model is fit 400 times and each time the training set grows by one day.

(18)

5. RESULTS

First of all, the model was fit with all of the variables and the default parameters. This gave the reference error for comparing the results of hyperparameter tuning. In total 54 combinations were tested and the combination with the lowest error along with the default hyperparameters are listed in Table 5.1. The upper row is the combination of hyperparameters that suit the model the best when all of the variables are considered and the lower row is the model with default parameters. All of the combinations yielded MAPE values between [6.82, 7.34].

Table 5.1.Hyperparameter tuning results

Changepoint prior scale

Seasonality prior scale

Holidays prior scale

Seasonality mode

MAPE (%)

0.5 10 10 multiplicative 6.82

0.05 10 10 additive 7.24

After the best hyperparameters were acquired for the model and the dataset, the variable selection was implemented. The results of the variable selection are listed in Table 5.2.

(19)

Table 5.2.Variable selection results

Variable left out MAPE (%) RMSE MAE

Daily peak occupancy 6.83 22.55 17.49

Hospital beds available 6.57 22.04 16.83

Weekday 6.80 22.49 17.43

Month 6.77 22.39 17.33

Days around holiday 6.89 22.61 17.64

Holiday name 6.85 22.70 17.53

Is working day 6.83 22.53 17.48

Weather 6.85 22.34 17.52

Website visits 6.83 22.54 17.47

Ekströms visits 6.86 22.59 17.55

Google trends 6.82 22.56 17.46

Number of public events per day

6.83 22.59 17.49

As the results show, the lowest errors were achieved by leaving out variableHospital beds available. Notable thing is that all of the other variables which were left out did neither increase nor decrease the errors almost at all.

Figure 5.1.A plot of the fitted Prophet model which yielded the lowest MAPE.

Figure 5.1 illustrates the result of the model fitted to the data. The y-axis represents the arrivals of the emergency department and the x-axis is the date. In the figure, black dots are the real arrivals on a certain date, the dark blue line is the arrivals predicted and finally, the light blue line is the uncertainty interval. As can be seen from the Figure 5.1 the model is able to predict the arrivals pretty efficiently and also is able to catch some of the peaks.

(20)

Figure 5.2.Components of the Prophet model.

Facebook Prophet has a built-in method for visualizing for instance trend and seasonality of the model. Figure 5.2 represents the decomposed components, which are trend, holidays and seasonality. As Figure 5.2 shows, the trend is fluctuating until July 2018 and after that linearly growing. From the holiday component, clear peaks can be seen around Christmas and New Year. Also, a little bit smaller peaks can be seen at the end of June, where Midsummer resides. Weekly seasonality appears to be stronger at the start and at the end of the week. Finally yearly seasonality is emphasized between July and September.

5.1 Comparison with previous results

Different machine learning models have been tested with the same dataset as in this thesis due to the research being multidisciplinary. Some of the models are for instance SARIMAX and also a fairly new neural network architecture Temporal Fusion Transformer.

Table 5.3 visualizes the results which each model achieved.

The first three models are neural networks: Long Short-Term Memory and TFT. The next two models are Prophets with the default hyperparameters and the first having calendar variables and the second having no variables. After that comes General Linear model and SARIMAX with calendar variables, then simple and seasonal naive models and finally the model used in this thesis.

(21)

Table 5.3.Results with different machine learning models on the same dataset.

Model MAPE (%)

LSTM 7.96

TFT Daily 6.68

TFT Hourly 6.37

Prophet:Calvars 6.90

Prophet:None 6.70

GLM 8.10

SARIMAX:Calvars 6.60

Simple Naive 8.40

Seasonal Naive 8.10

Prophet 6.57

From all of the models, TFT Hourly performed the best. Simple naive had the worst performance with GLM, seasonal naive and LSTM quite close to the last place. However, the training set used in the paper by Tuominen et al. [16] is slightly larger than the training set used for the other models, so that must be taken into account when comparing the error rates.

Notable thing is that there is not much variance between the models except for the four worst models in the results. Even the Prophet model with neither variables nor hyperparameter tuning performed quite decently. Deciding which model to use is heavily application dependant. For example, if one wants to make a forecast with quite a low effort, then considering Prophet without hyperparameter tuning can be a solid choice. On the other hand, if more training data is available, then the TFT might be a beneficial choice. The original papers can be found from the reference links [17][16].

(22)

6. CONCLUSIONS

Being able to predict the arrivals to an ED can help prevent overcrowding and ensure treatment for critical health status patients swiftly enough. This thesis aims to predict the number of arrivals to Tampere University ED Acuta for the next day which can aid in resource allocation.

Forecasting the number of arrivals for an ED can be tricky due to their unpredictability. In this paper, a machine learning library Facebook Prophet is utilized to predict the number of arrivals. Hyperparameter tuning and variable selection were also implemented for the model. The former was done in order to acquire the best parameters for the model and this particular dataset. The latter was implemented for gaining knowledge on which individual parameter had the most impact on the model. Both of these experiments managed to decrease the error rate.

The model used in this thesis performed second-best compared to the other models which used the same dataset. However, considering the extra computing time caused by hyperparameter tuning and variable selection, one must deliberate which is the most suitable model for a given application. If Facebook Prophet is used, and speed is the essence, then hyperparameter tuning and variable selection may be left out with the price of slight accuracy decrease.

Altogether, machine learning applications in the medical field are yet in an early stage.

The more data is produced and available, the more accurate the models become and this way the models can help medical staff in decision making.

(23)

REFERENCES

[1] Bernstein, S. L., Aronsky, D., Duseja, R., Epstein, S., Handel, D., Hwang, U., Mc- Carthy, M., John McConnell, K., Pines, J. M., Rathlev, N., Schafermeyer, R., Zwe- mer, F., Schull, M. and Asplin, B. R. The effect of emergency department crowding on clinically oriented outcomes.Acad Emerg Med 16.1 (2009), pp. 1–10.

[2] Hong, T., Gui, M., Baran, M. E. and Willis, H. L. Modeling and forecasting hourly electric load by multiple linear regression with interactions. IEEE PES General Meeting. 2010, pp. 1–8.DOI:10.1109/PES.2010.5589959.

[3] Dei, M. Facebook Prophet. Aug. 23, 2020. URL:https : / / medium . com / swlh / facebook-prophet-426421f7e331(visited on 03/13/2021).

[4] Sosnovshchenko, A. Machine Learning with Swift. Packt Publishing, 2018. ISBN: 1-78712-151-8.

[5] Nair, A. An Introductory Guide To Time Series Forecasting. Aug. 19, 2019. URL: https : / / analyticsindiamag . com / an - introductory - guide - to - time - series-forecasting/(visited on 03/13/2021).

[6] Brownlee, J. What Is Time Series Forecasting? Dec. 2, 2016. URL: https : / / machinelearningmastery.com/time-series-forecasting/(visited on 03/13/2021).

[7] Lim, B., Arik, S. O., Loeff, N. and Pfister, T.Temporal Fusion Transformers for Inter- pretable Multi-horizon Time Series Forecasting. 2020.

[8] Yang, D., Jirutitijaroen, P. and Walsh, W. M. Hourly solar irradiance time series forecasting using cloud cover index.Solar energy 86.12 (2012), pp. 3531–3543.

[9] Qing, X. and Niu, Y. Hourly day-ahead solar irradiance prediction using weather forecasts by LSTM. eng. Energy (Oxford) 148 (2018), pp. 461–468. ISSN: 0360- 5442.

[10] Cao, J., Li, Z. and Li, J. Financial time series forecasting model based on CEEM- DAN and LSTM.Physica A519 (2019), pp. 127–139.

[11] Khashei, M. and Hajirahimi, Z. Performance evaluation of series and parallel strate- gies for financial time series forecasting.Financial innovation (Heidelberg)3.1 (2017), pp. 1–24.

[12] Sagheer, A. and Kotb, M. Time series forecasting of petroleum production using deep LSTM recurrent networks.Neurocomputing323 (2017), pp. 203–213.

[13] Whitt, W. and Zhang, X. Forecasting arrivals and occupancy levels in an emergency department.Operations Research for Health Care21 (2019), pp. 1–18.

(24)

[14] Toharudin, T., Pontoh, R. S., Caraka, R. E., Zahroh, S., Lee, Y. and Chen, R. C.

Employing long short-term memory and Facebook prophet model in air temperature forecasting.Communications in statistics. Simulation and computation(2021).

[15] Park, J. C., Chang, B. P. and Mok, N. 144 Time Series Analysis and Forecasting Daily Emergency Department Visits Utilizing Facebook’s Prophet Method.Annals of Emergency Medicine74.4 (2019).

[16] Tuominen, J., Roine, A., Sauviak, T., Seppo, A., Pihlaja, M., Ovaska, J., Pauniaho, S.-L. and Oksala, N. Forecasting Daily Arrivals and Peak Occupancy in a Combined Emergency Department.Research Square(2021).

[17] Pulkkinen, E. Forecasting emergency department arrivals with neural networks.

URL:https://trepo.tuni.fi/handle/10024/124390.

[18] Facebook. Python API. 2017. URL:https://facebook.github.io/prophet/

docs/quick_start.html#r-api(visited on 03/17/2021).

[19] Facebook. Facebook Prophet documentation. 2017. URL:https : / / facebook . github.io/prophet/(visited on 03/17/2021).

[20] Taylor, S. J. and Letham, B. Forecasting at scale.PeerJ Preprints(2017).

[21] Stan.Stan documentation.URL:https://mc-stan.org/(visited on 03/31/2021).

[22] Facebook. GitHub Prophet. URL: https : / / github . com / facebook / prophet (visited on 03/31/2021).

[23] Parmar, R.Common Loss functions in machine learning. Sept. 2, 2018.URL:https:

/ / towardsdatascience . com / common - loss - functions - in - machine - learning-46af0ffc4d23(visited on 03/24/2021).

[24] Gul, M. and Celik, E. An exhaustive review and analysis on applications of statistical forecasting in hospital emergency departments. Health systems 9.4 (2020), pp. 263–284.

[25] de Myttenaere, A., Golden, B., Le Grand, B. and Rossi, F. Mean Absolute Percent- age Error for regression models.Neurocomputing 192 (2016), pp. 38–48.

[26] Chai, T. and Draxler, R. Root mean square error (RMSE) or mean absolute error (MAE)? -Arguments against avoiding RMSE in the literature. Geoscientific model development7.3 (2014), pp. 1247–1250.

[27] Lever, J., Krzywinski, M. and Altman, N. Points of Significance: Model selection and overfitting. eng.Nature methods13.9 (2016), pp. 703–.ISSN: 1548-7091.

[28] Mäkipää, A.-J. GitHub repository. URL:https : / / github . com / makipaa / BSc - Thesis(visited on 05/05/2021).

[29] Agrawal, T.Hyperparameter Optimization in Machine Learning Make Your Machine Learning and Deep Learning Models More Efficient. eng. 1st ed. 2021. Apress, 2021.

[30] Facebook. Diagnostics. 2017. URL:https://facebook.github.io/prophet/

docs/diagnostics.html#hyperparameter-tuning(visited on 03/30/2021).

(25)

[31] DeepAI. What is a hyperparameter? URL: https : / / deepai . org / machine - learning-glossary-and-terms/hyperparameter(visited on 03/31/2021).

[32] Svetunkov, I.Time Series Analysis and Forecasting with ADAM. 2020.URL:openforecast.

org/adam(visited on 04/11/2021).

Forecasting Emergency Department arrivals with Facebook Prophet library