Evaluating and predicting developers' contribution in software projects

(1)

Evgeniia Onosovskaia

EVALUATING AND PREDICTING DEVELOPERS’ CONTRIBUTION IN SOFTWARE PROJECTS

Bachelor’s thesis Faculty of Engineering and Natural Sciences Examiners: Outi Sievi-Korte April 2021

(2)

ABSTRACT

Evgeniia Onosovskaia: Evaluating and predicting developers’ contribution in software projects Bachelor’s thesis

Tampere University

International Bachelor’s Degree in Science and Engineering April 2021

Nowadays the software development industry is growing rapidly as well as the popularity of the distance work. Hence there are a lot of big software teams scattered all over the world and it might be very importantfor a team leader to monitor the performance and general working style of their team members. In this thesis work, the system to evaluate and predict software developers’ performance based on the pure information from a version control system is presented. Dur- ing development of the system three main tasks have been solved: creation of an environment, which retrieves information from the version control system and visualizes it, development of a formula to evaluate developers’ contribution to the project, and creation of a prediction tool. The developer’s contribution formula creation included two steps. Сollecting possible metrics which might indicate developer’s performance and then series of experiments on different projects to construct a formula from the most suitable ones. After the formula finalization it became possible to calculate precise developer’s contributions and visualize them as plots over a custom period.

The prediction tool was required to construct a continuation of this curve over the next period of the same length. Two hypotheses have been tested: fitting the curve and the neural network approach. The neural network appeared to be the best solution, as it works more stable and guesses the curve scale and form properly. The final system takes two parameters as input: a git repository and time period we are interested in. As a result, it produces the plots for all developers:

constructed from their contribution over the target period and a predicted contribution for the future period of the same length. As further research options, different additional metrics, that could be included to make contribution evaluation formula more precise can be proposed. However, the prediction part can be considered good enough and doesn’t require additional development or research, as it already fits the initial requirements rather good. Overall, the research was considered successful as an initial goal of creating a basic version of a software to evaluate and predict developer’s contribution was achieved.

Keywords: machine learning, software development, software analytics

The originality of this thesis has been checked using the Turnitin OriginalityCheck service.

(3)

PREFACE

The general concept of visualizing aspects of the software development process was inspired by VISDOM project. The precise idea of this thesis came to me while I was working in JetBrains because it is an international company with rather big teams and a free working style, where it might be really helpful to have some general picture of the developers working styles.

I would like to thank my supervisor Outi Sievi-Korte for her valuable guidance and com- ments throughout the whole process. Also, I highly appreciate the support from my family and friends which makes the thesis writing more interesting than stressful.

Tampere, 20^th April 2021 Evgeniia Onosovskaia

(4)

LIST OF FIGURES

Figure 1. Possible basis functions ... 4

Figure 2. RMSE 1.24 * 10^-6 ... 6

Figure 3. RMSE 0.065 ... 7

Figure 4.General neural network structure. ... 8

Figure 5.Possible activation functions ... 9

Figure 6.Example of a rather bad (left) and a quite good (right) fit. ... 22

Figure 7. Example of the created system output. ... 24

(6)

LIST OF SYMBOLS AND ABBREVIATIONS

LOCs - lines of code

MAE - Mean Absolute Error

R² - R-squared or coefficient of determination Radj2 - Adjusted R squared

RMSE - Root Mean Squared Error

(7)

1. INTRODUCTION

Software development is a complex process. A person working in software development contributes to projects through different activities. Moreover, working styles may vary significantly within one team. One person has a constant productivity rate, while another works extremely hard in some periods and is relatively relaxed otherwise. One writes a lot of code doing some simple front-end task, while another deals with a smaller but more complicated and essential back-end part.

When the software team is small enough, the team leader can keep in mind all processes and developers within the team. However, the bigger it becomes, the harder it to gather this information and follow what the current productivity level is. In the big international companies, there might be big teams scattered in different countries. This may become an essential problem for managing a team when you can’t get familiar with each developer’s unique working manner and hence unable to assign to him the most suitable task or find out that current workload specifics aren’t optimal.

The goal of this thesis is to provide a basic version of a system that can do three things.

1. Gather development process information from a version control system (i.e. git).

2. Based on that evaluate in some appropriate way what is the developer’s current contribution to the project.

3. Predict how this particular developer is going to work in the future.

This would help to get a general picture of the current state for each developer and predict how he or she is going to work in the future taking into account their working style, productivity level, etc.

Chapter 2 briefly explains the general concepts of machine learning and goes through the most applicable approaches for this type of task. Chapter 3 covers background related to neural networks and practicalities on using them. Chapter 4 goes through software development process methodologies and analytics. Chapter 5 states the problem to be solved and describes the research process. Chapter 6 contains experiment description and results. Finally, in Chapters 7 and 8 possible future research and conclusions are discussed.

(8)

2. MACHINE LEARNING BACKGROUND

According to Tom M.Mitchell “Machine learning is the study of computer algorithms that allow computer programs to automatically improve through experience” (Mitchell T., 1997). In other words, the ultimate goal of machine learning is to “teach” computers to solve some problems without any pre-programmed instructions. Thereby this leads to a possibility of solving some problems which are too complicated for humans to find appropriate solutions. Some real-life examples of such tasks are speech synthesis and recognition, face detection, context advertisement targeting. However, nowadays machine learning has just started rising and many challenges are still being solved, such as natural language processing. (Rus, 2013)

Machine learning can be classified into three different methods: supervised,

unsupervised, and reinforcement learning. (Stinis, 2019). Unsupervised (or descriptive) learning is a type of task when a machine is supposed to find patterns in an unknown set of objects by grouping similar objects together. For example, it takes a data set and tries to find natural groupings or patterns within the data elements or records

(Chandramouli S., Dutt S., Das A.K., 2018). Reinforcement learning means, that a machine learns to act on its own to achieve the given goals. In other words, the learner is not told which actions to do, but instead is supposed to find out which actions lead to the best outcome. (Richard S. Sutton, 2018)

The problem we are facing when trying to predict developers’ contribution is related to supervised learning, so in this thesis, we are going to focus on it.

2.1 Supervised machine learning

In supervised learning, a machine predicts the class of unknown objects based on prior class-related information of similar objects (Chandramouli S., Dutt S., Das A.K., 2018).

In other words, in this type of task, there is a training data set, with predefined mapping

“question” to “answer”, which could be analyzed in any desired way. Question and answer could be different depending on the type of the task.

For example, if the goal is to learn how to detect animals from the image, the pair would be “picture” - “animal type”. The goal is to study this set as well as possible to provide the most precise solution when the same question will be asked in a real situation. The problem that we aim to solve in this thesis – predict developer’s contribution – fits the

(9)

criteria of supervised learning. The general task from the machine learning point of view looks like the following: we have a plot that represents the developer's behavior during some period and our goal is to construct the curve based on some part of the initial plot which will be rather close to the ”right answer” (first plot) in terms of form and scale. Here, we have already predefined answers for a current period and in the future, we’ll be able to check whether we were wrong or right.

Program 1 provides a pseudocode example that illustrates the manner the supervised learning is working.

There are two main types of supervised learning problems: classification and regression.

(Hastie T., Tibshirani R., Friedman J., 2009) In the classification task, the computer attains to answer the question “Which category does this object belong to?”.

(Chandramouli S., Dutt S., Das A.K., 2018)The regression is going to be discussed in more detail in the following.

2.2 Regression

In the same way as with the classification task, in the regression problems, the computer is required to predict labels associated with example data. The main difference is that for regression this label is real-valued. (Ratsch G, 2004) It can be easier to think about regression as a task to fit the curve (predict some real-valued numerical data). A simple example of such a task could be temperature prediction during the next month. It means that we need to introduce a real-valued continuous function, representing the

1 3 5 7 9 11 12 13 14

do_training(train_data){

for label in labels{

features[label] = find_common_features(train_data[label]) }

return features }

do_prediction(train_data, test_data){

calculated_features = do_training(train_data) for sample in test_data{

features = get_features(sample)

labels[sample] = find_label_corresponding_to_these_fea- tures(calculated_features) }

return labels }

Program 1. Supervised learning pseudocode illustration

(10)

temperature changes. This is exactly the type of task that should be solved to predict developers’ contribution. Let’s go through one of the popular approaches to solving this type of task.

2.2.1 Fitting the curve

Curve fitting is a solution to the problem to find a mathematical relationship between points in a given set of data when we don’t know exactly a formula producing this set (M.

Beitollahi, S. A. Hosseini, 2016). Generally speaking, it is a type of mathematical optimization problem, where the optimal set of parameters for the concrete function should be found. This approach is widely used in machine learning in terms of regression problems. Thinking about curve fitting might be easier if consider it as a task of fitting a 2-dimensional set of dots with some curve. Solving a problem using the curve fitting method could be divided into two main steps: choosing the form of the mapping (or basis) function and searching for the parameter to the function that results in a minimum error.

The error can be calculated by comparing expected and actual outputs. The basis function may be chosen freely and only depends on the particular set of data. In Figure 1, some of the possible basis functions are presented.

Figure 1. Possible basis functions

(11)

2.2.2 Regression performance evaluation.

As it was previously discussed, in the tasks where regression is used the goal is to predict a continuous value, in other words, it is considered as a curve. To evaluate how good is the performance of the particular regression model, some evaluation metrics should be introduced. The most popular and common metrics to evaluate regression performance are: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), R Squared / Co- efficient of Determination, and Adjusted R Squared (Mann, 2020).

Each of these metrics is made to compare how precisely the set of predicted values fits the set of initial observed values.

Mean Absolute Error represents an average of the absolute distance between the ac- tual and predicted datasets. This measures the average of the residuals. (Willmott C. J., Matsuura K., 2005)

𝑀𝐴𝐸 = ¹

𝑁∑^𝑁_𝑖=1|𝑦_𝑖− 𝑦̂| Where, 𝑦_𝑖− 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 𝑦 y̅ −𝑚𝑒𝑎𝑛 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 𝑦

Root Mean Squared Error represents the square root of the average of the square dif- ference between the original and predicted datasets. It indicates the standard deviation of the residuals.

𝑅𝑀𝑆𝐸 = √¹

𝑁∑^𝑁_𝑖=1|𝑦_𝑖− 𝑦̂| Where, 𝑦_𝑖− 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 𝑦 y̅ −𝑚𝑒𝑎𝑛 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 𝑦

The value is always not less than zero and the lower the value of RMSE is, the higher the accuracy is. (Willmott C. J., Matsuura K., 2005)

R-squared or coefficient of determination indicates the proportion of the variance in the independent variable which is explained by the linear regression model.

𝑅²= 1 − ^{∑(𝑦−𝑦̂)}_{∑(𝑦−𝑦̅)} Where, 𝑦_𝑖 − 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 𝑦 y̅ −𝑚𝑒𝑎𝑛 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 𝑦

Its value is always less than one. This metric is used to evaluate how precise the independent variables in the linear regression model explain the variability in the dependent variable. (Chugh A., 2020)

Adjusted R-squared is an improved version of R²and it is adjusted for the number of independent variables in the model, and its value is always less or equal to R². N – number of observations, k – number of independent variables in the data.

(12)

𝑅_𝑎𝑑𝑗² = 1 − [(1 − 𝑅²)(𝑛 − 1) 𝑛 − 𝑘 − 1 ]

It solves the problem that occurred with the R²model when the value increases with the addition of independent variables which might lead to the addition of redundant variables in the model. (Chugh A., 2020)

2.2.3 Root mean squared error values meaning.

As it was mentioned in the previous section only MAE and RMSE are suitable for non- linear regressions. RMSE is going to be used as the main metric in this thesis.

RMSE is a great metric to compare different models, however, there are no absolute values that can be considered as a “good” or a “bad” result, because this completely depends on the context of the concrete task. (Linear regression models, 2021) Hence even knowing the general requirements for the particular task it might be complicated to understand the validity of the results, as these values are not associated with any

“physical meaning”. Hence, here we are going to overview two examples of fitting the curve to a different degree and their corresponding RMSE values.

In both examples, the first half of the curve is used to train the machine learning model and the second half is the predicted result on which the RMSE is calculated. This is done in the same manner as in the experiment description part to provide a better understanding of the interpretation of the results.

In Figure 2, RMSE is roughly equal to 1.24 * 10^-6. This corresponds to the almost perfect curve form fit in a proper scale.

Figure 2. RMSE 1.24 * 10^-6

(13)

At the same time, in Figure 3, the RMSE value is roughly equal to 0.065. This corresponds to the wrong curve form fit.

Figure 3. RMSE 0.065

(14)

3. NEURAL NETWORKS

One of the most powerful instruments in modern machine learning is neural networks, which are used to solve different types of problems, including regressions. In this section general concepts of neural networks are going to be covered and then discussed key approaches related to solving practical tasks using neural networks. (Chandramouli S., Dutt S., Das A.K., 2018)

3.1.1 Basic notions and neural network architecture

The concept of neural networks in machine learning directly relates to the biological neural networks in the human brain. It could be considered as a mathematical way to model the process of human thinking.

A neural network normally consists of multiple neurons. The neuron is formed of three major pillars: a set of synapses, a summation junction, and a threshold activation function. Each synapse has weight 𝑤_𝑖, which is applied to the input signal 𝑥_𝑖. The summation junction is weighted using synapsis weights. A threshold activation function is applied to results in an output signal only when an input signal exceeding a specific threshold value comes as an input. Figure 4 illustrates this structure.

Figure 4.General neural network structure.

Also, an important notion in the world of neural networks is overfitting, a phenomenon, when a neural network has too many parameters and hence becomes too “flexible” trying to follow all the points in the set, which produces worse results than with a less-parameter

(15)

configuration. (Dreyfus G., 2005) This is an important metric to take into account in further sections when speaking about optimal neural network configuration.

3.1.2 How to configure neural network

When using neural networks in practice it is crucial to use appropriate configurations, because this is what influences the result at most. In this subsection, the choice of the three main aspects that should be specified is going to be discussed: number of output nodes, number of layers, and an activation function.

The number of output nodes in the neural network strictly depends on the activation function being chosen and the general problem context (what are we getting as an output). For example, in the regression when the goal is to map x to y value, only one output node will be used. (Aggarwal C., 2018)

Neural networks that contain more than one layer are called multilayer networks and all layers between input and output layer are called ”hidden layers”. The more layers pre- sent in the network, the fewer units per layer it tends to require, this might be used when there is a limited amount of data available. Also, increasing the number of layers, reduces the number of problems related to overfitting. (Aggarwal C., 2018)

Depending on the type of output data different activation functions should be used. There are six most commonly used activation functions, which are illustrated in Figure 5.

Figure 5.Possible activation functions

(16)

For example, linear (or identity) function is commonly used when the target is a real value, sign function can be used to map binary outputs at prediction time. In terms of this thesis, we are mostly interested in the sigmoid, tanh, reLU, and hard tanh functions. All of them have a relatively similar shape but sigmoid and reLU are used to operate with values between 0 and 1, also they are not symmetric about zero.Tanh and hard tanh have values between [-1, 1], more steep and symmetric about the origin. As in this thesis, we are predicting contribution, which is a positive value between 0 and 1, the most suitable activation functions would be reLU and sigmoid functions. (Sharma S., Sharma S., 2010)

3.1.3 Practicalities on using neural networks

Neural networks are currently implemented in different programming languages. Hence, there is no need to implement it on your own. In this section, a brief overview of how neural networks can be used with Python libraries is going to be provided.

Neural networks with custom configuration in Python can be constructed using the Keras API with Teserflow. (Singh P., Manure A., 2019)

Even though, the main things are already pre-programmed and stored in the library, constructing a proper model still depends on the concrete task.

The general steps of constructing a neural network model using TensorFlow are:

1. Building the model: at this step, the general model configuration should be specified.

2. Training the model: at this stage, the training data is taken and the already pre- programmed library back-end starts to perform calculations to get an optimum result.

3. Testing the model: this part validates and checks the accuracy of the model.

In terms of the developer’s impact, the first step requires work to be done, while the last ones are almost handled by library methods. (Silaparasetty N., 2021)

To implement the neural network using Python it is required to develop a proper architecture of the machine learning model. To do this, the following things should be specified: the number of hidden layers, the number of neurons in each layer, the weights and biases, the activation function. The preferred configuration of the neural network might be found after careful study of data. (Silaparasetty N., 2021)

(17)

Let’s now go through a real code example in Program 2, to see how will it look like.

This is a model, which has three layers, with eight nodes on the first layer, four nodes on the second layer, and one output node. No activation function is specified, hence no activation function will be applied, which basically means a linear activation function (a(x)

= x). (tf.keras.layers.Dense, 2021) After that, this model is compiled with a MeanSquaredError as a loss function, which means it computes the mean of squares of errors between labels and predictions. (Regression losses, 2021) And then the model is trained, with 2500 iterations and no logging. (Model training APIs, 2021)

1 3 5 7 9 11

import tensorflow as tf

model = tf.keras.Sequential([

tf.keras.layers.Dense(8), tf.keras.layers.Dense(4), tf.keras.layers.Dense(1) ])

model.compile(loss='mean_squared_error')

model.fit(x_train, y_train, epochs=2500, verbose=False)

Program 2. Example of creating and training model using TensorFlow

(18)

4. SOFTWARE DEVELOPMENT PROCESS MODELING AND ANALYSIS

In this section, we provide a general understanding of what is a software development process, which parts might it consist of, and based on which metrics can it be

analyzed.

4.1 Business process modelling

Before going deeper into software development specifics let’s cover some basic concepts related to the business process and business process modeling, to know which components are they consist of in the general case.

Business process modeling is used to provide a good representation of both management and engineering processes within the same project. This gives a possibility to follow the workflow, determine current problems and improve eventual behavior (e.g.

find some ineffective strategies and enhance them). (Ruth S., Aguilar-Savén, 2004) A business process is the combination of activities within an enterprise with a structure describing their logical order and dependence whose objective is to produce the desired result. (Ruth S., Aguilar-Savén, 2004) Further in this thesis, by the process, only the software development process is meant.

The business process lifecycle can be divided into four phases:

1. Design phase - design of a process.

2. Configuration phase - the process is programmed.

3. Monitoring phase - the process is monitored to notice if any changes are needed.

4. Diagnostic phase - process evaluation phase, when some new requirements may be added. (Wei W., Hongwei D., Jin D., Changrui R., 2006)

In terms of the thesis topic, we are mostly interested in the configuration and monitoring phases.

(19)

4.2 Methodologies

Software development methodology is a way of managing a software development process. (Young D., 2013) The chosen methodology may influence significantly the types and styles of processes occurring in the project. In terms of this thesis, it is important to consider different methodologies, as they might significantly influence the working style and various types should be taken into account.

 Waterfall – the most traditional and ”linear” developing approach, which puts a lot of attention to the initial prerequisites, and all programming is done ”step by step”

only after everything has been planned, designed, and prepared. (S.Balaji, Dr.M.Sundararajan Murugaiyan, 2012)

 Agile – can be considered as a family of methodologies the most popular is SCRUM. The general idea behind is that all development process is divided into periods (sprints) to be more flexible for changes and all team members are taking part in product planning. (Srivastava A., Bhardwaj S., Saraswat S., 2017)

 Rapid Application Development (RAD) – the basic idea behind this approach is to give a much faster development and higher quality results, by taking maximum advantage of software development that has evolved recently. (Martin J , 1991) During the development process programmer focuses on each requirement at a time, hence it is easier to implement. Also, users are involved in the development process, and in the end, this increases their final satisfaction. (Daud N. M. N., Bakar N. A. A. A., Rusli H. M., 2010)

 Lean Development – the goal is to achieve a minimal absolute cost. The main focus is put on customer current satisfaction (provide them what they want right now) and getting rid of all unnecessary tasks. (Rossi M., Taisch M., Terzi S., 2012)

4.3 Software analytics

According to Menzies and Zimmermann (Menzies T., 2018), software analytics is used to avoid misconceptions between developers who develop their own “good” and “bad”

software. In other words, it provides some set of notations, chunks of information, and features, which prospect to evaluate software quality in some objective way.

The initial reason behind software analytics is fairly simple: software development is a rather buggy and complicated process and one of the first goals was to find the factors which influence this at most. One of the most important requirements of the software

(20)

analytics approaches is to be real-time and actionable. This means that information provided by analytics should lead to some appropriate action and be relevant regarding the current project situation. For instance, it doesn’t matter how to optimize a bit review process when the ultimate project goal is not likely to be achieved by time.

According to Menzies and Zimmermann (Menzies T., Zimmermann T., 2013), there are seven main principles of software analytics.

 Take usability goals and user experience about particular software into account to focus on the most important issues while evaluating the software.

 Create automated systems for software analytics.

 Early feedback from users leads to an opportunity to provide deeper analytics.

 It is important to stay open-minded even though there are already many common approaches invented and proceed to find new ways.

 Use smart learning, validate information more times, and check how well your conclusions fit already existing data.

 Work with the data you have. Not always there is a perfect set of data including all features you are interested in, but it is important to get the best possible outcome from what you have.

 Use all available tools which might help you achieve better analytical results.

After considering all these requirements let’s go through the practicalities: which metrics can be used to analyze the developer’s contribution based on the version control system.

One set of metrics is proposed by Parizi, Spoletini, and Singh (Parizi R. M., Spoletini P., Singh A., 2018): a number of commits per month, a number of merges per month, number of files per month, total lines of code (LOCs) per month, time spent on the project per day.

The number of commits and merges per month is related to the developer’s speed and intensity of their work. A number of files and LOCs per month indicate the real ”physical”

impact made by the developer – the amount of added code and contribution to the project. Time spent on the project per day relates mostly to the working style and performance stability (Parizi R. M., Spoletini P., Singh A., 2018).

These metrics are rather popular and can be found in some other sources, however sometimes in a little bit different form.

(21)

Here are some examples:

1. How many files have been changed by the commit? This metric intends to indicate the impactfulness of the changes made by the commit.

2. Total number of LOCs written by the developer.

3. Frequency of the commits. This is a metric to measure the speed of the developer.

(5 Developer Metrics Every Software Manager Should Care About, 2021)

Metrics covered in this section are going to be discussed and used to construct the developer’s contribution formula further in the practical part of this thesis.

(22)

5. RESEARCH PROCESS

5.1 Problem description.

The ultimate goal of this thesis is to create a basic prototype of a unified system that can evaluate developers’ contribution to the project and predict their future performance. In this context “unified” means that the goal is to be able to get all this information based on the version control only, without any additional external information about the project (what is it about, which specific management system does it have, and so on). However, it is important to be aware of the information about the software development process specifics and possible methodologies to interpret the system output and evaluate how well does it work on a particular project.

This requires solving at least two ”research” problems (besides the engineering ones):

1. How to evaluate developer’s performance based on information found in version control.

2. How to predict with this particular type of data. It is important to find out, what type of problem are we dealing with, which machine learning algorithms are suitable for this case and how should they be configured.

What is essential to understand about this task – there is no aim to provide a perfect solution for any of these aspects. The goal is to deliver some basic reasonable enough way of solving each of these tasks. As a result, if somebody wants to create such a system, this software could be used in a pure form or modified to more narrow purposes.

Future developers will be able to get from this work:

1. Basic reasonable considerations to start with about developer’s contribution evaluation.

2. An understanding of what is essential in the prediction system and some reasonable approaches for prediction on this type of data for these particular goals.

5.2 Research process description

The key goals were: create an environment that might be independently used with different projects, invent a good enough developer’s contribution formula and find a reasonable way to predict future contribution.

(23)

The research was divided into four main stages:

1. Creating an infrastructure that is able to parse git, extract all necessary information and finally visualize it. At this phase, there was no prediction system and for the developer’s contribution value number of LOCs was used.

2. After the infrastructure was created, it became possible to do experimentations on different projects, modifying a formula in a more complicated way and trying to predict future contribution. So, a set of projects was collected to be able to see how this system is going to behave on various project types: small and big open-source projects, small students’ project.

3. Then, formula development was done in an iterative approach: check how the current formula works on one project, if there are some problems – do some modifications to fix them, if there are no problems – add some new improvements, see how it influence and then decide to leave it or not. Repeat with the next project.

4. In parallel with the previous step, the developers’ contribution prediction part was developed. In contrast, there wasn’t an iterative approach, as there were two possible solutions defined from the very beginning, and after they were implemented the goal was to see which one of them has better performance and which should be used in a resulting system.

Projects that were used during the development process:

1. A small open-source project, 30 developers: https://github.com/gliderlabs/ssh 2. Team Khronos, small students’ project, 3 developers:

https://github.com/vern97/Khronos

3. A small open-source project, 21 developers:

https://github.com/OperationCode/START_HERE

4. Hoodie, medium open-source project, 160 developers:

https://github.com/hoodiehq/hoodie

(24)

6. PROTOTYPE IMPLEMENTATION

The created prototype consists of four parts: tool which parses, and processes developers’ information based on git; an algorithm that evaluates their per week contribution on the given period; machine learning part which predicts their future contribution and the drawing part, which displays all calculated information. As project information extraction and results displaying tool were just instruments to make the research process and software functioning possible, we are not going to discuss them in the further parts and falling back to focusing on developers’ contribution formula and prediction approaches.

6.1 Developer’s contribution evaluation.

The developer’s contribution is a rather subjective notion, which involves various facets.

In this section, we are going to focus on the question: how to take into account various criteria and arrange them into one formula.

6.1.1 Developer’s contribution formula construction

The research was organized in the following way: starting with the simplest version (just calculating LOCs percentage as a measure of contribution) the formula was tested on different developers on several projects in an iterative manner: moving to the better version by adding some modifications. When the result produced by the algorithm contradicts what can be seen with the naked eye from version control, the formula would be modified to achieve more realistic results. For example, someone gets the best score, in a situation when he just has deleted a lot of code or added a huge file with automatically generated code.

At a certain point, it could be seen that no significant errors appeared anymore. This was a challenging moment because there were still a lot of possible improvements, which could be added, but no evaluation system to measure how better or worse they would make the formula. So, the following algorithm of testing improvements was introduced:

 Think about any possible situations when these improvements might cause problems and find out if there any.

 If yes, try to test them on an available set of projects and if it doesn’t cause any new problems, check they don’t make the formula worse.

(25)

 If based on the previous two points these additional features don’t make the formula worse, consider that they make the formula better as they take into account a bigger number of facets.

This is just the basic evaluation approach, which is good enough for use in a basic prototype implementation to maintain validity while adding new components to the formula, however, there still are ways to develop an evaluation system.

Components and facets taken into account in this formula are gathered from the metrics introduced in section 4.3 Software analytics. The finally used metrics are:

 Number of LOCs written by the developer.

 Frequency of the commits.

 The number of files being modified by commit.

6.1.2 Steps of research

As a basic evaluatiom the formula (1) was used.

1. 𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠[𝑖] 𝑐𝑜𝑛𝑠𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛 = 𝐿𝑂𝐶𝑠 𝑤𝑟𝑖𝑡𝑡𝑒𝑛 𝑏𝑦 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟[𝑖]

∑^𝑁_{𝑖 = 0}𝐿𝑂𝐶𝑠 𝑤𝑟𝑖𝑡𝑡𝑒𝑛 𝑏𝑦 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟[𝑖]

Quite rapidly after testing, the following problem has occurred: in the Team Khronos project, there was a developer, who has deleted quite a lot of lines of code. As in the formula, the deleted LOCs counted as a negative number of LOCs, this leads to the negative contribution value. This produces the error: according to this formula deleting LOCs negotiated the positive impact which came from the code adding (the total contribution from adding 10 LOCs and then deleting some other 10 LOCs turned out to be 0).

Taking into account previous considerations, the modified formula version (2) was created in such a way, that LOCs deletion leads to a small positive impact on contribution and doesn’t negotiate impact from the added code.

2. 𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑒𝑎𝑐ℎ 𝑐𝑜𝑚𝑚𝑖𝑡 = 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑖𝑚𝑝𝑎𝑐𝑡 + 𝐿𝑂𝐶𝑠 𝑎𝑑𝑑𝑒𝑑 + 𝐿𝑂𝐶𝑠 𝑑𝑒𝑙𝑒𝑡𝑒𝑑 ∗ 0.3

𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠[𝑖] 𝑐𝑜𝑛𝑠𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛 = 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 [𝑖] 𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑙𝑎𝑠𝑡 𝑐𝑜𝑚𝑚𝑖𝑡

∑^𝑁_{𝑖 = 0}𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠[𝑖] 𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑙𝑎𝑠𝑡 𝑐𝑜𝑚𝑚𝑖𝑡 What was interesting, after this modification nothing special was taken into account yet, but the calculated contribution became consistent with what can be seen with a naked eye from version control.

(26)

The main consideration for further modifications was: if the formula takes into account only the number of LOCs (in any combination) it is not stable for situations when LOCs are not equivalently valuable in different parts of the project. For example, usually programming languages used on the front end are not that concise as the backend ones and this might also lead to disbalance. So, the new goal was to add some additional metrics not purely related to LOCs.

To get away from this instability the number of files modified by the commit metric was introduced, formula (3). The idea behind this is simple: the more files were changed the more impactful the change is.

3. 𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑒𝑎𝑐ℎ 𝑐𝑜𝑚𝑚𝑖𝑡 = 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑖𝑚𝑝𝑎𝑐𝑡 + ((𝐿𝑂𝐶𝑠 𝑎𝑑𝑑𝑒𝑑 + 𝐿𝑂𝐶𝑠 𝑑𝑒𝑙𝑒𝑡𝑒𝑑 ∗ 0.3) ∗ 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑜𝑑𝑖𝑓𝑖𝑒𝑑 𝑓𝑖𝑙𝑒𝑠

𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠[𝑖] 𝑐𝑜𝑛𝑠𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛 = 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 [𝑖] 𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑙𝑎𝑠𝑡 𝑐𝑜𝑚𝑚𝑖𝑡

∑^𝑁_{𝑖 = 0}𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠[𝑖] 𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑙𝑎𝑠𝑡 𝑐𝑜𝑚𝑚𝑖𝑡 The current formula still didn’t take into account any information about the development process. So, the new metric was introduced: developers’ commit frequency, it indicates the developer’s productivity rate. This formula (4) is a final one.

4. 𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑒𝑎𝑐ℎ 𝑐𝑜𝑚𝑚𝑖𝑡 = 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑖𝑚𝑝𝑎𝑐𝑡 + ((𝐿𝑂𝐶𝑠 𝑎𝑑𝑑𝑒𝑑 + 𝐿𝑂𝐶𝑠 𝑑𝑒𝑙𝑒𝑡𝑒𝑑 ∗ 0.3) ∗ 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑚𝑜𝑑𝑖𝑓𝑖𝑒𝑑 𝑓𝑖𝑙𝑒𝑠 𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠[𝑖] 𝑐𝑜𝑛𝑡𝑟𝑖𝑏𝑢𝑡𝑖𝑜𝑛

= 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠 [𝑖] (𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑙𝑎𝑠𝑡 𝑐𝑜𝑚𝑚𝑖𝑡 ∗ ℎ𝑖𝑠 𝑐𝑜𝑚𝑚𝑖𝑡 𝑓𝑟𝑒𝑞𝑢𝑒𝑐𝑦)

∑^𝑁_{𝑖 = 0}𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟′𝑠[𝑖] (𝑖𝑚𝑝𝑎𝑐𝑡 𝑎𝑓𝑡𝑒𝑟 𝑙𝑎𝑠𝑡 𝑐𝑜𝑚𝑚𝑖𝑡 ∗ ℎ𝑖𝑠 𝑐𝑜𝑚𝑚𝑖𝑡 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦)

6.2 Developer’s contribution prediction

The prediction problem can be generalized to the following form: the goal is to fit the curve with minimum information about this curve. From a machine learning point of view, this is not an interesting question, because there is an absolute lack of input information, so there are not so many things to analyze, and, hence, high accuracy is unlikely to be fit. However, in terms of a prototype being created, it is absolutely fine. There is no need for high accuracy when the goal is to see the general trend of how this particular developer is going to work in the future. So, the task is mostly to guess the form of the curve and properly scale it.

Based on the previously discussed formula for developer’s contribution we can say that data is positive values between 0 and 1, which can be considered a monotonously non- descending function. From a machine learning point of view, this could be classified as

(27)

a regression problem. There are multiple approaches to solving this type of task, but here it was decided to choose two of them with a crucial difference: neural network as an example of deep learning and curve fitting as an alternative and more mathematical approach. What is essential to remember: there is no need for high accuracy prediction, so as a solution we are just looking for the models which can guess the general form of the curve and scale it properly, this would mean reasonable accuracy. In this section, we are going to first describe both approaches and then compare them to find out which one of them is more suitable for this task.

6.2.1 Curve fitting approach

Solving a curve-fitting problem normally consists of two parts: choosing the mapping function and finding the optimal parameters. All mathematics related to this task (particularly, finding the parameters calculations) can be easily done with pre-built python function curve_fit, which, in the basic version, accepts the set of dots and a callable function. As a result, there are two things returned:

 Optimal values for the parameters so that the sum of the squared residuals of the predicted and real set is minimized.

 The estimated covariance of these parameters. (scipy.optimize.curve_fit, 2021)

Hence, to fit a set of data using python it is enough to choose the basis function, call curve_fit(set of data, basis function), and then construct this function with returned parameters.

To predict developer’s contribution in the future with this method we can do the following:

take data about their performance throughout the given period (in the past) and fit it with some curve. After we get optimal parameters for fitting this set of existing data let’s construct this curve with these parameters over the next period. This will be a predicted developer’s performance.

So, the key question is to find the best basis function. As the goal is to create a unified system, it can be assumed that there is no additional information on the form of the curve, except the fact that it is monotonously non-descending, which is defined by the contribution formula. Then a possible solution was so: choose some of the most common and appropriate in this case (monotonously non-descending) functions, try them on each particular data set, choose the one which fits the training set the best and use it for the predicted curve construction. But how should it be measured: which one fits the best?

(28)

RMSE was used to measure error and chose the one which is the most similar to the output.

In this implementation, we used the following functions: logarithmic, sigmoid, power, and polynomial, given as formulas 1-4 in the following, respectively.

1. 𝑦 = 𝑎 ∗ 𝑒^𝑥+ 𝑏 2. 𝑦 = ¹

(1+ 𝑒^{−𝛼∗(𝑥− 𝛽)})

3. 𝑦 = 𝑎 ∗ 𝑥^𝑏

4. 𝑦 = 𝑎 ∗ 𝑥⁵+ 𝑏 ∗ 𝑥⁴ + 𝑐 ∗ 𝑥³+ 𝑑 ∗ 𝑥²+ 𝑒 ∗ 𝑥 + 𝑓

It turns out that the curve fitting approach in this implementation was not stable. If there was a curve in the set of basic curves which could correspond to the final form with some parameters and it differs enough from all others to be chosen after fitting the training set its predictions were close to perfect. However, if there was no such function it might provide something completely wrong. In Figure 6 there are two examples of a rather bad and quite good prediction using this method.

Figure 6.Example of a rather bad (left) and a quite good (right) fit.

Finally, the average accuracy tested on 64 developers from 4 different projects is 0.386.

This isn’t a high accuracy value. This means that there were several situations when it hasn’t found an appropriate function and provides some inconvenient result (much worse than the one presented above).

This method could be improved by adding a bigger set of different base functions.

However, in the current form, it provided a bad quality in terms of the task, because in some of the cases the form of the curve wasn’t matched totally and this is one of the key things required to be achieved.

(29)

6.2.2 Neural network approach

The other hypothesis was to take a neural network and see whether it is going to be better or worse than the curve fitting.

Tensorflow python library was used to construct and train the neural network. The ways to use this library to configure, compile and train neural network models are discussed in section 2.2.4 Practicalities on using neural networks.

Program 3 illustrates the model configuration: Sequential model with 3 layers and sigmoid activation function, trained with 2500 epochs.

Different combinations of layers number and epochs were tried, as a result, the maximum average accuracy was achieved in this combination. When adding more layers or increasing the number of epochs the overfitting occurred.

Average accuracy achieved by the neural network tested on 4 projects and 64 developers: 0.00091514482. This is an RMSE value and by itself it isn’t supposed to tell anything about fitting the initial requirement of predicting the curve form in a proper scale.

But, as it was illustrated in section 2.2.3 Root mean squared error values meaning, RMSE equal to 0.065 corresponds to the wrong curve form fit and RMSE equal to 1.24

* 10^-6 corresponds to the precise curve form in a proper scale. Hence, it can be considered as a satisfactory result because this accuracy is rather high and even in the situations when the neural network doesn’t fit ideally the numerical values, there are still no essential errors with the general form of the curve which is a key thing required in terms of this task. As a final prediction approach, the neural network was chosen.

1 3 5 7 9 11

def get_neural_network_model(x_train, y_train):

model = tf.keras.Sequential([

tf.keras. Layers.Dense(8, activation='sigmoid'), tf.keras.layers.Dense(8, activation='sigmoid'), tf.keras.layers.Dense(1)

])

model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(0.1))

model.fit(x_train, y_train, epochs=2500, verbose=False) return model

Program 3. Illustration of how the model was constructed.

(30)

6.3 Final results.

As a final result, the discussed system was developed and created.

As an input it gets

 Project local repository.

 Period of time (in the past) we are interested in to see the performance.

The output is the set of plots presenting developers’ performance during the given period and predictions done for the period of the same length in the future.

The developer’s contribution is calculated with the formula illustrated by Program 4.

The average prediction accuracy is 0.00091514482.

In Figure 7 there is an example of part of the output for the gliderlab-ssh project (small open-source project, 30 developers.

Figure 7. Example of the created system output.

1 3 5 7

Developer’s impact after each commit = developer’s current impact + ((LOCs added + LOCs deleted * 0.3) * number of files being modified).

Developer’s contribution = developer’s impact after his last commit * developer’s commit frequency / sum of all (developer’s[i] impact after his last commit * developer’s[i] commit frequency)

Program 4. Final developer’s contribution formula presentation.

(31)

7. FUTURE RESEARCH OPTIONS

As the research period wasn’t long, there was no possibility to elaborate on all versions and hypotheses that were suggested and discussed during system creation. However, as a prediction part achieves the required result (to fit the general curve form) there is no need to improve it and in this section, we are going to focus on possible future research for developer’s contribution formula.

The very first thing, which could be studied: based on the current results – what is parameters hierarchy, which of them influence at most? Lines of code, deletions, frequency of commits, or how many files have been changes applied to.

The second thing, a list of factors that has been discussed during the research, but there was no time to add them to the system and study properly:

 How many commits are made? Here, probably, could be added an “artificial”

consideration – one big or several smalls probably might be considered as a single task and then we can measure in some way number of tasks have been done by a developer.

 How many times code has been modified within 21 days after it had been added?

 Try to find a common “team commit style” and this also will give a possibility to better interpret different developer’s behavior.

 Adding some relative coefficient – how well did they work relative to others in the same period of time.

 Parse words such as “add” and “fix” in a commit message to consider that it was a single task.

The third thing is studying system behavior on a big number of different types of projects (big open-source, small pet projects, etc) and see if there are some specific characteristics as the general goal is to make a unified system.

(32)

8. CONCLUSIONS

The initial research question was to create a capable system, based on the project’s code repository (and only it), of evaluating developers’ contribution and predicting their further performance on the project. When the research just started it seemed that the most significant part during the implementation and research process will be contribution evaluation formula construction. However, it appeared to be just a single part of the com- pound task.

The main goal during the research process was to balance between all aspects involved:

practical environment implementation (parsing git, plotting the results), research related to developer’s contribution evaluation, and their future results prediction. On the one hand, they all had to be elaborated to such a degree, that the final system is useable in practice and its results are reasonable enough, on the other hand, due to the limited period it was important not to pay too much attention to one of them, depriving the others.

It can be considered that the prediction part at the current point is completely satisfactory and there is no need to significantly improve it, as it already fits the initial requirement:

provide a general understanding of future developer’s performance manner.

Contribution and performance evaluation was a trickier part, as it is a rather subjective notion, which involves different facets and activities and not all of them can be easily retrieved from a version control system. Hence, as it was mentioned in the ”Future research options” part there are still many criteria, that can be tested and probably included in the formula.

All in all, I suppose that research and development results can be considered successful.

The basic system version was created and even though there are still many ways and directions on how it can be improved, its performance is good enough with concise with initial requirements.

(33)

9. REFERENCES

[1] 5 Developer Metrics Every Software Manager Should Care About. (30. March 2021). Noudettu osoitteesta pluralsight.com:

https://www.pluralsight.com/blog/teams/5-developer-metrics-every-software- manager-should-care-about

[2] Aggarwal C. (2018). An Introduction to Neural Networks. Neural Networks and Deep Learning, 1-52.

[3] (2011). Business Process Model and Notation. OMG.

[4] Chandramouli S., Dutt S., Das A.K. (2018). Machine Learning. Pearson Education India.

[5] Chugh A. (8. Dec 2020). MAE, MSE, RMSE, Coefficient of Determination, Adjusted R Squared — Which Metric is Better? Noudettu osoitteesta

medium.com: https://medium.com/analytics-vidhya/mae-mse-rmse-coefficient- of-determination-adjusted-r-squared-which-metric-is-better-cd0326a5697e [6] Daud N. M. N., Bakar N. A. A. A., Rusli H. M. (2010). Implementing rapid

application development (RAD) methodology in developing practical training application system. International Symposium on Information Technology. Kuala Lumpur: 1664-1667.

[7] Dreyfus G. (2005). Neural Networks. Springer, Berlin, Heidelberg.

[8] gliderlabs. (ei pvm). Noudettu osoitteesta github.com:

https://github.com/gliderlabs/ssh

[9] Hastie T., Tibshirani R., Friedman J. (2009). Overview of Supervised Learning.

The Elements of Statistical Learning, 9-41.

[10] Linear regression models. (5. April 2021). Noudettu osoitteesta people.duke.edu: https://people.duke.edu/~rnau/compare.htm

[11] M. Beitollahi, S. A. Hosseini. (2016). Using curve fitting for spectral reflectance curves intervals in order to hyperspectral data compression. 10th International Symposium on Communication Systems (ss. 1-5). Prague: Networks and Digital Signal Processing (CSNDSP).

[12] Mann. (2020). Quick Guide to Evaluation Metrics for Supervised and Unsupervised Machine Learning. Teoksessa Data Science Blogathon.

[13] Martin J . (1991). Rapid application development. Macmillan Publishing Co.

[14] Menzies T., Z. T. (September/October 2018). Software Analytics: What’s Next?

IEEE Software, ss. 64-70.

[15] Menzies T., Zimmermann T. (2013). Software Analytics: So What? IEEE Software, ss. 31-37.

[16] Mitchell T. (1997). Machine learning. McGraw-Hill Science/Engineering.

(34)

[17] Model training APIs. (23. March 2021). Noudettu osoitteesta Keras API reference: https://keras.io/api/models/model_training_apis/

[18] OMG. (2013). BPMN2.0.2.

[19] Parizi R. M., Spoletini P., Singh A. (2018). Measuring Team Members’

Contributions in Software Engineering Projects using Git-driven Technology.

IEEE Frontiers in Education Conference (FIE) (ss. 1-5). San Jose: IEEE.

[20] Ratsch G. (2004). A Brief Introduction into Machine Learning. Noudettu osoitteesta https://events.ccc.de/congress/2004/fahrplan/files/105-machine- learning-paper.pdf

[21] Regression losses. (23. March 2021). Noudettu osoitteesta Keras:

https://keras.io/api/losses/regression_losses/#meansquarederror-class [22] Richard S. Sutton, A. G. (2018). Reinforcement Learning, second edition: An

Introduction. The MIT Press.

[23] Rossi M., Taisch M., Terzi S. (2012). Lean product development: A five-steps methodology for continuous improvement. 18th International ICE Conference on Engineering (ss. 1-10). Munich: Technology and Innovation.

[24] Rus. (2013). Natural Language Processing. Encyclopedia of Sciences and Religions.

[25] Ruth S., Aguilar-Savén. (2004). Business process modelling: Review and framework. International Journal of Production Economics, 129-149.

[26] S.Balaji, Dr.M.Sundararajan Murugaiyan. (2012). WATEERFALLVs V-MODEL Vs AGILE: A COMPARATIVE STUDY ON SDLC. International Journal of Information Technology and Business Management.

[27] scipy.optimize.curve_fit. (13. March 2021). Noudettu osoitteesta SciPy.org:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.ht ml#scipy-optimize-curve-fit

[28] Sharma S., Sharma S. (2010). ACTIVATION FUNCTIONS IN NEURAL.

International Journal of Engineering Applied Sciences and Technology.

[29] Silaparasetty N. (16. March 2021). Machine Learning Programming with Tensorflow 2.0. Noudettu osoitteesta oreilly:

https://www.oreilly.com/library/view/machine-learning-

concepts/9781484259672/html/491271_1_En_11_Chapter.xhtml [30] Singh P., Manure A. (2019). Learn TensorFlow 2.0: Implement Machine

Learning and Deep Learning Models with Python. Apress.

[31] Srivastava A., Bhardwaj S., Saraswat S. (2017). SCRUM model for agile methodology,. International Conference on Computing (ss. 864-869). Greater Noida: Communication and Automation (ICCCA).

[32] Stinis, P. (2019). Enforcing constraints for time series prediction in supervised, unsupervised and reinforcement learning. Advanced Computing, Mathematics and Data Division.

(35)

[33] tf.keras.layers.Dense. (17. March 2021). Noudettu osoitteesta tensorflow.org:

tensorflow

[34] Wei W., Hongwei D., Jin D., Changrui R. (2006). A Comparison of Business Process Modeling Methods. Beijing: IBM Research Report.

[35] White, S. A. (2004). Introduction to BPMN. BPTrends.

[36] Willmott C. J., Matsuura K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Center for Climatic Research, Department of Geography.

[37] Young D. (2013). Software Development Methodologies.

Evaluating and predicting developers' contribution in software projects

Evgeniia Onosovskaia