Deep learning for biomarker and outcome prediction in cancer

(1)

AND OUTCOME PREDICTION IN CANCER

Dmitrii Bychkov, MSc (Tech.)

Institute for Molecular Medicine Finland – FIMM Helsinki Institute of Life Science – HiLIFE

Faculty of Medicine Doctoral Programme in Biomedicine

University of Helsinki Finland

Academic dissertation

To be publicly discussed, with the permission of the Faculty of Medicine of the University of Helsinki,

in Biomedicum Helsinki 1, Lecture Hall 3, Haartmaninkatu 8, Helsinki, on February 18th, 2022 at 14:00.

Helsinki 2022

(2)

(3)

Institute for Molecular Medicine Finland (FIMM) Faculty of Medicine, University of Helsinki Helsinki, Finland

Docent, Nina Linder, MD, PhD

Reviewed by

Associate Professor, Esa Rahtu, PhD Tampere University of Technology Tampere, Finland

Associate Professor, Mattias Rantalainen, PhD Karolinska Institutet

Stockholm, Sweden Opponent

Associate Professor, Lee A Cooper, PhD Feinberg School of Medicine

Northwestern University Chicago Illinois, USA Custos

Docent Nina Linder, MD, PhD

The Faulty of Medicine uses the Urkund system (plagiarism recognition) to examine all doctoral dissertations.

Dissertationes Scholae Doctoralis Ad Sanitatem Investigandam Universitatis Helsinkiensis (13/2022)

ISBN 978-951-51-7889-3 (Print) ISBN 978-951-51-7890-9 (Online) ISSN 2342-3161 (Print)

ISSN 2342-317X (Online) http://ethesis.helsinki.fi Unigraﬁa Oy, Helsinki 2022

(4)

(5)

(6)

(7)

Machine learning in the form of deep learning (DL) has recently transformed how computer vision tasks are solved in numerous domains, including image-based medical diagnostics.

DL-based methods have the potential to enable more precise quantitative characterisation of cancer tissue specimens routinely analysed in clinical pathology laboratories for diagnostic purposes. Computer-assisted tissue analysis within pathology is not restricted to the quantification and classification of specific tissue entities. DL allows to directly address clinically relevant questions related to the prediction of cancer outcome and efficacy of cancer treatment.

This thesis focused on the following crucial research question: is it possible to predict cancer outcome, biomarker status, and treatment efficacy directly from the tissue morphology using DL without any special stains or molecular methods? To address this question, we utilised digitised hematoxylin-eosin-stained (H&E) tissue specimens from two common types of solid tumours – breast and colorectal cancer. Tissue specimens and corresponding clinical data were retrieved from retrospective patient series collected in Finland. First, a DL-based algorithm was developed to extract prognostic information for patients diagnosed with colorectal cancer, using digitised H&E images only. Computational analysis of tumour tissue samples with DL demonstrated a superhuman performance and surpassed a consensus of three expert pathologists in predicting five-year colorectal cancer-specific outcomes.

Then, outcome prediction was studied in two independent breast cancer patient series.

Particularly, generalisation of the trained algorithms to previously unseen patients from an independent series was examined on the large whole-slide tumour specimens. In breast cancer outcome prediction, we investigated a multitask learning approach by combining outcome and biomarker-supervised learning. Our experiments in breast and colorectal cancer show that tissue morphological features learned by the DL models supervised by patient outcome provided prognostic information independent of established prognostic factors such as histological grade, tumour size and lymph nodes status. Additionally, the accuracy of DL- based predictors was compared to other prognostic characteristics evaluated by pathologists in breast cancer, including mitotic count, nuclear pleomorphism, tubules formation, tumour necrosis and tumour-infiltrating lymphocytes. We further assessed if molecular biomarkers such as hormone receptor status andERBB2gene amplification can be predicted from H&E- stained tissue samples obtained at the time of diagnosis from patients with breast cancer and showed that molecular alterations are reflected in the basic tissue morphology and can be captured with DL. Finally, we studied how morphological features of breast cancer can be linked to molecularly targeted treatment response. The results showed thatERBB2-associated morphology extracted with DL correlated with the efficacy of adjuvant anti-ERBB2treatment and can contribute to treatment-predictive information in breast cancer.

Taken together, this thesis shows the potential utility of DL in tissue-based characterisation of cancer for prediction of cancer outcome, tumour molecular status and efﬁcacy of molecularly targeted treatments. DL-based analysis of the basic tissue morphology can provide signiﬁcant predictive information and be combined with clinicopathological and molecular data to improve the accuracy of cancer diagnostics.

(8)

Koneoppiminen syväoppimisen (SO) muodossa on muuttanut, miten tietokonenäön tehtävät ratkaistaan monilla toimialueilla, kuten lääketieteellisessä kuvantamisdiagnostiikkassa. SO- perusteiset menetelmät mahdollistavat tarkemman kvantitatiivisen karakterisoinnin syöpäkas- vainnäytteistä, jotka rutiinisti analysoidaan kliinisen patologian laboratorioissa diagnosointia varten. Tietokoneavusteinen kudosanalyysi ei rajoitu ainoastaan tiettyjen kudosentiteettien määrittämiseen ja luokitteluun. SO:n avulla voidaan suoraan tutkia syövän ennustetta ja syöpähoitojen vastetta.

Tämä väitöskirja keskittyi tärkeään tutkimuskysymykseen: onko syövän ennuste, biomarkke- rien status ja hoidon tehokkuus mahdollista ennustaa SO:lla suoraan kudosmorfologiasta ilman erillisiä värjäyksiä tai molekyylibiologisia testejä? Vastataksemme tähän kysymykseen käytimme digitaalisia hematoksyliini-eosiini (H&E)-värjättyjä kudosnäytteitä kahdesta taval- lisesta kiinteästä kasvaimesta, rinta- ja paksusuolensyövästä. Kudosnäytteet ja niihin liittyvät kliiniset tiedot saatiin Suomessa kerätystä retrospektiivisestä potilassarjasta. Ensimmäiseksi kehitimme SO-algoritmin, jolla poimimme prognostisen tiedon paksusuolensyöpäpotilaista käyttäen ainoastaan digitalisoituja H&E-värjäyksiä. Kudosnäytteistä SO:lla tehty laskennalli- nen analyysi osoitti ihmisasiantuntijaa parempaa suorituskykyä ja ylitti kolmen patologian asiantuntijan antaman yksimielisen viiden vuoden ennusteen syövän lopputulemasta. Seu- raavaksi lopputuleman ennustamista tutkittiin kahdessa erillisessä rintasyöpäpotilassarjassa.

Erityisesti tutkimme koulutetun algoritmin kykyä yleistää syöpäkudosten kokoleikkeistä, jotka olivat peräisin erillisestä algoritmille aiemmin tuntemattomasta potilassarjasta. Rin- tasyövän ennusteen suhteen tutkimme ”multitask learning”-lähestymistapaa yhdistämällä eloonjäämis- ja biomarkkeri-valvotun oppimisen. Tutkimuksemme rinta- ja paksusuolen- syövän osalta osoittavat, että SO-mallien avulla, jotka ovat opetettu potilaan eloonjäämisen mukaan, voidaan kudosmorfologian perusteella saada ennuste, joka on rippumaton aiemmin saatavilla olevista ennustetekijöistä, kuten histologisesta luokittelusta, kasvaimen koosta ja imusolmukkeiden statuksesta. Lisäksi SO-perusteisten ennusteiden tarkkuutta rintasyövässä verrattiin patologien arvioimiin syovän, kuten mitoosien lukumäärä, tuman pleomorﬁsmiin, tubulusten tiehyeiden erilaistumisasteeseen, kasvaimen nekroosiin ja kasvaimen inﬁltroiviin lymfosyytteihin. Tutkimme myös, voiko rintasyöpäpotilailta syöpädiagnosoinnin yhteydessä saaduista H&E-värjätyistä kudosnäytteistä ennustaa molekulaarisia biomarkkereita, kuten hormonireseptoristatusta jaERBB2-geenin monistumista. Tutkimuksemme osoitti, että mo- lekulaariset muutokset löytyvät myös kudosmorfologiasta ja ne voi tunnistaa SO:n avulla.

Lopuksi tutkimme, miten rintasyövän morfologiset piirteet voidaan yhdistää hoitovasteeseen.

Tutkimuksemme osoitti, että SO:n tunnistamaERBB2-positiivisen kasvaimen morfologia kor- reloi anti-ERBB2-liitännäishoitojen tehokkuuden kanssa ja SO:ta voi käyttää ennustamaan rintasyövän lääkevastetta.

Tämän väitöskirjatyön tulokset osoittavat, että SO:n syöpäkudoksen karakterisointi voi olla hyödyllinen syövän ennusteen arvioinnissa sekä, molekulaarisen statuksen ja lääkevas- teen ennustamisessa. SO-perusteinen kudosmorfologinen analyysi voi antaa merkittävää tietoa syövän ennusteesta ja se voidaan yhdistää kliiniseen patologiaan ja molekulaariseen informaatioon tarkemman syöpädiagnosoinnin mahdollistamiseksi.

(9)

Abbreviations xi

List of original publications xiii

1 Introduction 1

2 Review of the literature 3

2.1 Deep learning in computer vision . . . 3

2.1.1 Introduction to artiﬁcial neural networks . . . 3

2.1.2 Network architectures for computer vision . . . 4

2.1.3 Supervised learning . . . 4

2.1.4 Regularisation and model selection . . . 6

2.2 Preparation and visual examination of tissue specimens for research and diagnostic purposes . . . 8

2.2.1 Sample preparation and staining . . . 8

2.2.2 Visual tissue examination in cancer diagnostics . . . 9

2.3 Deep learning for cancer tissue analysis . . . 11

2.3.1 Applications in cancer diagnostics . . . 11

2.3.2 Outcome prediction . . . 11

2.3.3 Biomarker and treatment response prediction . . . 13

2.4 Methodological aspects of survival modelling . . . 14

3 Aims of the study 17 4 Materials and methods 18 4.1 Patient series and samples . . . 18

4.1.1 Colorectal cancer tissue microarray series (I) . . . 18

4.1.2 Breast cancer FinProg tissue microarray series (II, III) . . . 20

4.1.3 Breast cancer FinHer whole-slide image series (II, III) . . . 21

4.2 Digitisation of samples . . . 22

4.3 Computer vision methods . . . 23

4.3.1 Image pre-processing and augmentation . . . 23

4.3.2 Outcome prediction for colorectal cancer . . . 24

4.3.3 Outcome and biomarker prediction in breast cancer . . . 25

4.3.4 Activation maps for biomarker prediction . . . 26

4.4 Performance evaluation and statistical analysis . . . 26

5 Results 28 5.1 Colorectal cancer outcome prediction (I) . . . 28

5.2 Breast cancer outcome prediction (II) . . . 30

5.3 Biomarker prediction in breast cancer (III + unpublished) . . . 35

5.3.1 Receptor status prediction . . . 35

5.3.2 ERBB2-linked morphology and prediction of trastuzumab treatment efﬁcacy . . . 36

5.3.3 Activation maps forERBB2gene ampliﬁcation . . . 37

6 Discussion 39

7 Conclusions 44

(10)

Bibliography 47

(11)

AP Average precision

AUC Area under the receiver operating characteristic curve BCSS Breast cancer-speciﬁc survival

CI Conﬁdence interval

CISH Chromogenicin situhybridisation CNN Convolutional neural network DDFS Distant disease-free survival

DL Deep learning

ECW Enhanced compressed wavelet

ER Estrogen receptor

ERBB2 Erb-B2 receptor tyrosine kinase 2 gene ERM Empirical risk minimisation

FFPE Formalin-ﬁxed parafﬁn-embedded GPU Graphics processing unit

H&E Hematoxylin and eosin

HER2 Human epidermal growth factor receptor 2

HR Hazard ratio

IFV Improved Fisher vector

IHC Immunohistochemistry

LSTM Long short-term memory

MAP Maximum a posteriori probability estimate

ML Machine learning

PH Proportional hazards

PR Progesterone receptor

ROC Receiver operating characteristic curve SVM Support vector machine

TILs Tumour-inﬁltrating lymphocytes

TMA Tissue microarray

WS Whole-slide

WSI Whole-slide image

(12)

" "!!# '! /:7630 + !" "!!# '! /:32680 + !" %.!"!!# !"! /:8340

""#" " "' "!!#' " !! %")

!" #"

" "!" ""!! &""!")

!"!!" " %" " " !""#! "' "!!# ')

!"!!""' !!"!"%

!" '

'#' " "" "")

3,!!"%" $#"9 # "# "% " " .!!# $$ 4,& !!. # $!" $#"""-!# ' 3,#""!$#"# "% ""' " .!!# $$*

%"" " !""#! 4,& !! . # $! "$#"""-!#' 3,$#"# "% " #"'

$ & !! 4,& !! . # $! "!!!!""!"%" .!!" '!"" !!.!# $$ 5," ""# "% .! #""%" "

.!#" "#" !$!#!" !!!!" '" !" """%.. !""! #""!"

!# $!'"""#"

!""#!("" "#"" ""! ! %.!"!!#!!," $ !" "" ! !!"! !"" ! "! " "!!# '"# %" , !#"!!#!""" #!""'" %#""!" #$" " !"#(#,

'!!"!!# !!" "!* & ""!"" ! !" "" !""!!# '!!"!" &" #"" ""!!! !""% ! "!, ""'*" !"!!" !! #" !!#""" " ""

(13)

Publication I Bychkov D, Linder N, Turkki R, Nordling S, Kovanen PE, Verrill C, Wal- liander M, Lundin M, Haglund C & Lundin J "Deep learning based tissue analysis predicts outcome in colorectal cancer"Scientiﬁc Reports8, 3395 (2018).

Publication II Bychkov D, Joensuu H, Nordling S, Tiulpin A, Kücükel H, Lundin M, Sihto H, Isola J, Lehtimäki T, Kellokumpu-Lehtinen PK, von Smitten K, Lundin J

& Linder N "Outcome and Biomarker Supervised Deep Learning for Survival Prediction in Two Multicenter Breast Cancer Cohorts"Journal of Pathology Informatics13:9 (2022).

Publication III Bychkov D, Linder N, Tiulpin A, Kücükel H, Lundin M, Nordling S, Sihto H, Isola J, Lehtimäki T, Kellokumpu-Lehtinen PK, von Smitten K, Joensuu H &

Lundin J "Deep Learning Identiﬁes Morphological Features in Breast Cancer Predictive of CancerERBB2Status and Trastuzumab Treatment Efﬁcacy"

Scientiﬁc Reports11, 4037 (2021).

The publications are referred to in the text by their roman numerals. The original publications are reprinted with the permission of their copyright holders.

(14)

(15)

1 Introduction

Advances in deep learning (DL) techniques start to play an increasingly important role in healthcare in general, and image-based medical diagnostics in particular [1, 2, 3, 4, 5]. Computational image analysis empowered by DL has been used to address several tissue-based diagnostic tasks performed on digitised hematoxylin- and-eosin (H&E) tissue specimens. Examples include detection of cancerous tissue on whole-slide images (WSIs) [6, 7, 8, 9], histological grading of tumours [10, 11], quantification of tissue entities such as mitotic cells [12, 13], necrosis [14] and tumour-infiltrating lymphocytes (TILs) [15, 16, 17]. Addressing these tasks with supervised DL requires a significant amount of image annotations that are typically performed manually by human experts, e.g. pathologists. This approach is prone to potential human bias and does not allow to extract information that is not readily discernible by a human eye [18].

Recent studies go beyond expert-guided algorithms for the analysis of H&E tumour samples [18]. For example, genetic alterations in cancer can cause phenotypic changes in tumours and their surrounding tissue that can be captured with DL [19, 20, 21]. This approach eliminates potential human biases introduced through manual data annotations, thus reducing human labour work and, more importantly, allowing to discover how tumour morphology reﬂects molecular perturbations in cancer.

In the same way, as DL can be trained to predict genetic changes based on tissue morphology, DL algorithms have been applied to predict disease outcome, biomarker status and response to treatment. Studies have shown how DL trained with digitised H&E tissue specimens as input can predict outcome in patients with brain [22], breast [23], colorectal [24, 25], and other cancers [26, 27]. Moreover, existing molecular biomarkers can also be captured with DL [18]. For example, estrogen (ER) and progesterone (PR) receptor status in breast cancer are well- established therapy-specific prognostic biomarkers that predict whether a patient is likely to respond to hormone therapy. Similarly, breast cancer patients with ERBB2 gene amplification are more likely to respond to anti-ERBB2 targeted therapy [28, 29]. While prognostic biomarkers aid in patient stratification according to their risk of disease progression or death, predictive biomarkers also provide information on how the patients should be treated and the effect of the therapeutic intervention [30, 31]. Thus, with DL it could be possible to develop novel tissue- based prognostic and predictive biomarkers.

(16)

In this thesis, we study the use of DL algorithms for biomarker and outcome prediction directly from tissue morphology of breast and colorectal tumours. Pre- dictions of disease outcome and the efficacy of molecularly targeted treatments are essential for the decision-making process, management and counselling of patients with cancer. We studied whether DL algorithms trained to predict the therapy- specific prognostic biomarkerERBB2based on morphology only, also can predict the efficacy of anti-ERBB2targeted therapy with trastuzumab in patients with breast cancer. We demonstrate how DL can complement expert-based visual tissue analysis to provide independent prognostic information in breast and colorectal cancer.

(17)

2 Review of the literature

2.1 Deep learning in computer vision

DL has during the last decade emerged as the state-of-the-art method within machine learning (ML) and artificial intelligence (AI) [1]. Recent DL methods have dramatically improved the the accuracy and efficiency of pattern recognition and representation learning in various domains, including self-driving cars [32], machine translation [33], finance [34], arts [35], and healthcare [36]. DL have demonstrated particular success in computer vision tasks related to image classification [9], object detection [12, 16] and segmentation [37, 17], and shown great potential in image-based medical diagnostics [4].

2.1.1 Introduction to artiﬁcial neural networks

Essentially, DL is a reincarnation of artificial neural networks – a broad family of ML models composed of simple computational units called neurons. The basic idea behind artificial neuron dates back to 1958 when a Perceptron was first conceived as a simplified mathematical model of how neurons function in the human brain [38]. Mathematically, an artificial neuron is a scalar product of two vectors or a weighted sum of inputs, followed by an activation function:

a=f

w^Tx+b

. (1)

Here,x∈Rⁿis ann-dimensional input vector andwis a set of weights such that eachx_iis associated with its weightw_i, and f(·)is an activation function. Popular activation functions have been sigmoid and hyperbolic tangent nonlinearities.

The former maps neuron outputs to[0,1]range and has a natural probabilistic interpretation. The latter scales the output values to[−1,1]. Both functions, though, may lead to a problem called vanishing gradients and make deep networks difﬁcult to train [39]. To surpass the problem, other activation functions have been proposed, including the currently popular rectiﬁed linear unit (ReLU) [40].

A neuron in equation (1) can either take raw data values or outputs of other neurons as inputs, suggesting that neurons can be organised in an interconnected structure and constitute artiﬁcial neural networks. Importantly, neural connections have to be organised in an acyclic manner. Typically, neurons are arranged in layers of three types:inputlayers,hiddenlayers andoutputlayers. Networks with at least one hidden layer are often referred to as the Multi-Layer Perceptron (MLP). As

(18)

the number of hidden layers grows, the networks become deep and consequently give rise to DL [1]. Vanilla MLP, also known as a fully-connected network - is the simplest and most generic architecture yet not optimal in, e.g. computer vision applications. A variety of network architectures have been designed to efﬁciently handle different data types, such as image data or data that exhibit temporal dynamic behaviour [41, 42].

2.1.2 Network architectures for computer vision

In computer vision,convolutional neural networks(CNNs) [41, 43] – a special type of feed-forward neural networks that efficiently handle the grid-like structure of images have a central role. As the name suggests, the CNNs are based on the operation called convolution or cross-correlation. This operation performs pattern matching through the multiplication of inputs at each spatial location with a kernel – a three-dimensional matrix of weights [44]. Important properties of the CNNs are sparse connectivity and parameter sharing [44]. These properties significantly reduce the number of model parameters, resulting in improved computational efficacy and reduced memory requirements compared to the fully-connected architecture [44]. Typically, CNNs represent a feature pyramid, where blocks of layers learn intermediate feature representations. Many variants of the original CNN architecture have been proposed in recent years. Popular examples include, AlexNet [45], VGG [46], ResNets [47], Inception [48].

2.1.3 Supervised learning

Many computer vision tasks, including image classification, pixel-level segmentation or bounding-box object detection, can be formalised, for example either as classification or regression and solved in a supervised fashion. In supervised learning [44], a datasetDis represented by pairs{(x⁽ⁱ⁾,y⁽ⁱ⁾)}^N_i=0⁻¹, where each data pointx⁽ⁱ⁾has a corresponding target orlabel y⁽ⁱ⁾, andNis a the size of the dataset drawn from a joint distributionp(x,y). A DL algorithm, e.g. a neural network with a fixed structure is defined as a parametric functionf(x;θ)that provides a mapping between observationsx⁽ⁱ⁾ and corresponding labelsy⁽ⁱ⁾. The goal of learning is to find an optimal set of parametersθ of the model by minimising an objective functionJ(θ):

J(θ) =E(x,y)∼p(x,y)[L(f(x;θ),y)]→min

θ , (2)

whereL(·)is the distance between model predictions and labels – a measure of the

(19)

algorithm’s performance. Since the underlying distributionp(x,y)of the data is typically unknown, the expectationEis calculated across anempiricaldistribution of the observed data ˆp(x,y)that we call atraining set:

E_(x,y)∼p(x,y)ˆ [L(f(x;θ),y)] = 1 N

N−1 i=0

∑

L(f(x⁽ⁱ⁾;θ),y⁽ⁱ⁾). (3)

Minimising the average training error is known asempirical risk minimisation (ERM) [44]. Other approaches to estimateθ exist, e.g. a maximum a posteriori probability estimate (MAP) and maximum likelihood estimate (MLE) [49].

The choice of distance measure L(·) depends on the task. For classiﬁcation problems,cross-entropyis a standard function. A binary version of the cross- entropy (BCE) loss for model f(x;θ)that outputs predictions ˆy∈[0,1]looks as follows:

L_BCE(f(x;θ),y) =−1 N

N−1

∑

i=0

y_ilog(yˆ_i) + (1−y_i)log(1−yˆ_i). (4)

Extensions of the original cross-entropy were proposed, for example afocal loss was introduced to address class imbalance problem for CNN-based object detectors [50]. When f(x;θ)solves a regression problem such that ˆy∈R,mean squared error(MSE) loss is a standard choice:

L_MSE(f(x;θ),y) =−1 N

N−1

∑

i=0

(y_i−yˆ_i)². (5)

As the loss function is deﬁned, the training procedure aims to minimise the loss by iteratively updating the parameters of the model. The gradient of the loss function with respect to the parameters of the model deﬁnes the best direction along which the parameters should be changed:

θL(f(x;θ),y) =∂L

∂θ = ∂L

∂θ1, ∂L

∂θ2,..., ∂L

∂θn

. (6)

Calculating ﬁrst-order partial derivatives of the loss with respect toθis done using thebackpropagationalgorithm [51] or the chain rule [49]. Once the gradients can be computed, the Gradient Descent algorithm is applied by repeatedly calculating the gradient and performing a parameter update until a speciﬁc stop criterion is met. In practical applications with large-scale datasets, which is often the case in computer vision, computing the loss function on the entire training set becomes

(20)

problematic. To overcome this challenge, the loss function can be approximated on batches - smaller portions of training data. This is referred to as mini-batch or stochastic mini-batch gradient descent (SGD). The algorithm for implementing mini-batch SGD is given below:

Algorithm 1:Mini-batch Gradient Descent Algorithm Requre: D- training data;

θ←initialise model parameters;

η←initialise learning rate;

whilestop criterion not metdo

{(x^(batch),y^(batch))} ← sample a mini-batch ofmpairs fromD;

Compute outputs: yˆ^(batch) ← f(x^(batch);θ);

Compute gradient: gˆ ← _m¹θL(yˆ^(batch),y^(batch)); Apply update: θ ← θ−ηgˆ

end

The result of the training procedure heavily depends on a hyperparameterηcalled learning rate. Typically, the learning rate is initialised with a small positive value.

e.g. 10⁻³, which deﬁnes the size of a step that SGD takes along the gradient (downhill) at each iteration. Original SGD have seen many modiﬁcations meant to improve the speed of convergence [52]. Some recent and popular versions of SGD include Adagrad [53], Adadelta [54] and Adam [55]. Most of them leverage the idea of the adaptive learning rate for the individual parameters, which is claimed to improve the speed and convergence of the models.

2.1.4 Regularisation and model selection

DL models are often overparameterised, which can lead tooverﬁtting- an undesired effect when a model demonstrates high accuracy on a training set but fails to generalise to a new, unseen set of data. Model generalisation is often assessed by splitting the dataset at hands into three non-overlapping parts:

• Training set – used to estimate model parametersθ;

• Validation set – used for hyperparameter (e.g. number of hidden layers, learning rate) tuning and for early stopping;

• Test set – used to obtain an unbiased estimate of model performance on unseen data.

Often,cross-validationis used for hyperparameter tuning and model selection [56, 57]. Particularly, the dataset is split intoK equally-sized non-overlapping

(21)

folds. Then,K−1 folds are used for training, and thek_thfold is used for validation.

The process is repeatedKtimes, shufﬂing the folds in such a way that a different validation set is used at each iteration.

A common approach used for decades to improve model generalisation is to add a regularisation or parameterpenalty termR(θ)to the equation (3), such that ERM parameter estimates ˆθare obtained by solving equation (7):

θˆ=arg min

θ

1 N

N−1

∑

i=0

L(f(x⁽ⁱ⁾;θ),y⁽ⁱ⁾) +γR(θ), (7)

whereγ∈[0,∞]is a hyperparameter that deﬁnes the strengths of regularisation.

Two popular choices ofR(θ)in neural networks and other ML models arelasso regression[58] andridge regression[59]. The former is know asL1 regularisation and takes the form of∑|θ|, the later is calledL2 and deﬁned as∑θ². Intuitively, L1 leads to sparseθduring training, whereasL2 enforces diffuse values ofθby pushing model parameters towards 0. A combination of both is calledelastic net [60]. It is possible to demonstrate that usingL1 in equation (7) is equivalent to MAP estimate forθ with a Laplace prior [61], whereasL2 becomes equivalent to Gaussian prior [62].

In a supervised setting,multitask learning[63] has been proven effective to yield better generalisation of the models. The "multitasking" is achieved by introducing additional output nodes to the network to predict different but related targets, i.e.

solving several tasks simultaneously. The tasks still share common inputs and hidden layers, hence imposing additional constraints on the parameters of the model [64].

More recently, a dropout technique was introduced to address regularisation speciﬁcally in deep neural networks [65]. It keeps individual neurons inactive during training with some probabilityp(a hyperparameter) and can effectively complementL1 andL2.

Image augmentation[66] methods have an important role in computer vision.

These methods are particularly effective when data labelling for supervised training is laborious, time-consuming and expensive. In that situation, the training set can be significantly enlarged by augmenting existing labelled data and creating new artificial observations. Image transformations often used to generate augmented data include rotations, flipping, shears, adding noise and performing contrast and brightness perturbations [67].

(22)

2.2 Preparation and visual examination of tissue specimens for research and diagnostic purposes

Visual microscopic assessment of tissue morphology, supplemented by molecular methods, such as immunohistochemistry andin situhybridization, are standard methods in cancer diagnostics and research [68]. The cancer tissue morphology, protein and gene expression pattern, characteristics of the tumour microenvironment are associated with the aggressiveness of the disease, risk of recurrence and efficacy of specific treatments [69]. Below we briefly summarise the main steps for tissue preparation, staining and visual examination used for diagnostic and research purposes.

2.2.1 Sample preparation and staining

After a tissue sample is obtained from the patient during surgery or as a biopsy, it requires preparation to prevent tissue degradation and preserve morphological structures, and molecular composition [70]. First, the tissue is immersed and fixed in a formaldehyde (formalin) solution, then dehydrated using ethanol and finally embedded into paraffin [70]. The resulting formalin-fixed paraffin-embedded (FFPE) tissue blocks are then typically cut into 3-5 μm thick sections with a microtome and mounted onto microscopic glass slides for diagnostic and research purposes, and the remaining tissue blocks can be archived for later use [70].

Pathology laboratories typically prepare one or more tissue blocks per tissue specimen, and multiple slides are cut from each block, which subsequently are subject to different stainings. A technique called tissue microarrays (TMA) was developed to gather tissue from multiple specimens into a single tissue block [71].

That is achieved by punching 0.6 - 1 mm cylinder-shaped tissue cores from different FFPE blocks and transferring those to a densely and precisely arrayed recipient block. As a result, up to 100-150 individual patient samples can be represented on the same slide cut from a TMA block and analysed simultaneously [72].

Tissue sections are transparent and almost colourless; thus, they need to be stained or dyed to make tissue components visible. Various staining procedures have been developed to increase contrast, visualise the morphological structures and highlight speciﬁc tissue entities [73]. The most common staining protocol is based on a combination of hematoxylin and eosin (H&E) dyes [68]. Hematoxylin stains the DNA of the cell nuclei dark blue, whereas eosin stains the cytoplasm and extracellular components pink. Other tissue structures take up a combination of those colours with different shades, hues, and textures [68]. Another commonly

(23)

used staining technique is immunohistochemistry (IHC) [74]. In contrast to the H&E, IHC relies on the specific interaction between an antigen and an antibody directed against it. The antibody-antigen interaction is visualised using chromogenic or fluorescent methods. The IHC method is beneficial in identifying and localising specific proteins of interest, individual cells and cell populations [74].

2.2.2 Visual tissue examination in cancer diagnostics

Pathologists examine tissue samples under a microscope to assess morphological characteristics of tumour tissue, including the exact size of lesions, patterns of growth, surgical margins, histological type and grade, the presence and quantity of specific cell types, as well as expression of proteins and genes through molecular methods. Specifically, the histological grade is a measure of the degree of tumour differentiation. Tumours that closely resemble normal tissue with regards to morphological features are considered well differentiated, and tumours that have lost their resemblance with healthy tissue are considered poorly differentiated. Poorly differentiated are more aggressive and likely to metastasise [75]. Grading systems differ depending on the cancer type, e.g., a three-tier grading system in breast cancer is based on a semiquantitative assessment of the following morphological features: percentage of tubule formation, the degree of nuclear pleomorphism, and a mitotic count within a defined tissue area defined by fields-of-view analysed by microscopy [75]. A four-tier standard used for grading colorectal cancer is defined by the degree of glandular structures formation and the least differentiated tumour areas [76, 77]. Tumour size and type, lymph node status and presence of distant metastases constitute the stage of the disease [78]. The TNM classification is the most widely used cancer staging system [78]. Other tissue features that can be assessed by pathologists include the presence of tissue necrosis, immune cell infiltration, cancer-induced angiogenesis, and various stromal features associated with cancer [69].

Molecular pathology is an essential component of cancer diagnostics that sup- plements the examination of tissue morphological features [79]. It is typically performed through IHC to evaluate specific proteins that serve as prognostic and therapy-specific biomarkers. Prognostic biomarkers provide information about patient outcome regardless of treatment, whereas therapy-specific prognostic biomarkers give information on the likelihood that a patient will respond to a certain therapeutic intervention [30]. Molecular characterisation of tumours allows to better understand underlying molecular pathways that drive the disease and tailor the therapy for individual cancer patients. For example, hormone receptor status as assessed by IHC analysis of estrogen receptor (ER) and progesterone receptor (PR)

(24)

status is routinely determined in the diagnosis of breast cancer [28]. Overexpression of HER2 protein – human epidermal growth factor receptor 2, encoded by amplified ERBB2gene, is an established prognostic and therapy-specific biomarker. Amplified ERBB2gene is a molecular alteration that can be targeted with anti-ERBB2therapy, such as trastuzumab [29]. Other biomarkers that complement the diagnosis of breast cancer include proliferative activity of tumours evaluated through IHC analysis of the Ki67 protein [80]. Ki67 is expressed in dividing cells and serves as a prognostic biomarker [80, 81] in the management of various malignancies.

(25)

2.3 Deep learning for cancer tissue analysis

2.3.1 Applications in cancer diagnostics

Deep learning-based approaches within digital pathology and cancer research have already shown promising results in a substantial series of image-based quantiﬁ- cation and classiﬁcation tasks and have the potential to address issues related to reproducibility and throughput in tissue examination within pathology [82]. The development of microscopy scanners, capable of digitising entire tissue specimens and produce so-called whole-slide images (WSIs), allow both visualisation and image analysis of the digital samples for diagnostic and research purposes [83].

Digitisation of large retrospective series of tissue samples combined with comprehensive clinical information, such as information on outcome and treatment [84, 85, 86] provide opportunities for mining knowledge about the disease, improve cancer diagnostics and support clinical decision-making [4].

Conventional tasks in tissue examination within pathology that have been success- fully tackled with DL methods include counting mitosis [87, 13], quantifying tumour-infiltrating immune cells [15, 16, 17], assessing the grade of tumour differentiation [88, 89], segmenting specific tissue entities such as glands [90, 91], stroma [92], vascular structures and other tissue components [93]. Quantification of these tissue entities often serves as intermediate steps to address “higher level”

biological and clinical questions such as response to treatment, need for surgical intervention, and disease outcome prediction. With DL, it has become possible to go beyond expert-supervised tasks in tissue analysis [94, 5] and directly predict clinical endpoints, such as disease outcome [23] and response to treatment [18].

Recent studies demonstrate that using digitised tumour samples stained for basic morphology, it is possible to extract information not readily discernible by the human eye. For example, molecular and genomic alterations can be reﬂected in the tissue morphology and identiﬁed with DL algorithms [95].

2.3.2 Outcome prediction

Early studies that addressed cancer outcome prediction directly from the tumour morphology as revealed by H&E staining relied on conventional image feature extraction followed by a machine learning classiﬁer such as Support Vector Machine (SVM) [96] or logistic regression. Examples include survival prediction in the lung [97, 26, 98] and breast cancer [99]. With the advances in DL methods, researchers have started to adapt convolutional neural networks (CNNs) for feature extraction as a powerful alternative to hand-crafted features.

(26)

CNNs have been trained using tissue entity labels made by experts and supervised learning approaches to quantify tissue entities that are known to be predictive of survival. For example, the tumour-stroma ratio has been shown to be an independent prognostic factor in various solid tumours, including breast, colorectal and lung cancer [100]. A study on colorectal cancer used CNNs to classify tumour and stroma regions in TMAs and demonstrated that a high stroma to tumour ratio was associated with a worse prognosis than a low stroma ratio in patients with rectal cancer [101]. Similarly, in a study on outcome prediction in colorectal cancer, CNNs were used to segment cancer-associated stroma, tumour epithelium, and lymphocytes in WSIs, followed by survival analysis [25]. The results showed that a “deep stroma score” was an independent prognostic factor for patient overall survival with a hazard ratio (HR) of 1.63. In prostate cancer, a pre-trained CNN (Inception-V3) [102] was used for patch-level Gleason pattern [103] segmentation which was then used in risk stratiﬁcation for disease progression [104].

Another group of studies used CNN-derived features to perform outcome prediction with no direct domain expertise involved. One of the first purely outcome- supervised methods used a pre-trained VGG-16 [46] architecture to extract image features from colorectal cancer TMA samples [24]. A recurrent architecture was then trained on VGG feature vectors to jointly aggregate patch-level information and predict five-year cancer-specific survival [24]. A related approach for TMA-based breast cancer outcome prediction combined pre-trained VGG-16 feature extraction, feature pooling with Improved Fisher Vector encoding and classification with SVM [23]. A similar study on breast cancer also combined VGG-16 features with SVM to predict the risk of recurrence based on H&E-stained TMA samples [105]. Fully end-to-end (without intermediate steps) outcome-supervised methods have been studied for survival prediction in patients with brain [22, 106], lung cancer [106], mesotheliomas [107], colorectal cancer [108], and across ten different cancer types [27] based on data from The Cancer Genome Atlas (TCGA) [86]. Moreover, the study on TCGA Low-Grade Glioma (LGG) and Glioblastoma (GBM) cohorts integrated information on genomic biomarkers with morphology-derived features to improve the prognostic accuracy of DL models [22].

Some unsupervised approaches for tissue image subtyping, i.e. clustering, based on visual similarity, were used on digitised tissue samples of cholangiocarcinoma [109], and various cancer types [110] from the TCGA archive. Histological tissue subtypes/clusters were then evaluated using Cox survival regression. A comprehensive study across 28 TCGA cancer types also utilised transfer learning with the Inception architecture to predict patients’ survival [111].

(27)

2.3.3 Biomarker and treatment response prediction

Several studies have investigated the prediction of molecular biomarkers in breast cancer directly from H&E samples. Prediction of ER status was approached with DL [105] and using nuclear morphometric pattern analysis [112] on breast cancer TMA samples. Later, morphological properties of breast cancer were analysed to ﬁnd associations with ER, PR, Ki67, and HER2 expression in TMA tissue specimens [113]. Recent studies expanded receptor status prediction in breast cancer by analysing WSIs of tissue sections [114, 115, 116, 95]. The performance of the DL algorithms used for biomarker prediction in breast cancer has varied depending on the size of the datasets used for training and validation.

In gastrointestinal cancers, microsatellite instability was predicted directly from WSIs of tissue specimens [19]. Moreover, it was demonstrated that messenger RNA expression can be predicted from WSIs of H&E samples [81], and across 28 different tumour types, including breast and colorectal cancers [21].

With DL, it becomes possible to identify morphological features of tumour tissue that could be predictive of a positive response to both chemotherapy [117, 118] and targeted therapy [18]. Most of the cancer therapies are effective only in a subset of patients; thus, a more accurate segregation of responders from non-responders can help to minimise side effects of the treatments [18]. One way to predict treatment response from the H&E tumour tissue samples is through already known molecular biomarkers, e.g. hormone-receptor status andERBB2gene amplification in breast cancer patients [116]. Alternatively, treatment response can be predicted directly from the basic morphology images, without intermediate identifications of predefined molecular biomarkers [18].

Since treatment response information is typically not readily available, few studies explored the feasibility of tissue-based therapy-speciﬁc prognostic biomarkers.

Prediction of response to neoadjuvant chemotherapy using histological tissue images has been investigated in breast cancer patients [117, 118]. Pathological complete response prediction with DL achieved a ROC AUC of 0.847 on a validation set of 117 breast cancer patients [118]. Prediction of response to immunotherapy has been addressed in patients with malignant melanoma and non-small cell lung cancer directly from H&E-stained tissue samples [119, 120].

Similarly to outcome prediction, prediction of treatment response can lead to the identiﬁcation of novel tissue-based biomarkers and have an impact on clinical decision-making [18].

(28)

2.4 Methodological aspects of survival modelling

Statistical methods that model expected time before a particular event of interest occurs, as a function of covariates, are calledSurvival Models[121]. In the context of cancer outcome prediction, examples of such events could be local or regional disease recurrence, development of distant metastasis or death. Each patient is treated as an individual observation followed for a speciﬁc duration of time. We refer to this time as a follow-up time, during which the event may or may not occur. Time- to-event data present challenges that stem from incomplete observations, known as censoredobservations [122].Censoringrefers to the situation when an individual is lost to follow-up for whatever reason, or situations when information, whether the event of interest occurred or not, is not available [122]. Established and widely used methods that deal with censored data include the non-parametric Kaplan- Meier estimator [123], and Cox survival regression [124]. The Cox regression is a linear model with a restrictive assumption that the effect of the covariates does not change over time, i.e. is independent of time. Therefore, the original Cox method is frequently referred to as a proportional hazards regression. Violating the assumption of proportional hazards (PH) may lead to faulty conclusions, though it is rarely checked for in practical applications [125, 126, 127]. Variations of the original Cox method have been proposed with relaxed assumptions [128, 129], as well as alternative approaches such as the accelerated failure time model [130].

However, these approaches are less frequently used.

Extensions of some traditional machine learning algorithms such as survival SVM [131], and Random Survival Forest [132] have been developed to handle censored data. None of the methods described above are designed to work with raw image data and require preliminary image feature extraction. This is where convolutional neural networks have prominent advantages and researchers have started to combine feature extraction with CNNs with established statistical models to perform image- based survival modelling. The most relevant studies in the ﬁeld of digital pathology [105, 24, 104, 101] have been cited in the previous chapters.

More advanced neural network-based approaches have adopted the original Cox Partial Likelihoodfor end-to-end modelling of time-to-event data. This method was ﬁrst described as a DeepSurv neural network [133] and based on the work that dates back to 1995 [134]. In the following paragraphs, we will take a closer look at this method.

Formally, in survival analysis each observationi can be expressed as a triple (x_i,t_i.δi), wherex_i is a set of covariates,t_i>0 is the follow-up time or time-to- event, andδi∈0,1 is the binary event indicator or censoring status. The Cox PH

(29)

model estimates the risk or a probability of the event to occur at timet, given that the individual has survived until that time. This is called a hazard function, and in the Cox framework, it is expressed as follows:

λ(t,x) =λ0(t)exp(β^Tx). (8)

The first term of the hazard function in equation (8) is called the baseline hazard and depends only on time. The baseline hazard remains unspecified; thus, no particular survival time distribution is assumed in the model. That is one of the reasons why the Cox PH model has been commonly used. The second term depends only on the covariatesx_ibut not time. That fact defines the proportional hazard assumption, i.e. the effects of covariates on survival are constant over time;β is a vector of regression coefficients.

It is essential to brieﬂy explain the estimation of the Cox PH model as it directly affects how the model can be adapted for image-based time-to-event modelling with DL. The full maximum likelihood requires to specify the baseline hazardλ0(t). In 1972 a partial likelihood [124] that depends only on the parameters of the model and does not depend on the underlying hazard functionλ0(t)was proposed:

L(β,x) =

∏

i:δi=1

e^hⁱ

∑j:tj≥tie^h^j

, (9)

whereh_i=β^Tx_i- predicted risk for individuali; The corresponding log partial likelihood is:

l(β,x) =

∑

i:δi=1

h_i−log

∑

j:t_j≥ti

e^h^j

. (10)

Minimising the log-likelihood in equation (10) turns the outcome prediction task into a ranking problem. Several studies have used this method for direct outcome prediction from tumour tissue morphology [110, 22, 107, 135].

The likelihood function in equation (10) only considers the subjects that experienced the event by the end of follow-up. Censored observations are included only in the risk set, i.e. in the denominator of the equation (9). Importantly, equation (9) is valid only for continuous-time survival data, which is not the case in most practical applications where multiple events may occur at the same (discrete) time, i.e. resulting in ties. The most common approaches to handle tied data are Breslow [136] and Efron [137] approximations to the discrete likelihood and the exact

(30)

method that considers all possible orders of events that occurred at the same time.

The exact approach becomes computationally intensive as the number of ties grows.

Efron’s method is considered to give a better approximation to the original partial likelihood [138]:

L(β,x) =

∏

i:δi=1

∏j∈Hie^h^j

∏^m−1_l=0 ∑j:t_j≥tie^h^j−_m^l ∑j∈Hie^h^j

, (11)

whereH_iis a set of individuals that failed at timei, andmis a number of ties; The corresponding log likelihood looks as follows:

l(β,x) =

∑

i:δi=1

j∈H

∑

i

h_j−^m−

∑

¹

l=0

log

∑

j:t_j≥ti

e^h^j−(l/m)

∑

j∈Hi

e^h^j

. (12)

Still, most practical applications within cancer research and pathology have relied on the Breslow approximation or original likelihood, ignoring ties. A non-vision- based study on patient-speciﬁc kidney graft survival analysis with DL-adapted Efron’s method [139].

Cox Partial Likelihood adaptation in DL has its limitations, e.g. lack of hazard proportionality [127]. AComplete Hazard Ranking(Guan Rank) method was proposed to address some of the limitations [140]. The Guan Rank algorithm assigns hazard ranks to all observations in the training set, including the censored ones. That provides a complete set of labels for subjects and allows to train arbitrary regression algorithms, e.g. a DL model in an end-to-end fashion [140].

(31)

3 Aims of the study

The overall aim of this doctoral thesis was to investigate whether patient outcome and tumour biomarker status can be predicted from cancer tissue morphology by deep learning applied to digitised H&E-stained tumour specimens.

Speciﬁcally, the aims were:

1. To assess whether patient outcome can be predicted from tissue morphology of breast and colorectal cancer samples using outcome and biomarker supervised deep learning

2. To study ifERBB2gene ampliﬁcation can be predicted directly from tissue morphology in breast cancer

3. To assess if the efﬁcacy of a molecularly targeted treatment in breast cancer can be predicted based on tissue morphological features learned through biomarker supervised deep learning

(32)

4 Materials and methods

4.1 Patient series and samples

4.1.1 Colorectal cancer tissue microarray series (I)

In Study I, we investigated whether patient outcome can be predicted based on digitised H&E-stained tissue samples of primary colorectal cancer using an outcome supervised DL approach. The samples originated from a retrospective cohort of 641 consecutive patients diagnosed with colorectal cancer. All patients underwent primary tumour resection at the Helsinki University Central Hospital in 1989-1998 [141]. Tissue cores were punched from the most representative areas of the original formalin-ﬁxed and parafﬁn-embedded (FFPE) tumour blocks, i.e. typically from the least differentiated parts of the tumour and assembled into tumour TMA blocks.

Then, the TMA blocks were cut into four-micrometre thick sections, stained for basic morphology (H&E) and digitised with a WS scanner.

Patient survival data, i.e. follow-up time and disease outcome were available for each of the patients. This information was obtained from the Finnish Population and Register Centre and Statistics Finland. Clinicopathological characteristics related to the patients were extracted from pathology reports and included histological grade and Dukes’ stage of the disease. Additionally, each TMA spot, representing a patient’s tumour, was classiﬁed by three pathologists low-risk and high-risk groups. The experts were blind to patient outcome and were guided purely by the morphology of each TMA sample. A consensus score deﬁned by a majority vote among the three experts was derived and referred to as a Visual Risk Score. The Visual Risk Scoring was performed to allow direct comparison of expert-based and DL-based patient outcome prediction (Table 1).

A total of thirty-nine patients were excluded from the analysis. Twenty-four patient samples were detached or had no tumour tissue. Fifteen patients were excluded due to misdiagnosis or postoperative death.

Ethical approvals were obtained from The Hospital District of Helsinki and Uusimaa (Dnro HUS 226/E6/06, extension TMK02 §66 17.4.2013) and the National Super- visory Authority for Welfare and Health (Valvira Dnro 10041/06.01.03.01/2012).

Written informed consent was not required because patient consent could not be obtained since the study was retrospective and the number of specimens was extensive.

(33)

Table 1: Clinicipathological characteristics of the colorectal cancer patients.

Included Patients All patients

No. of patients 420 641

Age at diagnosis (years)

< 50 53 (12.6%) 77 (12%)

50 - 64 123 (29.3%) 189 (29.5%)

65 - 74 145 (34.5%) 216 (33.7%)

75 99 (23.6%) 159 (24.8%)

Average 65.4 65.9

Gender

Male 227 (54%) 340 (53%)

Female 193 (46%) 301 (47%)

Stage

Dukes’ A 51 (12.1%) 93 (14.5%)

Dukes’ B 141 (33.6%) 231 (36%)

Dukes’ C 114 (27.1%) 166 (25.9%)

Dukes’ D 114 (27.1%) 149 (23.2%)

NA 0 (0%) 2 (0.3%)

Histological grade (WS)

Low (I-II) 285 (67.9%) 439 (68.4%)

High (III-IV) 135 (32.1%) 200 (32.5%)

NA 0 (0%) 2 (0.3%)

Visual Risk (TMA)

Low 173 (41.2%) -

High 225 (53.6%) -

NA 22 (5.2%) -

(34)

4.1.2 Breast cancer FinProg tissue microarray series (II, III)

In Study II and III, breast cancer H&E-stained FFPE TMA tumour samples were collected from the FinProg original series [85] and from the FinProg validation series [142]. The original FinProg series is a nationwide cohort that consists of 93%

of all breast cancer cases diagnosed in 1991 and 1992 within ﬁve selected geograph- ical regions in Finland [143]. The FinProg validation cohort included 565 women treated for breast cancer at the Departments of Surgery and Oncology, Helsinki University Hospital in 1987-1990. The combined FinProg and FinProg validation set included 2,313 patients. A total of 1,047 patient samples were available for analysis after the exclusions (Table 2) (described in Study II, supplementary ﬁgure 1).

Table 2: Clinicipathological characteristics of the FinProg patients.

Training and tuning

N = 693

Test set patients N = 354

Included patients N = 1047

All patients N = 1299

N % N % N % N %

Histological grade (WS)

I 98 14.1 68 19.2 166 15.9 226 17

II 244 35.2 127 35.9 371 35.4 450 35

III 168 24.2 68 19.2 236 22.5 273 21

NA 183 26.4 91 25.7 274 26.2 350 27

ERBB2status (CISH)

Negative 557 80.4 288 81.4 845 80.7 944 73

Positive 136 19.6 66 18.6 202 19.3 216 17

Na 139 10

Estrogen receptor

Negative 221 31.9 111 31.4 332 31.7 364 28

Positive 472 68.1 243 68.6 715 68.3 812 63

NA 123 9

Progesterone receptor

Negative 326 47.0 163 46.0 489 46.7 539 41

Positive 367 53.0 191 54.0 558 53.3 638 49

NA 122 9

Cancer-speciﬁc survival

Censored 483 69.7 254 71.8 737 70.4 979 75

Uncensored 210 30.3 100 28.2 310 29.6 205 16

Patient clinical information, tumour characteristics and patient outcome were retrieved from the hospital records, the Finnish Cancer Registry and Statistics Finland. Those included histological grade, tumour size, stage, axillary lymph

(35)

node status, estrogen (ER) and progesterone (PR) receptor expression,ERBB2 gene ampliﬁcation and patient survival (Table 2). Additionally, the following characteristics regarding the patient tumours were assessed by a pathologist via visual examination of the digitized tumour TMA samples:

• Degree of immune cell inﬁltration: Low / High

• Pleomorphism: Minimal / Moderate / Marked

• Mitotic events per 1 high-power ﬁeld: <1 / 1 / >1

• Tubule formation: <10% / 10 - 75% / >75%

• Necrosis: absent / present

Similar to the colorectal cohort described above, a Visual Risk Score (low vs. high) was assigned to each patient based on the visual assessment of the corresponding H&E-stained TMA samples. Follow-up time with disease-speciﬁc outcome was available for each patient.

ER and PR receptor expression was determined for each tumour sample with immunohistochemistry [85]. Quantiﬁcation ofERBB2gene ampliﬁcation was performed by chromogenicin situhybridisation (CISH). Ethical approvals were obtained from The Hospital District of Helsinki and Uusimaa (Dnro 94/13/03/02/2012) and the National Supervisory Authority for Welfare and Health (Valvira Dnro 7717/06.01.03.01/2015).

4.1.3 Breast cancer FinHer whole-slide image series (II, III)

Studies II and III also comprised 712 H&E-stained FFPE WS tumour sections from the FinHer trial (ISRCTN76560285) [144]. The trial included 1,010 women with primary breast cancer that had undergone breast cancer surgery [84]. Expression of ER, PR andERBB2was determined with IHC andERBB2gene ampliﬁcation was conﬁrmed with CISH (Table 3). Patients withERBB2-positive cancer (N=232) were randomly assigned either to receive or not to receive adjuvant anti-ERBB2 treatment (trastuzumab; Herceptin). The study was approved by the institutional review board (HUS 177/13/03/02/2011). Patients’ written informed consent was acquired for further research to be carried out on in their tissue material.