Machine Learning Models

Currently, the most frequently used methods for named entity recognition are machine learning approaches. While some studies focus on purely machine learning-based mod-els, others utilize hybrid systems that combine machine learning with rule-based or dictionary-based approaches. Overall these present state-of-the-art methods.

This section discusses three principal machine learning methodologies utilizing super-vised, semi-supersuper-vised, and unsupervised learning. These also include Deep Neural Net-works (DNN) and Conditional Random Fields (CRF) because newer studies are focused on using LSTM/Bi-LSTM coupled with Conditional Random Fields (CRF). Furthermore, in Section 2.4.4, We will discuss a few hybrid approaches.

Supervised methods

The first supervised machine learning methods used were Support Vector Machines (Kazama et al. 2002), Hidden Markov models (Shen et al. 2003), Decision trees, and Naive Bayesian methods (Nobata, Collier et al. 1999). However, the milestone publication by (Lafferty et al. 2001) about Conditional Random Fields (CRF) taking the probability of contextual dependency of words into account shifted the focus away from independence assumptions made in Bayesian inference and directed to graphical probability models.

Conditional Random Fields

CRFs are an exceptional case of conditionally-trained finite-state machines, where the final result is a statistical-graphical model. These models perform well with sequential data, making it ideal for language modeling tasks such as NER (Settles 2004), in both general and biomedical domains. While conditional random fields are similar to Hidden Markov Models (HMM), they have several differences, the most significant being that CRFs are undirected and conditional probability models. In contrast, HMMs are directed-independent-probability models that do not account for dependencies between the in-put data. Figure 2.2 illustrates the main differences between CRF, HMM, and MEMM (Maximum-entropy Markov Models) in the linear chain context. MEMM, similar to CRF, models conditional probability; however, it is prone to label-bias problems caused by the model’s directed nature. However, CRF models conditional probabilities normalized by local state transition and considers the whole model such that the label-bias problem will not occur. As such, modeling for text sequences with succeeding and preceding contex-tual information, CRF performs much better than MEMM and HMM in NLP tasks.

Figure 2.2. Hidden Markov Model, Maximum-entropy Markov Model and Conditional Random fields represented as a linear chain to illustrate learning with context.

Deep learning

In the last five years, there is a general shift in the literature towards deep neural network models in the machine learning domain, and consequently, in biomedical NLP (Perera et al. 2020; LeCun et al. 2015; Emmert-Streib, Yang et al. 2020). For instance, there have already been BioNER models trained with feed-forward neural networks (FFNN) (Furrer et al. 2019), recurrent neural networks (RNN), and convolution neural networks (CNN) (Zhu et al. 2017) with state-of-the-art performances. Among these, RNNs have been in more limelight due to their ability to model sequential data well. Consequently, several variations of RNNs such as, e.g., Elman-type, Jordan-type, unidirectional, and bidirectional LSTM models have been explored in NER domain (L. Li, Jin and D. Huang 2015).

The Neural Network (NN) language models are essential since they excel at dimension reduction of word representations and help improve NLP application performances im-mensely (Jing et al. 2019). Consequently, (Bengio et al. 2003) introduced the earliest NN language model as a feed-forward neural network architecture focusing on "fighting the curse of dimensionality." This FFNN, which first learns a distributed continuous space of word vectors, is also the inspiration behind CBOW and Skip-gram models of feature space modeling. The generated distributed word vectors are then fed into a neural net-work that estimates each word’s conditional probability in context to the others. However, this model has several drawbacks, the first being that it is limited to pre-specifiable con-textual information. Secondly, it is impossible to use timing and sequential information in FFNNs, which would facilitate language to be represented in its natural state, as a sequence of words instead of probable word space (Jing et al. 2019).

Similarly, convolutional neural networks (CNN) are used in literature to extract contextual information from embedded word and character spaces. In Y. Kim et al. 2016, such a CNN has been applied to a general English language model, with each word represented as character embeddings. The CNN then filters the embeddings and creates a feature vector to represent the word. Extending this approach to Biomedical text processing, Zhu et al. 2017, generates embeddings for characters, words, and POS tagging, which are then combined to represent words and fed to a CNN level with several filters. The CNN, eventually, outputs a vector representing the local feature of each term, which can then be tagged by a CRF layer.

Researchers have later started exploring recurrent neural networks for language modeling to facilitate language to be represented as a collection of sequential tokens. Elman-type and Jordan-Elman-type networks are simple recurrent neural networks, where contextual information is fed into the system as weights either in the hidden layers in the former type or the output layer in the latter-type. The main issue with these simple RNNs is that they face the vanishing gradient problem, making it difficult for the network to retain temporal

information long-term, as benefited by a recurrent language model.

Long Short-Term Memory (LSTM) neural networks were introduced to compensate for both of the weaknesses mentioned in previous DNN models (CNN and simple RNN).

Hence, they are most common and often used in language modeling tasks. LSTMs can learn long-term dependencies through a singular unit called a memory cell, which not only can retain information long time but has gates to control which input, output, and data in the memory to preserve and which to forget. Extension to the above model is bi-directional LSTM: where learning can be done with both past and future information (hence both directions), allowing more freedom to build a contextual language model. In contrast, unidirectional LSTM models learn based on only past data. (L. Li, Jin, Jiang et al. 2016).

For achieving the best results, Bi-LSTM and CRFs models are generally combined with a word-level and character-level embedding in a structure, as illustrated in Fig. 2.3 (Yoon et al. 2019; Habibi et al. 2017; X. Wang et al. 2018; Ling et al. 2019; Giorgi and Bader 2019; Weber et al. 2019). Here a pre-trained lookup table produces word embeddings, and secondary Bi-LSTM is trained to render character-level embedding, both of which are then combined to acquire x₁, x₂, ...., x_n as word representation (Habibi et al. 2017).

These vectors then become the input to a bi-directional LSTM, and the output of both forward and backward paths, h_b, h_f, are then combined through an activation function and inserted into a CRF layer. This layer is ordinarily configured to predict the class of each word using an IBO-format (Inside-Beginning-Outside).

Semi-Supervised methods

Semi-supervised learning is usually used when a small amount of labeled data and a larger amount of unlabeled data are available, which is often the case when it comes to Biomedical collections. If labeled data is expressed asX(x₁, x₂, ...., x_n)−> L(l₁, l₂, ..., l_n) where X is the set of data and L is the set of labels, the task is to develop a model that accurately mapsY(y₁, y₂, ..., y_m)− > L(l₁, l₂, ..., l_m)where m > nand Y is the set of unlabeled data that needs mapping to labels.

Whereas literature using a semi-supervised approach is lesser in BioNER, Munkhdalai et al. 2015 describes how domain knowledge has been incorporated into chemical and biomedical NER using semi-supervised learning by extending the existing BioNER sys-tem BANNER. The pipeline runs the labeled and unlabeled data in two parallel pipelines;

wherein one pipeline, labeled data is processed through NLP techniques to extract rich features such as word and character n-grams, lemma, and orthographic information as in BANNER. In the second pipeline, the unlabeled data corpus is cleaned, tokenized, and run through brown hierarchical clustering and word2vec algorithms to extract word representation vectors, and are clustered using k-means. All of the extracted features

Figure 2.3.Structure of the Bi-LSTM-CRF architecture for Named Entity Recognition.

from labeled and unlabeled data are then used to train a BioNER model using conditional random fields. This system’s authors emphasize that the system does not use lexical features or dictionaries and performs well in the BioCreative II gene-mention task.

Unsupervised methods

While unsupervised machine learning has potent in organizing new high throughput data without previous processing and improving the existing system’s ability to process previ-ously unseen information, it is not often the first choice for developing BioNER systems.

However, S. Zhang and Elhadad 2013 introduced a system, which uses an unsupervised approach to BioNER with the concepts of seed knowledge and signature similarities between entities.

In the above approach, first, semantic types and semantic groups are collected from UMLS (Unified Medical Language System) for each entity type, e.g., protein, DNA, RNA, Cell type, and cell line as seed concepts to represent the domain knowledge. Second, the candidate corpora are processed using a noun phrase chunker and an inverse document frequency filter, which formulates word sense disambiguation vectors for a given named entity using a clustering approach. The next step generates the signature vectors for each entity class to indicate that the same classes tend to have contextually similar words. The final step compares the candidate named entity signatures and entity class signatures by

calculating similarities. The method, however, performs only with the highest F-score of 67.2for protein entities. However, Sabbir et al. 2017 using a similar approach to imple-ment a word sense disambiguation with an existing knowledge base of concepts extracted through UMLS, managed to achieve over 90% accuracy in their BioNER model. These unsupervised methods tend to work well when dealing with ambiguous Biomedical enti-ties as well.