Entity relationship extraction from Chinese electronic medical records based on feature augmentation and cascade binary tagging framework

1.   Introduction

With the progress of information technology and the internet, the utilization of electronic medical records (EMRs) has skyrocketed worldwide. In the United States, this percentage rose drastically from 10% to almost 96% within a decade (2008 to 2017). Similarly, in China, the increase reached 85% ^[1]. Electronic medical records, serving as a crucial source of clinical data, encompass rich medical information, including patient's chief complaints, medical history, examination items and results, medication details, treatment plans, and more. Extracting such valuable information from unstructured text to construct triplet data constitutes a pivotal prerequisite for subsequent related research tasks, such as disease risk prediction ^[2,3], clinical knowledge bases ^[4,5], and assisted diagnosis ^[6,7]. In the past, the extraction of entity relations from electronic medical records was a manually intensive task that was not only time-consuming but also relied highly on physician's experience, leading to potential omissions and errors. Therefore, efficient and accurate methods for electronic medical record entity relationship extraction are of immense importance.

Until now, the extraction of entity relations from Chinese electronic medical records has continued to pose significant challenges. First, there is a scarcity of high-quality medical datasets in the Chinese domain compared to English EMRs, posing a limitation for comprehensive research. Second, Chinese semantic expression exhibits exceptional complexity. On one hand, with the same entity, Chinese can potentially convey vastly different meanings in varying contexts. On the other hand, Chinese has few strict entity boundaries, such as spaces between words, which complicates accurate extraction. Third, medical text tends to be longer and more information-dense than ordinary text, leading to an increased difficulty in extracting triplets as their quantity within the text increases.

To address the aforementioned challenges, we constructed a modest-scale document-level dataset, named VSCMeD, utilizing real Chinese electronic medical records. Furthermore, we introduced a network model based on a feature augmentation and cascade binary tagging framework (FA-CBT). This model divides the task into two parts: subject decoding and relation-object joint decoding. Initially, to enhance the encoding of lengthy electronic medical records, we incorporated a pre-trained model that is specifically tailored for Chinese natural language processing tasks. The subject classifier then predicts all potential subjects within the given sequence. Subsequently, our model combines subject features, sequential features, and semantic features through a dedicated feature augmentation layer. This comprehensive, fused information is then processed by the relation-specialized object decoder, ultimately yielding the predictive outcome. This approach aims to enhance the precision and efficiency of entity relation extraction from complex Chinese electronic medical records.

Our main contributions can be summarized as follows:

1) We built a real Chinese document-level electronic medical record dataset named VSCMeD, including genuine EMRs from over 500 vascular surgery patients. The maximum length of the data entries exceeded 1,000 characters.

2) We proposed a network model based on feature augmentation and cascade binary tagging framework (FA-CBT), for feature deterioration in lengthy texts and entity overlapping. Experimental results on VSCMeD demonstrated that the highest achieved precision is 87.82%, with a recall rate of 88.47%.

2.   Related works

Relation extraction has become one of the most important tasks in natural language processing due to its significant implications. In recent years, a series of studies on relation extraction have been published.

The relation extraction task is often decomposed into a pipeline of two independent tasks: entity recognition (NER), which extracts all entities from the text, and relationship classification (RC), which predicts whether a relation exists between the extracted entities [8−13]. Most studies have used traditional methods like conditional random fields (CRF) ^[14], recurrent neural networks (RNN), and RNN-based improvements like long short-term memory (LSTM) ^[15]. Wei et al. ^[16] pioneered a triplet extraction approach based on a cascade binary tagging framework. Although the method is still a two-step pipeline, it treats relation extraction as a mapping from subject to object and relation and can handle entity overlapping issues. This paper also adopts this tagging and decoding framework.

The two models in the pipeline approach are independent of each other, ignoring potential correlations between the tasks, and can cause error propagation issues. Therefore, recent researchers have started to view relation extraction as a complete task, known as the joint approach, which directly extracts entities and relations between them from text without determining entity types. At the same time, pre-trained BERT ^[17]-derived models have also been widely adopted, and many pre-training models targeting the medical domain have emerged, such as PubMedBERT ^[18], ClinicalBERT ^[19], and MedBERT ^[20], which have greatly facilitated the development of medical natural language processing. Wang et al. ^[21] designed an entity-aware self-attention layer based on BERT, showing great performance on standard benchmarks. Shang et al. ^[22] viewed relation extraction as a table-filling task, building a matrix for each relation and using four types of tags to extract subjects and objects under specific relations, which can even handle nested entities. Lu et al. ^[23] proposed a unified framework for general information extraction, enabling unified modeling of tasks including entity extraction, relation extraction, event extraction, sentiment analysis, etc., with good transferability and generalization between different tasks. However, the pipeline approach is not necessarily inferior to the joint approach. Zhong and Chen ^[24] proposed a very simple span-based pipelined approach for entity and relation extraction and established new state-of-the-art results on standard benchmarks.

Specifically in the medical information field, relevant foreign research started earlier, and there are high-quality datasets like i2b2 ^[25], CDR ^[26], MIMIC ^[27,28], etc. Li et al. ^[29] proposed a novel edge-oriented graph neural network based on document structure and external knowledge called SKEoG, which achieved an F1 score of 70.7 on the CDR dataset. Chen et al. ^[30] combined BERT with a one-dimensional convolutional neural network (1d-CNN) to fine-tune the pre-trained model for relation extraction, achieving F1 scores of 70.85 on the i2b2 dataset and 71.56 on the CDR dataset. Sun et al. ^[31] constructed document-level syntactic dependency graphs and encoded them using a graph convolution network (GCN), and then extracted relations by using weighted context features and long-range dependency features, achieving an F1 score of 73.3 on the CDR dataset.

Due to the lack of Chinese medical datasets, domestic related research is relatively scarce. Medical institutions often use internal data for research, which is very difficult to obtain. The China Health Information Processing Conference (CHIP) initiated the Chinese biomedical language understanding evaluation (CBLUE ^[32]) and released the Chinese medical named entity recognition dataset (CMeEE) and the Chinese medical entity relation extraction dataset (CMeIE), attracting more researchers to Chinese information extraction. Chang et al. ^[33] constructed a Chinese dataset consisting of stroke and diabetes electronic medical records and validated their model's performance in extracting overlapping triplets on this dataset and CMeIE. Pang et al. ^[34] constructed a specific relation attention-guided graph neural networks (SRAGNNs) model to jointly extract entities and their relations in Chinese EMRs. They demonstrated improved performance for Chinese medical entity and relation extraction on two Chinese datasets. Zhang et al. ^[35] utilized GCN to capture inter-sentence dependency and studied the impact of different pruning operations on performance. Ye et al. ^[36] proposed a method of subsequence and distant supervision based active learning which was annotated by selecting information-rich subsequences as a sampling unit. They achieved the best performance on the CMeIE dataset.

3.   Method

In this section, we describe our relation extraction approach, which involves extracting triplets— < subject, relation, object > —from electronic medical records based on predefined relationships. We adopt the cascade binary tagging framework. In prior studies, a single subject and its corresponding triplet were randomly selected from a given data entry as a training sample. In our approach, we include all triplets from each data entry, which are distributed together for training, thereby reducing the number of epochs and shortening the overall training time. Moreover, we performed pre-decoding feature processing and augmenting, elevating the performance of the approach. The method can be summarized into two steps: extracting subjects, then extracting relations and objects based on the subjects. It can be denoted as:

Taking the sentence from Figure 1 as an example, "With a medical history of hypertension for five years, the individual self-administers one Compound Hypotensive Tablet per day." The model initially predicts the potential subject as "hypertension" and subsequently forecasts possible objects under predefined relationships. This process results in the formation of the triplet < hypertension, pharmacological treatment for the disease, compound hypotensive tablet > .

The overall model architecture is shown in Figure 1, and the process is described in detail in the following sections.

3.1.   Encoder

The encoder is used to map the input tokens into a low-dimensional vector space, which is the foundation of natural language processing (NLP). In the past, we often used Word2Vec ^[37] to implement this step, but the resulting word vectors were static and performed poorly on specific tasks. The emergence of pretrained models, especially BERT ^[17], has brought milestone changes to the NLP field and been rapidly and widely applied to various NLP tasks. However, BERT and its derived pretrained models can encode at most 512 tokens, while real Chinese EMRs often exceed this length. Due to the uniqueness of the text, simple splitting cannot be applied. ERNIE ^[38] models the lexical, syntactic, and semantic information in the text. Similar to BERT, it employs a multi-layered transformer as its fundamental encoder. However, it enhances pre-trained language models with prior knowledge, avoiding the direct addition of knowledge embeddings. Moreover, it introduces a multi-stage knowledge masking strategy, integrating phrase-level and entity-level knowledge into language representations, and supporting up to 2048 input tokens. And ERNIE 3.0 outperformed the state-of-the-art models on 54 Chinese NLP tasks. Therefore, we choose ernie-3.0-base-zh as the encoder layer, as shown in Eq (2):

where $\left({t}_{1}, {t}_{2}, \dots, {t}_{n}\right)$ represents the input token sequence and $({h}_{1}, {h}_{2}, \dots, {h}_{n})$ represents the feature vector of the corresponding token.

3.2.   Feature augmentation layer

Gate recurrent unit (GRU) 39 is a variant of LSTM with fewer parameters and a simpler structure, enabling greater efficiency. It can effectively capture long-distance dependencies, making it well-suited for relation extraction from electronic medical records. The inputs to the GRU unit comprise the current time step input vector ${x}_{t}$ and previous time step hidden state ${h}_{t-1}$. The implementations of a GRU unit are as follows:

where ${W}_{xz}, {W}_{xr}, \mathrm{a}\mathrm{n}\mathrm{d}\;{W}_{xh}$ are weight matrices for input vector ${x}_{t}$; ${W}_{hz}, {W}_{hr}, {\mathrm{a}\mathrm{n}\mathrm{d}\;W}_{hh}$ are weight matrices for hidden state ${h}_{t-1}$; ${b}_{z}, {b}_{r}, \mathrm{a}\mathrm{n}\mathrm{d}\;{b}_{h}$ are bias vectors; and $z$ and $r$ denote the update gate and reset gate respectively. The output vectors from forward and backward GRU units are concatenated to obtain feature vectors with bidirectional semantics:

UNet 40 has demonstrated astounding performance in medical imaging. Inspired by it, we designed a model for medical text feature augmentation. We perform convolution with 1*3 kernels and stride 1 to capture inter-semantic features in sequences and augment long-distance characteristics. The entire model has 13 convolutional layers, with padding to keep input and output feature vector dimensions identical.

3.3.   Decoder

We utilize two binary classifiers to identify the start and end positions of the subject respectively. For each token, the corresponding positions are two binary classification problems, described as follows:

where ${p}_{i}^{star{t}_{s}}\;\mathrm{a}\mathrm{n}\mathrm{d}\;{p}_{i}^{en{d}_{s}}$ denote the probabilities of the $i$-th token in the input sentence being the start and end positions of the subject respectively, ${W}_{start}, {W}_{end}$ are weight matrices, ${p}_{i}$ is the position feature, and ${b}_{start}\;\mathrm{a}\mathrm{n}\mathrm{d}\;{b}_{end}$ are biases. If the probability exceeds the set threshold, the token is tagged as 1, otherwise 0. In this paper, the threshold is 0.5. Subject $s$ can be denoted as:

where $L$ denotes sequence length; $I\left\{x\right\}$ indicates 1 if $x$ is true, otherwise 0; $star{t}_{s}, en{d}_{s}$ represent binary tags for the start and end positions of the subject; and $\theta$ denotes model parameters. As multiple $star{t}_{s}$ and $en{d}_{s}$ of subjects will inevitably be detected, we follow the nearest match principle, matching the most recent $star{t}_{s}$ and $en{d}_{s}$, marking tokens within the interval as one entity.

The relation-specialized object decoder builds upon previously extracted subjects, performing object extraction given the subject for each relation separately.

where $GRU\left(x\right)$ denotes feeding $x$ through the GRU layer, $CNN\left(x\right)$ denotes passing $x$ through the convolutional layers, ${p}_{i}^{star{t}_{o}}, {p}_{i}^{en{d}_{o}}$ represent the probabilities of the $i$-th token being the start and end positions of the object, and ${v}_{sub}$ denotes subject feature information. Similarly, object o can be denoted as:

where $star{t}_{o}, en{d}_{o}$ denote binary tags for the start and end positions of the object, and ${\mathrm{\Phi }}_{r}$ represents relation-specialized object decoder parameters.

4.   Experiment

4.1.   Dataset

4.1.1.   VSCMeD

The VSCMeD dataset contains 595 authentic electronic medical records from a Chinese top tertiary hospital's vascular surgery department. The maximum sequence length is 1207, with an average of 581. Each EMR encompasses patients' details, chief complaints, present illness history, past medical history, physical examination findings, and preliminary diagnosis. We defined 8 types of relations and had a professional physician annotate them using the open-source tool doccano. The annotated triplet structure is represented as < (sub_s, sub_e), rel_idx, (obj_s, obj_e) > , where sub_s, sub_e, obj_s, and obj_e represent the start and end positions of the subject and object in the original text, and rel_idx indicates the relationship type. Tables 1 and 2 show the statistics. The dataset is split into training and test sets with the ratio of 8:2 for experiments.

4.1.2.   CMeIE-V2

CMeIE-V2¹ is the latest version of CMeIE, with increased task difficulty. CMeIE-V2 has a total of 44 relation types, with maximum sequence length of 300. The training, validation, and test sets contain 14339, 3585 and 4482 instances respectively. As the test set remains unannotated, one must tender model predictions to the CBLUE 32 competition to obtain scores. Table 3 summarizes the distribution of relation quantities per instance in the dataset.

¹ https://tianchi.aliyun.com/dataset/95414

4.2.   Evaluation index

Triplet extraction necessitates identifying the entity span and corresponding relationship type. Hence, a triplet is deemed accurate only when subject, relation and object are correct. We utilize the universally adopted precision (P), recall (R), and micro F1 score (F1) as evaluation metrics. The formulae are as follows:

where $correct$, $recognized,$ and $triplets$ denote the number of triplets predicated correctly, the total number of triplets predicated, and the total number of triplets labeled respectively.

4.3.   Experimental environment and hyperparameters

The experimental environment employed in this work comprised CPU AMD 5900x, GPU RTX3090, Python 3.9.16, PyTorch 1.13.1. In the experiments, the model was trained for 150 epochs with batch size of 4, utilizing the Adam optimizer and dropout ratio of 0.5, GRU hidden size 384. The initial learning rate was set at 3 × 10^-5, linearly decaying to 1 × 10^-8.

4.4.   Baseline

We compared FA-CBT with several other methods:

LSTM-CRF+RC is a widely adopted classic model for relation extraction, first identifying plausible entities before classifying relations. We employ it as our baseline model, reproducing the model using ernie-3.0-base-zh as the encoder, with results derived from our own experiments.

CasRel 16 is an extraction model grounded in cascade binary tagging framework, first extracting all candidate subject entities before identifying the associated relations and object entities. We reproduced the model utilizing ernie-3.0-base-zh as the encoder, with results derived from our own experiments.

OneRel 22 transforms the task into a table-filling task, employing four special tags to determine subject and object boundaries. We reproduced the model using ernie-3.0-base-zh as the encoder, with results derived from our experiments.

UIE 23 is a universal information extraction framework enabling unified modeling across entity extraction, relation extraction, event extraction, sentiment analysis, and other tasks, facilitating robust generalization capabilities between different tasks. We reproduced the model utilizing uie-base as the default encoder, with results derived from our experiments.

BiRTE 41 represents an enhancement over CasRel, devising a bidirectional extraction framework extracting $(subject, relation, object)$ from both directions ${f}_{r}\left(s\right)\to o$ and ${f}_{r}\left(o\right)\to s$, ameliorating propagation error. We reproduced the architecture leveraging ernie-3.0-base-zh as the encoder, with results stemming from our experiments.

FA-CBT is the model devised in this work. It markedly boosts performance through feature augmentation.

4.5.   Results

Table 4 exhibits the outcomes across different methods on VSCMeD. The results demonstrated our proposed approach surpasses all models, as evidenced by the data in Table 4.

To ascertain the impact of the GRU and CNN in the feature augmentation layer, ablation experiments were conducted as well. The results of the ablation studies are delineated in Table 5.

Furthermore, to verify the generalization of our method, we also tested it on the CMeIE-V2 dataset, along with testing OneRel and BiRTE which performed poorly on VSCMeD compared against the best CBLUE baseline. The results are shown in Table 6.

5.   Discussion

We have devised an NLP model for extracting entity relations from Chinese electronic medical records. Compared to baseline models, it achieves over 50% tremendous F1 score improvement on small document-level datasets and surpasses baseline models in both precision and recall on large sentence-level datasets.

Existing research often seeks innovation in entity tagging and decoder to enable one-step extraction of entity relations, as with OneRel 22 and BiRTE 41. Admittedly, these studies have achieved outstanding results on public datasets, but model complexity also increases, making it difficult to learn adequate features from small datasets. As seen in Table 4, the three-dimensional table constructed in OneRel becomes extremely large when learning long sequences, incurring severe performance degradation. BiRTE attempts to improve upon the extraction mode of CasRel but shows no improvement or complementary effect from bidirectional extraction on small datasets. Only the unidirectional extraction CasRel 16 demonstrates decent performance on small datasets. UIE constructs some negative samples during training, with authors affirming through ablation studies that inserting negative samples is effective under small sample settings. In our tests, UIE attained the second highest F1 score on VSCMeD after our model.

On large datasets, as shown in Table 6, OneRel achieves excellent results on short sequence datasets, owing to its unique Rel-Spec Horns Tagging allowing highly accurate boundary identification and thus the highest precision. Meanwhile, the bidirectional extraction framework of BiRTE obtains the highest recall through complementary effects.

The results in Table 5 show that both BiGRU and CNN layers lead to considerable performance improvements. The BiGRU layer benefits the model nicely by extracting contextual features. CNN layer extracts semantic features on top of sequential features, while dampening the negative impact of ineffective features, boosting the model even more.

However, certain limitations persist in this research. As Chen et al. 43 pointed out, all records originate from the same hospital and the relatively consistent writing style may differentially impact different models. We aim to introduce electronic medical records from multiple collaborating hospitals across various departments in follow-up studies, to test model generalization and scalability.

6.   Conclusions

In this paper, we present a novel approach based on feature augmentation and cascade binary tagging framework to extract relations from Chinese electronic medical records, which is utilized to derive useful information from clinical texts. Experimental results indicate that the proposed model exhibits remarkable generalization ability, delivering impressive outcomes on both small and large datasets, and it is capable of handling long text sequences, making it highly suitable for the practical application scenario in Chinese hospitals. This exploration holds significant practical value for clinical inquiries in China.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This research was supported by Zhejiang Provincial Natural Science Foundation of China (LQ22F010006); Basic Public Welfare Research Project of Zhejiang Province (LGF20H150006); National Natural Science Foundation of China (No. 81971688); Medical and Public Health Projects in Zhejiang Province (No. 2020386297); Science Foundation of Zhejiang Sci-Tech University (ZSTU) (20032311-Y).

Conflict of interest

The authors declare there is no conflict of interest.