A transformer-driven framework for multi-label behavioral health classification in police narratives

1.   Introduction

An individual's behavioral health is crucial because it determines how they interact with others in their community. The substance abuse and mental health services administration (SAMHSA) ^[9] estimates that 53 million adults had a mental disorder in 2020. It is critical to understand that behavioral health includes not just mental health, but also substance use, lifestyle choices, and habits. Behavioral health disorders affect millions of people, from mild to severe cases ^[10].

The rising number of behavioral health-related incidents has put an immense pressure on community resources. With over 240 million 911 calls made yearly, law enforcement personnel spend a significant percentage of their time responding to service calls ^[26]. The standard procedure for each reported incident is for first responders to arrive, provide required assistance to the involved parties, and to file an incident report. Each month, these reports are manually evaluated to identify cases of behavioral health issues, which can often result in incorrect identification ^[12]. This process is not only time-consuming, but also error-prone due to a lack of expertise. Early detection of behavioral and mental health issues is essential for preventing harm to both individuals and others ^[5].

To address this issue, machine learning approaches have been utilized for identifying behavioral and mental health issues. However, these efforts have predominantly focused on identifying specific subclasses of behavioral health, such as schizophrenia, bipolar disorder, anxiety, depression, post-traumatic stress disorder, and other mental health conditions ^{[7,16,18,19,20,36]}, and detecting mental health issues from social media data ^[3,20,33]. LLMs are widely used in several tasks, such as vulnerability detection ^[31], mental health detection via online text ^[37], and domain-specific machine translation ^[41].

Since its inception in 2018, the bidirectional encoder representations from transformers (BERT) model ^[8] was developed as a pretrained transformer trained on a large corpus of text, enabling fine-tuning for specific natural language processing (NLP) tasks. BERT's use of stacking transformers for bi-directional processing led to various variants of the original model, including the robustly optimized BERT pretraining approach (RoBERTa) ^[22], a generalized autoregressive pretraining method (XLNet) ^[39], decoding-enhanced BERT with disentangled attention (DeBERTa) ^[13], and a lite BERT for self-supervised learning of language representations (ALBERT) ^[21]. Transformer-based LLMs, such as generative pre-trained transformer 4 (GPT-4) ^[1], Mistral ^[17], and large language model meta artificial intelligence (LLaMA) ^[34], consist of millions or even billions of parameters, making them highly robust and powerful. The LLaMA ^[34] family includes models with different parameter numbers and configurations. While the family's architecture is similar, each model is unique in its configuration, providing a wide range of capabilities. Currently, the LLaMA model family has just a handful but rising number of models ^[34]. However, pretrained LLMs struggle when assigned a domain-specific task. To achieve the best results on a particular task, LLMs must be fine-tuned using domain-specific data. For our work, we collaborated with subject matter experts (SMEs) to extract ground data from police narrative reports. In the first configuration, we utilized a parameter-frozen RoBERTa model to generate feature embeddings from the dataset, which we then fed into trainable fully connected layers for classification. In the second configuration, we fine-tuned a pretrained LLaMA-3.2-3B model using instruction sets and integrated its feature embeddings with trainable fully connected layers for multi-label classification. We summarize our contributions as follows:

● We analyzed deep learning models, including long short-term memory (LSTM) ^[14], bi-directional long short-term memory (BiLSTM) ^[30], and convolutional neural network (CNN)+LSTM (a hybrid model), for behavioral health multi-label classification in various configurations, utilizing a number of word embeddings such as model-generated embeddings, Word2Vec ^[24], GloVe ^[29], and fastText ^[4].

● We propose a trainable framework with a configurable architecture, providing the following capabilities:

– Classifies police narrative reports into one or more behavioral health classes through multi-label classification.

– Integrates transformer-based models to generate semantic feature embeddings relevant to the behavioral health context.

● We created prompts/instructions sets leveraging police narrative reports to fine-tune LLMs.

● We demonstrated the effectiveness of transformer-based models, in this case, RoBERTa and LLaMa-3.2-3B, were used in our experiments for extracting domain-specific representations.

● Finally, our evaluations show that assessing a model merely on accuracy does not accurately reflect its performance in sensitive domains like behavioral health. A model may excel in detecting a specific class yet fail to capture all of the dataset's variables, resulting in a partial representation of its performance.

2.   Intersection of behavioral health, machine learning and transformers

In this section, we review existing work on the intersection of behavioral health and machine learning in Subsection 2.1, and transformers in Subsection 2.2.

2.1.   Behavioral/mental health and machine learning

According to SAMHSA, approximately 53 million adults in the United States suffered from a mental disorder in 2020 ^[9]. This staggering statistic underscores the crucial need of effective efforts to address mental health issues, which are a major source of public health concerns all over the world. During the same time period, the COVID-19 pandemic began to exacerbate a variety of disorders. According to the study ^[11], the pandemic has undoubtedly increased the number of cases with behavioral health issues. Behavioral health is a critical problem that requires immediate attention since, if not medically diagnosed, it can have an impact on people without being aware of it. In more recent times, machine learning has provided several techniques for detecting a broad variety of mental health issues.

Several studies have focused on detecting mental health categories such as depression, anxiety, schizophrenia, and suicide ^{[16,18,19,20,36]}. Previous research on NLP classification approaches has revealed useful frameworks for dealing with this issue. Recurrent neural networks (RNNs) are commonly used for processing sequential data in NLP tasks, however, they struggle to retain information over extended text sequences due to the vanishing gradient problem, resulting in a tendency to "forget". To address this issue, more advanced variations have been developed, which include LSTM ^[14], BiLSTM ^[30], and gated recurrent unit (GRU) ^[6]. LSTMs tackle the "forgetting" problem in RNNs by utilizing memory cells and gates (input, forget, and output) to maintain long-term dependence. BiLSTMs improve this by processing sequences in both directions, capturing context from previous and future tokens. GRUs simplify LSTMs by combining gates into an update gate, reducing complexity while accurately representing long-range relationships. CNNs ^[27], unlike RNNs, are designed for pattern recognition rather than sequential processing. However, CNNs can capture local features and patterns within text, such as n-grams, making them beneficial to RNN-based models like LSTMs, BiLSTMs, and GRUs for tasks that need both local and sequential context knowledge.

The work ^[23] compared neural network architectures (CNNs, GRUs, LSTMs, and Bi-LSTMs) for classification of text in unstructured medical data. The study ^[2] created a model utilizing self-attention techniques in order to learn contextual features from unstructured data and includes domain expert knowledge by leveraging expert-provided keywords. The integration of context and domain-specific data increases the model's understanding of social and behavioral health cues. While existing machine learning models for detecting behavioral/mental health concerns are reliable, they frequently lack the transparency needed in sensitive domains like behavioral/mental health and healthcare in general.

2.2.   Transformer-based LLMs

Prior to the the introduction of transformers, deep learning models struggled with adequately capturing the context of a document. The transformer model introduced the attention mechanism to focus on the most relevant elements of a document to grasp the key idea ^[35]. This breakthrough has resulted in widespread success in a variety of applications. LLMs are transformer-based models with hundreds of billions of parameters that thrive at understanding natural language and accomplishing complex tasks ^[40]. LLMs can be fine-tuned for domain-specific tasks, as demonstrated by the success of zero-shot approaches in domain-specific machine translation ^[41]. Similar research like ours used online text to fine-tune an LLM for mental health prediction ^[37]. The work (MentaLLaMa) ^[38] proposed an interpretable model for mental health analysis based on ChatGPT's generating capabilities. Another interesting work ^[33] proposes a model with reasoning abilities comparable to human experts in the field of medicine.

Research has shown that fine-tuning LLMs can improve their performance on specific tasks. Typically, fine-tuning has focused on LLMs tailored for specific downstream tasks or settings, which often requires separate fine-tuning for each task. This approach demands extensive training resources, and has deployment and maintenance challenges. Several researchers have investigated LLMs as encoders. Another work ^[28], they utilized a frozen transformer block from pretrained LLMs as an encoder layer to decode visual tokens. This is an interesting approach considering that LLMs are typically used to text data, making this application in the visual domain particularly intriguing. Other researchers have used LLMs to encode clinical data and developed an automated evaluation framework to benchmark LLM performance in the clinical domain ^[32]. A neural semantic encoder was implemented in a different work, with a novel memory update rule and a variable-sized encoding memory that changes over time to preserve comprehension of input sequences via read, compose, and write operations ^[25]. Such an encoder is useful for a variety of natural language tasks, including machine translation and document sentiment analysis.

3.   Transformer-enhanced lightweight framework.

In this section, we will discuss the architecture of the framework for behavioral health analysis, shown in Figure 1.

We leveraged parameter-frozen transformer-based models as an encoder to extract high-dimensional contextual embeddings for each text input. Our design has various components.

3.1.   Instruction set

A hard prompt/instruction is generated to fine-tune the LLM based on the annotations. The input is organized in this prompt generation phase, as shown in Table 1. Usually, it entails developing a set of instructions to guide the LLM to classify the input text into one of more behavioral health classes. Formally, we define the instruction as

where $x$ is the input text (narrative), and $A = \{a_1, a_2, \ldots, a_k\}$ represents the set of annotations or behavioral health class labels provided by SMEs. The function $f$ constructs an instruction set that organizes the input and specifies the task for the LLM, such as

A practical instance of a prompt based on the dataset is presented in Table 1.

3.2.   Feature embedding extraction

A transformer-based model is fundamental to the architecture. The model transforms the input text into high-dimensional contextual embeddings that capture the narrative's semantic meaning, this model acts as an encoder. For the transformer models, we employed RoBERTa ^[22], known for its lightweight design and higher performance over BERT and other models within the family. LLaMA-3.2-3B, based on hardware constraints, is capable of extracting domain-specific details with fewer parameters.

● Input Encoding: Upon tokenizing and embedding the input text $x$, a sequence of token embeddings is generated

where $x_i$ represents the $i$-th token in the input sequence.

● Contextual Embedding Generation: For each token, we compute contextual embeddings using the transformer-based LLM, which produces

where $h_i$ is the high-dimensional contextual embedding for token $x_i$.

● Sentence Embedding: Token embeddings are usually pooled to generate the final embedding $h$ for the input narrative

The output of the transformer-based model is a set of feature embeddings, which are high-dimensional representations of the input text. These embeddings $h$ combine the semantic context of the narrative in a high-dimensional space, enabling the downstream components to leverage this information for classification.

3.3.   Dimensionality reduction and trainable layers

The framework's initial layer performs dimensionality reduction, which optimizes computing efficiency while preserving essential features. This layer uses a linear transformation to reduce the dimensionality of the embedding to 256. The reduced embeddings are then passed to a trainable layer, allowing the model to learn task-specific patterns efficiently.

where $W_{\text{reduce}} \in \mathbb{R}^{256 \times d}$ and $b_{\text{reduce}} \in \mathbb{R}^{256}$ are the learned weights and biases of the layer.

The resulting $h_{\text{reduced}} \in \mathbb{R}^{128}$ is a lower-dimensional embedding that retains the relevant information for multi-label classification. Now, it is then passed to additional dense layers, which further refine and capture relationships between different features within the narrative report text. These layers help in capturing complex interactions between features, enhancing the model's capacity to distinguish between behavioral health classes.

3.4.   Behavioral health classifier

The output layer (classifier) applies a linear transformation to the reduced and refined embedding vector obtained from the last hidden layer. Let $h_{\text{reduced}} \in \mathbb{R}^{256}$ be the output of the last hidden layer. The output layer computes the logits for each behavioral health class as follows:

where

● $W_{\text{output}} \in \mathbb{R}^{\text{num_labels} \times 256}$ is the weight matrix of the output layer.

● $b_{\text{output}} \in \mathbb{R}^{\text{num_labels}}$ is the bias vector of the output layer.

● $\text{num_labels} = \text{len}(\text{bhr_type_classes})$ is the number of behavioral health classes.

Remember that our goal is to classify the input into one or more behavioral health classes, resulting in a multi-label classification task.

where $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid function applied element-wise.

The resulting $\text{output} \in [0, 1]^{\text{num_labels}}$ vector has probabilities showing the likelihood of the input belonging to each behavioral health class.

where $\hat{Y}_i$ is the predicted label for class $i$, and $\theta$ is the threshold.

To optimize the framework during training, we used binary cross-entropy (BCE) loss as the loss function:

where $Y_i \in \{0, 1\}$ is the ground truth label for class $i$.

4.   Experimental setup

In this section, we will provide an overview of the dataset, alongside details about its creation, implementation settings, and the evaluation metrics used to assess model performance in our experiments.

4.1.   Data description and analysis

The data utilized for this work was annotated by an SME who used an annotation tool to extract ground truths from police narrative reports. These reports are classified into one of several behavioral health classes as defined by SMEs. We utilized around 2000 annotated reports for the experiments that we conducted. The classes are presented in the Table 2.

Table 3 provides the statistics of the annotated report done by SME. These statistics allow for a more in-depth analysis of the dataset's representation.

4.2.   Instruction dataset construction

We constructed the instruction sets to adhere to the structure of instruction, input, and output. Where the instruction specifies an action to be performed, the text is the public narrative report, and the output is the class label(s).

To construct a dataset suitable for classification learning in behavioral health analysis, we formatted it in an instruction-like format as shown in Table 1, which allows the model to classify inputs. We achieved this format using a customized function, ensuring consistency and adaptability across a wide range of data.

4.3.   Evaluation

To evaluate the effectiveness of our study, we used evaluation criteria such as precision, recall, f1-score, and accuracy. These metrics have been selected as they provide useful insights into the model's ability to effectively identify and classify behavioral health classes within reports.

4.3.1.   Accuracy

The model's ability to correctly identify the class. It can be expressed as a ratio of precise predictions to total predictions. Mathematically, it is calculated as the ratio of the sum of true positive (TP) and true negative (TN) to the sum of TP, TN, false positive (FP), and false negative (FN).

4.3.2.   Precision

It measures how well the model can identify positive values.

4.3.3.   Recall

It measures the model's ability to properly identify the TP value.

4.3.4.   F1-score

This metric provides a balance between the recall and precision score. It is also known as the harmonic mean of precision and recall.

4.4.   Implementation details

We implemented the framework with PyTorch, a versatile deep learning library that supports dynamic computational graphs and powerful graphics processing unit (GPU), making it ideal for dealing with complex neural network architectures. We fine-tuned the LLM on a single high-performance NVIDIA A4500 GPU with CUDA. To improve efficiency and reduce memory usage without sacrificing performance, we fine-tuned LLaMA using low-rank adaptation (LoRA) ^[15], leveraging LLM's feature embeddings as learnable weights for the framework. First, we reduced the dimensionality of embeddings from 1024 to 256. In addition, we bound the length of the narrative report (text) to n characters due to computational constraints.

To extensively assess model performance, we split the dataset into an 80:20 training:testing split. The training set was used to fine-tune the model's parameters and the test set to evaluate the model's generalization to unseen data.

5.   Results

This section presents the findings of our experiments, focusing on key research questions. These questions emphasize on the model's performance and applicability.

● RQ1: How effectively does our proposed framework perform in comparison to state-of-the-art (SOTA) models?

● RQ2: Can the proposed framework classify cases with complex and overlapping classes?

● RQ3: How does the use of the transformer-based models to generate feature embeddings impact the semantic understanding of narrative reports?

● RQ4. What are the common errors made by the framework?

5.1.   SOTA models (RQ1)

We compared SOTA models to our fine-tuned LLM encoder for multi-label classification of behavioral health cases. We compared performance using deep learning models, including LSTM, BiLSTM, and CNN+LSTM. We additionally evaluated the results of the classifier enhanced with transformer-based models. Each deep learning model was evaluated across four different configurations, as summarized in Table 4. Our experiments utilized widely adopted word embeddings: Word2Vec ^[24], GloVe ^[29], and fastText ^[4]. Word embedding is an essential technique in NLP that represents text and vocabulary as numerical vectors in a continuous vector space. This representation captures semantic relationships, with comparable words having closer vector representations which help with contextual interpretation. Word2Vec uses predictive models to learn word associations, GloVe uses co-occurrence statistics to capture global relationships, and fastText improves conventional embeddings by taking into account subword information, which efficiently handles uncommon and out-of-vocabulary words.

Table 4 shows the evaluation results of various models for identifying behavioral health classes in police narrative reports. We categorized the models into two categories: deep learning models and transformer-based models, with each demonstrating its efficacy.

The deep learning models, with and without pretrained embeddings, were evaluated. Across all configurations, models with pretrained embeddings outperformed those without, demonstrating the benefit of pretrained embeddings. LSTM with GloVe embeddings outperformed other LSTM models in terms of F1-score (75.1%) and recall (67.8%), while LSTM without embeddings performed poor (F1-score of 66.9%). The BiLSTM model using Word2Vec had the highest F1-score (81.2%), followed by fastText. CNN+LSTM models had reduced performance, compared to BiLSTM, suggesting that the added CNN layer did not provide significant improvements in this task.

The RoBERTa model exhibited strong performance across every metric for transformer-enabled models, achieving 53% accuracy, 76% precision, 69% recall, and 72% F1-score, while utilizing shorter sequence lengths and a two-layer network. In comparison, the fine-tuned behavioral health LLM performed similarly to RoBERTa, which excels in encoding contextual embeddings for classification rather than text generation.

5.2.   Class-wise performance (RQ2)

Each narrative report in the dataset is labeled with one or more behavioral health classes, enabling multi-label classification. Figure 2 shows the distribution of behavioral health in the dataset. "Domestic/social" class is the most common class, with 1,394 cases. "Substance abuse" class has 561 cases, while "mental health" class has 331 cases. There are 169 cases of "nondomestic/social" class. "Mental health crisis and substance abuse" class includes 97 cases, which are a composite of two (2) classes. The disparity between classes highlights the necessity for techniques to ensure balanced model performance.

As shown in the Figure 3, the LSTM is much more better at identifying the "domestic/social" and "substance abuse" classes compared to other classes, whereas the BiLSTM model performs comparatively good across the classes other than the "nondomestic/social" class, shown in Figure 4. This could be due to the low availability of class samples in the test set. Figure 5 shows that the CNN+LSTM model performed poorly in identifying classes other than "domestic/social" and "substance abuse".

Figure 6 shows that RoBERTa accurately captured the contextual representation of the embeddings, as demonstrated by its ability to distinguish between behavioral health classes. The lightweight framework had high F1-scores and accuracy in well-represented classes including "domestic/social" and "substance abuse", showing excellent results in these classes. However, the results for classes with low representation, such as "mental health" and "mental health crisis and substance abuse", show significantly lower F1-scores compared to accuracy, indicating difficulties in dealing with class imbalances.

The lightweight framework, enhanced by feature embeddings from the fine-tuned LLM, performed well in distinguishing the "domestic/social" class. However, as shown in Figure 7, it performed fairly in the "substance abuse" class and struggled with other classes. This low performance in less prevalent classes can be attributed to the nature of LLaMA, which is designed primarily for text generation tasks rather than encoding fine-grained contextual embeddings for classification. In contrast, RoBERTa, which was specifically built for encoding tasks, is better suited to capture nuanced distinctions across classes, making it more effective for balanced multi-label classification. The findings in Figures 3–7 and Table 4 suggest that, while fine-tuned LLaMA embeddings benefit dominant classes, they are less effective at handling the peculiarities of imbalanced datasets.

5.3.   t-SNE plot analysis of the classes (RQ3)

First, we tried to address the class imbalance. We applied oversampling to the extracted features and labels, ensuring balanced representation. Figure 8 depicts the reduced embeddings for test data across classes, giving insight on the distinction of class features. Each point in Figure 8 represents a sample and the colors indicate certain classes, such as "domestic/social" (blue) and "substance abuse" (brown). Some classes, including "domestic/social" and "mental health crisis and substance abuse" (green), form distinct clusters, demonstrating the model's capacity to distinguish attributes. However, overlaps between classes such as "mental health" (orange) and "substance abuse" (brown) indicate feature similarity or ambiguity, which affects classification. While oversampling corrected class imbalance during training, more improvements in feature distinctiveness are necessary for better separation and performance.

5.4.   Error analysis (RQ4)

We will look into the errors in the multi-label classifier enhanced by fine-tuned LLM for behavioral health analysis. There are some test cases incorrectly identified from our experiment.

Figures 9 and 10 demonstrate that the LLM + classifier could not identify the correct class. For the first sample, it misclassified the narrative report as "domestic/social" rather than the correct label: "mental health". In addition, it wrongly identified a report from the "nondomestic/social" class as "domestic/social. It is possible that training the classifier for additional epochs may improve its ability to distinguish across classes.

6.   Conclusions

In this work, we provide an approach that employs transformer-based models as an encoder to extract feature embeddings for training fully connected layers for multi-label classification. Our results demonstrate that a lightweight network enhanced with parameter-frozen transformer-based LLMs, while having fewer layers and shorter sequence lengths, can match or rival traditional and hybrid models such as LSTM, BiLSTM, CNN+LSTM in terms of contextual embedding quality and classification performance. In this case, the RoBERTa + classifier achieved high F1-scores and accuracy in a number of behavioral health classes, demonstrating its usefulness and robustness despite its streamlined architecture. Furthermore, our findings show that accuracy alone is insufficient for evaluating model performance in sensitive areas such as behavioral health, where a model may perform well in some classes but fail to provide a complete representation of the dataset.

6.1.   Discussion and limitations

We evaluated several models alongside a trainable framework enhanced with transformer-based models, providing crucial insights into the efficacy of our proposed approach. Our study was constrained by a number of variables. First, class imbalance was a significant concern, which could affect the accuracy of classifications and stability of the transformer-based LLMs, as shown in Section 5.4. To address this, we utilized an oversampling technique in which positive samples from each class were reproduced in proportion to the number of negative samples, resulting in a balanced representation across the dataset. For better computation performance, we applied this to extracted embeddings rather than individual sets (training and testing). This eliminated the need for repeated tokenization. In addition, we refined the instruction sets to improve clarity and granularity, which reduced ambiguity and increased model output stability to some degree. These improvements increased the ability of the model to learn nuanced patterns and improve robustness and performance by fairly representing all classes across the dataset. Also due to hardware constraints, we lowered some parameters to their minimal values to reduce computing demands and training time. With better hardware, we would like to explore more powerful transformer-based models and fully use their capabilities to generate richer feature embeddings.