Improving Pain Assessment by Fine-tuning a Language Model to a Specific Hospital System

Research Objectives: Building a tailored language model to predict Cerebral Palsy (CP) pain scores utilizing unstructured electronic health records (EHR).

Design: Cohort study

Setting: Shriners Children's (Multisite) Setting.

Participants: We have collected unstructured EHRs from the Shriners Children's Research Data Warehouse. The inclusion criteria are children with CP who were admitted for femoral osteotomy surgery. We identified 824 subjects across four types of notes: Physical Therapy Evaluation/Treatment Note PowerForms, Discharge Instructions PowerForms, Post-Anesthesia Care Unit Assessment Powerform, and Occupational Therapy Evaluation/Treatment Note PowerForms. Among these 2527 notes have a Vocalization Ocular Pain Assessment Survey (OPAS) score.

Interventions: Not Applicable.

Main Outcome Measures: Prediction of the presence of pain based on the information contained within the EHRs.

Results: We pre-trained the language model Tiny Clinical Bidirectional Encoder Representations from Transformers (BERT) and fine-tuned it for Vocalization OPAS prediction. We predicted pain (greater than 0) or no pain (score 0). To validate the model we split the dataset into a 20% test set and an 80% training set. The test set was from subjects that were not used for training. In three random runs, we achieved the following metrics with mean and standard deviation: recall of 0.91 ± 0.002, precision of 0.68 ± 0.02, F-1 score of 0.78 ± 0.01, and accuracy of 0.78 ± 0.02.

Conclusions: We demonstrate a natural language processing modeling approach to a specific context that holds significant potential for broader integration across various healthcare settings.

Author(s) Disclosures: NA