Structuring Radiology Reports: Challenging LLMs with Lightweight Models

Johannes Moll1,2, Louisa Fay1, Asfandyar Azhar1,3, Sophie Ostmeier1, Tim Lueth2, Sergios Gatidis1, Curtis Langlotz1, Jean-Benoit Delbrouck1,4
1Stanford AIMI   |   2Technical University of Munich  |  3Carnegie Mellon University  |  4HOPPR
arXiv Code HuggingFace

Abstract

Radiology reports are critical for clinical decision-making but often lack a standardized format, limiting both human interpretability and machine learning (ML) applications. While large language models (LLMs) have shown strong capabilities in reformatting clinical text, their high computational requirements, lack of transparency, and data privacy concerns hinder practical deployment. To address these challenges, we explore lightweight encoder-decoder models (<300M parameters)—specifically T5 and BERT2BERT—for structuring radiology reports from the MIMIC-CXR and CheXpert Plus datasets. We benchmark these models against eight open-source LLMs (1B–70B parameters), adapted using prefix prompting, in-context learning (ICL), and low-rank adaptation (LoRA) finetuning. Our best-performing lightweight model outperforms all LLMs adapted using prompt-based techniques on a human-annotated test set. While some LoRA-finetuned LLMs achieve modest gains over the lightweight model on the Findings section (BLEU 6.4%, ROUGE-L 4.8%, BERTScore 3.6%, F1-RadGraph 1.1%, GREEN 3.6%, and F1-SRR-BERT 4.3%), these improvements come at the cost of substantially greater computational resources. For example, LLaMA-3-70B incurred more than 400 times the inference time, cost, and carbon emissions compared to the lightweight model. These results underscore the potential of lightweight, task-specific models as sustainable and privacy-preserving solutions for structuring clinical text in resource-constrained healthcare settings.

Overview

Lightweight Models

Model Variant HuggingFace Link
BERT2BERT RoBERTa-base 🤗 StanfordAIMI/SRR-BERT2BERT-RoBERTa-base
RoBERTa-biomed 🤗 StanfordAIMI/SRR-BERT2BERT-RoBERTa-biomed
RoBERTa-PM-M3 🤗 StanfordAIMI/SRR-BERT2BERT-RoBERTa-PM-M3
RadBERT 🤗 StanfordAIMI/SRR-BERT2BERT-RadBERT
T5 T5-Base 🤗 StanfordAIMI/SRR-T5-Base
Flan-T5 🤗 StanfordAIMI/SRR-StanfordAIMI/SRR-T5-Flan
SciFive 🤗 StanfordAIMI/SRR-T5-SciFive

Dataset

Dataset HuggingFace Link
SRRG-Findings 🤗 StanfordAIMI/srrg_findings_impression

Results

We benchmarked our best lightweight model against open-source LLMs (1B–70B parameters), adapted using prefix prompting, in-context learning (ICL), and low-rank adaptation (LoRA) finetuning. The results are summarized in the figure below:

Results

The result for the LLaMA-3-70B model with LoRA finetuning is indicated with a dashed line, as this configuration was trained for only one epoch—compared to five epochs for the other models—due to computational constraints.

BibTeX

@article{structuring-2025,
            title={Structuring Radiology Reports: Challenging LLMs with Lightweight Models},
            author={Moll, Johannes and Fay, Louisa and Azhar, Asfandyar and Ostmeier, Sophie and Lueth, Tim and Gatidis, Sergios and Langlotz, Curtis and Delbrouck, Jean-Benoit},
            journal={arXiv preprint arXiv:2506.00200},
            url={https://arxiv.org/abs/2506.00200},
            year={2025}
        }