LLM-based OCR for Handwritten Text Recognition

Deakin University | Geelong, Victoria, Australia | August 2025 - October 2025

This project developed an Optical Character Recognition (OCR) pipeline to transcribe the handwritten 1885 diary of Beatrice Moxon, in collaboration with Deborah Lee-Talbot at Deakin University. Historical handwriting is difficult for conventional OCR, so the project combined a pre-trained visual text recognition model with a Large Language Model (LLM) that refines the raw output, corrects recognition errors, and enforces diplomatic transcription rules.

The pipeline was built in four stages: image preprocessing, initial OCR with TrOCR, LLM-based refinement, and rule enforcement to preserve the integrity of the original manuscript. Transcription quality was evaluated against hand-transcribed ground truth using Character Error Rate (CER) and Word Error Rate (WER).

Key Technical Achievements:

  • Built image preprocessing for the scanned diary pages, including rotation, noise reduction, contrast adjustment, and binarisation.
  • Generated baseline transcriptions with the pre-trained TrOCR transformer model.
  • Designed a structured LLM prompt (Google Gemini) to correct the baseline text while producing a faithful, line-by-line diplomatic transcript.
  • Encoded diplomatic transcription rules for struck-through words (<del>), underlined words (<u>), doubtful letters ([?]), and illegible text ([UNSURE]).
  • Reduced the Character Error Rate from 28.43% to 18.95% and from 16.68% to 9.39% on the two evaluated pages, with corresponding reductions in Word Error Rate.

Software and Tools:

  • Python, PyTorch, Hugging Face Transformers (TrOCR), OpenCV, and PIL
  • LLM APIs (Google Gemini) and LangChain
  • Jupyter Notebook and Anaconda

This project demonstrated how LLMs can be combined with visual recognition models to digitise historical handwritten archives accurately while respecting scholarly transcription standards.