Machine learning · vision-to-sequence

Reading sheet music with a neural network.

An optical music recognition model that turns a photograph of a melody into machine-readable ABC notation. A pretrained vision encoder (Google DeepMind's SigLIP-2, partially frozen) feeds a transformer decoder (trained on roughly 150,000 synthetic scores using a single consumer GPU).

The problem

Photo in, music notation out.

The input is a flat image (phone photo, scan, PDF clipping). The output is structured notation a computer can render, transpose, search, or play back.

Interactive · photo in, notation out Drag to compare ↔
Output: score re-engraved from the predicted ABC Input: photographed, degraded score
Input Output
Predicted ABC · re-engraved as “Output” above ↑
T: Banks Of The Ness
M: 3/4
L: 1/8
K: Gmajor
|:z dc|B3 A Bd|G4 dc|B3 A Bd|E4- EG|A3 B cd|B3 G FG|EF G2 A2|A3:|
|:AGF|E4 B2|B3 B AG|F4 B2|B4 GF|E4 c2|c3 c dc|B3 A Bd|A3:|
Data

A synthetic dataset.

There is no large corpus of photographed sheet music paired with ground truth ABC scores. Instead, I collected ~50,000 tunes from various archives, engraved each to a pristine PNG with abcm2ps using several music fonts, then applied physically-motivated augmentations to simulate photographs. About 150,000 image-score pairs were used in total.

Interactive · augmentation explorer Pick an effect ↓
Clean engraved score Clean render · source, abcm2ps

Eighteen effects in all: noise, blur, brightness, vignette, rotation, shear, JPEG artifacts, ink bleed, lighting hotspots, resolution loss, and more, were sampled in random combinations.

Architecture

Encoder-decoder transformers.

The architecture follows the state of the art image-encoder → projector → autoregressive decoder OMR recipe (LEGATO, 2025). A pre-trained large vision model knows how to extract general features from an image. It's kept mostly frozen, but the final four layers are fine-tuned to optimize for musical scores; the rest of the training budget goes into a small decoder that learns to output ABC notation from these features.

frozen · pretrained fine-tuned trained from scratch SCORE native ratio ≈ 384 patches first 23 layers frozen final 4 layers fine-tuned SigLIP-2 So400m NaFlex 27-layer ViT 384 × 1152 PROJECTOR 1152 → 512 image memory cross-attention every 3rd decoder layer TRANSFORMER DECODER 12 layers · trained from scratch RoPE · RMSNorm pre-norm self + cross · L2 L5 L8 L11 self-attn + FFN 512-token BPE T: Banks Of The Ness K:Gmajor M:3/4 L:1/8 |: z dc | B3 A Bd | G4 dc | B3 A Bd | E4- EG | ABC beam search autoregressive · all previous tokens
Vision tower (left) hands off to the decoder tower (right). Green marks trainable weights.
Native aspect ratios

SigLIP-2's NaFlex encoder handles variable image dimensions directly, dropping the overlapping-strip segmentation earlier OMR pipelines relied on.

A tiny, tuned vocabulary

ABC has a ~97-character alphabet, so a 512-token byte-level BPE vocabulary captures common note patterns while keeping the embedding table small and decoding short.

Training

Training details and loss curve.

0.02 0.05 0.1 0.2 0.5 1 2 5 0 10k 20k 30k 40k Val minimum 0.263 · step 25k Train Validation Loss Training step
Train vs validation loss · log scale

The objective is token-level cross-entropy, the next-token negative log-likelihood that is standard for autoregressive decoders.

Training used AdamW, a cosine rate schedule with warmup, bf16, batch size 24, ten epochs and ~50,000 steps. Training loss continues toward zero, but validation loss bottoms out around step 25,000 (and then drifts up due to mild overfitting beyond this point).

Keeping most of the 400M-parameter encoder frozen holds the memory footprint small enough to train the whole model overnight on a single consumer GPU.

Results

Half of all test images decode with under 5% character error.

Evaluated on 14,970 held-out scores, two-thirds of which carry synthetic photo damage. The headline metric is character error rate (CER): the fraction of characters you'd have to add, delete, or change to turn the prediction into the correct ABC.

4.7%
Median CER

The middle of the pack: half of the 14,970 test scores decode with under 4.7% character error. I lead with the median because on a typical score the model is either nearly perfect or perfect; the median reflects the common case, not the tail.

11.2%
Mean CER

The average over every score. It runs well above the median because a long tail of the longest scores drags it up. There the autoregressive decoder can lose length-sync (the 90th-percentile CER is ~21%). The gap between mean and median is that tail.

9.5%
Exact match

The strictest bar: predictions matching the reference byte-for-byte, headers and all: roughly 1 in 11. No partial credit, so a single stray character fails the whole score.

Next steps

A clear path to improvement.

A long, distorted score, 'Above And Beyond', under perspective, shear, and arc warping

For now, the project has already returned what I wanted from it: a working image-to-ABC pipeline, a clear map of where encoder-decoder OMR strains, and a model that is genuinely usable in practice.

The long score (465 tokens) shown here scored at 17% CER. The augmentations barely matter (they correlate with error at just ~0.03); length is driving error rate. Median CER climbs steadily with sequence length, roughly quadrupling from the shortest quarter of scores to the longest as the decoder accumulates error over hundreds of tokens.
Richer training data

Every training pair is engraved from a handful of music fonts and then photo-degraded. Limited data diversity holds back the model: more engravers, more notation styles, and real captured photographs would allow training beyond what the existing renderer has produced.

A notation-aware metric

CER punishes differences the eye cannot see. For example, extra whitespace or a key labelled "A dorian" instead of "G major" yield the same output. Scoring the rendered notation rather than the raw text would be a better measure of fidelity.

Titles and lyrics

The decoder is not optimizing for free text such as titles and lyrics ("Above And Beyond" becomes "Abre And Beynand"). Two options: mask it out of the loss so it stops competing with notes, or handle it properly with a text-aware objective.

Runaway generation

The catastrophic tail is length divergence: CER tracks the predicted-to-reference length ratio at r ≈ 0.96, as long inputs send the decoder into loops or over-generation. A no-repeat and length penalty at decode time, beam search, and a length curriculum in training all aim straight at it.