1. Introduction

When a judge on a music competition panel listens to a contestant, they are doing something remarkably sophisticated. They track whether the singer got the words right, whether the melody stayed true to the song’s key, and whether the timing locked onto the rhythm, all while forgiving, even rewarding, the artistic liberties that make a performance feel alive: a held note stretched for effect, a graceful melisma, a deliberate change of vocal texture between a verse and a chorus.

Getting a machine to listen this way is surprisingly hard. Most automatic singing quality assessment (SQA) systems today evaluate either the sound (pitch accuracy, acoustic quality) or the words (lyric transcription accuracy), but rarely both together, and almost never in a way that understands musical structure. The result is systems that penalize a singer for a beautiful, intentional improvisation, or that praise a pitch-perfect performance in which half the lyrics are wrong.

In our paper accepted at Interspeech 2026, we present MusicJudge, a music-aware framework for automatic singing performance evaluation that couples lyric correctness with pitch-rhythm fidelity, analyzed at the level of musically meaningful segments. This blog post gives a brief tour of the paper, shares some of the engineering and research challenges we hit along the way, and offers a few additional insights from the journey.

2. Background

Singing quality assessment sits at the intersection of two research threads that have historically evolved separately.

On one side, acoustic SQA methods, from early handcrafted-feature models to modern neural MOS (mean opinion score) predictors, judge how a performance sounds. They capture intonation and audio quality well, but they are blind to what is being sung: a singer could hum flawlessly through the wrong lyrics and still score highly.

On the other side, lyric-based approaches use automatic speech recognition (ASR) to transcribe the vocals and compare them against reference lyrics. This captures content correctness but breaks down in exactly the places where singing differs from speech. Sustained vowels, vibrato, melisma (one syllable stretched across several notes), and tempo elasticity all distort the acoustic patterns that ASR models were trained on, producing transcription errors that have nothing to do with the singer’s actual competence.

Human experts do neither of these things in isolation. They evaluate lyrical accuracy and adherence to the melodic and rhythmic structure of the piece (for instance, the tonal framework of a Raag in Indian classical music) while accommodating singer-to-singer differences in timbre, range, and expressive style. Bridging that gap between how machines and humans listen is the motivation behind MusicJudge.

3. Methodology

MusicJudge is built around one central idea: evaluate a performance block by block, where each block is a musically coherent unit (a verse, a chorus, a bridge, an alaap), and judge each block along two complementary dimensions: content fidelity (are the right words sung, in the right order?) and musical fidelity (do pitch and rhythm respect the music?).

Figure 2 shows the overall pipeline.

The main stages are as follows.

Source separation. The input performance is first separated into a vocal stream and an accompaniment stream. The vocals drive lyric and pitch analysis, while the accompaniment provides the beat and tonal context.

Singing-adapted transcription with MG-LoRA. Standard ASR models struggle with singing, so we fine-tune a Whisper-based model [1] on curated singing data using what we call Modality-Guided LoRA (MG-LoRA), built on low-rank adaptation [2]. The key novelty is in the training objective: alongside the usual sequence-to-sequence loss, we add music-aware regularization terms that (a) penalize unstable token durations in sustained segments, (b) discourage spurious token boundaries inside acoustically smooth pitch regions, (c) enforce monotonic alignment consistency, and (d) encourage token boundaries to coincide with detected vocal onsets. In effect, the model is taught to respect the music while decoding the words.

Multi-signal block detection. ASR-derived segment boundaries rarely align with musical phrase boundaries, especially with melisma and vowel elongation in play. We therefore form overlapping sliding-window candidates over the transcribed segments and match them to the reference lyrics using three complementary signals: semantic embedding similarity, fuzzy lexical matching, and phonetic (grapheme-to-phoneme) similarity. This lets the system find the right lyric block even when the surface transcription is imperfect or the singer pronounces words differently. Within each detected block, line-level ordered matching flags missing, repeated, or out-of-order lines, and the block receives a content score combining coverage, correctness, and flow.

Music-aware pitch and rhythm scoring. For the musical dimension, a single global key is estimated from the performance’s own accompaniment. This is an intentional choice: a singer who transposes the song to suit their range is not penalized, while consistency within the performance is still enforced. Pitch fidelity per block combines in-key deviation, short-term stability, and voiced-frame rate computed from the vocal pitch contour. Rhythm fidelity measures how vocal onsets align with the accompaniment’s beat grid via timing error, bias, and stability statistics.

Aggregation and feedback. Block-level content and musical scores are fused with a slight emphasis on lyrical fidelity (chosen empirically against human expert ratings) and aggregated across blocks with duration weighting to produce a single interpretable performance score. Because the block structure is preserved, the system can also generate structured natural-language feedback rather than only emitting a number.

4. Results

We evaluated MusicJudge on SwaraLyrics, a corpus of 420 samples that we curated for this work, together with the public SingMOS-Pro [3] and Jamendo [4] datasets. SwaraLyrics pairs singing performances with authoritative playback audio and native-script lyrics, and consists primarily of Indian solo music spanning diverse genres, moods, eras, and singer demographics.

Agreement with human experts. On a set of 120 vocal performances independently scored by three or more human expert judges, MusicJudge’s ranking agreed with the expert ranking at a Spearman correlation of 0.683 (Kendall’s τ of 0.499). That is roughly +31.9% over a lyric-only evaluation and +38.0% over a music-only evaluation: clear evidence that jointly modeling the two modalities is what makes the difference (Figure 3).

Singing-robust transcription. MG-LoRA fine-tuning improved singing transcription accuracy by about 29.87% relative over the second-best system, averaged across SwaraLyrics, SingMOS-Pro, and Jamendo. The gains hold up across singing genres, including melisma-heavy classical and folk styles, and across five languages representing Whisper’s performance extremes, with word error rate reductions of 20.1 ± 7.52% across genres and 27.7 ± 10.87% across languages (Figure 4). Qualitatively, the fine-tuned model correctly transcribes passages with sustained note prolongation, ornamentation such as gamakas, and portamento that the base model garbles.

Component contributions. Ablations confirm that each design choice earns its place: multi-signal block detection outperforms any individual matching signal (reaching a correlation of 0.626 for intra-song boundary localization, +2.96% over the second-best configuration; Figure 5), and each music-aware regularization term in the MG-LoRA objective adds a measurable increment, with the pitch-region boundary term contributing the largest single gain after the base fine-tuning.

From scores to feedback. Because every score is anchored to a musically coherent block, the final output reads like a judge’s scorecard: per-section content and musical ratings that roll up into one overall score, accompanied by section-aware natural-language feedback (Figure 6).

5. Discussion: Challenges and Insights

Building MusicJudge taught us a few lessons that we believe are worth passing on.

Singing breaks ASR in structured, predictable ways. Our error analysis showed that transcription failures cluster exactly where singing departs from speech: melisma, long sustained vowels, vibrato. This predictability is good news: it is what made it possible to design targeted regularization terms (duration stability, pitch-region boundary suppression, onset alignment) instead of simply throwing more data at the problem. The insight that modality structure can guide adaptation may transfer to other cross-domain ASR problems, such as chanted or highly expressive speech.

Data was as hard as modeling. Existing singing datasets often lack pitch-rich annotation, contain only fragmentary lyrics, or cannot pair performances with their authoritative reference tracks due to licensing and usage restrictions. Curating SwaraLyrics, with its clean reference lyrics in native script, authoritative playback audio, and realistic performance artifacts such as audience noise and judge commentary, took a significant share of the project effort. The portions used for fine-tuning were either locally recorded by our institutional band or appropriately licensed.

Respecting artistic freedom is a design decision, not an afterthought. Two concrete examples: estimating the musical key from the performance’s own accompaniment (so transposition is never penalized), and treating song structure as latent rather than requiring verse/chorus labels at inference time (so that reordered or partial performances, such as a live cover that starts from the chorus, are evaluated fairly). Both choices came directly from watching where simplistic metrics disagreed with human judges.

Block-level scoring is what makes feedback useful. Because scores are computed per musically coherent block, the system can localize praise and criticism (“the bridge drifted from the key; the final chorus was your strongest”). In our experience, this is far more actionable for a learner than a single global number, and it is also what allows a language model to generate faithful, section-aware natural-language feedback instead of generic commentary.

Limitations and future work. MusicJudge currently targets solo performances. Extending MG-LoRA with diarization awareness for multi-singer settings (duets, choirs, call-and-response forms) is a natural next step we highlight in the paper.

6. Conclusion

MusicJudge shows that automatic singing evaluation gets substantially closer to human expert judgment when it listens the way judges do: attending to the words and the music together, at the level of musically meaningful segments, while leaving room for legitimate artistic expression. Beyond the benchmark numbers, we see practical potential in assistive vocal-training tools, quality assessment for synthetic singing, and scalable pre-screening support for music competitions.