Introduction
This demo page showcases the results of our research on transforming symbolic music scores into expressive piano performance audio. The approach combines a Transformer-based Expressive Performance Rendering (EPR) model with a fine-tuned neural MIDI synthesiser, offering a streamlined method for converting inexpressive score MIDI files into rich, expressive piano performances.
Our integrated system is designed to directly generate expressive audio performances from score inputs by combining MIDI-to-MIDI (M2M) and MIDI-to-Audio (M2A) models. The M2M model is responsible for rendering expressive MIDI files, while the M2A model, which has been fine-tuned for this task, generates the corresponding audio outputs. This demo page illustrates the improvement achieved through the fine-tuning of the M2A model and presents a comparison of the proposed system (M2M + M2A) with other existing systems.
The evaluation conducted in this study highlights the system's effectiveness in reconstructing human-like expressiveness and capturing the acoustic ambiance of environments such as concert halls and recording studios. The proposed system is the first of its kind to seamlessly convert inexpressive score MIDI files into expressive piano performance audios using purely deep learning models.
Fine-tuned M2A Model
This section demonstrates the improvements achieved by fine-tuning the M2A model. The audio samples (all from test set) before and after fine-tuning are presented below along with the groundtruth audio recordings, allowing for a direct comparison of the model's performance.
| Model | Sample 1 | Sample 2 | Sample 3 | Sample 4 |
|---|---|---|---|---|
| Groundtruth | ||||
| M2A (Before Fine-tuning) | ||||
| M2A (After Fine-tuning) |
- Sample1: "Fantasiestücke, Op. 12: VI. Ende vom Lied", composed by Robert Schumann, performed by Sviatoslav Richter
- Sample2: "English Suite No. 3 in G Minor, BWV 808: VI. Gigue", composed by Johann Sebastian Bach, performed by Friedrich Gulda
- Sample3: "12 Etudes, Op. 10: No. 2. in A Minor - Chromatique", composed by Frédéric Chopin, performed by Sviatoslav Richter
- Sample4: "24 Préludes, Op. 28: Prélude No. 13 in F-Sharp Major", composed by Frédéric Chopin, performed by Claudio Arrau
Comparison of Different Systems with Proposed M2M + M2A
In this section, we present a comparison between the proposed M2M + M2A system and other systems. Audio samples generated by different methods, including:
- Groundtruth audio recording,
- Groundtruth midi + Pianoteq,
- Groundtruth midi + M2A,
- M2M output + Pianoteq,
- M2M output + M2A (proposed),
- Baseline (proposed),
- Score + Pianoteq,
| System | Sample 1 | Sample 2 | Sample 3 | Sample 4 |
|---|---|---|---|---|
| Groundtruth Audio Recording | ||||
| Groundtruth midi + Pianoteq | ||||
| Groundtruth midi + M2A | ||||
| M2M output + Pianoteq | ||||
| M2M output + M2A | ||||
| Baseline | ||||
| Score + Pianoteq |
- Sample1 & 2: "Piano Sonata No. 18 in E-Flat Major, Op. 31 No. 3 'The Hunt'", composed by Ludwig van Beethoven, performed by Daniel Barenboim
- Sample3 & 4: "Piano Sonata No.30 in E, Op.109: 2. Prestissimo", composed by Ludwig van Beethoven, performed by Claudio Arrau
More and Longer Generation Samles from the Proposed M2M + M2A and Baseline Models
In this section, we present more and longer samples generated with our proposed M2M + M2A system and the baseline systems. The results demonstrate how the integrated system effectively balances musical expressiveness with audio quality, outperforming the baseline models.
| Model | Sample 1 | Sample 2 | Sample 3 | Sample 4 |
|---|---|---|---|---|
| M2M+M2A | ||||
| Baseline | ||||
| Groundtruth |
- Sample1: "Piano Sonata No.3 in C, Op.2 No.3: 2. Adagio", composed by Ludwig van Beethoven, performed by Alfred Brendel
- Sample2: "Piano Sonata No.9 in E, Op.14 No.1: 1. Allegro", composed by Ludwig van Beethoven, performed by Alfred Brendel
- Sample3: "Piano Sonata No. 31 in A-Flat Major, Op. 110: 2. Allegro molto", composed by Ludwig van Beethoven, performed by Alfred Brendel
- Sample4: "Piano Sonata No. 18 in E-Flat Major, Op. 31 No. 3: IV. Presto con fuoco", composed by Ludwig van Beethoven, performed by Alfred Brendel