A challenge that pushes audio-visual speech enhancement beyond simulated-speech, clean-video benchmarks toward real-world robustness.
Schedule update: the leaderboard freeze has been further postponed to August 5, and the final rankings will be announced on August 6. The paper submission deadline has been extended to August 9.
Schedule update: the leaderboard freeze has been postponed to July 29, and the final rankings will be announced on July 31. The paper submission deadline remains unchanged on August 3.
Registration is now closed. Thank you to everyone who signed up!
The testing set has been released and the leaderboard is now open for submissions. The relevant links have been sent to the email address you provided in the registration form. Please check your inbox. If you have any questions, feel free to contact us at any time.
We have updated the baseline! Please see GitHub for details.
Registration is now open! The dataset link will be sent to your registered email shortly. If you do not receive it, please contact us immediately via email or WeChat.
The Real-World AVSE Challenge has been accepted by ISCSLP 2026.
The Real-World AVSE Challenge focuses on Audio-Visual Speech Enhancement (AVSE) under genuinely real-world conditions. Unlike conventional benchmarks built on the "clean video plus additive mixing" paradigm, it emphasizes naturally occurring speech overlap, authentic reverberation, and ambient noise, while also confronting the visual degradations common in practice: occlusion, low quality, missing frames, blur and far-field. Through two complementary tracks on real-world mixed scenarios and visual degradation, the Challenge aims to move AVSE from the "ideal video" setting toward real-world deployment.
Please register via this Google Form.
This track provides multi-speaker audio-visual data captured naturally in real environments, with speech mixed organically rather than synthetically. It offers a comprehensive test of model robustness and practicality in realistic settings, aiming to bridge the gap between simulation and reality.
Built on relevant public datasets, this track constructs a set of visually degraded samples, covering occlusion, low quality, missing frames, and blur, to systematically assess how well models hold up when the visual modality becomes unreliable.
The Real-World AVSE Challenge provides an official development set and testing set for each track. The provided data falls into two cases: single-speaker speech and two-speaker simultaneous speech, and all audio files are naturally recorded. The recordings involve seven groups of distinct speakers (two per group, with no gender restriction). For both Track 1 and Track 2, the speakers in the development and test sets are completely non-overlapping. We place no restrictions on the training data.
This Challenge does not specify any official training set. Participants are free to use any open-source datasets, as well as any pre-trained models and data augmentation methods. (Regardless of which datasets, models, or methods are used, participants must clearly describe them in the final paper submission.)
The speech falls into two types. One is mix, which comes from naturally recorded real-world mixtures; it has no clean reference, so only non-intrusive (reference-free) metrics are provided. The other is remix, formed by summing two clean single-speaker segments, which supports both reference-based and reference-free metrics. (Both mix and remix contribute to the final evaluation. remix plays a key role, since its ground truth makes reference-based metrics possible.) The dev set is drawn from three of the seven speaker groups.
The development set reuses the Track 1 development data, likewise divided into mix and remix, with visual degradations applied to the target speakers' face videos. There are five types of visual degradation: low quality, occlusion, frame freezing, desynchronization, and blackout. It also includes the original 3-meter far-field visual material from the data. The data comes from three of the seven speaker groups.
The testing set follows the same mixing scheme as the development set, with data drawn from the remaining four unseen speaker groups, ensuring no overlap between the development and testing sets.
The testing set likewise reuses the Track 1 testing data, with visual degradations applied to the target speakers' face videos. The data again comes from the remaining four unseen speaker groups, ensuring no overlap between the development and testing sets.
The baseline system is AV-ConvTasNet, a two-stream audio-visual separator that recovers a single target speaker from a mixture, conditioned on that speaker's lip movements. The audio branch is a Conv-TasNet separator operating on the raw waveform, while the video branch is a frozen ResNet-34 lip-reading encoder (a 3D convolutional stem followed by ResNet-34). These visual features are temporally aligned to the audio bottleneck and fused with the audio representation, allowing the model to use the target speaker's lip video to guide separation. Given a mixture waveform and the corresponding lip frames, the model directly outputs the enhanced target speech.
Submissions are assessed with a suite of complementary metrics covering signal quality, perceptual quality, intelligibility, and speaker identity. Each metric is computed on the scenes indicated above using the listed backend.
| Metric | Type | Scenes | Backend |
|---|---|---|---|
| SI-SDR / PESQ / STOI | Reference-based | Remix only | torchmetrics |
| UTMOS | No-reference MOS | Mix + remix | UTMOSv2 |
| DNSMOS | No-reference P.835 | Mix + remix | torchmetrics ONNX |
| CER | Chinese character error rate | Mix + remix | Fun-ASR-Nano-2512 |
| Speaker similarity | Identity preservation | Mix + remix | WeSpeaker ResNet34 |
All dates are tentative and may be subject to change. Times are in AoE (Anywhere on Earth).
The testing set has been released and leaderboard submissions are now open. Submissions are scored offline by the organizers using the full metric suite of the baseline evaluation protocol, and the official ranking is based on OVRL, the mean of a team's per-metric ranks (lower is better).
If you have any questions, feel free to contact us at:
