Audio-visual deepfake detection: every result

Design choices and fusion strategies

Fine-tuned (EfficientNet-B0 + audio CNN) and frozen (CLIP, XLS-R, AASIST + small head) models, each as a single modality, with feature-level fusion, and with late fusion, without and with augmentation. "+ SyncNet" adds six lip-sync numbers to the step above it. "Extra training step" is the same logistic regression without SyncNet, so the SyncNet comparison is fair.

Precision depends on how many real and fake videos a test set has. Most test videos are fake, so fake precision is high for every model (a detector that calls every video fake would already score 0.977, 0.947 and 0.876), and real precision is low. Recall and macro F1 are the fairer comparison across datasets.

Which modality was faked? Per-class recall

FakeAVCeleb test set, learned late fusion. Each cell is the share of that class's videos given the correct four-class label.

Per generator

The two best detectors overall: A = frozen learned late fusion + SyncNet, without augmentation; B = fine-tuned feature-level fusion + SyncNet, with augmentation. Each generator's fakes are scored against all real videos of that dataset ("All" is the whole test set), so real recall is the same for every generator of a dataset. When both detectors are shown, the better one is outlined.