Instruction Following: IFLONG26 Leaderboard

Models Average
ACHAP-CollarF1 ACHAP-COMET ACHAP-GC-BERTScore ACHAP-TM-BERTScore ACHAP-TM-MATCHED ACHAP-WER ASR-WER QA-BERTScore QE-accuracy QE-format-accuracy SUM-BERTScore TRANS-COMET
Qwen3-Omni 0.304 0.283 0.708 0.741 0.381 0.552 0.540 0.294 0.454 0.500 0.103 0.446
KIT_IWSLT26_IF_LONG_ unconstrained_ contrastive1 0.421 0.532 0.693 0.743 0.467 0.023 0.016 0.344 0.361 0.498 0.268 0.632
Phi4-Multimodal 0.104 0.248 0.666 0.749 0.224 0.985 0.820 0.402 0.394 0.253 0.170 0.411
KIT_IWSLT26_IF_ LONG_ unconstrained_ primary 0.474 0.566 0.737 0.772 0.525 0.078 0.067 0.427 0.250 0.001 0.275 0.564
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_HYB 0.000 0.516 0.611 0.799 0.132 0.050 0.032 0.368 0.290 0.351 0.202 0.526
FBK_IWSLT26_IF_LONG_ CONSTRAINED_PRIMARY 0.000 0.523 0.604 0.801 0.130 0.090 0.049 0.376 0.290 0.351 0.201 0.514
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_VAD 0.000 0.513 0.625 0.805 0.135 0.050 0.044 0.369 0.290 0.351 0.202 0.531
Models en → en
ACHAP-CollarF1 ACHAP-COMET ACHAP-GC-BERTScore ACHAP-TM-BERTScore ACHAP-TM-MATCHED ACHAP-WER ASR-WER QA-BERTScore QE-accuracy QE-format-accuracy SUM-BERTScore TRANS-COMET
Qwen3-Omni 0.609 N/A 0.877 0.882 0.607 0.208 0.158 0.327 N/A N/A 0.187 N/A
KIT_IWSLT26_IF_LONG_ unconstrained_ contrastive1 0.583 N/A 0.877 0.889 0.634 0.093 0.064 0.385 N/A N/A 0.218 N/A
Phi4-Multimodal 0.084 N/A 0.837 0.844 0.247 0.941 0.281 0.420 N/A N/A 0.183 N/A
KIT_IWSLT26_IF_ LONG_ unconstrained_ primary 0.436 N/A 0.869 0.882 0.481 0.311 0.269 0.412 N/A N/A 0.212 N/A
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_HYB 0.000 N/A 0.803 0.884 0.135 0.200 0.126 0.377 N/A N/A 0.156 N/A
FBK_IWSLT26_IF_LONG_ CONSTRAINED_PRIMARY 0.000 N/A 0.804 0.883 0.135 0.359 0.196 0.390 N/A N/A 0.152 N/A
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_VAD 0.000 N/A 0.803 0.884 0.135 0.200 0.175 0.377 N/A N/A 0.160 N/A
Models en → zh
ACHAP-CollarF1 ACHAP-COMET ACHAP-GC-BERTScore ACHAP-TM-BERTScore ACHAP-TM-MATCHED ACHAP-WER ASR-WER QA-BERTScore QE-accuracy QE-format-accuracy SUM-BERTScore TRANS-COMET
Qwen3-Omni 0.000 0.286 0.571 0.599 0.135 N/A N/A 0.246 0.957 1.000 0.021 0.812
KIT_IWSLT26_IF_LONG_ unconstrained_ contrastive1 0.103 0.451 0.511 0.556 0.221 N/A N/A 0.360 0.739 0.993 0.378 0.847
Phi4-Multimodal 0.051 0.333 0.617 0.640 0.195 N/A N/A 0.412 0.917 0.085 0.160 0.521
KIT_IWSLT26_IF_ LONG_ unconstrained_ primary 0.503 0.785 0.685 0.679 0.565 N/A N/A 0.452 1.000 0.004 0.383 0.789
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_HYB 0.000 0.683 0.496 0.757 0.135 N/A N/A 0.373 0.658 0.819 0.326 0.685
FBK_IWSLT26_IF_LONG_ CONSTRAINED_PRIMARY 0.000 0.681 0.496 0.757 0.135 N/A N/A 0.380 0.658 0.819 0.319 0.655
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_VAD 0.000 0.698 0.496 0.757 0.135 N/A N/A 0.374 0.658 0.819 0.325 0.699
Models en → it
ACHAP-CollarF1 ACHAP-COMET ACHAP-GC-BERTScore ACHAP-TM-BERTScore ACHAP-TM-MATCHED ACHAP-WER ASR-WER QA-BERTScore QE-accuracy QE-format-accuracy SUM-BERTScore TRANS-COMET
Qwen3-Omni 0.512 0.557 0.716 0.745 0.596 N/A N/A 0.354 N/A N/A 0.110 0.649
KIT_IWSLT26_IF_LONG_ unconstrained_ contrastive1 0.489 0.842 0.709 0.768 0.499 N/A N/A 0.322 N/A N/A 0.269 0.841
Phi4-Multimodal 0.207 0.353 0.668 0.759 0.289 N/A N/A 0.404 N/A N/A 0.175 0.515
KIT_IWSLT26_IF_ LONG_ unconstrained_ primary 0.456 0.737 0.703 0.777 0.504 N/A N/A 0.439 N/A N/A 0.267 0.732
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_HYB 0.000 0.735 0.528 0.767 0.123 N/A N/A 0.373 N/A N/A 0.174 0.695
FBK_IWSLT26_IF_LONG_ CONSTRAINED_PRIMARY 0.000 0.723 0.532 0.773 0.123 N/A N/A 0.385 N/A N/A 0.185 0.707
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_VAD 0.000 0.719 0.586 0.790 0.135 N/A N/A 0.376 N/A N/A 0.171 0.702
Models en → de
ACHAP-CollarF1 ACHAP-COMET ACHAP-GC-BERTScore ACHAP-TM-BERTScore ACHAP-TM-MATCHED ACHAP-WER ASR-WER QA-BERTScore QE-accuracy QE-format-accuracy SUM-BERTScore TRANS-COMET
Qwen3-Omni 0.094 0.288 0.667 0.736 0.185 N/A N/A 0.247 0.858 1.000 0.092 0.323
KIT_IWSLT26_IF_LONG_ unconstrained_ contrastive1 0.508 0.836 0.676 0.761 0.514 N/A N/A 0.308 0.705 1.000 0.208 0.840
Phi4-Multimodal 0.074 0.308 0.543 0.753 0.165 N/A N/A 0.370 0.658 0.927 0.162 0.607
KIT_IWSLT26_IF_ LONG_ unconstrained_ primary 0.500 0.740 0.690 0.748 0.551 N/A N/A 0.405 0.000 0.000 0.238 0.733
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_HYB 0.000 0.647 0.616 0.789 0.135 N/A N/A 0.350 0.501 0.584 0.153 0.723
FBK_IWSLT26_IF_LONG_ CONSTRAINED_PRIMARY 0.000 0.690 0.584 0.793 0.127 N/A N/A 0.348 0.501 0.584 0.149 0.694
FBK_IWSLT26_IF_LONG_ CONSTRAINED_CONTRASTIVE_VAD 0.000 0.635 0.616 0.789 0.135 N/A N/A 0.351 0.501 0.584 0.153 0.722