Instruction Following: IFSHORT26 Leaderboard
| Models | Average | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACHAP-CollarF1 | ACHAP-COMET | ACHAP-GC-BERTScore | ACHAP-TM-BERTScore | ACHAP-TM-MATCHED | ACHAP-WER | ASR-WER | QA-BERTScore | QE-accuracy | QE-format-accuracy | SUM-BERTScore | TRANS-COMET | |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive3 | N/A | N/A | N/A | N/A | N/A | N/A | 0.032 | 0.299 | 0.199 | 0.178 | N/A | 0.397 |
| Qwen3-Omni | N/A | N/A | N/A | N/A | N/A | N/A | 0.549 | 0.325 | 0.454 | 0.500 | 0.000 | 0.635 |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive | N/A | N/A | N/A | N/A | N/A | N/A | 0.032 | 0.400 | 0.421 | 0.415 | N/A | 0.584 |
| Phi4-Multimodal | N/A | N/A | N/A | N/A | N/A | N/A | 0.767 | 0.421 | 0.394 | 0.253 | 0.000 | 0.601 |
| BSC_IWSLT26_IF_SHORT_unconstrained_primary | N/A | N/A | N/A | N/A | N/A | N/A | 0.034 | 0.440 | 0.429 | 0.476 | N/A | 0.591 |
| KIT_IWSLT26_IF_SHORT_ unconstrained_ contrastive1 | N/A | N/A | N/A | N/A | N/A | N/A | 0.042 | 0.448 | 0.361 | 0.498 | N/A | 0.626 |
| KIT_IWSLT26_IF_ SHORT_unconstrained_ primary | N/A | N/A | N/A | N/A | N/A | N/A | 0.019 | 0.484 | 0.000 | 0.000 | N/A | 0.634 |
| NLE_IWSLT26_IF_SHORT_UNCONSTRAINED_CONSTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | 0.033 | 0.486 | 0.208 | 0.005 | N/A | 0.559 |
| NLE_IWSLT26_IF_SHORT_CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | 0.034 | 0.486 | 0.420 | 0.499 | N/A | 0.580 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_CONTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | 0.036 | 0.496 | 0.290 | 0.351 | N/A | 0.551 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | 0.031 | 0.507 | 0.419 | 0.484 | N/A | 0.570 |
| Models | en → de | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACHAP-CollarF1 | ACHAP-COMET | ACHAP-GC-BERTScore | ACHAP-TM-BERTScore | ACHAP-TM-MATCHED | ACHAP-WER | ASR-WER | QA-BERTScore | QE-accuracy | QE-format-accuracy | SUM-BERTScore | TRANS-COMET | |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive3 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.383 | 0.796 | 0.712 | N/A | 0.799 |
| Qwen3-Omni | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.330 | 0.858 | 1.000 | N/A | 0.844 |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.425 | 0.810 | 0.997 | N/A | 0.798 |
| Phi4-Multimodal | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.363 | 0.658 | 0.927 | N/A | 0.812 |
| BSC_IWSLT26_IF_SHORT_unconstrained_primary | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.467 | 0.785 | 0.953 | N/A | 0.808 |
| KIT_IWSLT26_IF_SHORT_ unconstrained_ contrastive1 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.423 | 0.705 | 1.000 | N/A | 0.830 |
| KIT_IWSLT26_IF_ SHORT_unconstrained_ primary | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.466 | 0.000 | 0.000 | N/A | 0.840 |
| NLE_IWSLT26_IF_SHORT_UNCONSTRAINED_CONSTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.462 | 0.333 | 0.005 | N/A | 0.749 |
| NLE_IWSLT26_IF_SHORT_CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.470 | 0.786 | 0.997 | N/A | 0.765 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_CONTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.471 | 0.501 | 0.584 | N/A | 0.739 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.477 | 0.762 | 0.974 | N/A | 0.762 |
| Models | en → en | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACHAP-CollarF1 | ACHAP-COMET | ACHAP-GC-BERTScore | ACHAP-TM-BERTScore | ACHAP-TM-MATCHED | ACHAP-WER | ASR-WER | QA-BERTScore | QE-accuracy | QE-format-accuracy | SUM-BERTScore | TRANS-COMET | |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive3 | N/A | N/A | N/A | N/A | N/A | N/A | 0.127 | 0.420 | N/A | N/A | N/A | N/A |
| Qwen3-Omni | N/A | N/A | N/A | N/A | N/A | N/A | 0.197 | 0.325 | N/A | N/A | N/A | N/A |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive | N/A | N/A | N/A | N/A | N/A | N/A | 0.127 | 0.383 | N/A | N/A | N/A | N/A |
| Phi4-Multimodal | N/A | N/A | N/A | N/A | N/A | N/A | 0.069 | 0.463 | N/A | N/A | N/A | N/A |
| BSC_IWSLT26_IF_SHORT_unconstrained_primary | N/A | N/A | N/A | N/A | N/A | N/A | 0.134 | 0.425 | N/A | N/A | N/A | N/A |
| KIT_IWSLT26_IF_SHORT_ unconstrained_ contrastive1 | N/A | N/A | N/A | N/A | N/A | N/A | 0.170 | 0.450 | N/A | N/A | N/A | N/A |
| KIT_IWSLT26_IF_ SHORT_unconstrained_ primary | N/A | N/A | N/A | N/A | N/A | N/A | 0.074 | 0.495 | N/A | N/A | N/A | N/A |
| NLE_IWSLT26_IF_SHORT_UNCONSTRAINED_CONSTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | 0.134 | 0.501 | N/A | N/A | N/A | N/A |
| NLE_IWSLT26_IF_SHORT_CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | 0.136 | 0.531 | N/A | N/A | N/A | N/A |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_CONTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | 0.145 | 0.505 | N/A | N/A | N/A | N/A |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | 0.123 | 0.507 | N/A | N/A | N/A | N/A |
| Models | en → zh | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACHAP-CollarF1 | ACHAP-COMET | ACHAP-GC-BERTScore | ACHAP-TM-BERTScore | ACHAP-TM-MATCHED | ACHAP-WER | ASR-WER | QA-BERTScore | QE-accuracy | QE-format-accuracy | SUM-BERTScore | TRANS-COMET | |
| Qwen3-Omni | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.273 | 0.957 | 1.000 | N/A | 0.856 |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.398 | 0.872 | 0.663 | N/A | 0.750 |
| Phi4-Multimodal | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.443 | 0.917 | 0.085 | N/A | 0.807 |
| BSC_IWSLT26_IF_SHORT_unconstrained_primary | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.413 | 0.929 | 0.950 | N/A | 0.782 |
| KIT_IWSLT26_IF_SHORT_ unconstrained_ contrastive1 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.471 | 0.739 | 0.993 | N/A | 0.845 |
| KIT_IWSLT26_IF_ SHORT_unconstrained_ primary | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.456 | 0.000 | 0.000 | N/A | 0.852 |
| NLE_IWSLT26_IF_SHORT_UNCONSTRAINED_CONSTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.466 | 0.500 | 0.014 | N/A | 0.755 |
| NLE_IWSLT26_IF_SHORT_CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.487 | 0.894 | 1.000 | N/A | 0.794 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_CONTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.482 | 0.658 | 0.819 | N/A | 0.734 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.520 | 0.915 | 0.961 | N/A | 0.777 |
| Models | en → it | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ACHAP-CollarF1 | ACHAP-COMET | ACHAP-GC-BERTScore | ACHAP-TM-BERTScore | ACHAP-TM-MATCHED | ACHAP-WER | ASR-WER | QA-BERTScore | QE-accuracy | QE-format-accuracy | SUM-BERTScore | TRANS-COMET | |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive3 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.395 | N/A | N/A | N/A | 0.787 |
| Qwen3-Omni | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.373 | N/A | N/A | N/A | 0.839 |
| BSC_IWSLT26_IF_SHORT_unconstrained_contrastive | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.395 | N/A | N/A | N/A | 0.787 |
| Phi4-Multimodal | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.414 | N/A | N/A | N/A | 0.787 |
| BSC_IWSLT26_IF_SHORT_unconstrained_primary | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.454 | N/A | N/A | N/A | 0.773 |
| KIT_IWSLT26_IF_SHORT_ unconstrained_ contrastive1 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.449 | N/A | N/A | N/A | 0.830 |
| KIT_IWSLT26_IF_ SHORT_unconstrained_ primary | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.519 | N/A | N/A | N/A | 0.841 |
| NLE_IWSLT26_IF_SHORT_UNCONSTRAINED_CONSTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.514 | N/A | N/A | N/A | 0.733 |
| NLE_IWSLT26_IF_SHORT_CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.456 | N/A | N/A | N/A | 0.763 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_CONTRASTIVE | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.527 | N/A | N/A | N/A | 0.733 |
| FBK_IWSLT26_IF_SHORT_ CONSTRAINED_PRIMARY | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.524 | N/A | N/A | N/A | 0.742 |