1. Home
  2. Benchmarks
  3. Machine Translation on WMT2014 English-German
Dataset Machine Translation on WMT2014 English-German

Machine Translation on WMT2014 English-German

#
MODEL
REPOSITORY
BLEU SCORE SACREBLEU SPEED
PAPER
ε-REPRODUCES PAPER
1
Facebook-FAIR
(ensemble)
pytorch / fairseq
36.72 36.04 1.7
--
2
Facebook-FAIR
(single)
pytorch / fairseq
36.18 35.56 9.2
--
3
Transformer Big + BT
pytorch / fairseq
33.89 33.58 8.1
4
Local Joint Self-attention
jarfo / joint
29.59 28.90 28.1
5
DynamicConv
(without GLUs)
pytorch / fairseq
29.17 28.42 68.6
--
6
LightConv
pytorch / fairseq
29.17 28.62
7
DynamicConv
pytorch / fairseq
29.10 28.49 57.9
8
LightConv
(without GLUs)
pytorch / fairseq
28.44 27.80 69.4
--
9
ConvS2S
pytorch / fairseq
25.70 25.17 76.6
10
Transformer Big
pytorch / fairseq
25.39 24.77 68.4
Models on Papers with Code for which code has not been tried out yet.
MODEL
PAPER
BLEU SCORE SACREBLEU
PWC RANK
depth growing
Depth Growing for Neural Machine Translation
30.07
2
Transformer Big + MoS
Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
29.60
5
Transformer Big + adversarial MLE
Improving Neural Language Modeling via Adversarial Training
29.52
6
Evolved Transformer Big
The Evolved Transformer
29.30
8
SB-NMT
Synchronous Bidirectional Neural Machine Translation
29.21
9
Transformer
(big)
Self-Attention with Relative Position Representations
29.20
10
Transformer Big with FRAGE
FRAGE: Frequency-Agnostic Word Representation
29.11
11
adequacy-oriented NMT
Neural Machine Translation with Adequacy-Oriented Learning
28.99
12
Weighted Transformer
(large)
Weighted Transformer Network for Machine Translation
28.90
13
universal transformer base
Universal Transformers
28.90
14
KERMIT
KERMIT: Generative Insertion-Based Modeling for Sequences
28.70
16
RNMT+
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
28.50
17
Evolved Transformer Base
The Evolved Transformer
28.40
20
Transformer + SRU
Simple Recurrent Units for Highly Parallelizable Recurrence
28.40
18
Transformer Big
Attention Is All You Need
28.40
19
Transformer Base
Attention Is All You Need
27.30
21
ConvS2S
(ensemble)
Convolutional Sequence to Sequence Learning
26.40
22
GNMT+RL
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
26.30
23
SliceNet
Depthwise Separable Convolutions for Neural Machine Translation
26.10
24
MoE
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
26.03
25
DenseNMT
Dense Information Flow for Neural Machine Translation
25.52
26
ByteNet
Neural Machine Translation in Linear Time
23.75
28
FlowSeq-large
(NPD n = 30)
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
23.64
29
FlowSeq-large
(NPD n = 15)
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
23.14
30
FlowSeq-large
(IWD n = 15)
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
22.94
31
Denoising autoencoders
(non-autoregressive)
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
21.54
32
FlowSeq-large
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
20.85
33
Deep-Att
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation
20.70
34
PBSMT + NMT
Phrase-Based & Neural Unsupervised Machine Translation
20.23
35
NAT +FT + NPD
Non-Autoregressive Neural Machine Translation
19.17
36
FlowSeq-base
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
18.55
37
Seq-KD + Seq-Inter + Word-KD
Sequence-Level Knowledge Distillation
18.50
38
Unsupervised PBSMT
Phrase-Based & Neural Unsupervised Machine Translation
17.94
39
NSE-NSE
Neural Semantic Encoders
17.90
40
Unsupervised NMT + Transformer
Phrase-Based & Neural Unsupervised Machine Translation
17.16
41
SMT + iterative backtranslation
(unsupervised)
Unsupervised Statistical Machine Translation
14.08
42

This benchmark is evaluating models on the test set of the WMT 2014 English-German news (full) dataset.

Step 1: Evaluate models locally

First, use our public benchmark library to evaluate your model. sotabench-eval is a framework-agnostic library that implements the WMT2014 Benchmark. See sotabench-eval docs here.

Once you can run the benchmark locally, you are ready to connect it to our automatic service.

Step 2: Login and connect your GitHub Repository

Connect your GitHub repository to automatically start benchmarking your repository. Once connected we'll re-benchmark your master branch on every commit, giving your users confidence in using models in your repository and helping you spot any bugs.