huggingface / transformers

EM F1
MODEL CODE PAPER
ε-REPR
CODE PAPER
ε-REPR
PAPER
GLOBAL RANK
BERT large
(whole word masking, cased)
86.3% -- 92.8% -- #3
BERT large
(whole word masking, uncased)
86.9% -- 93.2% -- #2
DistilBERT
79.1% 77.7%
86.9% 85.8%
#4
TEST PERPLEXITY
SPEED
MODEL CODE PAPER
ε-REPR
PAPER
GLOBAL RANK
DistilGPT2
49.91 -- 38425.0 #8
GPT1
90.61 -- 29444.9 #9
GPT-2 Large
22.61 22.05 4482.9 #5
GPT-2 Medium
26.51 26.37 9316.7 #6
GPT-2 Small
36.53 37.50
25200.1 #7
Transformer-XL Large
18.19 18.30 2970.8 #3
See Full Build Details +get badge code
[![SotaBench](https://img.shields.io/endpoint.svg?url=https://sotabench.com/api/v0/badge/gh/PiotrCzapla/transformers)](https://sotabench.com/user/piotr.czapla-priv/repos/PiotrCzapla/transformers)

How the Repository is Evaluated

The full sotabench.py file - source
from impl import language_modelling as lm
from impl import squad
def main():
   squad.main()
   lm.main()
   
if __name__ == '__main__':
    main()
STATUS
BUILD
COMMIT MESSAGE
RUN TIME
Merge branch 'master' of https://github.com/huggingface/transfor…
PiotrCzapla   7c4cc1c  (+297 commits )  ·  Nov 08 2019
0h:19m:15s
0h:16m:16s
0h:17m:40s
Remove `.eval` and use evaluation loop explicitly
PiotrCzapla   df27a94  ·  Oct 09 2019
0h:17m:53s
0h:17m:57s
0h:14m:03s
Clean up sotabench_setup.sh
PiotrCzapla   d2c7d85  ·  Oct 08 2019
0h:26m:28s
Move sotabench files to sotabench/
PiotrCzapla   f54f739  ·  Oct 04 2019
0h:06m:48s
Add distilgpt2 + model selection via env var MODEL
PiotrCzapla   ad2e7b4  ·  Oct 04 2019
0h:17m:46s
Fix distilbert and rename langauge_model[l]ing
PiotrCzapla   c35b5fe  (+42 commits )  ·  Oct 04 2019
0h:17m:33s
disable DistilBert temporarliy
PiotrCzapla   ae970ba  ·  Oct 04 2019
0h:12m:58s
fix squad
PiotrCzapla   009b71b  ·  Oct 03 2019
0h:20m:38s
Add cache support & more squad models
PiotrCzapla   f2b9f75  ·  Oct 03 2019
0h:12m:18s
Update to latest sotabench-eval
PiotrCzapla   af884ec  ·  Oct 03 2019
0h:21m:10s
fully disable cache & first batch calc.
PiotrCzapla   35e8751  ·  Oct 02 2019
0h:20m:09s
Disable cache in squad for the time being
PiotrCzapla   6b50a88  ·  Oct 02 2019
0h:20m:38s
Fix broken pretrained model names
PiotrCzapla   0a319de  ·  Oct 02 2019
0h:16m:33s
add fixed sotabench squad
PiotrCzapla   e57d472  ·  Oct 02 2019
0h:09m:40s
Remove squad from sotabench.py to see wikitext results
PiotrCzapla   326c8b9  ·  Oct 02 2019
0h:09m:40s
Temporarliy copy original uils_squad files.
PiotrCzapla   a5f0969  (+2 commits )  ·  Oct 02 2019
0h:07m:36s
Remove TensorbordX deps
PiotrCzapla   919da8b  ·  Oct 02 2019
0h:07m:28s
Fix missing force_full_run
PiotrCzapla   c689aff  ·  Oct 02 2019
0h:06m:50s
Fix missing modules in squad.py
PiotrCzapla   9436a67  ·  Oct 02 2019
0h:07m:59s
Add crud quad evaluator
PiotrCzapla   0ed3267  ·  Oct 02 2019
0h:03m:14s
0h:12m:39s
Rename sotabench.sh back to _setup.sh
PiotrCzapla   5bebbaf  ·  Oct 02 2019
0h:11m:37s
change sotabench_setup to sotabench
PiotrCzapla   af01520  ·  Oct 02 2019
0h:05m:50s
Temp sotabenc_scripts folder
PiotrCzapla   abe74c1  ·  Oct 02 2019
0h:05m:07s
unknown
0h:02m:19s
move sotabench.py to sotabench folder
PiotrCzapla   426204e  ·  Oct 02 2019
0h:02m:03s
Clean up imports and remove XLNet
PiotrCzapla 9ae6f57    0eb7d2f  ·  Oct 02 2019
0h:02m:23s
XLNet fixes
PiotrCzapla   6bf7f73  ·  Oct 01 2019
0h:14m:22s
Final fix to the wikitext extraction?
PiotrCzapla   a82971e  ·  Oct 01 2019
0h:15m:06s
Dataset setup fix, gpt2 large batch size=1
PiotrCzapla   1934d82  ·  Oct 01 2019
0h:05m:33s
fix sotabench setup + add more models
PiotrCzapla   8663259  ·  Oct 01 2019
0h:05m:26s
fixes to sotabench_setup.sh
PiotrCzapla   ea3fbb4  ·  Oct 01 2019
0h:04m:49s
Fix to sotabench setup to unzip data and install torch
PiotrCzapla   a652120  ·  Oct 01 2019
0h:04m:09s
add torch installation
PiotrCzapla   41c2da5  ·  Oct 01 2019
0h:03m:50s
Fix sotabench setup
PiotrCzapla   573cbe4  ·  Oct 01 2019
0h:03m:57s
Add downloading of datasets
PiotrCzapla   76c010a  ·  Oct 01 2019
0h:03m:49s
0h:04m:11s