Explore Benchmarks
Latest Results
How it Works
INTEGRATED WITH GITHUB
Connect your repo and benchmark models for freeMODELS RUN ON BENCHMARKS
Free GPUs to run your code on public benchmarksCOMPARED TO PAPERS
Results compared to papers to test reproducibilityHow to Add Your Repository
Step One : Evaluate Locally
Each public benchmark has its own instructions on how to use.
For example, to use the Image Classification on ImageNet benchmark
on your model in a framework-independent way, create a
sotabench.py
file like this:
from sotabencheval.image_classification import ImageNetEvaluator
evaluator = ImageNetEvaluator(
# automatically compare to this paper
model_name='ResNeXt-101-32x8d',
paper_arxiv_id='1611.05431'
)
predictions = ... # use your model to make predictions
evaluator.add(predictions)
evaluator.save()
Step Two : Connect Git and Showcase Results
Sotabench is like Continuous Integration, but instead of running
unit tests, it benchmarks models in sotabench.py
on every commit.
For developers, this is an easier way to continuously test their ML models, allowing for direct comparison with other repositories and papers.
For the community, this is a free and always up-to-date reference of ML model implementations.


