DAWNBench is a benchmark suite for end-to-end deep learning training and inference. Computation time and cost are critical resources in building deep models, yet many existing benchmarks focus solely on model accuracy. DAWNBench provides a reference set of common deep learning workloads for quantifying training time, training cost, inference latency, and inference cost across different optimization strategies, model architectures, software frameworks, clouds, and hardware.

In my test, 33 out of 50 runs reached 94% test set accuracy. Runtime for 35 epochs is roughly 29sec using Kakao Brain BrainCloud V4.XLARGE Type (V100 4GPU, 56CPU, 488GB).

In my test, 30 out of 50 runs reached 94% test set accuracy. Runtime for 25 epochs is roughly 57sec using Kakao Brain BrainCloud V1.XLARGE Type (V100 1GPU, 56CPU, 488GB).

In my test, runs reached 94% test set accuracy. Runtime latency per sample is roughly 0.9 milliseconds using Kakao Brain BrainCloud V1.XLARGE Type (V100 1GPU, 56CPU, 488GB).

accuracy:94.20%, 0.85933us per sample

• pre requirements

$ apt update
$ apt install -y libsm6 libxext-dev libxrender-dev libcap-dev
$ pip install torch torchvision

• clone and init. the repository

$ git clone {THIS_REPOSITORY} && cd torchskeleton
$ git submodule init
$ git submodule update
$ pip install -r requirements.txt

• run dawnbench image classification on CIFAR10

$ # train
$ python bin/dawnbench/cifar10.py --seed 0xC0FFEE --download > log_dawnbench_cifar10.tsv
$ python bin/dawnbench/cifar10_multigpu.py --num-gpus 4 --seed 0x00CAFE --download > log_dawnbench_cifar10_multigpu.tsv
$ # infer
$ python bin/dawnbench/cifar10_infer.py --model assets/kakaobrain_custom-resnet9_single_cifar10.pth  --download

• prepare resized imagenet

$ # original : ./data/imagenet
$ # resized : ./data/imagenet-sz/160
$ python scripts/resize.py

This project is developed by Woonhyuk Baek at Kakao Brain. It is distributed under Apache License2.0.