download | home | papers | performance | projects | team | user guide | users

This page compares the performance of the following SML compilers on a range of benchmarks.

There are tables for run time ratio, absolute run time, compile time, and code size.

All of the benchmarks are available here. Some of the benchmarks were obtained from the SML/NJ benchmark suite. Some of the benchmarks use data files in the DATA subdirectory.

Setup

All benchmarks were compiled and run on a 1.6 GHz Pentium 4 with 4G of RAM. The benchmarks were compiled with the default settings for all the compilers, except for Moscow ML, which was passed the -orthodox -standalone -toplevel switches. The Poly/ML executables were produced by useing the file, followed by a PolyML.commit. The SML/NJ executables were produced by wrapping the entire program in a local declaration whose body performs an SMLofNJ.exportFn.

For more details, or if you want to run the benchmarks yourself, please see the benchmark directory of the MLton sources.

Run time ratio

The following table gives the ratio of the run time of each benchmark when compiled by another compiler to the run time when compiled by MLton. That is, the larger the number, the slower the generated code runs. A number larger than one indicates that the corresponding compiler produces code that runs more slowly than MLton. If an entry is *, that means that the corresponding compiler failed to compile the benchmark or that the benchmark failed to run.

benchmark ML-Kit Moscow-ML Poly/ML SML/NJ
barnes-hut * 11.8 12.5 1.1
boyer * 7.8 1.9 2.6
checksum 9.0 * * 4.0
count-graphs 6.3 37.1 6.6 2.4
DLXSimulator 1.8 * * *
fft 2.4 * 42.7 0.9
fib 1.3 5.4 0.9 1.4
hamlet * 10.6 2.3 1.8
imp-for 4.0 62.7 10.2 5.6
knuth-bendix * 15.1 7.0 2.9
lexgen 2.0 5.7 2.0 1.5
life 2.4 25.0 9.7 1.4
logic * 5.2 1.2 0.8
mandelbrot 13.3 42.7 67.1 1.4
matrix-multiply * 71.2 22.4 5.4
md5 * * * 5.6
merge * * 1.5 5.7
mlyacc * 6.2 1.3 1.4
model-elimination * 7.5 3.0 1.9
mpuz 3.8 62.9 6.6 4.0
nucleic * 12.3 22.5 0.4
peek 28.9 169.3 28.5 20.8
psdes-random 8.4 * * 6.1
ratio-regions 3.1 36.6 4.0 6.7
ray * 21.1 34.5 1.3
raytrace * * 38.1 2.7
simple 1.8 14.2 7.5 1.6
smith-normal-form * * * >1090
tailfib 1.4 35.1 2.3 1.4
tak 2.2 9.4 0.8 1.5
tensor * * * 20.6
tsp 2.9 22.6 * 13.7
tyan * 13.3 1.6 0.9
vector-concat 1.6 16.2 1.7 9.4
vector-rev 2.1 22.3 3.3 75.4
vliw 1.8 8.6 2.0 1.3
wc-input1 18.4 * 16.8 10.0
wc-scanStream 18.0 * 340.1 7.6
zebra 6.2 27.4 6.6 7.9
zern * * * 2.8

Absolute run time

The following table gives the run time of each benchmark in seconds. If an entry is *, that means that the corresponding compiler failed to compile the benchmark or that the benchmark failed to run.

benchmark MLton ML-Kit Moscow-ML Poly/ML SML/NJ
barnes-hut 32.8 * 386.9 411.6 34.6
boyer 37.3 * 291.2 71.2 97.9
checksum 46.4 417.3 * * 186.8
count-graphs 32.3 202.3 1197.7 212.7 76.2
DLXSimulator 74.1 136.2 * * *
fft 42.1 101.8 * 1796.7 39.2
fib 26.4 34.4 143.2 23.4 36.0
hamlet 36.1 * 382.1 84.4 64.8
imp-for 37.6 148.8 2353.6 381.8 210.9
knuth-bendix 31.4 * 475.0 219.3 92.7
lexgen 32.4 66.0 185.3 64.6 50.1
life 37.4 89.4 935.8 361.6 52.4
logic 40.5 * 212.5 48.0 34.0
mandelbrot 36.3 483.5 1549.4 2435.0 49.0
matrix-multiply 34.2 * 2430.9 766.6 183.3
md5 95.9 * * * 542.0
merge 57.1 * * 88.4 323.9
mlyacc 32.1 * 200.6 42.2 46.1
model-elimination 46.9 * 349.8 142.4 87.9
mpuz 24.6 92.2 1545.8 161.0 97.2
nucleic 39.0 * 480.1 877.9 16.2
peek 18.8 541.9 3176.5 535.5 390.5
psdes-random 17.9 150.2 * * 109.5
ratio-regions 31.2 98.3 1145.3 125.6 210.2
ray 16.1 * 340.1 555.1 20.6
raytrace 22.6 * * 861.7 62.1
simple 31.3 56.3 443.9 234.2 49.0
smith-normal-form 33.0 * * * >36,000
tailfib 29.6 42.7 1040.6 69.3 42.0
tak 56.5 122.5 530.7 46.2 86.6
tensor 20.1 * * * 415.1
tsp 36.0 106.2 815.3 * 494.1
tyan 37.2 * 492.6 57.8 34.2
vector-concat 63.9 104.2 1033.7 106.2 601.7
vector-rev 75.0 157.7 1671.5 245.4 5655.5
vliw 31.1 55.7 267.5 60.6 40.1
wc-input1 33.5 615.5 * 562.0 333.0
wc-scanStream 30.8 555.2 * 10487.8 233.0
zebra 36.6 227.9 1000.9 239.7 287.7
zern 40.0 * * * 112.2

For SML/NJ, the smith-normal-form benchmark was killed after running for 36,000 seconds.

Compile time

The following table gives the compile time of each benchmark in seconds. A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit Moscow-ML Poly/ML SML/NJ
barnes-hut 1.6 5.5 0.5 0.2 1.1
boyer 3.2 9.3 0.4 0.1 3.3
checksum 0.3 1.0 * * 0.2
count-graphs 1.0 2.0 0.1 0.1 0.7
DLXSimulator 2.5 7.3 * * *
fft 0.8 1.5 0.1 0.1 0.6
fib 0.4 0.8 0.0 0.0 0.2
hamlet 33.1 * 21.1 12.8 48.6
imp-for 0.4 1.0 0.0 0.0 0.2
knuth-bendix 1.4 4.2 0.2 0.1 1.4
lexgen 3.6 6.4 0.4 0.4 3.3
life 0.8 2.4 0.1 0.1 0.5
logic 1.9 4.8 0.2 0.1 1.4
mandelbrot 0.4 1.0 0.0 0.0 0.2
matrix-multiply 0.5 1.0 0.1 0.0 0.2
md5 0.8 1.8 * * 1.6
merge 0.4 1.1 0.0 0.0 0.2
mlyacc 16.3 37.1 3.7 1.6 14.9
model-elimination 14.4 * 2.3 2.8 24.8
mpuz 0.5 1.2 0.1 0.0 0.3
nucleic 3.7 17.6 1.1 0.3 1.5
peek 0.6 0.9 0.1 0.0 0.2
psdes-random 0.4 1.0 * * 0.2
ratio-regions 1.5 3.8 0.2 0.1 1.4
ray 2.2 3.2 0.1 0.1 0.8
raytrace 7.0 * * 0.6 4.7
simple 4.5 11.9 0.4 0.3 3.3
smith-normal-form 4.9 * * 0.1 2.1
tailfib 0.4 0.9 0.0 0.0 0.2
tak 0.4 0.9 0.0 0.0 0.2
tensor 2.0 * * * 2.1
tsp 0.9 1.9 0.1 * 0.5
tyan 2.5 5.8 0.3 0.2 1.9
vector-concat 0.4 0.9 0.0 0.0 0.2
vector-rev 0.4 0.9 0.0 0.0 0.2
vliw 8.1 27.0 1.4 1.3 12.2
wc-input1 1.0 0.9 0.0 0.0 0.2
wc-scanStream 1.0 0.9 0.1 0.0 0.2
zebra 4.3 2.6 0.1 0.1 0.6
zern 0.7 * * * 0.5

Code size

The following table gives the code size of each benchmark in bytes. The size for MLton and the ML Kit is the sum of text and data for the standalone executable as reported by size. The size for Moscow ML is the size in bytes of the executable a.out. The size for Poly/ML is the difference in size of the database before the session start and after the commit. The size for SML/NJ is the size of the heap file created by exportFn and does not include the size of the SML/NJ runtime system (approximately 100K). A * in an entry means that the compiler failed to compile the benchmark.

benchmark MLton ML-Kit Moscow-ML Poly/ML SML/NJ
barnes-hut 103,312 171,033 94,945 507,904 332,792
boyer 140,287 156,737 116,301 122,880 421,856
checksum 43,567 72,413 * * 320,512
count-graphs 64,567 88,617 84,613 98,304 359,472
DLXSimulator 102,224 162,261 * * *
fft 52,747 85,689 84,094 65,536 332,808
fib 43,575 16,125 79,878 49,152 294,880
hamlet 1,227,872 * 277,168 5,316,608 1,275,080
imp-for 43,591 16,853 80,041 57,344 295,904
knuth-bendix 86,096 97,177 88,440 180,224 326,624
lexgen 172,765 215,745 104,884 188,416 396,280
life 61,951 79,253 83,390 90,112 309,216
logic 103,623 115,217 87,252 114,688 335,840
mandelbrot 43,671 77,905 81,340 57,344 300,000
matrix-multiply 44,111 96,121 82,417 57,344 326,640
md5 52,712 91,441 * * 335,888
merge 44,895 25,585 80,091 49,152 295,912
mlyacc 536,605 502,081 148,287 2,908,160 701,480
model-elimination 636,000 * 175,885 2,187,264 896,136
mpuz 47,751 75,925 82,383 81,920 308,192
nucleic 82,623 221,121 207,154 204,800 355,312
peek 51,784 60,829 81,619 57,344 303,120
psdes-random 44,615 25,617 * * 301,024
ratio-regions 62,367 98,345 87,485 73,728 337,904
ray 104,080 112,309 89,860 147,456 387,144
raytrace 277,789 * * 524,288 508,008
simple 199,499 202,561 94,397 475,136 643,104
smith-normal-form 181,612 * * 131,072 488,520
tailfib 43,303 16,285 79,939 57,344 294,880
tak 43,735 16,041 79,926 57,344 290,784
tensor 104,347 * * * 345,120
tsp 59,296 99,497 86,147 * 323,576
tyan 107,152 146,117 91,587 204,800 371,728
vector-concat 44,079 24,485 80,193 49,152 305,136
vector-rev 43,967 24,681 80,075 57,344 305,136
vliw 323,593 477,129 135,387 696,320 628,816
wc-input1 66,045 132,749 85,772 49,152 300,000
wc-scanStream 66,717 133,261 85,948 49,152 301,024
zebra 142,296 44,709 83,419 90,112 314,352
zern 50,450 * * * 339,992

MLton
Last modified: Sat Nov 2 14:15:57 PST 2002