Synthetic Benchmarks (And How You Can Contribute Yours)

In order to assess the performance of Blosc in a variety of scenarios, a benchmark is provided so as to enable a fair comparison between different platforms. In this page, a series of plots about the performance of Blosc on a selected set of platforms are shown. It is fun to see the evolution of the hardware/software in the last years in terms of the speed of Blosc compared to a plain OS memcpy(). These conclusions can be applied to the evolution of the ratio of computing power versus memory bandwidth in general.

In case you want to compare your own platform against the ones listed here, after the plots there are instructions on how to compile, run and report back the results of this benchmark. Also you will be presented the opportunity to contribute testing Blosc by running the hard or extreme suites. Please read the sections below for the gory details.

Contributed outputs

Processor model: Intel Core i7 3930K Unlocked (6 x 3.2 GHZ) 12 MB Cache

Compiler: GCC version 4.6.1 20110908 (Red Hat 4.6.1-9)

OS: Fedora 15 2.6.42.9-1.fc15.x86_64 #1 SMP (64 bit)

Contributed by: Alvaro Tejero

Suite output (12 threads)

---

Processor model: 2 x Six-Core Intel Xeon CPU X5690 @ 3.47GHz

(12 physical cores, hyperthreading disabled)

Compiler: GCC 4.4.5 (Red Hat 4.4.5-6)

OS: Red Hat Enterprise Linux 6 (2.6.32-131.0.15.el6.x86_64), 64-bit

Contributed by: Valentin Haenel

Suite output (12 threads)

---

Processor model: 4 x Six-Core AMD Opteron(tm) Processor 8431 (24 cores @ 2.4 GHz)

Compiler: GCC (Debian 4.4.5-8) 4.4.5

OS: Debian 6.0.2 (Squeeze), 64-bit

Contributed by: Valentin Haenel

Suite output (24 threads)

---

Processor model: Six-Core Intel i7 (Bloomfield) @ 3.33 GHz / 1 processors

Compiler: GCC 4.4.5

OS: Debian 6.0.2 (Squeeze), 64-bit

Contributed by: Francesc Alted

Suite output (6 threads)

---

Processor model: Quad-Core Intel Xeon (Nehalem) @ 2.93 GHz / 2 processors

Compiler: GCC 4.5

OS: Mac OSX Snow Leopard (10.6.3), 64-bit

Contributed by: Louis Wicker

Suite output (8 threads)

---

Processor model: AMD Phenom II X6 @ 3.7 GHz

Compiler: MINGW64/GCC version 4.4.5

OS: Windows 7, 64-bit

Contributed by: Francesc Alted

Suite output (6 threads)

---

Processor model: Intel Core2 Quad @ 3 GHz

Compiler: MSVC 2008

OS: Windows, 64-bit

Contributed by: Christoph Gohlke

Suite output (4 threads)

---

Processor model: Intel Core2 Quad Q8400 @ 2.66 GHz

Compiler: GCC version 4.4.1

OS: openSUSE 11.2, 64-bit

Contributed by: Francesc Alted

Suite output (4 threads)

---

Processor model: Intel Core2 Duo E8400 @ 3 GHz

Compiler: GCC version 4.4.3

OS: OpenSUSE Linux 11.2, 64-bit

Contributed by: Francesc Alted

Suite output (2 threads)

---

Processor model: Dual-Core AMD Opteron 1214 @ 2.2 GHz

Compiler: GCC version 4.4.3

OS: Ubuntu Linux 10.04, 64-bit

Contributed by: Tony Theodore

Suite output (2 threads)

---

Processor model: Intel Pentium4 @ 3.2 GHz (with hyper-threading)

Compiler: GCC version 4.4.3

OS: Ubuntu Linux 10.04, 32-bit

Contributed by: Gabriel Beckers

Suite output (2 threads)

---

Processor model: Intel Atom 330 @ 1.6 GHz (2 physical cores, with hyper-threading)

Compiler: GCC version 4.5.2

OS: Ubuntu Linux 11.04, 64-bit

Contributed by: Valentin Haenel

Suite output (4 threads)

---

Processor model: Intel Atom N270 @ 1.6 GHz (with hyper-threading)

Compiler: GCC version 4.4.3

OS: Ubuntu Linux 10.04, 32-bit

Contributed by: Francesc Alted

Suite output (2 threads)

---

Processor model: PowerPC G4 @ 1.2 GHz / 512 KB L2 cache

Compiler: GCC version 4.0.1

OS: Mac OSX Tiger, 32-bit

Contributed by: Ivan Vilata

Suite output (1 thread)

---

How to compile (or get binaries for) the benchmark suite for Blosc

First, checkout the master version from:

 https://github.com/FrancescAlted/blosc

Then, compile the sources:

GCC (Unix) or MINGW (Windows):

$ cd your_blosc_sources/bench
$ gcc -O3 -msse2 -o bench bench.c ../blosc/*.c -lpthread

MINGW (Windows):

> gcc -O3 -msse2 -o bench bench.c blosc\*.c

MSVC 2008 or higher (Windows):

> cd your_blosc_sources\bench
> cl /Ox /Febench.exe bench.c ..\blosc\*.c

For those that are using Windows and do not have a compiler installed (I strongly recommend using the free MINGW), you can use the next binary for Windows 64-bit or binary for Windows 32-bit .

Running and plotting the different suites in benchmark

Now that you have the executable benchmark, you can run it by passing the 'suite' parameter followed by the number of cores in your machine to the bench program, i.e. something like:

$ ./bench suite [nthreads]

then a small suite will be run that checks the speed of Blosc for the specified number of threads. Given this output, you can convert it into a plot by using the bench/plot-speeds.py scripts (you will need the  matplotlib library installed). You can print a small online help for this script usage:

$ python plot-speeds.py -h
Usage: plot-speeds.py [-o outfile] [-t title ] [-d|-c] filename

Options:
  -h, --help            show this help message and exit
  -o OUTFILE, --outfile=OUTFILE
                        filename for output (many extensions supported, e.g.
                        .png, .jpg, .pdf)
  -t TITLE, --title=TITLE
                        title of the plot
  -l LIMIT, --limit=LIMIT
                        expression to limit number of threads shown
  -x XMAX, --xmax=XMAX  limit the x-axis
  -d, --decompress      plot decompression data
  -c, --compress        plot compression data

For example, if you have, say, 4 cores in your machine, and want to get the plots interactively, proceed like this:

$ ./bench suite 4 > mysuite.out
$ python plot-speeds.py -c mysuite.out   # get the compression plot
$ python plot-speeds.py -d mysuite.out   # get the decompression one

Alternatively, you can straight get a plot file by using the -o flag:

$ python plot-speeds.py -o plot.png -c mysuite.out

Sometimes the legend may cover some of the data in this case you can increase the limit of the x-axis (compression ratdio) using the -x switch (10 is quite a good value):

$ python plot-speeds.py -x 10 -c mysuite.out

If you have many, many threads, the output can become quite confusuing and you may want to take a look at the -l switch. This can limit the number of displayed threads using an arbitrary Python expression, like a list or an iterator over ints (indexing starts at 1, not 0):

$ python plot-speeds.py -l '[1]' -c mysuite.out
$ python plot-speeds.py -l 'range(1, 8)' mysuite.out
$ python plot-speeds.py -l 'range(1, 8, 2)' mysuite.out
$ python plot-speeds.py -l '[1, 3, 28]' mysuite.out

Also, if you have spare CPU cycles available, you may want to run the hardsuite, which is a series of tests that are much more comprehensive (and costly) than the suite above. It will take between 1 and 6 hours to run, depending on your machine and the number of cores, and will compress/decompress around 4 TB of data, checking that it has had a good round-trip. Running it is easy:

$ ./bench hardsuite 4 > myhardsuite.out
$ gzip -9 < myhardsuite.out > myhardsuite.out.gz    # use zip or 7z compressors if on Windows

IMPORTANT: In order to get fine results, please be sure that you are not running other heavy process while running the suites.

You can look into the output for the FAILED string in order to see if something went wrong. If FAILED does not appear anywhere, you can pretty sure that Blosc works well for your platform. If failures appear, please report this to me.

NOTE: You cannot use plot-speeds.py to plot the results of the hardsuite, as it is only meant for plotting suite output purposes.

[Incidentally, I've added a new suite called extremesuite that performs a crazy check on many, many possible inputs to Blosc. It works similarly than the hardsuite, but it can take between 2 and 3 days to finish on a relatively recent CPU, and can account up to 60 TB of data checked. Really, this is not everyone but in case you are brave enough you might want to have it a try.]

Reporting your results back

If you want to help with the fine-tuning of Blosc for other processors, please send your own output of these suites (either suite, hardsuite or both) to  me. That info will be extremely useful for allowing allow better compression ratios and performance in future versions. Please be sure that you also provide the next information:

CPU info: (vendor, model or cache sizes)
Operating System: (e.g. Linux/Windows/MacOSX/Solaris and version)
Compiler used: (e.g. GCC/ICC/MSVC/MINGW and version)

Thanks!

-- Francesc Alted