| 1 | Blosc supports threads now |
|---|
| 2 | ========================== |
|---|
| 3 | |
|---|
| 4 | It just happened: Blosc can be run in threaded mode for both |
|---|
| 5 | compressing and decompressing. However, threaded Blosc doesn't work |
|---|
| 6 | better than the serial version in all cases and the reason is that |
|---|
| 7 | threads do add a non-negligible overhead (most specially, the cost of |
|---|
| 8 | synchronization between them). |
|---|
| 9 | |
|---|
| 10 | In order to reduce the overhead of threads as much as possible, I've |
|---|
| 11 | decided to implement a pool of threads (the workers) that are waiting |
|---|
| 12 | for the main process (the master) to send them jobs (basically, |
|---|
| 13 | compressing and decompressing small blocks of the initial buffer). |
|---|
| 14 | |
|---|
| 15 | Despite this and many other internal optimizations in the threaded |
|---|
| 16 | code, it does not work faster than the serial version for buffer sizes |
|---|
| 17 | around 64/128 KB or less. This is for Intel Quad Core2 (Q8400 @ 2.66 |
|---|
| 18 | GHz) / Linux (openSUSE 11.2, 64 bit), but your mileage may vary (and |
|---|
| 19 | will vary!) for other processors / operating systems. |
|---|
| 20 | |
|---|
| 21 | In contrast, for buffers larger than 64/128 KB, the threaded version |
|---|
| 22 | starts to perform significantly better, being the sweet point at 1 MB |
|---|
| 23 | (again, this is with my setup). For larger buffer sizes than 1 MB, |
|---|
| 24 | the threaded code slows down, but it is still considerably faster than |
|---|
| 25 | serial code. |
|---|
| 26 | |
|---|
| 27 | This is why Blosc falls back to use the serial version for such a |
|---|
| 28 | 'small' buffers. So, you don't have to worry too much about deciding |
|---|
| 29 | whether you should set the number of threads to 1 (serial) or more |
|---|
| 30 | (parallel). Just set it to the number of cores in your processor and |
|---|
| 31 | your are done! |
|---|
| 32 | |
|---|
| 33 | Francesc Alted |
|---|
| 34 | 2010-05-03 |
|---|