[hdf-forum] Severe slowdown when using chunks

[Andrew Collette]

Hi,

I noticed a serious issue this morning while tracking down some
performance problems in my code, which might affect other people here.
 It seems that writing performance deteriorates significantly (roughly
a factor of 6) when the following conditions are met:

(1) a dataset is stored in a chunked layout,
(2) the memory dataspace and file dataspace have different ranks

I'm not aware of any warnings in the end-user documentation about
this.  My application frequently performs writes from a single buffer
(say 10000 floats) to one "row" of a dataset (e.g. of dataspace (500,
10000)).  If, in chunked mode, I use a rank-2 memory dataspace with
shape (1, 10000), it takes about 6 seconds to write, but if I use a
rank-1 memory dataspace with shape (10000), it takes 34 seconds (!).

I've verified this behavior occurs with HDF5 1.6.6 and 1.8.3.  An
example C file is attached.

Andrew Collette
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ctest.c
Type: text/x-csrc
Size: 1377 bytes
Desc: not available
URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/20090609/71c14395/attachment.bin>
-------------- next part --------------
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.