[Hdf-forum] Slow conversion to binary using h5dump

[Peter Cao]

Mark,

the default buffer size used in h5dump is 1MB. Setting it to 4MB will 
improve
the performance but it may be still slow because the buffer size is much 
less
than the chunk size.

If you use h5repack to change the chunk size to (1x64x3072, a little 
less than 1MB)
and try h5dump again, you will see the difference.

Thanks
--pc


Mark Howison wrote:
> I tried using BE and LE and both are equally slow. Here is the header
> info. Also, I should note that the dataset is roughly 108GB, but it
> does fit into local memory (196GB is available). Also, it seems to
> continuously write at 4MB/s, instead of sitting and processing for a
> while and then bursting at 100MB/s or something. It is also chunked.
> Maybe this is causing problems, because h5dump has to jump around to
> non-contiguous offsets to contiguously assemble the binary output?
>
> Thanks,
> Mark
>
> mhowison at davinci:/project/projectdirs/vacet/mark> h5dump -p -H -d
> /Step#0/Block/Analyze7.5/0 combustion.h5part
> HDF5 "combustion.h5part" {
> DATASET "/Step#0/Block/Analyze7.5/0" {
>    DATATYPE  H5T_IEEE_F32LE
>    DATASPACE  SIMPLE { ( 3072, 3072, 3072 ) / ( 3072, 3072, 3072 ) }
>    STORAGE_LAYOUT {
>       CHUNKED ( 1024, 768, 768 )
>       SIZE 115964116992
>     }
>    FILTERS {
>       NONE
>    }
>    FILLVALUE {
>       FILL_TIME H5D_FILL_TIME_IFSET
>       VALUE  0
>    }
>    ALLOCATION_TIME {
>       H5D_ALLOC_TIME_EARLY
>    }
> }
> }
>
>
> On Tue, Jan 12, 2010 at 3:40 PM, Elena Pourmal <epourmal at hdfgroup.org> wrote:
>   
>> Mark,
>>
>> h5dump performance may be affected by many factors (size of the h5dump default read buffer, chunking sizes of the dataset, compression, etc.)
>> Would it be possible for you to do h5dump -p -H -d .... to print the header information for the dataset you are trying to export? We may have a better idea what may go wrong.
>>
>> Thank you!
>>
>> Elena
>> On Jan 12, 2010, at 12:15 PM, Jonathan Kim wrote:
>>
>>     
>>> Hi,
>>>
>>> It's different from h5copy.
>>>
>>> What the original data format? BE or LE?
>>> If it's BE, could you try -b BE and see if any performance difference?
>>>
>>> And could you try with smaller size of HDF5 file?  (under 10GB)
>>>
>>> Also could you try on other filesystems?  (non-parallel as well)
>>>
>>> Since it's the performance issue not a specific bug, more testing results
>>> would be helpful.
>>>
>>> Thanks.
>>>
>>> - Jonathan
>>>
>>> -----Original Message-----
>>> From: hdf-forum-bounces at hdfgroup.org [mailto:hdf-forum-bounces at hdfgroup.org]
>>> On Behalf Of Mark Howison
>>> Sent: Tuesday, January 12, 2010 10:26 AM
>>> To: HDF forum
>>> Subject: [Hdf-forum] Slow conversion to binary using h5dump
>>>
>>> Hi, I tried converting a 108GB HDF5 file to binary using the "-b LE"
>>> flag in h5dump, but it ran at a crawling pace, only about 4MB/s. This
>>> is in comparison to an h5copy I did on the same machine (our SGI
>>> Altix) that ran at 600MB/s. The filesystem is GPFS. Any ideas why
>>> h5dump is having so much trouble? Is there a conversion phase (to LE)
>>> that is bogging things down? Thanks, Mark
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum at hdfgroup.org
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum at hdfgroup.org
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>       
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum at hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>>     
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>