[Hdf-forum] Slow conversion to binary using h5dump

[Mark Howison]

I tried using BE and LE and both are equally slow. Here is the header
info. Also, I should note that the dataset is roughly 108GB, but it
does fit into local memory (196GB is available). Also, it seems to
continuously write at 4MB/s, instead of sitting and processing for a
while and then bursting at 100MB/s or something. It is also chunked.
Maybe this is causing problems, because h5dump has to jump around to
non-contiguous offsets to contiguously assemble the binary output?

Thanks,
Mark

mhowison at davinci:/project/projectdirs/vacet/mark> h5dump -p -H -d
/Step#0/Block/Analyze7.5/0 combustion.h5part
HDF5 "combustion.h5part" {
DATASET "/Step#0/Block/Analyze7.5/0" {
   DATATYPE  H5T_IEEE_F32LE
   DATASPACE  SIMPLE { ( 3072, 3072, 3072 ) / ( 3072, 3072, 3072 ) }
   STORAGE_LAYOUT {
      CHUNKED ( 1024, 768, 768 )
      SIZE 115964116992
    }
   FILTERS {
      NONE
   }
   FILLVALUE {
      FILL_TIME H5D_FILL_TIME_IFSET
      VALUE  0
   }
   ALLOCATION_TIME {
      H5D_ALLOC_TIME_EARLY
   }
}
}


On Tue, Jan 12, 2010 at 3:40 PM, Elena Pourmal <epourmal at hdfgroup.org> wrote:
> Mark,
>
> h5dump performance may be affected by many factors (size of the h5dump default read buffer, chunking sizes of the dataset, compression, etc.)
> Would it be possible for you to do h5dump -p -H -d .... to print the header information for the dataset you are trying to export? We may have a better idea what may go wrong.
>
> Thank you!
>
> Elena
> On Jan 12, 2010, at 12:15 PM, Jonathan Kim wrote:
>
>> Hi,
>>
>> It's different from h5copy.
>>
>> What the original data format? BE or LE?
>> If it's BE, could you try -b BE and see if any performance difference?
>>
>> And could you try with smaller size of HDF5 file?  (under 10GB)
>>
>> Also could you try on other filesystems?  (non-parallel as well)
>>
>> Since it's the performance issue not a specific bug, more testing results
>> would be helpful.
>>
>> Thanks.
>>
>> - Jonathan
>>
>> -----Original Message-----
>> From: hdf-forum-bounces at hdfgroup.org [mailto:hdf-forum-bounces at hdfgroup.org]
>> On Behalf Of Mark Howison
>> Sent: Tuesday, January 12, 2010 10:26 AM
>> To: HDF forum
>> Subject: [Hdf-forum] Slow conversion to binary using h5dump
>>
>> Hi, I tried converting a 108GB HDF5 file to binary using the "-b LE"
>> flag in h5dump, but it ran at a crawling pace, only about 4MB/s. This
>> is in comparison to an h5copy I did on the same machine (our SGI
>> Altix) that ran at 600MB/s. The filesystem is GPFS. Any ideas why
>> h5dump is having so much trouble? Is there a conversion phase (to LE)
>> that is bogging things down? Thanks, Mark
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum at hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum at hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>