[hdf-forum] speeding up h5repack

[Brock Palen]

On Oct 28, 2008, at 7:21 PM, George N. White III wrote:

> On Tue, Oct 28, 2008 at 3:20 PM, Brock Palen <brockp at umich.edu> wrote:
>
>> Is there any tweaks that can be done to speed up compressing  
>> already created
>> hdf5 files?
>>
>> For example
>>
>> h5repack -v -i rt_3d_71nm_5micron_hdf5_plt_cnt_0010 -o  
>> lt_cnt_0010_zipped -f
>> GZIP=1
>>
>> Takes 129 Minutes
>>
>> While:
>> gzip rt_3d_71nm_5micron_hdf5_plt_cnt_0010
>>
>> Takes 1.5 Minutes
>>
>> hdf5-1.6.7
>>
>> We don't have szip enabled, but would be interested in trying  
>> (academic work
>> so licensing should not be a problem).
>>
>> Just seemed strange that it took so long,  the uncompressed hdf5  
>> file is
>> from FLASH2.5.
>>
>> Any insight would be nice.
>
> There is overhead processing the structural information in hdf5 files,
> and there is startup overhead for the compression library (setting up
> the structures) for each
> chunk to be compressed.   It would be interesting to see the time  
> for some
> trivial h5repack operation (-f NONE, scaling?).   In principle,  
> h5repack should
> be able to take advantage of parallel processing, so if you could  
> get 1000
> processors going you mght beat gzip by a large factor.
>
> What were the file sizes?

about 2GB uncompressed.

>
> The gzip program supports levels 1--9 (fast, less compressed to slow,
> more compressed), with default 6, so your gzip run should have been
> doing more

Yes thats what I thought,

> compression work than h5repack.  The question is how much of the  
> overhead
> is dealing with the hdf5 structure and how much from the  
> compression library
> startup.   Function call profiles would give you the number of  
> calls to deflate
> and deflateInit for the two runs.

I don't have the time to compile it with -pg  but I should point out  
that 160minutes to 1.5 minutes in a huge spread.

What does h5repack do if the file is not chunked?  These files were  
written using parallel hdf5, but I did not write the app and am only  
starting to learn hdf5.

>
> -- 
> George N. White III <aa056 at chebucto.ns.ca>
> Head of St. Margarets Bay, Nova Scotia
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum- 
> subscribe at hdfgroup.org.
> To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.
>
>
>


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.