[hdf-forum] Advice on way to write HDF file for optimal reading.

[Thomas, Brian (GE Infra, Energy)]

Andrew,

Hyperslabs are the best way to extract data when working with large
datasets and for selective read. You can use hyperslabs for 2
dimensional as well as 3 dimensional arrays too. Use optimal indexing
techniques and chunk size while you create the HDF5 dataset for optimal
speed while reading from it.

Thanks,
Brian

-----Original Message-----
From: Andrew Cunningham [mailto:andrewc at mac.com] 
Sent: Wednesday, October 29, 2008 5:41 PM
To: HDF Forum
Subject: [hdf-forum] Advice on way to write HDF file for optimal
reading.

Hi,
	 I am looking for advice on the optimal way to R/W a HDF file in
C/C+ 
+ under the following circumstances.
	Essentially my data is one large array of single precision data
of , say , 1 million rows by , say, 3000 columns. Sometimes it can be
smaller, of course, and may be larger, but that would be a typical
'large' problem.

	The data is "received" by columns 1,2,3......N. Obviously I
cannot hold all the data in memory so I write a column 'hyperslab' one
at a time as I get the data.
	
	The main requirement is I need to efficiently be able to read a
random selection of rows. So I need to read, say, the data from a 1000
rows (random indices) into memory. I do not know before hand which rows
will need to be read. Obviously the implementation is a to construct a
hyperslab read of those rows - but is that the most efficient way?
	
	Any tips before I dive in?

Andrew
	

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.


----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.