[Hdf-forum] HDF5 file as a binary stream ?

[Quincey Koziol]

Hi Cyril,

On Jan 20, 2010, at 6:21 AM, Giraudon Cyril wrote:

> Thanks you Werner,
> 
> Must I really use a new driver ?
> I thought I could use the hdf5_core driver and then manipulate a C
> structure.
> What should I do if I would to put a memory HDF5 file into a cache system ?

	You can use the core driver to "store" an HDF5 file in memory, along with the H5Fget_vfd_handle to retrieve a pointer to the memory buffer where the file is stored.  However, we are missing the final piece of the puzzle: a way to "open" a file from a buffer in memory with the core driver.  So, at the moment, you need to store the buffer in a file on the receiving side and then open it with a file-oriented VFD (sec2, stdio, mpio, etc).  However, if you'd like to work on a patch for opening a file from a memory buffer, I'd be happy to work with you on how I think you should proceed, etc.

	Quincey

> Regards,
> 
> Cyril.
> 
> 
> 
> Le 20/01/2010 12:26, Werner Benger a écrit :
>> Hi Cyril,
>> 
>> this has been done as part of the German TIKSL project
>> ( http://www.zib.de/visual/projects/TIKSL/ ) with the creation
>> of the HDF5 Streaming VFD driver, which as of now is available
>> through
>> 
>>    http://hdf5-addons.origo.ethz.ch/
>> 
>> It is however currently not actively worked on, so might require
>> some code cleanup and revision, though it was recently working with
>> HDF5 1.8 .
>> 
>> It does not use MPI, but directly operates on TCP sockets. Thereby
>> it does not conflict with an MPI application and can be used within
>> some MPI-parallelized code without affecting that one.
>> 
>> Cheers,
>>    Werner
>> 
>> 
>> On Wed, 20 Jan 2010 11:47:08 +0100, Giraudon Cyril
>> <cyril.giraudon at free.fr> wrote:
>> 
>>> Hello,
>>> 
>>> I have read HDF5 files can be in-memory structure.
>>> 
>>> That said, is it be possible to the retrieve this HDF5 file as a binary
>>> array and send it by the network (as a float array with MPI for
>>> instance) ?
>>> 
>>> Thanks a lot,
>>> 
>>> Cyril.
>>> 
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum at hdfgroup.org
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>> 
>>> 
>> 
>> 
> 
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org