[Hdf-forum] HDF5 file as a binary stream ?

[Giraudon Cyril]

Hi Wermer,

I work at the moment on a scientific format based on HDF5.
The format permits to describe an electromagnetic simulation and
electromagnetic objects like materials, mesh, result data ...
(if you are interested : http://groups.google.com/group/amelet-hdf)

The idea would be to write data in memory and cache them in a cache
system like http://memcached.org/, to store and retore rapidly or to
share data with other processes.

I imagine a weak coupling system for scientific applications.

Regards,

Cyril.



Le 20/01/2010 15:38, Werner Benger a écrit :
> Hi Cyril,
>
>  there is a memory image driver in the standard HDF5 distribution, from
> which this streaming file driver is derived, but the streaming file
> driver
> is currently not part of the HDF5 distribution.
>
> What do you mean by putting a "file into a cache system" ?
>
>     Werner
>
>
> On Wed, 20 Jan 2010 13:21:15 +0100, Giraudon Cyril
> <cyril.giraudon at free.fr> wrote:
>
>> Thanks you Wermer,
>>
>> Must I really use a new driver ?
>> I thought I could use the hdf5_core driver and then manipulate a C
>> structure.
>> What should I do if I would to put a memory HDF5 file into a cache
>> system ?
>>
>> Regards,
>>
>> Cyril.
>>
>>
>>
>> Le 20/01/2010 12:26, Werner Benger a écrit :
>>> Hi Cyril,
>>>
>>>  this has been done as part of the German TIKSL project
>>> ( http://www.zib.de/visual/projects/TIKSL/ ) with the creation
>>> of the HDF5 Streaming VFD driver, which as of now is available
>>> through
>>>
>>>     http://hdf5-addons.origo.ethz.ch/
>>>
>>> It is however currently not actively worked on, so might require
>>> some code cleanup and revision, though it was recently working with
>>> HDF5 1.8 .
>>>
>>> It does not use MPI, but directly operates on TCP sockets. Thereby
>>> it does not conflict with an MPI application and can be used within
>>> some MPI-parallelized code without affecting that one.
>>>
>>> Cheers,
>>>     Werner
>>>
>>>
>>> On Wed, 20 Jan 2010 11:47:08 +0100, Giraudon Cyril
>>> <cyril.giraudon at free.fr> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have read HDF5 files can be in-memory structure.
>>>>
>>>> That said, is it be possible to the retrieve this HDF5 file as a
>>>> binary
>>>> array and send it by the network (as a float array with MPI for
>>>> instance) ?
>>>>
>>>> Thanks a lot,
>>>>
>>>> Cyril.
>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> Hdf-forum at hdfgroup.org
>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>
>>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum at hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
>>
>
>