[Hdf-forum] HDF5 file as a binary stream ?

Thu Jan 21 07:21:28 EST 2010

Hi Cyril,

  the aforementioned HDF5 streaming VFD already provides a complete server, all
the socket communication is done internally. Only the TCP socket needs to be opened
by the  application, and the socket descriptor is passed to the HDF5 streaming VFD,
like otherwise a filename is passed to it. You can write to a stream or read from it,
and if you include an HTTP header in this stream - which is supported by the most
recent version of the streaming VFD - then it easily becomes its standalone webserver
that produces HDF5 files on demand in memory, just for streaming data, never
a physical file created anywhere. If your application can write to an HDF5 file,
then it can also write to a stream, no change required on the writing code,
only the "file opening" part needs to know about the streaming VFD.

However, I don't want to promise too much, only saying that it has worked
this way some time ago, and it *should* still work.

Cheers,
	Werner

On Thu, 21 Jan 2010 11:28:25 +0100, Giraudon Cyril <cyril.giraudon at free.fr> wrote:

> Wermer,
>
> I don't express my mind correctly.
> I 'd like to do strong coupling between applications (different
> applications, different processes) without being too intrusive into the
> applications. There would be some data exchanges at each step of the
> algorithm (time domain computation, FDTD for instance) as with an MPI
> solution.
>
> The amount of data is relatively low but I have to perform exchange the
> fastest as possible. Amelet-HDF can be seen as a cross language IDL of
> kind for electromagnetic computations. I 'd like a simple solution for
> C/Fortran languages.
>
> I will consider your proposal, although I did not want to write a
> complete server.
>
> Regards,
>
> Cyril.
>
>
> Le 20/01/2010 16:11, Werner Benger a écrit :
>> Hi Cyril,
>>
>>  this sounds like a nice project, similar to what we're doing as well
>> (file format: http://www.fiberbundle.net has similar intentions).
>>
>> It seems you would not use memcached itself, as it is only designed for
>> "small" chunks:
>>
>> "Memcached is an in-memory key-value store for small chunks of
>> arbitrary data
>> (strings, objects) from results of database calls, API calls, or page
>> rendering." ?
>>
>> Rather you'd had something like a server that retrieves complete HDF5
>> files
>> via socket and make them available to other clients, like on some NFS
>> or SSHFS
>> where the file written is closed after each operation, and thereby
>> "published"
>> for other clients to read the same file?
>>
>>     Werner
>>
>>
>> On Wed, 20 Jan 2010 15:58:33 +0100, Giraudon Cyril
>> <cyril.giraudon at free.fr> wrote:
>>
>>> Hi Wermer,
>>>
>>> I work at the moment on a scientific format based on HDF5.
>>> The format permits to describe an electromagnetic simulation and
>>> electromagnetic objects like materials, mesh, result data ...
>>> (if you are interested : http://groups.google.com/group/amelet-hdf)
>>>
>>> The idea would be to write data in memory and cache them in a cache
>>> system like http://memcached.org/, to store and retore rapidly or to
>>> share data with other processes.
>>>
>>> I imagine a weak coupling system for scientific applications.
>>>
>>> Regards,
>>>
>>> Cyril.
>>>
>>>
>>>
>>> Le 20/01/2010 15:38, Werner Benger a écrit :
>>>> Hi Cyril,
>>>>
>>>>  there is a memory image driver in the standard HDF5 distribution, from
>>>> which this streaming file driver is derived, but the streaming file
>>>> driver
>>>> is currently not part of the HDF5 distribution.
>>>>
>>>> What do you mean by putting a "file into a cache system" ?
>>>>
>>>>     Werner
>>>>
>>>>
>>>> On Wed, 20 Jan 2010 13:21:15 +0100, Giraudon Cyril
>>>> <cyril.giraudon at free.fr> wrote:
>>>>
>>>>> Thanks you Wermer,
>>>>>
>>>>> Must I really use a new driver ?
>>>>> I thought I could use the hdf5_core driver and then manipulate a C
>>>>> structure.
>>>>> What should I do if I would to put a memory HDF5 file into a cache
>>>>> system ?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Cyril.
>>>>>
>>>>>
>>>>>
>>>>> Le 20/01/2010 12:26, Werner Benger a écrit :
>>>>>> Hi Cyril,
>>>>>>
>>>>>>  this has been done as part of the German TIKSL project
>>>>>> ( http://www.zib.de/visual/projects/TIKSL/ ) with the creation
>>>>>> of the HDF5 Streaming VFD driver, which as of now is available
>>>>>> through
>>>>>>
>>>>>>     http://hdf5-addons.origo.ethz.ch/
>>>>>>
>>>>>> It is however currently not actively worked on, so might require
>>>>>> some code cleanup and revision, though it was recently working with
>>>>>> HDF5 1.8 .
>>>>>>
>>>>>> It does not use MPI, but directly operates on TCP sockets. Thereby
>>>>>> it does not conflict with an MPI application and can be used within
>>>>>> some MPI-parallelized code without affecting that one.
>>>>>>
>>>>>> Cheers,
>>>>>>     Werner
>>>>>>
>>>>>>
>>>>>> On Wed, 20 Jan 2010 11:47:08 +0100, Giraudon Cyril
>>>>>> <cyril.giraudon at free.fr> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have read HDF5 files can be in-memory structure.
>>>>>>>
>>>>>>> That said, is it be possible to the retrieve this HDF5 file as a
>>>>>>> binary
>>>>>>> array and send it by the network (as a float array with MPI for
>>>>>>> instance) ?
>>>>>>>
>>>>>>> Thanks a lot,
>>>>>>>
>>>>>>> Cyril.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Hdf-forum is for HDF software users discussion.
>>>>>>> Hdf-forum at hdfgroup.org
>>>>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Hdf-forum is for HDF software users discussion.
>>>>> Hdf-forum at hdfgroup.org
>>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum at hdfgroup.org
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>
>>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>

-- 
___________________________________________________________________________
Dr. Werner Benger                Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809                        Fax.: +1 225 578-5362