[hdf-forum] HDF5 multiple servers writing at once

[Robert Latham]

On Thu, Nov 27, 2008 at 10:23:42AM +0100, Francesc Alted wrote:
> No, neither PyTables nor HDF5 are thought for doing this safely.  What 
> you can do is synchronize your programs for doing writes to the file by 
> turns (providing some kind of file locking for the three processes).  
> If you follow that path, you must ensure that the file is effectively 
> opened and *closed* before and *after* each writing operation.

Normally I'd say use MPI to coordinate multiple writers, but the
amount of data is so small that it would be overkill to move to
parallel i/o.

Why not have a single designated writer thread or process which
consumes a work queue?  Depending on how clever you want your work
queue to be and what your consistency requirements are for the
underlying file, you could either batch updates to the HDF5 file or
ignore updates that are subsequently rendered out-of-date by newer
information.  

It's a little more work than passing a writer token, but gives you
some optimization opportunities.  

==rob

-- 
Rob Latham
Mathematics and Computer Science Division    A215 0178 EA2D B059 8CDF
Argonne National Lab, IL USA                 B29D F333 664A 4280 315B

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.