[Hdf-forum] writing data as fast as possible

Tue Jan 19 17:05:12 EST 2010

Hi Ger,

On Jan 15, 2010, at 4:28 AM, Ger van Diepen wrote:

> 
> We have an instrument generating several GBytes/second. 
> We would like to dump these data as fast as possible to disk (of course
> in a parallel way), preferably with O_DIRECT to bypass the OS file cache
> and to avoid seeks to maintain meta data. 
> Yet, it would be nice if these data can be seen as an HDF5 dataset. 
> 
> My question is if it possible to do something like this. Either that
> HDF5 has the option to write data in this way or through some mechanism
> that HDF5 can treat such an external file as HDF5 data, possibly by
> dynamically loading a shared library that knows how to interpret the
> data. 
> 
> I'm asking this because we have the impression that something like an
> extendible dataset does not give us the performance/robustness we need.
> In case of a crash we would like to lose as few data as possible, which
> means that a regular flush has to be done which might kill performance.
> I guess that without a flush the indices are only written at the end, so
> the data cannot be found in case of a crash. 

	You might want the "external" dataset storage feature - try H5Pset_external().  However, I'll also mention that we are working on parallel I/O improvements (and extensible dataset improvements) currently, so it may not be necessary to use external storage and still get the performance you are looking for.

	Quincey

> Cheers, 
> Ger van Diepen 
> 
> begin 644 TEXT.htm
> M/&AT;6P^"B`@/&AE860^"@H@(#PO:&5A9#X*("`\8F]D>2!S='EL93TB9F]N
> M="UV87)I86YT.B!N;W)M86P[(&QI;F4M:&5I9VAT.B!N;W)M86P[(&UA<F=I
> M;BUR:6=H=#H at -'!X.R!M87)G:6XM=&]P.B`T<'@[(&UA<F=I;BUB;W1T;VTZ
> M(#%P>#L@;6%R9VEN+6QE9G0Z(#1P>"(^"B`@("`\<"!S='EL93TB;6%R9VEN
> M+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^"B`@("`@(#QF;VYT(&9A8V4]
> M(DQU8VED82!'<F%N9&4B('-I>F4](C,B/E=E(&AA=F4 at 86X@:6YS=')U;65N
> M="!G96YE<F%T:6YG('-E=F5R86P at 1T)Y=&5S+W-E8V]N9"X\+V9O;G0^("`@
> M(#PO<#X*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM
> M=&]P.B`P(CX*("`@("`@/&9O;G0 at 9F%C93TB3'5C:61A($=R86YD92(@<VEZ
> M93TB,R(^5V4@=V]U;&0@;&EK92!T;R!D=6UP('1H97-E(&1A=&$@87, at 9F%S
> M="!A<R!P;W-S:6)L92!T;R!D:7-K("8C-#`[;V8 at 8V]U<G-E(&EN(&$@<&%R
> M86QL96P@=V%Y)B,T,3LF(S0T.R!P<F5F97)A8FQY('=I=&@@3U]$25)%0U0@
> M=&\@8GEP87-S('1H92!/4R!F:6QE(&-A8VAE(&%N9"!T;R!A=F]I9"!S965K
> M<R!T;R!M86EN=&%I;B!M971A(&1A=&$N/"]F;VYT/B`@("`\+W`^"B`@("`\
> M<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^"B`@
> M("`@(#QF;VYT(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/EEE="8C
> M-#0[(&ET('=O=6QD(&)E(&YI8V4@:68@=&AE<V4 at 9&%T82!C86X at 8F4@<V5E
> M;B!A<R!A;B!(1$8U(&1A=&%S970N/"]F;VYT/B`@("`\+W`^"CQB<CX@("`@
> M("`*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P
> M.B`P(CX*("`@("`@/&9O;G0 at 9F%C93TB3'5C:61A($=R86YD92(@<VEZ93TB
> M,R(^37D@<75E<W1I;VX@:7,@:68@:70@<&]S<VEB;&4@=&\@9&\@<V]M971H
> M:6YG(&QI:V4@=&AI<RX at 16ET:&5R('1H870 at 2$1&-2!H87,@=&AE(&]P=&EO
> M;B!T;R!W<FET92!D871A(&EN('1H:7,@=V%Y(&]R('1H<F]U9V@@<V]M92!M
> M96-H86YI<VT@=&AA="!(1$8U(&-A;B!T<F5A="!S=6-H(&%N(&5X=&5R;F%L
> M(&9I;&4 at 87, at 2$1&-2!D871A)B,T-#L@<&]S<VEB;'D at 8GD@9'EN86UI8V%L
> M;'D@;&]A9&EN9R!A('-H87)E9"!L:6)R87)Y('1H870@:VYO=W,@:&]W('1O
> M(&EN=&5R<')E="!T:&4 at 9&%T82X\+V9O;G0^("`@(#PO<#X*/&)R/B`@("`@
> M(`H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T;VTZ(#`[(&UA<F=I;BUT;W`Z
> M(#`B/@H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S
> M(CY))B,S.3MM(&%S:VEN9R!T:&ES(&)E8V%U<V4@=V4@:&%V92!T:&4@:6UP
> M<F5S<VEO;B!T:&%T('-O;65T:&EN9R!L:6ME(&%N(&5X=&5N9&EB;&4 at 9&%T
> M87-E="!D;V5S(&YO="!G:79E('5S('1H92!P97)F;W)M86YC92]R;V)U<W1N
> M97-S('=E(&YE960N($EN(&-A<V4@;V8 at 82!C<F%S:"!W92!W;W5L9"!L:6ME
> M('1O(&QO<V4 at 87, at 9F5W(&1A=&$@87,@<&]S<VEB;&4F(S0T.R!W:&EC:"!M
> M96%N<R!T:&%T(&$@<F5G=6QA<B!F;'5S:"!H87,@=&\@8F4 at 9&]N92!W:&EC
> M:"!M:6=H="!K:6QL('!E<F9O<FUA;F-E+B!)(&=U97-S('1H870@=VET:&]U
> M="!A(&9L=7-H('1H92!I;F1I8V5S(&%R92!O;FQY('=R:71T96X at 870@=&AE
> M(&5N9"8C-#0[('-O('1H92!D871A(&-A;FYO="!B92!F;W5N9"!I;B!C87-E
> M(&]F(&$@8W)A<V at N/"]F;VYT/B`@("`\+W`^"CQB<CX@("`@("`*("`@(#QP
> M('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX*("`@
> M("`@/&9O;G0 at 9F%C93TB3'5C:61A($=R86YD92(@<VEZ93TB,R(^0VAE97)S
> M)B,T-#L\+V9O;G0^("`@(#PO<#X*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T
> M=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX*("`@("`@/&9O;G0 at 9F%C93TB3'5C
> M:61A($=R86YD92(@<VEZ93TB,R(^1V5R('9A;B!$:65P96X\+V9O;G0^("`@
> 8(#PO<#X*("`\+V)O9'D^"CPO:'1M;#X*
> `
> end
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org