[Hdf-forum] writing data as fast as possible

Tue Jan 19 17:07:27 EST 2010

Hi Ger,

On Jan 15, 2010, at 5:30 AM, Ger van Diepen wrote:

> 
> Hi Dimitris, 
> 
> Thanks for your suggestion. 
> I'll look into the VFL possibility. A disadvantage I see is that you
> have to make it known, so common tools won't recognize it. I guess it is
> not  possible to load it dynamically. In principle it should be possible
> by storing the driver name and use it as the name of the shared library.
> 
> 
> Does using a specific VFL mean that all data are written through that
> VFL and that it is not possible to store the meta data in the 'normal'
> way and only store the bulk data through that VFL? 
> Preferably I would like to bypass HDF5 entirely when writing (apart
> from telling the VFL) and use that VFL when reading. In that way I can
> write as I like and still use tools like h5view to look at the data
> (provided the VFL can be loaded dynamically).  
> 
> As I said writing happens in parallel, so each disk has to part of the
> job. But we would like to keep the amount of resources needed to a
> minimum, for which optimal write performance is needed. Reading is less
> of an issue. 

	The "split" file driver may also be a way to get what you'd like, although it stores all the raw datasets in one file, which may be difficult to get your application to play nicely with.  If possible I would try to stay with just using the MPI file driver and see where the bottlenecks are, so we can correct them.

	Quincey

> Cheers, 
> Ger
> 
>>>> Dimitris Servis <servisster at gmail.com> 1/15/2010 11:47 AM >>>
> Ger
> 
> couldn't you use the multi VFL driver? In any case, the nGB/sec rate
> will not be sustainable by any hard drive today in the market right?
> Even if you buffer in SSDs still the disk write speed is your bottleneck
> rather than hdf5...
> 
> HTH
> 
> -- dimitris
> 
> 
> 2010/1/15 Ger van Diepen <diepen at astron.nl>
> 
> 
> 
> 
> We have an instrument generating several GBytes/second.
> We would like to dump these data as fast as possible to disk (of
> course
> in a parallel way), preferably with O_DIRECT to bypass the OS file
> cache
> and to avoid seeks to maintain meta data.
> Yet, it would be nice if these data can be seen as an HDF5 dataset.
> 
> My question is if it possible to do something like this. Either that
> HDF5 has the option to write data in this way or through some
> mechanism
> that HDF5 can treat such an external file as HDF5 data, possibly by
> dynamically loading a shared library that knows how to interpret the
> data.
> 
> I'm asking this because we have the impression that something like an
> extendible dataset does not give us the performance/robustness we
> need.
> In case of a crash we would like to lose as few data as possible,
> which
> means that a regular flush has to be done which might kill
> performance.
> I guess that without a flush the indices are only written at the end,
> so
> the data cannot be found in case of a crash.
> 
> Cheers,
> Ger van Diepen
> 
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
> 
> 
> 
> 
> begin 644 TEXT.htm
> M/&AT;6P^#0H@(#QH96%D/@T*#0H@(#PO:&5A9#X-"B`@/&)O9'D@<W1Y;&4]
> M(F9O;G0M=F%R:6%N=#H@;F]R;6%L.R!L:6YE+6AE:6=H=#H@;F]R;6%L.R!M
> M87)G:6XM<FEG:'0Z(#1P>#L@;6%R9VEN+71O<#H at -'!X.R!M87)G:6XM8F]T
> M=&]M.B`Q<'@[(&UA<F=I;BUL969T.B`T<'@B/@T*("`@(#QP('-T>6QE/2)M
> M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QF;VYT
> M(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/DAI($1I;6ET<FES)B,T
> M-#L\+V9O;G0^("`@(#PO<#X-"CQB<CX@("`@("`-"B`@("`\<"!S='EL93TB
> M;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^#0H@("`@("`\9F]N
> M="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S(CY4:&%N:W, at 9F]R('EO
> M=7(@<W5G9V5S=&EO;BX\+V9O;G0^("`@(#PO<#X-"B`@("`\<"!S='EL93TB
> M;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^#0H@("`@("`\9F]N
> M="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S(CY))B,S.3ML;"!L;V]K
> M(&EN=&\@=&AE(%9&3"!P;W-S:6)I;&ET>2X at 02!D:7-A9'9A;G1A9V4 at 22!S
> M964@:7,@=&AA="!Y;W4@:&%V92!T;R!M86ME(&ET(&MN;W=N)B,T-#L@<V\@
> M8V]M;6]N('1O;VQS('=O;B8C,SD[="!R96-O9VYI>F4@:70N($D at 9W5E<W,@
> M:70@:7,@;F]T("9N8G-P.W!O<W-I8FQE('1O(&QO860@:70 at 9'EN86UI8V%L
> M;'DN($EN('!R:6YC:7!L92!I="!S:&]U;&0 at 8F4@<&]S<VEB;&4 at 8GD@<W1O
> M<FEN9R!T:&4 at 9')I=F5R(&YA;64 at 86YD('5S92!I="!A<R!T:&4@;F%M92!O
> M9B!T:&4@<VAA<F5D(&QI8G)A<GDN/"]F;VYT/B`@("`\+W`^#0H\8G(^("`@
> M("`@#0H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T;VTZ(#`[(&UA<F=I;BUT
> M;W`Z(#`B/@T*("`@("`@/&9O;G0 at 9F%C93TB3'5C:61A($=R86YD92(@<VEZ
> M93TB,R(^1&]E<R!U<VEN9R!A('-P96-I9FEC(%9&3"!M96%N('1H870 at 86QL
> M(&1A=&$@87)E('=R:71T96X@=&AR;W5G:"!T:&%T(%9&3"!A;F0@=&AA="!I
> M="!I<R!N;W0@<&]S<VEB;&4@=&\@<W1O<F4@=&AE(&UE=&$@9&%T82!I;B!T
> M:&4@)B,S.3MN;W)M86PF(S,Y.R!W87D at 86YD(&]N;'D@<W1O<F4@=&AE(&)U
> M;&L at 9&%T82!T:')O=6=H('1H870 at 5D9,)B,V,SL\+V9O;G0^("`@(#PO<#X-
> M"B`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@
> M,"(^#0H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S
> M(CY0<F5F97)A8FQY($D@=V]U;&0@;&EK92!T;R!B>7!A<W, at 2$1&-2!E;G1I
> M<F5L>2!W:&5N('=R:71I;F<@)B,T,#MA<&%R="!F<F]M('1E;&QI;F<@=&AE
> M(%9&3"8C-#$[(&%N9"!U<V4@=&AA="!61DP@=VAE;B!R96%D:6YG+B!);B!T
> M:&%T('=A>2!)(&-A;B!W<FET92!A<R!)(&QI:V4 at 86YD('-T:6QL('5S92!T
> M;V]L<R!L:6ME(&@U=FEE=R!T;R!L;V]K(&%T('1H92!D871A("8C-#`[<')O
> M=FED960@=&AE(%9&3"!C86X at 8F4@;&]A9&5D(&1Y;F%M:6-A;&QY)B,T,3LN
> M(#PO9F]N=#X@("`@/"]P/@T*/&)R/B`@("`@(`T*("`@(#QP('-T>6QE/2)M
> M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QF;VYT
> M(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/D%S($D@<V%I9"!W<FET
> M:6YG(&AA<'!E;G,@:6X@<&%R86QL96PF(S0T.R!S;R!E86-H(&1I<VL@:&%S
> M('1O('!A<G0@;V8@=&AE(&IO8BX at 0G5T('=E('=O=6QD(&QI:V4@=&\@:V5E
> M<"!T:&4 at 86UO=6YT(&]F(')E<V]U<F-E<R!N965D960@=&\@82!M:6YI;75M
> M)B,T-#L at 9F]R('=H:6-H(&]P=&EM86P@=W)I=&4@<&5R9F]R;6%N8V4@:7,@
> M;F5E9&5D+B!296%D:6YG(&ES(&QE<W,@;V8 at 86X@:7-S=64N/"]F;VYT/B`@
> M("`\+W`^#0H\8G(^("`@("`@#0H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T
> M;VTZ(#`[(&UA<F=I;BUT;W`Z(#`B/@T*("`@("`@/&9O;G0 at 9F%C93TB3'5C
> M:61A($=R86YD92(@<VEZ93TB,R(^0VAE97)S)B,T-#L\+V9O;G0^("`@(#PO
> M<#X-"B`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O
> M<#H@,"(^#0H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE
> M/2(S(CY'97(\+V9O;G0^/&)R/CQB<CXF9W0[)F=T.R9G=#L at 1&EM:71R:7,@
> M4V5R=FES("9L=#MS97)V:7-S=&5R0&=M86EL+F-O;29G=#L@,2\Q-2\R,#$P
> M(#$Q.C0W($%-("9G=#LF9W0[)F=T.SQB<CY'97(\8G(^/&)R/F-O=6QD;B8C
> M,SD[="!Y;W4@=7-E('1H92!M=6QT:2!61DP at 9')I=F5R)B,V,SL at 26X@86YY
> M(&-A<V4F(S0T.R!T:&4@;D="+W-E8R!R871E('=I;&P@;F]T(&)E('-U<W1A
> M:6YA8FQE(&)Y(&%N>2!H87)D(&1R:79E('1O9&%Y(&EN('1H92!M87)K970@
> M<FEG:'0F(S8S.R!%=F5N(&EF('EO=2!B=69F97(@:6X at 4U-$<R!S=&EL;"!T
> M:&4 at 9&ES:R!W<FET92!S<&5E9"!I<R!Y;W5R(&)O='1L96YE8VL@<F%T:&5R
> M('1H86X@:&1F-2XN+CQB<CX\8G(^2%1(/&)R/CQB<CXM+2!D:6UI=')I<SQB
> M<CX\8G(^("`@(#PO<#X-"B`@("`\9&EV(&-L87-S/2)G;6%I;%]Q=6]T92(^
> M#0H@("`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O
> M<#H@,"(^#0H@("`@("`@(#(P,3`O,2\Q-2!'97(@=F%N($1I97!E;B`\<W!A
> M;B!D:7(](FQT<B(^)FQT.SQA(&AR968](FUA:6QT;SID:65P96Y`87-T<F]N
> M+FYL(CYD:65P96Y`87-T<F]N+FYL/"]A/B9G=#L\+W-P86X^/&)R/B`@("`@
> M(#PO<#X-"B`@("`@(#QB;&]C:W%U;W1E('-T>6QE/2)P861D:6YG+6QE9G0Z
> M(#`[(&)O<F1E<BUL969T.B`Q<'@@<V]L:60@<F=B*#(P-"P@,C`T+"`R,#0I
> M.R!M87)G:6XM<FEG:'0Z(#!P=#L@;6%R9VEN+71O<#H@,'!T.R!M87)G:6XM
> M8F]T=&]M.B`P<'0[(&UA<F=I;BUL969T.B`P(B!C;&%S<STB9VUA:6Q?<75O
> M=&4B/@T*("`@("`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R
> M9VEN+71O<#H@,"(^#0H@("`@("`@("`@/&)R/@T*("`@("`@("`@(%=E(&AA
> M=F4 at 86X@:6YS=')U;65N="!G96YE<F%T:6YG('-E=F5R86P at 1T)Y=&5S+W-E
> M8V]N9"X\8G(^5V4@=V]U;&0@;&EK92!T;R!D=6UP('1H97-E(&1A=&$@87,@
> M9F%S="!A<R!P;W-S:6)L92!T;R!D:7-K("8C-#`[;V8 at 8V]U<G-E/&)R/FEN
> M(&$@<&%R86QL96P@=V%Y)B,T,3LF(S0T.R!P<F5F97)A8FQY('=I=&@@3U]$
> M25)%0U0@=&\@8GEP87-S('1H92!/4R!F:6QE(&-A8VAE/&)R/F%N9"!T;R!A
> M=F]I9"!S965K<R!T;R!M86EN=&%I;B!M971A(&1A=&$N/&)R/EEE="8C-#0[
> M(&ET('=O=6QD(&)E(&YI8V4@:68@=&AE<V4 at 9&%T82!C86X at 8F4@<V5E;B!A
> M<R!A;B!(1$8U(&1A=&%S970N/&)R/CQB<CY->2!Q=65S=&EO;B!I<R!I9B!I
> M="!P;W-S:6)L92!T;R!D;R!S;VUE=&AI;F<@;&EK92!T:&ES+B!%:71H97(@
> M=&AA=#QB<CY(1$8U(&AA<R!T:&4@;W!T:6]N('1O('=R:71E(&1A=&$@:6X@
> M=&AI<R!W87D@;W(@=&AR;W5G:"!S;VUE(&UE8VAA;FES;3QB<CYT:&%T($A$
> M1C4 at 8V%N('1R96%T('-U8V@@86X at 97AT97)N86P at 9FEL92!A<R!(1$8U(&1A
> M=&$F(S0T.R!P;W-S:6)L>2!B>3QB<CYD>6YA;6EC86QL>2!L;V%D:6YG(&$@
> M<VAA<F5D(&QI8G)A<GD@=&AA="!K;F]W<R!H;W<@=&\@:6YT97)P<F5T('1H
> M93QB<CYD871A+CQB<CX\8G(^228C,SD[;2!A<VMI;F<@=&AI<R!B96-A=7-E
> M('=E(&AA=F4@=&AE(&EM<')E<W-I;VX@=&AA="!S;VUE=&AI;F<@;&EK92!A
> M;CQB<CYE>'1E;F1I8FQE(&1A=&%S970 at 9&]E<R!N;W0 at 9VEV92!U<R!T:&4@
> M<&5R9F]R;6%N8V4O<F]B=7-T;F5S<R!W92!N965D+CQB<CY);B!C87-E(&]F
> M(&$@8W)A<V@@=V4@=V]U;&0@;&EK92!T;R!L;W-E(&%S(&9E=R!D871A(&%S
> M('!O<W-I8FQE)B,T-#L@=VAI8V@\8G(^;65A;G,@=&AA="!A(')E9W5L87(@
> M9FQU<V@@:&%S('1O(&)E(&1O;F4@=VAI8V@@;6EG:'0@:VEL;"!P97)F;W)M
> M86YC92X\8G(^22!G=65S<R!T:&%T('=I=&AO=70 at 82!F;'5S:"!T:&4@:6YD
> M:6-E<R!A<F4@;VYL>2!W<FET=&5N(&%T('1H92!E;F0F(S0T.R!S;SQB<CYT
> M:&4 at 9&%T82!C86YN;W0 at 8F4@9F]U;F0@:6X at 8V%S92!O9B!A(&-R87-H+CQB
> M<CX\8G(^0VAE97)S)B,T-#L\8G(^1V5R('9A;B!$:65P96X\8G(^/&)R/CQB
> M<CY?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?
> M7U]?7SQB<CY(9&8M9F]R=6T@:7, at 9F]R($A$1B!S;V9T=V%R92!U<V5R<R!D
> M:7-C=7-S:6]N+CQB<CX\82!H<F5F/2)M86EL=&\Z2&1F+69O<G5M0&AD9F=R
> M;W5P+F]R9R(^2&1F+69O<G5M0&AD9F=R;W5P+F]R9SPO83X\8G(^/&$@:')E
> M9CTB:'1T<#HO+VUA:6PN:&1F9W)O=7`N;W)G+VUA:6QM86XO;&ES=&EN9F\O
> M:&1F+69O<G5M7VAD9F=R;W5P+F]R9R(@=&%R9V5T/2)?8FQA;FLB/FAT='`Z
> M+R]M86EL+FAD9F=R;W5P+F]R9R]M86EL;6%N+VQI<W1I;F9O+VAD9BUF;W)U
> M;5]H9&9G<F]U<"YO<F<\+V$^/&)R/B`@("`@("`@/"]P/@T*("`@("`@/"]B
> M;&]C:W%U;W1E/@T*("`@(#PO9&EV/@T*("`@(#QP('-T>6QE/2)M87)G:6XM
> M8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QB<CX-"B`@("`\
> 9+W`^#0H@(#PO8F]D>3X-"CPO:'1M;#X-"G9A
> `
> end
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org