[Hdf-forum] workarounds for dynamically allocated multidimensional arrays?

[Darren Adams]

Hi,
I'm working with a mpi CFD code that uses dynamically allocated 3D arrays of structs to store data.  I'm trying to develop a scalable parallel I/O routine using HDF5, or I suppose PHDF5.  I've had some initial success using a 1D buffering approach, but I'm not crazy about this method. This approach leads to a H5Sselect_hyperspab and H5Dwrite call for every 1D "pencil" in the 3D space.  Performance is far from optimal.  My next attempt was to use HDF derived datatypes and try to write from the main data array directly.  This seems much simpler, as I only need to select a hyperslab from memory that excludes the "ghost" cells, and then one from the file dataspace that offsets to the correct mpi processes' section, then--a single H5Dwrite! 

Cool, except that I kept getting low level I/O errors (below) which, after much googling, finally landed me on this forum post:

http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/2008-June/001118.html

The jist of which is:

>  ... The HDF5 library expects to  
>receive a contiguous array of elements, not pointers to elements in  
>lower dimensions.

The above forum post includes an alternate approach which is basically a select hyperspab and write for every 1D column, which is exactly what I wanted to avoid.

Are there any other workarounds so that I can still only define a single hyperslab and call a single H5Dwrite call on each process?


-Darren

My error:

HDF5-DIAG: Error detected in HDF5 (1.8.3) MPI-process 5:
  #000: H5Dio.c line 266 in H5Dwrite(): can't write data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 578 in H5D_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dcontig.c line 539 in H5D_contig_write(): contiguous write failed
    major: Dataset
    minor: Write failed
  #003: H5Dselect.c line 306 in H5D_select_write(): write error
    major: Dataspace
    minor: Write failed
  #004: H5Dselect.c line 217 in H5D_select_io(): write error
    major: Dataspace
    minor: Write failed
  #005: H5Dcontig.c line 1127 in H5D_contig_writevv(): block write failed
    major: Low-level I/O
    minor: Write failed
  #006: H5Fio.c line 159 in H5F_block_write(): file write failed
    major: Low-level I/O
    minor: Write failed
  #007: H5FDint.c line 185 in H5FD_write(): driver write request failed
    major: Virtual File Layer
    minor: Write failed
  #008: H5FDmpio.c line 1820 in H5FD_mpio_write(): MPI_File_write_at failed
    major: Internal error (too specific to document in detail)
    minor: Some MPI function failed
  #009: H5FDmpio.c line 1820 in H5FD_mpio_write(): Other I/O error , error stack:
ADIOI_GEN_WRITECONTIG(189): Other I/O error Bad address
    major: Internal error (too specific to document in detail)
    minor: MPI Error String