[hdf-forum] h5py -- most efficient way to load a hdf5

[Mag Gam]

Hello All,

I have a very large csv file 14G and I am planning to move all of my
data to hdf5. I am using h5py to load the data. The biggest problem I
am having is, I am putting the entire file into memory and then
creating a dataset from it. This is very inefficient and it takes over
4 hours to create the hdf5 file.

The csv file has various types:
int4, int4, str, str, str, str, str

I was wondering if anyone knows of any techniques to load this file faster?

TIA

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe at hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe at hdfgroup.org.