From bug-request at octave dot org Wed Oct 13 07:14:20 2004 Subject: Re: Failure to import data from hdf5 files From: David Bateman To: Claude =?iso-8859-1?Q?Lacoursi=E8re?= Cc: bug at octave dot org Date: Wed, 13 Oct 2004 14:09:45 +0200 A group in an imported hdf5 file is treated as an octave structure. Each dataset of the group is another structure. You should even be able to import a dataset within a group with a group which would be interpreted as a probably a matrix in a structure in a structure. So there is no fundamental reason you can't have multiple datasets in a group. In fact I tried you code, with the following examples % ./claude -g 2 -s 1 -f f.h5 % h5ls -r -d f.h5 /group0 Group /group0/s0 Dataset {3, 3} Data: (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9 /group1 Group /group1/s0 Dataset {3, 3} Data: (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9 % ./claude -g 2 -s 2 -f f2.h5 % h5ls -r -d f2.h5 /group0 Group /group0/s0 Dataset {3, 3} Data: (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9 /group0/s1 Dataset {3, 3} Data: (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9 /group1 Group /group1/s0 Dataset {3, 3} Data: (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9 /group1/s1 Dataset {3, 3} Data: (0,0) 1, 2, 3, 4, 5, 6, 7, 8, 9 % octave -q octave:1> load f.h5 octave:2> whos *** dynamically linked functions: Prot Name Size Bytes Class ==== ==== ==== ===== ===== r-- dispatch 1x1 0 dynamically-linked function Total is 1 element using 0 bytes *** local user variables: Prot Name Size Bytes Class ==== ==== ==== ===== ===== rw- __nargin__ 1x1 8 scalar rwd group0 1x1 36 struct rwd group1 1x1 36 struct Total is 3 elements using 80 bytes octave:3> group0 group0 = { s0 = 1 4 7 2 5 8 3 6 9 } octave:4> load f2.h5 octave:5> whos *** dynamically linked functions: Prot Name Size Bytes Class ==== ==== ==== ===== ===== r-- dispatch 1x1 0 dynamically-linked function Total is 1 element using 0 bytes *** local user variables: Prot Name Size Bytes Class ==== ==== ==== ===== ===== rw- __nargin__ 1x1 8 scalar rwd group0 1x1 72 struct rwd group1 1x1 72 struct Total is 3 elements using 152 bytes octave:6> group0 group0 = { s0 = 1 4 7 2 5 8 3 6 9 s1 = 1 4 7 2 5 8 3 6 9 } So I can't reproduce your problem. However, although I have the latest octave CVS installed, I don't have the latest HDF5 library installed. "h5ls -V" reports my version as h5ls: Version 1.4.5-post2 which is in fact the version supplied with Mandrake 9.2, so might even have some other stuff included... From the error message it appears that you are using the very latest version of the HDF5 libraries, version 1.6.2. I imagine that some behaviour in 1.4.5, with which I wrote and tested this code, has changed in version 1.6.2 and has resulted in the problem you see. I don't really have the time to debug this at the moment, but would aid if you'd like to try and debug it yourself... Already I see that the problem is in the H5Giterate function, at least that is where the error is reported, and so I suspect if changes are needed, it will be in the load_hdf5 functions in ov-struct.cc, rather than elsewhere. First step would be either running in a debugger or using printf's to isolate exactly the line in ov-struct.cc that causes the problem, and then comapring the syntax of this line in version 1.6.2 to see what changed.. Regards David According to Claude Lacoursière (on 10/13/04): > Hi. > > This is a follow up on a bug report posted on August 15 > 2004. > > The files produced by the example code posted then by Shaun > Cloherty loads correctly thanks to the patch from David > Bateman but there are more problems. > > It seems that if there is only one group in the HDF5 file, > only one dataset is loaded correctly. Similarly, if there > are 'n' groups, only 'n' datasets are loaded properly. > > However, if there are 2 groups, each containing a single > dataset, octave-2.1.60 produces the following error message: > > octave:1> P=load('-hdf5', 'f.h5'); > HDF5-DIAG: Error detected in HDF5 library version: 1.6.2 > thread 1103743904. Back trace follows. > #000: ../../../src/H5G.c line 433 in H5Giterate(): invalid > index specified > major(01): Function arguments > minor(05): Bad value > error: load: error while reading hdf5 item group0 > error: evaluating assignment expression near line 1, column > 2 > octave:1> version > ans = 2.1.60 > > > The same is true as soon as there are more groups than > datasets in each group. > > Below is a simple C program which generates HDF5 files with > one or several groups, each containing one or several > datasets. > > I looked at the ls-hdf5.cc load-save.cc files but didn't > find anything obvious yet. > > Is anyone active in supporting the HDF5 code? > > -- > Claude Lacoursière, HPC2N/VRlab, Umeå Universitet, Sweden > > > ==========simple C file to produce HDF5 datasets ====== > /* > compile this with LDFLAGS="-lhdf5". Run the program with -h > flag to get info on how to run it to produce hdf5 files with > different number of groups, each containing a > number of datasets. > */ > > > > #include > #include > #include > #include > #include > /* > change the following macros to experiment with different > dataset sizes. Other arguments are given on the command > line. > */ > > #define SZ1 3 > #define SZ2 3 > > int main(int argc, char *argv[]) > { > int groups = 1; > int datasets = 1; > hid_t file_id, group_id, dataset_id, dataspace_id; > herr_t status; > int i, j, g, k; > int data[SZ1][SZ2]; > hsize_t dims[2] = {sizeof(data)/sizeof(data[0]), sizeof(data[0])/sizeof(data[0][0])}; > char buff[FILENAME_MAX] = "h5grp.h5"; > char *progname; > extern char *optarg; > extern int optind; > char c; > int errflg = 0; > > > progname = argv[0]; > while((c=getopt(argc, argv, "g:s:f:h"))!= EOF) > switch(c){ > case 'g': > groups = atoi(optarg); > break; > case 's': > datasets = atoi(optarg); > break; > case 'f': > strncpy(buff, optarg, FILENAME_MAX); > break; > case '?': > default: > errflg++; > break; > } > if(errflg){ > fprintf(stderr, "usage: %s [-g n_groups] [-s n_datasets] [-f filename] [-h].\n", progname); > fprintf(stderr, "Produce a simple HDF5 file with:\n" > "'n_groups' groups (defaults to 1), \n" > "each group containing 'n_datasets' datasets (defaults to 1)\n" > "and writing the result to 'filename' (defaults to 'h5grp.h5').\n" > "All datasets are identical: a simple %dx%d integer array with sequential values.\n" > "The groups are named 'group0, group1, group2 ...'\n", > "The datasets are named 's0, s1, s2, ...'\n", > SZ1, SZ2 ); > > exit(2); > } > > for (i = 0; i < dims[0]; i++) > for (j = 0; j < dims[1]; j++) > data[i][j] = i*dims[0] + j + 1; > > file_id = H5Fcreate(buff, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); > > for (g = 0; g< groups; ++g) > { > sprintf(buff, "group%d", g); > group_id = H5Gcreate(file_id, buff, 0); > > for (k=0; k { > sprintf(buff, "s%d", k); > dataspace_id = H5Screate_simple(2, dims, NULL); > dataset_id = H5Dcreate(group_id, buff, H5T_STD_I32LE, dataspace_id, H5P_DEFAULT); > status = H5Dwrite(dataset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, data); > status = H5Sclose(dataspace_id); > status = H5Dclose(dataset_id); > } > status = H5Gclose(group_id); > } > status = H5Fclose(file_id); > return 0 ; > } > > > > ------------------------------------------------------------- > Octave is freely available under the terms of the GNU GPL. > > Octave's home on the web: http://www.octave.org > How to fund new projects: http://www.octave.org/funding.html > Subscription information: http://www.octave.org/archive.html > ------------------------------------------------------------- -- David Bateman David dot Bateman at motorola dot com Motorola CRM +33 1 69 35 48 04 (Ph) Parc Les Algorithmes, Commune de St Aubin +33 1 69 35 77 01 (Fax) 91193 Gif-Sur-Yvette FRANCE The information contained in this communication has been classified as: [x] General Business Information [ ] Motorola Internal Use Only [ ] Motorola Confidential Proprietary ------------------------------------------------------------- Octave is freely available under the terms of the GNU GPL. Octave's home on the web: http://www.octave.org How to fund new projects: http://www.octave.org/funding.html Subscription information: http://www.octave.org/archive.html -------------------------------------------------------------