Hello,
I've attached a simple test case, which fails on my system (Linux 3.13)
with:
Value: 0
Error: Value should be 42
strace shows:
open("external-file", O_RDONLY) = 5
lseek(5, 0, SEEK_SET) = 0
read(5,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
2165309440) = 2147479552
close(5) = 0
However, the test will need >2GB RAM, so I'm not sure whether it is a
good idea to add it to the test suite.
The test case also verifies that bytes after EOF are read as '0'.
Best regards,
Steffen Kieß
On 2015-12-19 00:24, Elena Pourmal wrote:
Hi Steffen,
We reviewed the patch today. It looks good. Could you please send us a
test? We would love to have it in HDF5 1.10.0. For your reference the
issue number is HDFFV-9634.
Thanks a lot!
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Dec 18, 2015, at 6:04 AM, Steffen Kieß
<[email protected]
<mailto:[email protected]>> wrote:
Hello,
the code for handling external file reads and writes currently does
not handle the case when the read() or write() operation returns a
number smaller than the requested amount.
Linux currently transfers at most 0x7ffff000 bytes per read() or
write(), meaning that if more than 2GB are read from an external file
only 2GB are read and the rest is filled up with zeros (because HDF5
thinks that it has reached EOF).
http://man7.org/linux/man-pages/man2/read.2.html#NOTES
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=758170#10
(Note that the problem does not occur if the files datatype and the
memory datatype do not match, because in this case HDF5 will use
smaller reads into a buffer for converting the values.)
I've attached a patch which will restart the read() or write()
operation if a number smaller than the requested amount is returned
(similar to e.g. the code in H5FDcore.c). I've also added logic which
will restart the syscall after EINTR.
Best regards,
Steffen Kieß
<efl-short-read.patch>_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
#include <hdf5.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
int main () {
int fd;
fd = open ("external-file", O_WRONLY | O_CREAT | O_TRUNC, 0666);
if (fd < 0) {
perror ("open");
return 1;
}
if (lseek (fd, 0x81000000, SEEK_SET) < 0) {
perror ("lseek");
return 1;
}
char c = 42;
if (write (fd, &c, 1) < 0) {
perror ("write");
return 1;
}
hid_t file = H5Fcreate ("test.hdf5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
if (file < 0)
return 1;
hsize_t size = 0x81100000;
hid_t dspace = H5Screate_simple (1, &size, &size);
if (dspace < 0)
return 1;
hid_t dcpl = H5Pcreate (H5P_DATASET_CREATE);
if (dcpl < 0)
return 1;
if (H5Pset_external (dcpl, "external-file", 0, size) < 0)
return 1;
hid_t dset = H5Dcreate (file, "DataSet", H5T_STD_U8LE, dspace, H5P_DEFAULT, dcpl, H5P_DEFAULT);
if (dset < 0)
return 1;
char* data = malloc (size);
if (!data) {
perror ("malloc");
return 1;
}
memset (data, 255, size);
if (H5Dread (dset, H5T_STD_U8LE, dspace, dspace, H5P_DEFAULT, data) < 0)
return 1;
size_t i;
for (i = 0; i < size; i++) {
if (data[i] != 0 && i != 0x81000000) {
printf ("data[%zu] = %d\n", i, data[i]);
return 1;
}
}
printf ("Value: %d\n", data[0x81000000]);
if (data[0x81000000] != 42) {
printf ("Error: Value should be 42\n");
return 2;
}
printf ("Ok\n");
return 0;
}
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
