Hi Elena, thanks for investigating.
A beter error reporting would be definately great and helpful. On the lzf site they state: With HDF5 version 1.8.11 or later the filter can be dynamically loaded as a plugin. The filter is built as a shared library that is *not* linked against the HDF5 library: Does it make sense to raise this issue with them so they add a note that this won't work when using in conjuction with JNI ? On Sun, Dec 13, 2015 at 4:30 AM, Elena Pourmal <[email protected]> wrote: > Hi Ümit, > > We investigated little-bit more. Here is a high-level description of what > we found. > > The lzf filter uses calls to the HDF5 library for the callback functions. > > When HDFView tries to read compressed data, it calls JNI (shared) that > calls HDF5 (statically linked with JNI); the HDF5 library then issues > dlopen to find and load the lzf filter shared library. When the filter is > linked with HDF5, dlopen pulls in the filter library and dependents (i.e., > HDF5, etc.), and HDFView works. When the filter is not linked with HDF5, > dlopen pulls in the filter library but cannot find the required HDF5 > library since there is no dependency. The behavior makes sense. > > What puzzles us is that we can build h5dump that is statically linked with > HDF5, and the lzf filter (that is not linked with HDF5) still works. > Apparently having intermediate JNI makes a difference. May be someone on > the FORUM could explain such behavior? > > We recommend to link with the HD5 library when a filter uses HDF5 calls. > We will look into how to enhance HDF5 error reporting to incorporate the > status of dlopen call. We will also add such filter to our internal testing. > > Thank you! > > Elena > > > On Nov 25, 2015, at 9:20 AM, Ümit Seren <[email protected]> wrote: > > Hi Elena, > Thanks for confirming. > I remeber that it used to work in one of the older Java-HDF5 versions. > Unfortunately I can't remember which one. > > > > > On Wed, Nov 25, 2015 at 4:14 PM, Elena Pourmal <[email protected]> > wrote: > >> Hi Ümit, >> >> Allen and I confirmed the problem you and Dan see with HDFView. For some >> reason HDFView cannot find the lzf plugin while it finds, for example, >> bzip2 plugin (see https://svn.hdfgroup.uiuc.edu/hdf5_plugins/trunk/BZIP2/). >> The only difference is that lzf plugin uses HDF5 calls in the callback >> functions. We are puzzled why the HDF5 library is able to find the lzf >> plugin for the command-line tools and fails for HDFView. >> >> I opened the ticket (JAVA-1920) and we will investigate. >> >> Thanks for reporting the problem! >> >> Elena >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Elena Pourmal The HDF Group http://hdfgroup.org >> 1800 So. Oak St., Suite 203, Champaign IL 61820 >> 217.531.6112 >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> >> >> >> On Nov 23, 2015, at 5:00 PM, Ümit Seren <[email protected]> wrote: >> >> Hi Elena, >> >> Thanks for the response, >> my problem is not that h5dump or other hdf5 command line utilities do not >> work the lzf plugin filter (they work just fine) but the Java HDF5 and >> HDFView can't find the plugin_filter. >> I will check the symbols tomorrow but as far as I can tell the problem is >> somewhere with the java hdf5 library. >> >> >> >> >> >> >> >> On Mon, Nov 23, 2015 at 11:41 PM, Elena Pourmal <[email protected]> >> wrote: >> >>> Hello, >>> >>> What kind of error are you getting? I was able to get the filter to work >>> on Linux (CentOS 7) with h5dump from 1.8.16 distribution I built. Please >>> notice that one has to specify HDF5 include files when building >>> liblzf_filter.so. >>> >>> Details: >>> >>> Here is the output of the nm command and h5dump when include files are >>> not specified: >>> >>> [epourmal@platypus lzf]$ gcc -O2 -fPIC -shared lzf/*.c lzf_filter.c -o >>> liblzf_filter.so >>> epourmal@platypus lzf]$ nm liblzf_filter.so >>> U H5E_CALLBACK_g >>> U H5E_CANTREGISTER_g >>> U H5E_PLINE_g >>> U H5Epush1 >>> U H5Pget_chunk >>> U H5Pget_filter_by_id2 >>> U H5Pmodify_filter >>> U H5Tget_size >>> U H5Zregister >>> U H5open >>> [epourmal@platypus lzf]$ hdf5/hdf5/bin/h5dump test_lzf.hdf5 >>> HDF5 "test_lzf.hdf5" { >>> GROUP "/" { >>> DATASET "dset" { >>> DATATYPE H5T_IEEE_F32LE >>> DATASPACE SIMPLE { ( 100, 100, 100 ) / ( 100, 100, 100 ) } >>> DATA {h5dump error: unable to print data >>> >>> } >>> } >>> } >>> } >>> [epourmal@platypus lzf]$ hdf5/hdf5/bin/h5dump -pH test_lzf.hdf5 >>> HDF5 "test_lzf.hdf5" { >>> GROUP "/" { >>> DATASET "dset" { >>> DATATYPE H5T_IEEE_F32LE >>> DATASPACE SIMPLE { ( 100, 100, 100 ) / ( 100, 100, 100 ) } >>> STORAGE_LAYOUT { >>> CHUNKED ( 1, 100, 100 ) >>> SIZE 174288 (22.951:1 COMPRESSION) >>> } >>> FILTERS { >>> PREPROCESSING SHUFFLE >>> USER_DEFINED_FILTER { >>> FILTER_ID 32000 >>> COMMENT lzf >>> PARAMS { 4 261 40000 } >>> } >>> ... >>> >>> Now if I specify HDF5 include files, I will get (notice two symbols >>> H5PLget_plugin_info/type in the nm output!) >>> >>> [epourmal@platypus lzf]$ gcc >>> -I//scr/epourmal/h5py-master/lzf/hdf5/hdf5/include -O2 -fPIC -shared >>> lzf/*.c lzf_filter.c -o liblzf_filter.so >>> [epourmal@platypus lzf]$ nm liblzf_filter.so >>> U H5E_CALLBACK_g >>> U H5E_CANTREGISTER_g >>> U H5E_PLINE_g >>> U H5Epush1 >>> 0000000000001080 T H5PLget_plugin_info >>> 0000000000001070 T H5PLget_plugin_type >>> U H5Pget_chunk >>> U H5Pget_filter_by_id2 >>> U H5Pmodify_filter >>> U H5Tget_size >>> U H5Zregister >>> U H5open >>> >>> And here we are: >>> >>> [epourmal@platypus lzf]$ hdf5/hdf5/bin/h5dump test_lzf.hdf5 |more >>> HDF5 "test_lzf.hdf5" { >>> GROUP "/" { >>> DATASET "dset" { >>> DATATYPE H5T_IEEE_F32LE >>> DATASPACE SIMPLE { ( 100, 100, 100 ) / ( 100, 100, 100 ) } >>> DATA { >>> (0,0,0): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, >>> 17, >>> (0,0,18): 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, >>> 32, >>> (0,0,33): 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, >>> 47, >>> (0,0,48): 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, >>> 62, >>> (0,0,63): 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, >>> 77, >>> (0,0,78): 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, >>> 92, >>> (0,0,93): 93, 94, 95, 96, 97, 98, 99, >>> ……. >>> >>> Hope it will help! >>> >>> Elena >>> >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Elena Pourmal The HDF Group http://hdfgroup.org >>> 1800 So. Oak St., Suite 203, Champaign IL 61820 >>> 217.531.6112 >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> >>> >>> >>> On Nov 23, 2015, at 8:50 AM, Daniel Tetlow <[email protected]> >>> wrote: >>> >>> Hi, >>> >>> I had a similar problem on Windows. I needed to link the LZF plugin >>> against the hdf5 library (as expected) to get it to compile, and it worked >>> fine with C++, but it wouldn’t work with the Java H5 libraries. Attempting >>> to link it against the jhdf5 2.11 library instead failed. I couldn’t figure >>> out what I was doing wrong, so in the end I rebuilt jhdf5 with the plugin >>> sources built in. >>> >>> Cheers, >>> >>> Dan >>> >>> >>> *From:* Hdf-forum [mailto:[email protected] >>> <[email protected]>] *On Behalf Of *Ümit Seren >>> *Sent:* 23 November 2015 14:10 >>> *To:* [email protected] >>> *Subject:* [Hdf-forum] Problem with a filter as shared library in >>> Java-HDF5 and HDF5View >>> >>> I have been trying to compile the lzf compressor as a shared library >>> (not linked against hdf5) to be used as a filter plugin for HDF5. >>> >>> Basically I was following this: >>> https://github.com/h5py/h5py/tree/master/lzf >>> >>> I compiled lzf as a shared library using this compile command: >>> >>> >>> gcc -O2 -fPIC -shared lzf/*.c lzf_filter.c -o liblzf_filter.so >>> >>> >>> >>> This will create a lzf filter plugin that I copied into the >>> /usr/local/hdf5/lib/plugin folder. >>> >>> The normal command line tools (h5ls, h5dump) work fine with lzf compressed >>> dataset, so the plugin seems to work. >>> >>> >>> >>> However when I try to read the same lzf compressed dataset using Java HDF5 >>> or HDFView (2.11), I get the error that the lzf filter plugin can not be >>> found. >>> >>> >>> >>> The only workaround is to link it against the hdf5 library: >>> >>> >>> >>> gcc -O2 -fPIC -shared lzf/*.c lzf_filter.c -lhdf5 -o liblzf_filter.so >>> >>> >>> >>> Is this a known issue ? >>> >>> This was on Ubuntu 14.04 and HDF5 1.8.15patch1 >>> >>> thanks in advance >>> >>> cheers >>> Ümit >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 > > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
