Xiao Yu created BIGTOP-3535:
-------------------------------
Summary: Build libhadoop.so with zstd
Key: BIGTOP-3535
URL: https://issues.apache.org/jira/browse/BIGTOP-3535
Project: Bigtop
Issue Type: Bug
Components: hadoop
Affects Versions: 1.5.0
Reporter: Xiao Yu
[Zstandard compression|https://github.com/facebook/zstd] support was added to
Hadoop 2.9.0 with HADOOP-13578 it would be great if we bundled support for it
with native lib. As it stands now with Bigtop 1.5.0 we don't support it:
{code}
$ hadoop checknative -a
21/04/13 23:18:20 INFO bzip2.Bzip2Factory: Successfully loaded & initialized
native-bzip2 library system-native
21/04/13 23:18:20 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib/x86_64-linux-gnu/libz.so.1
snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd : false
lz4: true revision:10301
bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
21/04/13 23:18:20 INFO util.ExitUtil: Exiting with status 1: ExitException
{code}
At least with Debian, after installing headers for {{zstd}} with {{apt-get
install libzstd-dev}} and rebuilding the native lib with {{mvn package
-Pdist,native -DskipTests -Drequire.zstd ...}} support appears to be enabled:
{code}
$ hadoop checknative -a
21/04/13 23:25:46 INFO bzip2.Bzip2Factory: Successfully loaded & initialized
native-bzip2 library system-native
21/04/13 23:25:46 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib/x86_64-linux-gnu/libz.so.1
snappy: true /usr/lib/hadoop/lib/native/libsnappy.so.1
zstd : true /usr/lib/x86_64-linux-gnu/libzstd.so.1
lz4: true revision:10301
bzip2: true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
