Hi Ayush,
Upgrading to 2.6.0-hadoop3 worked, thanks so much!
On 2024-05-25 20:15, Ayush Saxena wrote:
Multiple things, the output of checknative only contains these stuff
only, not everything. From the code [1], So looking at your command
output everything is sorted there barring OpenSSL & PMDK which you
explicitly didn't ask for in your maven command & I believe you don't
need them either, in case you need them the instructions are there in
[2]
Looking at the trace:
org.apache.hadoop.io.compress.Lz4Codec.getLibraryName(Lz4Codec.java:73)
at
You mentioned building ver 3.3.6, Your exception trace is calling
getLibraryName, which isn't present in the Lz4Codec.java in ver-3.3.6
[3], this method got removed as part of HADOOP-17292 [4] that is post
hadoop ver 3.3.1+, So, If you read the release notes of this ticket
you can see for Hadoop-3.3.1+ the Lz4 thing works OOTB
So, mostly it isn't a Hadoop problem.
What could be possible is the HBase version that you are using is
pulling in an older Hadoop release which is messing things up, So, I
would say try using the hadoop-3 binary of the latest version 2.6.0
[5] & see how things go, else download the source tar of their latest
release 2.6.0 and build with -Phadoop-3.0
-Dhadoop-three.version=3.3.6, Looking at their source code they still
use 3.3.5 by default.
-Ayush
[1]
https://github.com/apache/hadoop/blob/1baf0e889fec54b6560417b62cada75daf6fe312/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java#L137-L144
[2] https://github.com/apache/hadoop/blob/branch-3.3.6/BUILDING.txt
[3]
https://github.com/apache/hadoop/blob/branch-3.3.6/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/Lz4Codec.java#L73
[4] https://issues.apache.org/jira/browse/HADOOP-17292
[5]
https://www.apache.org/dyn/closer.lua/hbase/2.6.0/hbase-2.6.0-hadoop3-bin.tar.gz
On Sat, 25 May 2024 at 22:41, <fe...@posteo.nl> wrote:
Hey Ayush, thanks for the advice!
Building 3.3.6 from an EL9.4 machine resulted in the following:
[root@localhost bin]# JAVA_HOME=/etc/alternatives/java_sdk_openjdk/
./hadoop checknative -a
2024-05-25 19:05:56,068 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-05-25 19:05:56,071 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-05-25 19:05:56,097 INFO nativeio.NativeIO: The native code was
built without PMDK support.
Native library checking:
hadoop: true
/root/build/hadoop-3.3.6-src/hadoop-dist/target/hadoop-3.3.6/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
zstd : true /lib64/libzstd.so.1
bzip2: true /lib64/libbz2.so.1
openssl: false EVP_CIPHER_CTX_block_size
ISA-L: true /lib64/libisal.so.2
PMDK: false The native code was built without PMDK support.
No mention of lz4, though lz4[-devel] packages were installed on the
compiling host as per the BUILDING instructions. Is there a build
option
I'm missing? I'm using:
* mvn -X package -Pdist,native -DskipTests -Dtar
-Dmaven.javadoc.skip=true
Unfortunately the hbase "org.apache.hadoop.util.NativeLibraryChecker",
using this freshly made hadoop native library, also failed to load lz4
in the same way as the initial message with no extra information from
debug:
2024-05-25T19:03:42,320 WARN [main] lz4.Lz4Compressor:
java.lang.UnsatisfiedLinkError: 'void
org.apache.hadoop.io.compress.lz4.Lz4Compressor.initIDs()'
Exception in thread "main" java.lang.UnsatisfiedLinkError:
'java.lang.String
org.apache.hadoop.io.compress.lz4.Lz4Compressor.getLibraryName()'
at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.getLibraryName(Native
Method)
at
org.apache.hadoop.io.compress.Lz4Codec.getLibraryName(Lz4Codec.java:73)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:109)
Thanks for your help!
On 5/25/24 4:16 PM, Ayush Saxena wrote:
> above things don't work then enable debug logging &
> then run the checknative command and capture the log & exception as
> here [2] & they might give you an an
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org
