Hi,
Curious if anyone has tips for building Hadoop on macOS Monterey, for Apple
Silicon? My goal is to be able to use native (compression) libraries. After
some gymnastics, I have been able to compile Hadoop 2.9.1 but problems arise
locating and loading dynamic libraries.
For example running hadoop checknative results in the following
22/03/19 07:57:00 WARN bzip2.Bzip2Factory: Failed to load/initialize
native-bzip2 library system-native, will use pure-Java version
22/03/19 07:57:00 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
22/03/19 07:57:00 ERROR snappy.SnappyCompressor: failed to load SnappyCompressor
java.lang.UnsatisfiedLinkError: Cannot load libsnappy.1.dylib
(dlopen(libsnappy.1.dylib, 0x0009): tried:
'/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libsnappy.1.dylib'
(no such file), 'libsnappy.1.dylib' (no such file),
'/usr/lib/libsnappy.1.dylib' (no such file),
'/Volumes/work/hadoop-2.9.1/libsnappy.1.dylib' (no such file))!
at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native
Method)
at
org.apache.hadoop.io.compress.snappy.SnappyCompressor.<clinit>(SnappyCompressor.java:57)
at
org.apache.hadoop.io.compress.SnappyCodec.isNativeCodeLoaded(SnappyCodec.java:82)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:92)
22/03/19 07:57:00 WARN zstd.ZStandardCompressor: Error loading zstandard native
libraries: java.lang.InternalError: Cannot load libzstd.1.dylib
(dlopen(libzstd.1.dylib, 0x0009): tried:
'/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libzstd.1.dylib'
(no such file), 'libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib'
(no such file), '/Volumes/work/hadoop-2.9.1/libzstd.1.dylib' (no such file))!
WARNING: /work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64//bin/java is loading
libcrypto in an unsafe way
Abort trap: 6
No matter what combination I try of setting LD_LIBRARY_PATH, DYLD_LIBRARY_PATH
and/or DYLD_FALLBACK_LIBRARY_PATH it will not find the necessary libraries. I
think this has to do with restrictions due to Apple’s System Integrity
Protection (SIP).
The only way I have figured out how to work around this so far is to symlink
all the dynamic libraries in one location then run hadoop from that working
directory, for example
lrwxrwxr-x 1 mock staff 59 Mar 18 17:55 libcrypto.dylib@ ->
/opt/homebrew/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.dylib
lrwxrwxr-x 1 mock staff 45 Mar 18 18:09 libhadoop.dylib@ ->
/work/hadoop-2.9.1/lib/native/libhadoop.dylib
lrwxrwxr-x 1 mock staff 53 Mar 18 17:55 libsnappy.1.dylib@ ->
/opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.dylib
lrwxrwxr-x 1 mock staff 51 Mar 18 18:05 libzstd.1.dylib@ ->
/opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.dylib
% $HADOOP_HOME/bin/hadoop checknative
22/03/19 08:05:55 WARN bzip2.Bzip2Factory: Failed to load/initialize
native-bzip2 library system-native, will use pure-Java version
22/03/19 08:05:55 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
Native library checking:
hadoop: true /Volumes/work/hadoop-2.9.1/lib/native/libhadoop.dylib
zlib: true /usr/lib/libz.1.dylib
snappy: true /opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.1.1.9.dylib
zstd : true /opt/homebrew/Cellar/zstd/1.5.2/lib/libzstd.1.5.2.dylib
lz4: true revision:10301
bzip2: false
openssl: false EVP_CIPHER_CTX_cleanup
What am really looking to do is use Spark (and Jupyter), with native libraries,
which adds even more wrinkles to it.
Any suggestions would be appreciated.
—joe
