Re: Unsubscribe

[Ayush Saxena]

Sending unsubscribe here won’t help.
Send a mail to: 

user-unsubscr...@hadoop.apache.org

Or click on the unsubscribe link in this page for the relevant mailing list:

https://hadoop.apache.org/mailing_lists.html

-Ayush

> On 27-Mar-2022, at 3:13 AM, Kasi Subrahmanyam  wrote:
> 
> 
> 
>> On Sat, Mar 26, 2022, 10:13 Ravikumar Govindarajan 
>>  wrote:

Unsubscribe

[Kasi Subrahmanyam]

On Sat, Mar 26, 2022, 10:13 Ravikumar Govindarajan <
ravikumar.govindara...@gmail.com> wrote:

>

Unsubscribe

[Dmitrii Kuzin]

Unsubscribe

Unsubscribe

[Ravikumar Govindarajan]

Re: Building Hadoop on macOS Monterey?

[Joe Mocker]

I realize we are getting off topic for the list, but yeah I had entertained the 
idea of just running with SIP disabled. After all, I use Linux instances all 
day…

As I investigated how macOS striped DYLD* & LD*, macOS would still load native 
libraries in places like your current working directory, so as an attacker I 
could still arrange for my code to be loaded with enough tinkering without the 
means I resorted to. ¯\_(ツ)_/¯ 

  —joe

> On Mar 25, 2022, at 10:55 PM, Hariharan  wrote:
> 
> The stripping of DYLD* and LD* variables is a "feature" that's part of 
> Apple's SIP. So another option to stop this is to disable SIP - 
> https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection
>  
> 
> 
> Apple doesn't recommend this, but I've been running different macbooks with 
> SIP disabled for years, and haven't noticed any bad side effects.
> 
> As an aside, Apple has a history of crippling various functionalities behind 
> SIP. For example, in the latest versions of Monterey, you can't run certain 
> `dscl` commands unless you either disable SIP or provide "Full Disk Access" 
> to bash.
> 
> Thanks, 
> Hariharan 
> 
> On Fri, 25 Mar 2022, 21:46 Andrew Purtell,  > wrote:
> Thank you for sharing that blog post.
> 
> > The TL;DR is that as soon as macOS executes one if its trusted executables, 
> > like /bin/sh or /usr/bin/env, it cripples anything you might have done like 
> > setting DYLD_LIBRARY_PATH to dynamic library folders, and results in 
> > failure to load them. 
> 
> On the one hand I can see the security requirements that led to this 
> decision, but this is so contrary to the UNIX philosophy, IMHO, it's no 
> wonder it violates the principle of least surprise here and I bet in many 
> many other situations. This reminds me why 'step 1' of setting up for dev on 
> my new M1 macbook was to install Parallels and a Linux aarch64 VM. That 
> environment is quite sane and the VM overheads are manageable. 
> 
> 
> On Fri, Mar 25, 2022 at 9:03 AM Joe Mocker  > wrote:
> Hi, Thanks...
> 
> It ended up being more involved than that due to all the shared library 
> dependencies, but I figured it out (at least with an older version of 
> Hadoop). I ended up writing a short post about it
> 
> https://creechy.wordpress.com/2022/03/22/building-hadoop-spark-jupyter-on-macos/
>  
> 
> 
>  --joe
> 
> On Thu, Mar 24, 2022 at 3:14 PM Andrew Purtell  > wrote:
> If you build with -Dbundle.snappy -Dbundle.zstd on the Maven command line 
> this would produce a tarball containing copies of the native shared libraries 
> in lib/native/ and this would be like your symlink workaround but perhaps 
> less hacky and something the build supports already. Does this work for you? 
> 
>> On Mar 19, 2022, at 8:09 AM, Joe Mocker  wrote:
>> 
>> Hi,
>> 
>> Curious if anyone has tips for building Hadoop on macOS Monterey, for Apple 
>> Silicon? My goal is to be able to use native (compression) libraries. After 
>> some gymnastics, I have been able to compile Hadoop 2.9.1 but problems arise 
>> locating and loading dynamic libraries.
>> 
>> For example running hadoop checknative results in the following
>> 
>> 22/03/19 07:57:00 WARN bzip2.Bzip2Factory: Failed to load/initialize 
>> native-bzip2 library system-native, will use pure-Java version
>> 22/03/19 07:57:00 INFO zlib.ZlibFactory: Successfully loaded & initialized 
>> native-zlib library
>> 22/03/19 07:57:00 ERROR snappy.SnappyCompressor: failed to load 
>> SnappyCompressor
>> java.lang.UnsatisfiedLinkError: Cannot load libsnappy.1.dylib 
>> (dlopen(libsnappy.1.dylib, 0x0009): tried: 
>> '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libsnappy.1.dylib'
>>  (no such file), 'libsnappy.1.dylib' (no such file), 
>> '/usr/lib/libsnappy.1.dylib' (no such file), 
>> '/Volumes/work/hadoop-2.9.1/libsnappy.1.dylib' (no such file))!
>>  at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native 
>> Method)
>>  at 
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.(SnappyCompressor.java:57)
>>  at 
>> org.apache.hadoop.io.compress.SnappyCodec.isNativeCodeLoaded(SnappyCodec.java:82)
>>  at 
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:92)
>> 22/03/19 07:57:00 WARN zstd.ZStandardCompressor: Error loading zstandard 
>> native libraries: java.lang.InternalError: Cannot load libzstd.1.dylib 
>> (dlopen(libzstd.1.dylib, 0x0009): tried: 
>> '/Volumes/work/zulu8.60.0.21-ca-jdk8.0.322-macosx_aarch64/zulu-8.jdk/Contents/Home/bin/./libzstd.1.dylib'
>>  (no such file), 'libzstd.1.dylib' (no such file), 
>> '/usr/lib/libzstd.1.dylib' (no such file), 
>> '/Volumes/work/hadoop-2