Hi,
> $ DYLD_FALLBACK_LIBRARY_PATH="$(brew --prefix)/lib:/usr/local/lib:/usr/lib" \
> ruby arrow-test.rb
Ah, I forgot that you use rbenv. rbenv runs ruby from
a wrapper script. DYLD_FALLBACK_LIBRARY_PATH isn't inherited
to a subprocess on macOS for security reason.
Could you try the following?
$ DYLD_FALLBACK_LIBRARY_PATH="$(brew --prefix)/lib:/usr/local/lib:/usr/lib" \
$(rbenv which ruby) arrow-test.rb
If you can disable SIP (System Integrity Protection) on the
machine, could you provide dtrace log?
$ sudo dtrace $(rbenv which ruby) arrow-test.rb
Thanks,
--
kou
In <ca+ebnme0tpxknapaxmeqjwppi8qi8ubmqi1m-yg6foqwcp4...@mail.gmail.com>
"Re: [Ruby] Cannot require 'parquet' on M1 Mac" on Thu, 14 Apr 2022 08:26:54
+0200,
Sten Larsson <[email protected]> wrote:
> Hi kou
>
> Thanks for trying to help me with this
>
>
> 1. Yes
>
>
> 2. Note that I skipped the -r flag to get the result.
>
> $ grep -A 4 '<namespace' $(brew --prefix)/share/gir-1.0/Arrow-1.0.gir
> <namespace name="Arrow"
> version="1.0"
> shared-library="libarrow-glib.700.dylib"
> c:identifier-prefixes="GArrow"
> c:symbol-prefixes="garrow">
>
>
> 3. Same here
>
> $ grep -A 4 '<namespace' $(brew --prefix)/share/gir-1.0/Parquet-1.0.gir
> <namespace name="Parquet"
> version="1.0"
> shared-library="libparquet-glib.700.dylib"
> c:identifier-prefixes="GParquet"
> c:symbol-prefixes="gparquet">
>
>
> 4. No matches
>
> $ env | grep LIBRARY_PATH | sort
>
>
> 5. Nothing found
>
> $ ls /usr/local/lib/lib*-glib.*.dylib
> zsh: no matches found: /usr/local/lib/lib*-glib.*.dylib
>
>
> 6. Nothing found
>
> $ ls /usr/lib/lib*-glib.*.dylib
> zsh: no matches found: /usr/lib/lib*-glib.*.dylib
>
>
> DYLD_FALLBACK_LIBRARY_PATH doesn't seem to have any effect
>
> $ DYLD_FALLBACK_LIBRARY_PATH="$(brew --prefix)/lib:/usr/local/lib:/usr/lib"
> ruby arrow-test.rb
> (null)-WARNING **: Failed to load shared library
> 'libparquet-glib.700.dylib' referenced by the typelib:
> dlopen(libparquet-glib.700.dylib, 0x0009): tried:
> 'libparquet-glib.700.dylib' (no such file),
> '/usr/local/lib/libparquet-glib.700.dylib' (no such file),
> '/usr/lib/libparquet-glib.700.dylib' (no such file),
> '/Users/stenlarsson/Documents/src/arrow-test/libparquet-glib.700.dylib' (no
> such file)
> [...]
>
> Unfortunately it is still a mystery.
>
> Thanks
> /Sten
>
>
> On Wed, 13 Apr 2022 at 23:47, Sutou Kouhei <[email protected]> wrote:
>
>> Hi,
>>
>> Could you tell the following?
>>
>> 1. Did you run the script in
>> /Users/stenlarsson/Documents/src/arrow-test/ ?
>>
>> 2. The output of
>> grep -r -A 4 '<namespace' $(brew --prefix)/share/gir-1.0/Arrow-1.0.gir
>>
>> 3. The output of
>> grep -r -A 4 '<namespace' $(brew --prefix)/share/gir-1.0/Parquet-1.0.gir
>>
>> 4. The output of
>> env | grep LIBRARY_PATH | sort
>>
>> 5. The output of
>> ls /usr/local/lib/lib*-glib.*.dylib
>>
>> 6. The output of
>> ls /usr/lib/lib*-glib.*.dylib
>>
>> The following command line may resolve this:
>>
>> $ DYLD_FALLBACK_LIBRARY_PATH="$(brew
>> --prefix)/lib:/usr/local/lib:/usr/lib" \
>> ruby arrow-test.rb
>>
>> Thanks,
>> --
>> kou
>>
>> In <ca+ebnmfqgyjwrxrdlhd4qwbnhz0n3p-cd_esse1+b5ctk3g...@mail.gmail.com>
>> "[Ruby] Cannot require 'parquet' on M1 Mac" on Wed, 13 Apr 2022 10:34:52
>> +0200,
>> Sten Larsson <[email protected]> wrote:
>>
>> > Hi
>> >
>> > I'm struggling to get Arrow working on my M1 MacBook Pro. The test
>> program
>> > simply consists of
>> >
>> > require 'parquet'
>> >
>> > This fails with
>> >
>> > $ ruby arrow-test.rb
>> > (null)-WARNING **: Failed to load shared library
>> > 'libparquet-glib.700.dylib' referenced by the typelib:
>> > dlopen(libparquet-glib.700.dylib, 0x0009): tried:
>> > 'libparquet-glib.700.dylib' (no such file),
>> > '/usr/local/lib/libparquet-glib.700.dylib' (no such file),
>> > '/usr/lib/libparquet-glib.700.dylib' (no such file),
>> > '/Users/stenlarsson/Documents/src/arrow-test/libparquet-glib.700.dylib'
>> (no
>> > such file)
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:234:in
>> > `load_object_info'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet/loader.rb:38:in
>> > `load_object_info'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:73:in
>> > `load_info'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:47:in
>> > `block (2 levels) in load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:34:in
>> > `block (2 levels) in each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:33:in
>> > `times'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:33:in
>> > `block in each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:32:in
>> > `each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:32:in
>> > `each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:46:in
>> > `block in load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:622:in
>> > `prepare_class'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:41:in
>> > `load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:25:in
>> > `load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet/loader.rb:22:in
>> > `load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet.rb:28:in
>> > `<module:Parquet>'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet.rb:24:in
>> > `<top (required)>'
>> > from
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:160:in
>> > `require'
>> > from
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:160:in
>> > `rescue in require'
>> > from
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:149:in
>> > `require'
>> > from arrow-test.rb:1:in `<main>'
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet/loader.rb:40:in
>> > `load_object_info': uninitialized constant Parquet::ArrowFileReader
>> > (NameError)
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:73:in
>> > `load_info'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:47:in
>> > `block (2 levels) in load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:34:in
>> > `block (2 levels) in each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:33:in
>> > `times'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:33:in
>> > `block in each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:32:in
>> > `each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/repository.rb:32:in
>> > `each'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:46:in
>> > `block in load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:622:in
>> > `prepare_class'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:41:in
>> > `load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/gobject-introspection-3.5.1/lib/gobject-introspection/loader.rb:25:in
>> > `load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet/loader.rb:22:in
>> > `load'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet.rb:28:in
>> > `<module:Parquet>'
>> > from
>> >
>> /Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/gems/3.0.0/gems/red-parquet-7.0.0/lib/parquet.rb:24:in
>> > `<top (required)>'
>> > from
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:160:in
>> > `require'
>> > from
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:160:in
>> > `rescue in require'
>> > from
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:149:in
>> > `require'
>> > from arrow-test.rb:1:in `<main>'
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
>> > `require': cannot load such file -- parquet (LoadError)
>> > from
>> >
>> <internal:/Users/stenlarsson/.rbenv/versions/3.0.3/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in
>> > `require'
>> > from arrow-test.rb:1:in `<main>'
>> >
>> > I have installed the apache-arrow-glib Homebrew package, but it seems
>> like
>> > it is looking for the library in the wrong place, because on M1 it is
>> > installed in /opt/homebrew. The libraries seems to be built correctly as
>> > far as I can tell:
>> >
>> > $ otool -L /opt/homebrew/lib/libparquet-glib.700.dylib
>> > /opt/homebrew/lib/libparquet-glib.700.dylib:
>> > /opt/homebrew/opt/apache-arrow-glib/lib/libparquet-glib.700.dylib
>> > (compatibility version 700.0.0, current version 700.0.0)
>> > /opt/homebrew/Cellar/apache-arrow-glib/7.0.0/lib/libarrow-glib.700.dylib
>> > (compatibility version 700.0.0, current version 700.0.0)
>> > /opt/homebrew/opt/apache-arrow/lib/libarrow.700.dylib (compatibility
>> > version 700.0.0, current version 700.0.0)
>> > /opt/homebrew/opt/apache-arrow/lib/libparquet.700.dylib (compatibility
>> > version 700.0.0, current version 700.0.0)
>> > /opt/homebrew/opt/glib/lib/libgobject-2.0.0.dylib (compatibility version
>> > 7001.0.0, current version 7001.3.0)
>> > /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib (compatibility version
>> > 7001.0.0, current version 7001.3.0)
>> > /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
>> > 1200.3.0)
>> > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
>> > 1311.0.0)
>> >
>> > $ otool -L /opt/homebrew/lib/libparquet.700.dylib
>> > /opt/homebrew/lib/libparquet.700.dylib:
>> > /opt/homebrew/opt/apache-arrow/lib/libparquet.700.dylib (compatibility
>> > version 700.0.0, current version 700.0.0)
>> > @rpath/libarrow.700.dylib (compatibility version 700.0.0, current version
>> > 700.0.0)
>> > /opt/homebrew/opt/thrift/lib/libthrift-0.16.0.dylib (compatibility
>> version
>> > 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/[email protected]/lib/libssl.1.1.dylib (compatibility
>> version
>> > 1.1.0, current version 1.1.0)
>> > /opt/homebrew/opt/[email protected]/lib/libcrypto.1.1.dylib (compatibility
>> > version 1.1.0, current version 1.1.0)
>> > /opt/homebrew/opt/brotli/lib/libbrotlienc.1.dylib (compatibility version
>> > 1.0.0, current version 1.0.9)
>> > /opt/homebrew/opt/brotli/lib/libbrotlidec.1.dylib (compatibility version
>> > 1.0.0, current version 1.0.9)
>> > /opt/homebrew/opt/brotli/lib/libbrotlicommon.1.dylib (compatibility
>> version
>> > 1.0.0, current version 1.0.9)
>> > /opt/homebrew/opt/protobuf/lib/libprotobuf.30.dylib (compatibility
>> version
>> > 31.0.0, current version 31.4.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-config.dylib
>> > (compatibility version 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-transfer.dylib
>> > (compatibility version 0.0.0, current version 0.0.0)
>> >
>> /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-identity-management.dylib
>> > (compatibility version 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-cognito-identity.dylib
>> > (compatibility version 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-sts.dylib (compatibility
>> > version 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-s3.dylib (compatibility
>> > version 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-cpp-sdk-core.dylib
>> (compatibility
>> > version 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-crt-cpp.dylib (compatibility
>> > version 0.0.0, current version 0.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-mqtt.1.0.0.dylib
>> (compatibility
>> > version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-event-stream.1.0.0.dylib
>> > (compatibility version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-checksums.1.0.0.dylib
>> > (compatibility version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-s3.0unstable.dylib
>> > (compatibility version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-auth.1.0.0.dylib
>> (compatibility
>> > version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-http.1.0.0.dylib
>> (compatibility
>> > version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-io.1.0.0.dylib (compatibility
>> > version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-compression.1.0.0.dylib
>> > (compatibility version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-cal.1.0.0.dylib (compatibility
>> > version 0.0.0, current version 1.0.0)
>> > /opt/homebrew/opt/aws-sdk-cpp/lib/libaws-c-common.1.dylib (compatibility
>> > version 1.0.0, current version 1.0.0)
>> > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
>> > 1311.0.0)
>> >
>> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
>> > (compatibility version 150.0.0, current version 1856.105.0)
>> > /System/Library/Frameworks/Security.framework/Versions/A/Security
>> > (compatibility version 1.0.0, current version 60157.60.19)
>> > /opt/homebrew/opt/utf8proc/lib/libutf8proc.2.dylib (compatibility version
>> > 2.0.0, current version 2.5.0)
>> > /opt/homebrew/opt/re2/lib/libre2.9.dylib (compatibility version 9.0.0,
>> > current version 9.0.0)
>> > /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
>> > 1200.3.0)
>> >
>> > The strange thing is that if I change this to
>> >
>> > require 'arrow'
>> >
>> > It works correctly. So for some reason it is able to find the Arrow
>> > library, but not the Parquet library. What is causing the difference?
>> >
>> > Thanks
>> > /Sten
>>
