[ https://issues.apache.org/jira/browse/ARROW-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17637494#comment-17637494 ]
Arjan van der Velde edited comment on ARROW-17374 at 11/22/22 10:56 PM: ------------------------------------------------------------------------ I have a conda environment file. I haven't stripped it down to the minimum requirements for arrow but it illustrates the issue. The issue happens on a linux (centos) install and not on macOS where it seems to be using imported libraries from brew. [^environment.yml] {code:java} # Set up conda environment and activate conda env create -n testme -f environment.yml conda activate testme # Build required 'decor' which doesn't exist in any conda repo R -e 'install.packages("decor", repos="http://cran.us.r-project.org")' # Build arrow mkdir -p /tmp/git && \ cd /tmp/git && git clone 'https://github.com/apache/arrow' arrow && cd /tmp/git/arrow/r && \ ARROW_S3=OFF LIBARROW_MINIMAL=FALSE ARROW_R_DEV=TRUE R CMD INSTALL . {code} The build fails with: {noformat} ......_JSON DARROW_R_WITH_GCS -I'/home/username/miniconda3/envs/testme/lib/R/library/cpp11/include' -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/username/miniconda3/envs/testme/include -I/home/username/miniconda3/envs/testme/include -Wl,-rpath-link,/home/username/miniconda3/envs/testme/lib -fpic -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/miniconda3/envs/testme/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/r-base-split_1665472261643/work=/usr/local/src/conda/r-base-4.1.3 -fdebug-prefix-map=/home/username/miniconda3/envs/testme=/usr/local/src/conda-prefix -c type_infer.cpp -o type_infer.ox86_64-conda-linux-gnu-c++ -std=gnu++17 -shared -L/home/username/miniconda3/envs/testme/lib/R/lib -Wl,-O2 -Wl,sort-common -Wl,as-needed -Wl,-z,relro -Wl,-z,now -Wl,disable-new-dtags -Wl,gc-sections -Wl,-allow-shlib-undefined -Wl,-rpath,/home/username/miniconda3/envs/testme/lib -Wl,-rpath-link,/home/username/miniconda3/envs/testme/lib -L/home/username/miniconda3/envs/testme/lib -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/tmp/git/arrow/r/libarrow/arrow-10.0.0.9000/lib -larrow_dataset -lparquet -Llibarrow/arrow-10.0.0.9000/lib -L/home/username/miniconda3/envs/testme/lib -larrow SNAPPY_LIB-NOTFOUND /home/username/miniconda3/envs/testme/lib/libre2.so.9.0.0 /usr/lib64/libbz2.so -pthread -larrow_bundled_dependencies -lbrotlidec -lbrotlienc -lbrotlicommon -lthrift -lz -llz4 -lzstd -lutf8proc -larrow -larrow_bundled_dependencies -larrow_dataset -lparquet -lssl -lcrypto -lcurl -L/home/username/miniconda3/envs/testme/lib/R/lib -lR/home/username/miniconda3/envs/testme/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find SNAPPY_LIB-NOTFOUND: No such file or directory collect2: error: ld returned 1 exit status make: *** [/home/username/miniconda3/envs/testme/lib/R/share/make/shlib.mk:10: arrow.so] Error 1 ERROR: compilation failed for package ‘arrow’ * removing ‘/home/username/miniconda3/envs/testme/lib/R/library/arrow’ * restoring previous ‘/home/username/miniconda3/envs/testme/lib/R/library/arrow’ {noformat} was (Author: JIRAUSER297783): I have a conda environment file. I haven't stripped it down to the minimum requirements for arrow but it illustrates the issue. The issue happens on a linux (centos) install and not on macOS where it seems to be using imported libraries from brew. [^environment.yml] {code:java} # Set up conda environment and activate conda env create -n testme -f environment.yml conda activate testme # Build required 'decor' which doesn't exist in any conda repo R -e 'install.packages("decor", repos="http://cran.us.r-project.org")' # Build arrow mkdir -p /tmp/git && \ cd /tmp/git && git clone 'https://github.com/apache/arrow' arrow && cd /tmp/git/arrow/r && \ ARROW_S3=OFF LIBARROW_MINIMAL=FALSE ARROW_R_DEV=TRUE R CMD INSTALL . {code} The build fails with: ...... ......_JSON -DARROW_R_WITH_GCS -I'/home/username/miniconda3/envs/testme/lib/R/library/cpp11/include' -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/username/miniconda3/envs/testme/include -I/home/username/miniconda3/envs/testme/include -Wl,-rpath-link,/home/username/miniconda3/envs/testme/lib -fpic -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/username/miniconda3/envs/testme/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/r-base-split_1665472261643/work=/usr/local/src/conda/r-base-4.1.3 -fdebug-prefix-map=/home/username/miniconda3/envs/testme=/usr/local/src/conda-prefix -c type_infer.cpp -o type_infer.ox86_64-conda-linux-gnu-c++ -std=gnu++17 -shared -L/home/username/miniconda3/envs/testme/lib/R/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/username/miniconda3/envs/testme/lib -Wl,-rpath-link,/home/username/miniconda3/envs/testme/lib -L/home/username/miniconda3/envs/testme/lib -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/tmp/git/arrow/r/libarrow/arrow-10.0.0.9000/lib -larrow_dataset -lparquet -Llibarrow/arrow-10.0.0.9000/lib -L/home/username/miniconda3/envs/testme/lib -larrow SNAPPY_LIB-NOTFOUND /home/username/miniconda3/envs/testme/lib/libre2.so.9.0.0 /usr/lib64/libbz2.so -pthread -larrow_bundled_dependencies -lbrotlidec -lbrotlienc -lbrotlicommon -lthrift -lz -llz4 -lzstd -lutf8proc -larrow -larrow_bundled_dependencies -larrow_dataset -lparquet -lssl -lcrypto -lcurl -L/home/username/miniconda3/envs/testme/lib/R/lib -lR/home/username/miniconda3/envs/testme/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find SNAPPY_LIB-NOTFOUND: No such file or directory collect2: error: ld returned 1 exit status make: *** [/home/username/miniconda3/envs/testme/lib/R/share/make/shlib.mk:10: arrow.so] Error 1 ERROR: compilation failed for package ‘arrow’ * removing ‘/home/username/miniconda3/envs/testme/lib/R/library/arrow’ * restoring previous ‘/home/username/miniconda3/envs/testme/lib/R/library/arrow’ > [R] R Arrow install fails with SNAPPY_LIB-NOTFOUND > -------------------------------------------------- > > Key: ARROW-17374 > URL: https://issues.apache.org/jira/browse/ARROW-17374 > Project: Apache Arrow > Issue Type: Bug > Components: R > Affects Versions: 8.0.0, 8.0.1, 9.0.0 > Environment: Amazon Linux 2 (RHEL) - 5.10.102-99.473.amzn2.x86_64 > Reporter: Shane Brennan > Priority: Blocker > Attachments: build-images.out, environment.yml > > > I've been trying to install Arrow on an R notebook within AWS SageMaker. > SageMaker provides Jupyter-like notebooks, with each instance running Amazon > Linux 2 as its OS, itself based on RHEL. > Trying to install a few ways, e.g., using the standard binaries, using the > nightly builds, setting ARROW_WITH_SNAPPY to ON and LIBARROW_MINIMAL all > still result in the following error. > {noformat} > x86_64-conda-linux-gnu-c++ -std=gnu++11 -shared > -L/home/ec2-user/anaconda3/envs/R/lib/R/lib -Wl,-O2 -Wl,--sort-common > -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags > -Wl,--gc-sections -Wl,--allow-shlib-undefined > -Wl,-rpath,/home/ec2-user/anaconda3/envs/R/lib > -Wl,-rpath-link,/home/ec2-user/anaconda3/envs/R/lib > -L/home/ec2-user/anaconda3/envs/R/lib -o arrow.so RTasks.o altrep.o array.o > array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o > compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o > expression.o extension-impl.o feather.o field.o filesystem.o imports.o io.o > json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o > recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o > schema.o symbols.o table.o threadpool.o type_infer.o > -L/tmp/Rtmpuh87oc/R.INSTALL67114493a3de/arrow/libarrow/arrow-9.0.0.20220809/lib > -larrow_dataset -lparquet -larrow -larrow_bundled_dependencies -lz > SNAPPY_LIB-NOTFOUND /home/ec2-user/anaconda3/envs/R/lib/libbz2.so -pthread > -larrow -larrow_bundled_dependencies -larrow_dataset -lparquet -lssl -lcrypto > -lcurl -lssl -lcrypto -lcurl -L/home/ec2-user/anaconda3/envs/R/lib/R/lib -lR > x86_64-conda-linux-gnu-c++: error: SNAPPY_LIB-NOTFOUND: No such file or > directory > make: *** [/home/ec2-user/anaconda3/envs/R/lib/R/share/make/shlib.mk:10: > arrow.so] Error 1{noformat} > Snappy is installed on the systems, and both shared object (.so) and cmake > files are there, where I've tried setting the system env variables Snappy_DIR > and Snappy_LIB to point at them, but to no avail. -- This message was sent by Atlassian Jira (v8.20.10#820010)