[jira] [Commented] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[ https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17327023#comment-17327023 ] Julian Hyde commented on ARROW-11135: - [~projjal], Thanks for stepping up to help. I really appreciate it. In case I didn't make it clear: We have used the ursa-labs nightly build, and in fact we checked one of the jars into a git branch during development. But in order to merge into Calcite's master branch (from which releases will be made), we can only depend on released artifacts in Maven central. Our criteria are: * master branch MUST NOT contain binary files (e.g. .jar); * a source release MUST NOT contain binary files; * master branch and release builds MUST NOT depend on snapshot or nightly artifacts, only released artifacts; * code SHOULD run on at least Linux and macOS (windows would be nice, but it's not feasible); * all platforms SHOULD use the same jar file; * artifacts SHOULD be in maven central (another repo might be possible). These are fairly typical for an ASF project, I think. They are designed to reduce friction for people who want to use or develop this feature. > Using Maven Central artifacts as dependencies produce runtime errors > > > Key: ARROW-11135 > URL: https://issues.apache.org/jira/browse/ARROW-11135 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Affects Versions: 2.0.0, 3.0.0 >Reporter: Michael Mior >Assignee: Projjal Chanda >Priority: Major > > I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the > integration is working well, but I'm having issues . As [suggested on the > mailing > list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E], > using Dremio's public artifacts solves the problem. Between two Apache > projects however, there would be strong preference to use Apache artifacts as > a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[ https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320525#comment-17320525 ] Julian Hyde edited comment on ARROW-11135 at 4/13/21, 8:37 PM: --- This issue is going to be a blocker for getting CALCITE-2040 merged into master as a "beta" feature. For our purposes, the artifact(s) have to be in Maven central, and have to work on one OS (either macOS or Linux) preferably both. Currently the Maven central artifacts don't work for either. [~kou] and [~wesm], Any suggestions who we could nag to get this fixed for 4.0.0? I know contributors are volunteers, but it's a shame to have Calcite contributions languishing, and the Calcite adapter would be a great way to improve adoption of Arrow/Gandiva. If we can't find anyone, that's fine, but I want to be able to say we tried. was (Author: julianhyde): This issue is going to be a blocker for getting CALCITE-2040 merged into master as a "beta" feature. For our purposes, the artifact(s) have to be in Maven central, and have to work on one OS (either macOS or Linux) preferably both. Currently the Maven central artifacts don't work for either. [~kou] and [~wesm], Any suggestions who we could nag to get this fixed for 4.0.0? I know contributors are volunteers, but it's a shame to have Calcite contributions languishing. If we can't find anyone, that's fine, but I want to be able to say we tried. > Using Maven Central artifacts as dependencies produce runtime errors > > > Key: ARROW-11135 > URL: https://issues.apache.org/jira/browse/ARROW-11135 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Affects Versions: 2.0.0, 3.0.0 >Reporter: Michael Mior >Priority: Major > > I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the > integration is working well, but I'm having issues . As [suggested on the > mailing > list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E], > using Dremio's public artifacts solves the problem. Between two Apache > projects however, there would be strong preference to use Apache artifacts as > a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[ https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320525#comment-17320525 ] Julian Hyde commented on ARROW-11135: - This issue is going to be a blocker for getting CALCITE-2040 merged into master as a "beta" feature. For our purposes, the artifact(s) have to be in Maven central, and have to work on one OS (either macOS or Linux) preferably both. Currently the Maven central artifacts don't work for either. [~kou] and [~wesm], Any suggestions who we could nag to get this fixed for 4.0.0? I know contributors are volunteers, but it's a shame to have Calcite contributions languishing. If we can't find anyone, that's fine, but I want to be able to say we tried. > Using Maven Central artifacts as dependencies produce runtime errors > > > Key: ARROW-11135 > URL: https://issues.apache.org/jira/browse/ARROW-11135 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Affects Versions: 2.0.0, 3.0.0 >Reporter: Michael Mior >Priority: Major > > I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the > integration is working well, but I'm having issues . As [suggested on the > mailing > list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E], > using Dremio's public artifacts solves the problem. Between two Apache > projects however, there would be strong preference to use Apache artifacts as > a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[
https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319551#comment-17319551
]
Julian Hyde commented on ARROW-11135:
-
[~kou], Agreed. Static linking would be better. In fact static linking seems to
work on the Linux jar (I did not need to install protobuf on my Linux machine).
The basic problem statement should be 'make the
{{/usr/local/opt/protobuf/lib/libprotobuf.24.dylib}} error go away'.
> Using Maven Central artifacts as dependencies produce runtime errors
>
>
> Key: ARROW-11135
> URL: https://issues.apache.org/jira/browse/ARROW-11135
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Michael Mior
>Priority: Major
>
> I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the
> integration is working well, but I'm having issues . As [suggested on the
> mailing
> list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E],
> using Dremio's public artifacts solves the problem. Between two Apache
> projects however, there would be strong preference to use Apache artifacts as
> a dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[ https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Hyde updated ARROW-11135: Affects Version/s: 3.0.0 > Using Maven Central artifacts as dependencies produce runtime errors > > > Key: ARROW-11135 > URL: https://issues.apache.org/jira/browse/ARROW-11135 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Affects Versions: 2.0.0, 3.0.0 >Reporter: Michael Mior >Priority: Major > > I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the > integration is working well, but I'm having issues . As [suggested on the > mailing > list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E], > using Dremio's public artifacts solves the problem. Between two Apache > projects however, there would be strong preference to use Apache artifacts as > a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[
https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318933#comment-17318933
]
Julian Hyde commented on ARROW-11135:
-
I think this issue boils down to two problems:
* The install documentation should state that you need to install protobuf on
macOS. That is the cause of the
{{/usr/local/opt/protobuf/lib/libprotobuf.24.dylib}} error.
* The artifacts in Maven Central only support macOS. They should support Linux
and macOS.
Do you agree?
> Using Maven Central artifacts as dependencies produce runtime errors
>
>
> Key: ARROW-11135
> URL: https://issues.apache.org/jira/browse/ARROW-11135
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java
>Affects Versions: 2.0.0
>Reporter: Michael Mior
>Priority: Major
>
> I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the
> integration is working well, but I'm having issues . As [suggested on the
> mailing
> list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E],
> using Dremio's public artifacts solves the problem. Between two Apache
> projects however, there would be strong preference to use Apache artifacts as
> a dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[
https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318931#comment-17318931
]
Julian Hyde edited comment on ARROW-11135 at 4/11/21, 7:33 PM:
---
There are no missing packages. But the install instructions should probably say:
* The Gandiva library only works on macOS, and requires that you manually
install protobuf 2.5.
By the way, I compared which files are in the 3.0.0 release jar (which works on
macOS) and the 3.0.0-SNAPSHOT jar (which works on Linux).
{noformat}
$ diff -u <(tar tf ./arrow-gandiva-3.0.0-SNAPSHOT.jar | sort) <(tar tf
./arrow-gandiva-3.0.0.jar
| sort)
--- /dev/fd/63 2021-04-11 12:25:09.0 -0700
+++ /dev/fd/62 2021-04-11 12:25:09.0 -0700
@@ -11,7 +11,10 @@
META-INF/maven/org.apache.arrow.gandiva/arrow-gandiva/pom.xml
Types.proto
git.properties
-libgandiva_jni.so
+libgandiva_jni.300.0.0.dylib
+libgandiva_jni.300.dylib
+libgandiva_jni.a
+libgandiva_jni.dylib
org/
org/apache/
org/apache/arrow/
@@ -188,3 +191,8 @@
org/apache/arrow/gandiva/ipc/GandivaTypes$TreeNode.class
org/apache/arrow/gandiva/ipc/GandivaTypes$TreeNodeOrBuilder.class
org/apache/arrow/gandiva/ipc/GandivaTypes.class
+release/
+release/libgandiva_jni.300.0.0.dylib
+release/libgandiva_jni.300.dylib
+release/libgandiva_jni.a
+release/libgandiva_jni.dylib
{noformat}
It would be awesome if, in the next release, the jar contained ALL of those
files, and then I suppose it would work on both Linux and macOS.
was (Author: julianhyde):
There are no missing packages. But the install instructions should probably say:
* The Gandiva library only works on macOS, and requires that you manually
install protobuf 2.5.
By the way, I compared which files are in the 3.0.0 release jar (which works on
macOS) and the 3.0.0-SNAPSHOT jar (which works on Linux).
{noformat}
$ diff -u <(tar tvf ./arrow-gandiva-3.0.0-SNAPSHOT.jar |awk '{print $NF}'|sort)
<(tar tvf ./arrow-gandiva-3.0.0.jar |awk '{print $NF}'|sort)
--- /dev/fd/63 2021-04-11 12:25:09.0 -0700
+++ /dev/fd/62 2021-04-11 12:25:09.0 -0700
@@ -11,7 +11,10 @@
META-INF/maven/org.apache.arrow.gandiva/arrow-gandiva/pom.xml
Types.proto
git.properties
-libgandiva_jni.so
+libgandiva_jni.300.0.0.dylib
+libgandiva_jni.300.dylib
+libgandiva_jni.a
+libgandiva_jni.dylib
org/
org/apache/
org/apache/arrow/
@@ -188,3 +191,8 @@
org/apache/arrow/gandiva/ipc/GandivaTypes$TreeNode.class
org/apache/arrow/gandiva/ipc/GandivaTypes$TreeNodeOrBuilder.class
org/apache/arrow/gandiva/ipc/GandivaTypes.class
+release/
+release/libgandiva_jni.300.0.0.dylib
+release/libgandiva_jni.300.dylib
+release/libgandiva_jni.a
+release/libgandiva_jni.dylib
{noformat}
It would be awesome if, in the next release, the jar contained ALL of those
files, and then I suppose it would work on both Linux and macOS.
> Using Maven Central artifacts as dependencies produce runtime errors
>
>
> Key: ARROW-11135
> URL: https://issues.apache.org/jira/browse/ARROW-11135
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java
>Affects Versions: 2.0.0
>Reporter: Michael Mior
>Priority: Major
>
> I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the
> integration is working well, but I'm having issues . As [suggested on the
> mailing
> list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E],
> using Dremio's public artifacts solves the problem. Between two Apache
> projects however, there would be strong preference to use Apache artifacts as
> a dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[
https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318931#comment-17318931
]
Julian Hyde commented on ARROW-11135:
-
There are no missing packages. But the install instructions should probably say:
* The Gandiva library only works on macOS, and requires that you manually
install protobuf 2.5.
By the way, I compared which files are in the 3.0.0 release jar (which works on
macOS) and the 3.0.0-SNAPSHOT jar (which works on Linux).
{noformat}
$ diff -u <(tar tvf ./arrow-gandiva-3.0.0-SNAPSHOT.jar |awk '{print $NF}'|sort)
<(tar tvf ./arrow-gandiva-3.0.0.jar |awk '{print $NF}'|sort)
--- /dev/fd/63 2021-04-11 12:25:09.0 -0700
+++ /dev/fd/62 2021-04-11 12:25:09.0 -0700
@@ -11,7 +11,10 @@
META-INF/maven/org.apache.arrow.gandiva/arrow-gandiva/pom.xml
Types.proto
git.properties
-libgandiva_jni.so
+libgandiva_jni.300.0.0.dylib
+libgandiva_jni.300.dylib
+libgandiva_jni.a
+libgandiva_jni.dylib
org/
org/apache/
org/apache/arrow/
@@ -188,3 +191,8 @@
org/apache/arrow/gandiva/ipc/GandivaTypes$TreeNode.class
org/apache/arrow/gandiva/ipc/GandivaTypes$TreeNodeOrBuilder.class
org/apache/arrow/gandiva/ipc/GandivaTypes.class
+release/
+release/libgandiva_jni.300.0.0.dylib
+release/libgandiva_jni.300.dylib
+release/libgandiva_jni.a
+release/libgandiva_jni.dylib
{noformat}
It would be awesome if, in the next release, the jar contained ALL of those
files, and then I suppose it would work on both Linux and macOS.
> Using Maven Central artifacts as dependencies produce runtime errors
>
>
> Key: ARROW-11135
> URL: https://issues.apache.org/jira/browse/ARROW-11135
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java
>Affects Versions: 2.0.0
>Reporter: Michael Mior
>Priority: Major
>
> I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the
> integration is working well, but I'm having issues . As [suggested on the
> mailing
> list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E],
> using Dremio's public artifacts solves the problem. Between two Apache
> projects however, there would be strong preference to use Apache artifacts as
> a dependency.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[ https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318605#comment-17318605 ] Julian Hyde commented on ARROW-11135: - If I need to install protobuf 2.5. and the Gandiva jar only works on macOS, it seems that the [one-line installation instruction|https://arrow.apache.org/install/#java-packages] is incomplete. > Using Maven Central artifacts as dependencies produce runtime errors > > > Key: ARROW-11135 > URL: https://issues.apache.org/jira/browse/ARROW-11135 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Affects Versions: 2.0.0 >Reporter: Michael Mior >Priority: Major > > I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the > integration is working well, but I'm having issues . As [suggested on the > mailing > list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E], > using Dremio's public artifacts solves the problem. Between two Apache > projects however, there would be strong preference to use Apache artifacts as > a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ARROW-11135) Using Maven Central artifacts as dependencies produce runtime errors
[ https://issues.apache.org/jira/browse/ARROW-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318599#comment-17318599 ] Julian Hyde commented on ARROW-11135: - An update. I am working on the same branch as [~mmior] (see CALCITE-2040). I have been able to successfully run on Linux using Michael's [commit e9954|https://github.com/julianhyde/calcite/commit/e99547f371f100014e981e5a500dc04887806708]. But I noticed that Apache Arrow Gandiva 3.0.0 was released on 2021/01/25 and the artifacts are on [Maven central|https://search.maven.org/artifact/org.apache.arrow.gandiva/arrow-gandiva/3.0.0/jar], so I have been trying to switch to that version and run on macOS. Some background: In order to merge CALCITE-2040 to Calcite's master branch as a beta feature, we must use artifacts from Maven central, rather than from Dremio's maven repo, or Usra labs' snapshots, or a .jar committed to Calcite's git repo; I don't mind as much whether the artifacts are for Linux or macOS. I am running into errors which I think mean I need to install protobuf2.5 on my Mac. I'll send another update when I make some progress. > Using Maven Central artifacts as dependencies produce runtime errors > > > Key: ARROW-11135 > URL: https://issues.apache.org/jira/browse/ARROW-11135 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Affects Versions: 2.0.0 >Reporter: Michael Mior >Priority: Major > > I'm working on connecting Arrow/Gandiva with Apache Calcite. Overall the > integration is working well, but I'm having issues . As [suggested on the > mailing > list|https://lists.apache.org/thread.html/r93a4fedb499c746917ab8d62cf5a8db8c93a7f24bc9fac81f90bedaa%40%3Cuser.arrow.apache.org%3E], > using Dremio's public artifacts solves the problem. Between two Apache > projects however, there would be strong preference to use Apache artifacts as > a dependency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
