[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557675#comment-14557675 ] Patrick Wendell commented on SPARK-6907: Hey [~ste...@apache.org] - my guess is that the way most packagers will use this is that they will just point to their existing Hive jars that are present using the relevant configs. As Michael said, the ivy downloading is convenient, but it's not the only mechanism. Create an isolated classloader for the Hive Client. --- Key: SPARK-6907 URL: https://issues.apache.org/jira/browse/SPARK-6907 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Michael Armbrust Assignee: Michael Armbrust Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550808#comment-14550808 ] Steve Loughran commented on SPARK-6907: --- Having just looked at this code, I'm a bit worried about the implications of the dynamic JAR download. # what happens if the first attempt to use the client takes place while the caller is off the internet (i.e. an isolated cluster)? # when Ivy pulls down JARs over HTTP, it checks the MD5 sums from the same server. It's not secure, merely verifies that the SHA1 and JAR is tampered with consistently. Maybe I've misunderstood something —if I haven't this strikes me as insecure Create an isolated classloader for the Hive Client. --- Key: SPARK-6907 URL: https://issues.apache.org/jira/browse/SPARK-6907 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Michael Armbrust Assignee: Michael Armbrust Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550869#comment-14550869 ] Michael Armbrust commented on SPARK-6907: - Steve, thanks for the feedback. The ivy integration is really only for easy testing / adding new versions. As documented [here|https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala#L115], by default it uses the same jars that are bundled with Spark. When you want to connect to another version of Hive, you can specify a classpath manually. [~pwendell] and I have also talked about building assembly jars that people could use for specific version. Any other thoughts you have about packaging would be great to hear. Create an isolated classloader for the Hive Client. --- Key: SPARK-6907 URL: https://issues.apache.org/jira/browse/SPARK-6907 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Michael Armbrust Assignee: Michael Armbrust Fix For: 1.4.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524161#comment-14524161 ] Apache Spark commented on SPARK-6907: - User 'marmbrus' has created a pull request for this issue: https://github.com/apache/spark/pull/5851 Create an isolated classloader for the Hive Client. --- Key: SPARK-6907 URL: https://issues.apache.org/jira/browse/SPARK-6907 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Michael Armbrust Assignee: Michael Armbrust -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522514#comment-14522514 ] Michael Armbrust commented on SPARK-6907: - I think the cost of maintaining/versioning/publishing a separate repo here is probably too high. At least for the initial version. Create an isolated classloader for the Hive Client. --- Key: SPARK-6907 URL: https://issues.apache.org/jira/browse/SPARK-6907 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Michael Armbrust Assignee: Michael Armbrust -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522573#comment-14522573 ] Punya Biswal commented on SPARK-6907: - Makes sense, thanks for clarifying. I guess a weaker version of my question is, can we write this in Java (rather than Scala) to set it up for future separation? Create an isolated classloader for the Hive Client. --- Key: SPARK-6907 URL: https://issues.apache.org/jira/browse/SPARK-6907 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Michael Armbrust Assignee: Michael Armbrust -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6907) Create an isolated classloader for the Hive Client.
[ https://issues.apache.org/jira/browse/SPARK-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14515066#comment-14515066 ] Punya Biswal commented on SPARK-6907: - Would it make sense to do this as a separate project (repository)? It seems like a generic problem that's applicable more broadly than just Spark. Create an isolated classloader for the Hive Client. --- Key: SPARK-6907 URL: https://issues.apache.org/jira/browse/SPARK-6907 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Michael Armbrust -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
