[GitHub] spark pull request #14720: SPARK-12868: Allow Add jar to add jars from hdfs/...
Github user Parth-Brahmbhatt closed the pull request at: https://github.com/apache/spark/pull/14720 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14720: SPARK-12868: Allow Add jar to add jars from hdfs/...
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/14720#discussion_r87908473
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
---
@@ -856,6 +856,17 @@ class HiveQuerySuite extends HiveComparisonTest with
BeforeAndAfter {
sql("DROP TABLE alter1")
}
+ test("SPARK-12868 ADD JAR FROM HDFS") {
+val testJar = "hdfs://nn:8020/foo.jar"
+// This should fail with unknown host, as its just testing the URL
parsing
+// before SPARK-12868 it was failing with Malformed URI
+val e = intercept[RuntimeException] {
--- End diff --
I think this test should be improved before merging this. Looking for a
RuntimeException to validate that the Jar was registered is brittle and can
easily pass when the registration doesn't actually work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14720: SPARK-12868: Allow Add jar to add jars from hdfs/...
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/spark/pull/14720#discussion_r76334577
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
---
@@ -865,6 +865,16 @@ class HiveQuerySuite extends HiveComparisonTest with
BeforeAndAfter {
sql("DROP TABLE alter1")
}
+ test("SPARK-12868 ADD JAR FROM HDFS") {
+val testJar = "hdfs://nn:8020/foo.jar"
+// This should fail with unknown host, as its just testing the URL
parsing
+// before SPARK-12868 it was failing with Malformed URI
+val e = intercept[RuntimeException] {
+ sql(s"ADD JAR $testJar")
+}
+assert(e.getMessage.contains("java.net.UnknownHostException: nn1"))
--- End diff --
Good catch, unintentional typo. fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14720: SPARK-12868: Allow Add jar to add jars from hdfs/...
Github user Parth-Brahmbhatt commented on a diff in the pull request:
https://github.com/apache/spark/pull/14720#discussion_r76334535
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -87,6 +88,9 @@ private[hive] class HiveClientImpl(
// Circular buffer to hold what hive prints to STDOUT and ERR. Only
printed when failures occur.
private val outputBuffer = new CircularBuffer()
+ // An object lock to ensure URL factory is registered exactly once.
+ object URLFactoryRegistrationLock{}
--- End diff --
Removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14720: SPARK-12868: Allow Add jar to add jars from hdfs/...
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/14720#discussion_r75601795
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
---
@@ -865,6 +865,16 @@ class HiveQuerySuite extends HiveComparisonTest with
BeforeAndAfter {
sql("DROP TABLE alter1")
}
+ test("SPARK-12868 ADD JAR FROM HDFS") {
+val testJar = "hdfs://nn:8020/foo.jar"
+// This should fail with unknown host, as its just testing the URL
parsing
+// before SPARK-12868 it was failing with Malformed URI
+val e = intercept[RuntimeException] {
+ sql(s"ADD JAR $testJar")
+}
+assert(e.getMessage.contains("java.net.UnknownHostException: nn1"))
--- End diff --
You sure it's `nn1` (with `1`)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14720: SPARK-12868: Allow Add jar to add jars from hdfs/...
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/14720#discussion_r75601779
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -87,6 +88,9 @@ private[hive] class HiveClientImpl(
// Circular buffer to hold what hive prints to STDOUT and ERR. Only
printed when failures occur.
private val outputBuffer = new CircularBuffer()
+ // An object lock to ensure URL factory is registered exactly once.
+ object URLFactoryRegistrationLock{}
--- End diff --
Why do you need `{}`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14720: SPARK-12868: Allow Add jar to add jars from hdfs/...
GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/spark/pull/14720 SPARK-12868: Allow Add jar to add jars from hdfs/s3n urls. ## What changes were proposed in this pull request? Add jar command fails when given s3n or hdfs urls. This changes fixes that. ## How was this patch tested? unit test added, it was just copied from a previously opened PR https://github.com/apache/spark/pull/10797 so credit should be given the person who originally posted the patch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Parth-Brahmbhatt/spark SPARK-12868 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14720.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14720 commit 1a8a8bc208b914c524bba34a010d32a97b605009 Author: Parth Brahmbhatt Date: 2016-08-19T17:38:31Z SPARK-12868: Allow Add jar to add jars from hdfs/s3n urls. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
