[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk closed the pull request at: https://github.com/apache/spark/pull/9313 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-180700970 Doesn't seem like there is interest in having this added right now - so I'll close. But if this is something we want to add to the API in the future more than happy to re-open and update. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168936205 **[Test build #48745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48745/consoleFull)** for PR 9313 at commit [`bf3e98f`](https://github.com/apache/spark/commit/bf3e98f07097b21066fcd681c437998ce65a1379). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168936339 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168936341 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48745/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168920668 jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168921438 **[Test build #48745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48745/consoleFull)** for PR 9313 at commit [`bf3e98f`](https://github.com/apache/spark/commit/bf3e98f07097b21066fcd681c437998ce65a1379). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168067522 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168067523 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48498/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168067457 **[Test build #48498 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48498/consoleFull)** for PR 9313 at commit [`bf3e98f`](https://github.com/apache/spark/commit/bf3e98f07097b21066fcd681c437998ce65a1379). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168068913 I think the closurecleaner suite failure is unrelated, jenkins retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168070644 **[Test build #48508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48508/consoleFull)** for PR 9313 at commit [`bf3e98f`](https://github.com/apache/spark/commit/bf3e98f07097b21066fcd681c437998ce65a1379). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168054026 **[Test build #48498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48498/consoleFull)** for PR 9313 at commit [`bf3e98f`](https://github.com/apache/spark/commit/bf3e98f07097b21066fcd681c437998ce65a1379). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168088080 **[Test build #48508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48508/consoleFull)** for PR 9313 at commit [`bf3e98f`](https://github.com/apache/spark/commit/bf3e98f07097b21066fcd681c437998ce65a1379). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168088128 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-168088131 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48508/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-161084997 @sryza Any additional thoughts? Should we try pinging some people in the core group? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-154876195 **[Test build #45307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45307/consoleFull)** for PR 9313 at commit [`d9d1375`](https://github.com/apache/spark/commit/d9d13758f4e0b4adaeec07bd2fc9b76a9cbbff0c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-154876220 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-154859316 **[Test build #45307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45307/consoleFull)** for PR 9313 at commit [`d9d1375`](https://github.com/apache/spark/commit/d9d13758f4e0b4adaeec07bd2fc9b76a9cbbff0c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-154858802 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-154858796 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/9313#discussion_r43909799
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1611,8 +1611,14 @@ class SparkContext(config: SparkConf) extends
Logging with ExecutorAllocationCli
* Adds a JAR dependency for all tasks to be executed on this
SparkContext in the future.
* The `path` passed can be either a local file, a file in HDFS (or
other Hadoop-supported
* filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on
every worker node.
+ * If addToCurrentThread is true, attempt to add the new class to the
current threads class
+ * loader.
*/
def addJar(path: String) {
+addJar(path, false)
+ }
+
+ def addJar(path: String, addToCurrentThread: Boolean) {
--- End diff --
In that case, maybe good to call this out explicitly in the doc? Also, the
naming `addToCurrentThread` makes it seem like it'll get added *only* to the
current thread, so maybe something like `addToContextClassLoader` would be more
clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/9313#discussion_r43726053
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1611,8 +1611,14 @@ class SparkContext(config: SparkConf) extends
Logging with ExecutorAllocationCli
* Adds a JAR dependency for all tasks to be executed on this
SparkContext in the future.
* The `path` passed can be either a local file, a file in HDFS (or
other Hadoop-supported
* filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on
every worker node.
+ * If addToCurrentThread is true, attempt to add the new class to the
current threads class
+ * loader.
*/
def addJar(path: String) {
+addJar(path, false)
+ }
+
+ def addJar(path: String, addToCurrentThread: Boolean) {
--- End diff --
Would it be correct to say that in nearly all cases, setting the second
argument to true will result in the jar being added to all of the application's
threads? Because Spark sets the context classloader to a MutableClassLoader
before loading the application's main class, and then all other app threads
will inherit this as the default unless they explicitly change the context
class loader?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/9313#discussion_r43726148
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1667,6 +1673,13 @@ class SparkContext(config: SparkConf) extends
Logging with ExecutorAllocationCli
if (key != null) {
addedJars(key) = System.currentTimeMillis
logInfo("Added JAR " + path + " at " + key + " with timestamp " +
addedJars(key))
+if (addToCurrentThread) {
+ val currentCL = Utils.getContextOrSparkClassLoader
+ currentCL match {
+case cl: MutableURLClassLoader => cl.addURL(new
URI(key).toURL())
+case _ => logWarning(s"Unsupported cl $currentCL will not
update jars for current thread")
--- End diff --
My opinion is that it's better to throw an exception here. Though as
that's an API behavior question, probably best to defer to a core maintainer?
@pwendell any thoughts? And thoughts on adding this option to `addJar` in
general?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/9313#discussion_r43788466
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1611,8 +1611,14 @@ class SparkContext(config: SparkConf) extends
Logging with ExecutorAllocationCli
* Adds a JAR dependency for all tasks to be executed on this
SparkContext in the future.
* The `path` passed can be either a local file, a file in HDFS (or
other Hadoop-supported
* filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on
every worker node.
+ * If addToCurrentThread is true, attempt to add the new class to the
current threads class
+ * loader.
*/
def addJar(path: String) {
+addJar(path, false)
+ }
+
+ def addJar(path: String, addToCurrentThread: Boolean) {
--- End diff --
I think that is correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153505891 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153505715 **[Test build #44952 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44952/consoleFull)** for PR 9313 at commit [`2467ce5`](https://github.com/apache/spark/commit/2467ce59b7a2cf7af71ef896752e88c148116c8e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153505892 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44952/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153474005 **[Test build #44952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44952/consoleFull)** for PR 9313 at commit [`2467ce5`](https://github.com/apache/spark/commit/2467ce59b7a2cf7af71ef896752e88c148116c8e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153473511 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153473535 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153131739 @sryza so I've added it to the scala API - what are your thoughts on this approach? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/9313#discussion_r43682628
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -175,6 +175,29 @@ private[spark] object Utils extends Logging {
}
/**
+ * Update the current threads class loader.
+ * Requires the current class loader is a MutableURLClassLoader,
otherwise skips updating with a
+ * warning. Intended for use by addJar(), although constructing an
instance of the class will
+ * still require:
+ *
sc._jvm.java.lang.Thread.currentThread().getContextClassLoader().loadClass("class
name")
+ * as described in SPARK-5185.
+ */
+ def updatePrimaryClassLoader(sc: SparkContext) {
--- End diff --
What's the advantage of this update method that compares the full contents
of `sc.addedJars` to those already registered with the current classloader, vs.
just calling `cs.addUrl` directly on the jar being added?
A possibly confusing effect of this approach is that if I have code like:
sc.addJar(jar1)
... other stuff blah blah blah ...
sc.addJar(jar2, true)
jar1 will get added to the classpath only when `sc.addJar` is called the
second time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/9313#discussion_r43683655
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -175,6 +175,29 @@ private[spark] object Utils extends Logging {
}
/**
+ * Update the current threads class loader.
+ * Requires the current class loader is a MutableURLClassLoader,
otherwise skips updating with a
+ * warning. Intended for use by addJar(), although constructing an
instance of the class will
+ * still require:
+ *
sc._jvm.java.lang.Thread.currentThread().getContextClassLoader().loadClass("class
name")
+ * as described in SPARK-5185.
+ */
+ def updatePrimaryClassLoader(sc: SparkContext) {
--- End diff --
Ah thats a good point, it made a bit more sense when this was being done
only for Python and there wasn't a flag. I'll switch it around.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153188705 **[Test build #44848 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44848/consoleFull)** for PR 9313 at commit [`72b9f36`](https://github.com/apache/spark/commit/72b9f36aa5c195cc3071aa040542527c863dd1a6). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153188707 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153187887 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153187919 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153188708 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44848/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9313#issuecomment-153188257 **[Test build #44848 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44848/consoleFull)** for PR 9313 at commit [`72b9f36`](https://github.com/apache/spark/commit/72b9f36aa5c195cc3071aa040542527c863dd1a6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
