[GitHub] spark pull request: [Build] Uploads HiveCompatibilitySuite logs
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2993#issuecomment-61359866 @pwendell Would you mind to help to review this? Thanks to Michael, these logs contain error logs, query plan details as well as reproduction steps for all failed Hive compatibility suites. Uploading these Hive compatibility suite logs can be very helpful for debugging Spark SQL Hive code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2189][SQL] Adds dropTempTable API
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3039#issuecomment-61359778 [Test build #22686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22686/consoleFull) for PR 3039 at commit [`54ae99f`](https://github.com/apache/spark/commit/54ae99f16087c6a23f6fd57cee27a1e604727b53). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2868#issuecomment-61359747 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22684/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2868#issuecomment-61359743 [Test build #22684 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22684/consoleFull) for PR 2868 at commit [`5f5a156`](https://github.com/apache/spark/commit/5f5a1564af1a8a1cbf6d257941ad969169295fe7). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4017] show progress bar in console and ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3029#issuecomment-61359713 @JoshRosen I had make some improvements: 1) finished the bar before print the result in scala shell 2) can interwave with logging better(will not overwrite each other) 3) will not show progress in jenkins (using console instead of stderr --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3001#issuecomment-61359628 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22683/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3001#issuecomment-61359626 [Test build #22683 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22683/consoleFull) for PR 3001 at commit [`9883918`](https://github.com/apache/spark/commit/9883918bda8c570f9bf8241dc3910ae4f3b1020e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4017] show progress bar in console and ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3029#issuecomment-61359587 [Test build #22685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22685/consoleFull) for PR 3029 at commit [`bc53d99`](https://github.com/apache/spark/commit/bc53d99d518d6fafd607c617d0915c7a2f9eee85). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3001#issuecomment-61359390 [Test build #22682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22682/consoleFull) for PR 3001 at commit [`3d62679`](https://github.com/apache/spark/commit/3d62679bc7bf74233d04fb7e134c4bb21b4bbec6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3001#issuecomment-61359392 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22682/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4151][SQL] Add string operation functio...
Github user OopsOutOfMemory commented on the pull request: https://github.com/apache/spark/pull/2998#issuecomment-61359340 @marmbrus test passed, this can be merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4037][SQL] Removes the SessionState ins...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2887#discussion_r19701845
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -288,6 +296,14 @@ class HiveContext(sc: SparkContext) extends
SQLContext(sc) {
val cmd_1: String = cmd_trimmed.substring(tokens(0).length()).trim()
val proc: CommandProcessor =
HiveShim.getCommandProcessor(Array(tokens(0)), hiveconf)
+ // Makes sure the session represented by the `sessionState` field is
activated. This implies
+ // Spark SQL Hive support uses a single `SessionState` for all Hive
operations and breaks
+ // session isolation under multi-user scenarios (i.e.
HiveThriftServer2).
+ // TODO Fix session isolation
+ if (SessionState.get() != sessionState) {
--- End diff --
For example, Spark SQL CLI uses a global `CliSessionState` instance, which
inherits from `SessionState`. Also, this can be useful to fix the session
isolation problem `HiveThriftServer2` currently suffers from.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...
Github user manishamde commented on the pull request: https://github.com/apache/spark/pull/2868#issuecomment-61359008 @codeleft I agree that local training should be a high priority. Just curious -- what's the depth of the tree in the failing case? I vote for merging this PR since there is no loss in performance for shallow trees and gain in performance for deep trees. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4166][Core][WebUI] Display the executor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3033#issuecomment-61358991 [Test build #22680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22680/consoleFull) for PR 3033 at commit [`ff4664c`](https://github.com/apache/spark/commit/ff4664ceb6b9e49fa282cfe4730967bc524921b0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ExecutorLostFailure(execId: String) extends TaskFailedReason ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4166][Core][WebUI] Display the executor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3033#issuecomment-61358992 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22680/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3968 Use parquet-mr filter2 api in spark...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2841#issuecomment-61358979 Thanks, closed it and assigned it to you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3032#issuecomment-61358932 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22681/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3032#issuecomment-61358931 [Test build #22681 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22681/consoleFull) for PR 3032 at commit [`0c07d1f`](https://github.com/apache/spark/commit/0c07d1fa8f472b88b65653b3471b5afba3ee3546). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` trait FetchResult ` * ` case class SuccessFetchResult(blockId: BlockId, size: Long, buf: ManagedBuffer)` * ` case class FailureFetchResult(blockId: BlockId, e: Throwable) extends FetchResult ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2942 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4152] [SQL] Avoid data change in CTAS w...
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/3013#discussion_r19701825
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -320,8 +326,9 @@ object HiveMetastoreTypes extends RegexParsers {
"double" ^^^ DoubleType |
"bigint" ^^^ LongType |
"binary" ^^^ BinaryType |
-"boolean" ^^^ BooleanType |
-HiveShim.metastoreDecimal ^^^ DecimalType |
+"boolean" ^^^ BooleanType | // TODO decimal Hive 0.12.0
+"decimal\\((\\d+),(\\d+)\\)".r ^^^ DecimalType | // TODO decimal Hive
0.13.1
--- End diff --
we need these todos here? will this both ok for hive 12 and 13, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61358857 LGTM. Merged into master. Thanks for adding streaming k-means! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4168][WebUI] web statges number should ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3035#issuecomment-61358810 [Test build #22679 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22679/consoleFull) for PR 3035 at commit [`4ea8fd1`](https://github.com/apache/spark/commit/4ea8fd1238a6e8116ce369f3fbdac9539cf320de). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4168][WebUI] web statges number should ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3035#issuecomment-61358812 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22679/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...
Github user codedeft commented on the pull request: https://github.com/apache/spark/pull/2868#issuecomment-61358798 @mengxr @jkbradley Can you merge this? This is the only way you can effectively train 10 large trees with the mnist8m dataset. With node Id cache, it took a very long time, but we were able to finish training 10 trees on mnist8m in 15 hours with 20 executors. SF with local training can finish this in 20 minutes, so local training would be a must in the next release. However, without node Id cache, it looks like it's not even possible. It's currently only 60% of the way there and it's already taken 13 hours and dozens of fetch failures. I feel that it might eventually just fail because the models are just too big to pass around. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2868#issuecomment-61358651 [Test build #22684 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22684/consoleFull) for PR 2868 at commit [`5f5a156`](https://github.com/apache/spark/commit/5f5a1564af1a8a1cbf6d257941ad969169295fe7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3001#issuecomment-61358524 [Test build #22683 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22683/consoleFull) for PR 3001 at commit [`9883918`](https://github.com/apache/spark/commit/9883918bda8c570f9bf8241dc3910ae4f3b1020e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3886] [PySpark] simplify serializer, us...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2920#issuecomment-61358467 [Test build #502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/502/consoleFull) for PR 2920 at commit [`1d557fc`](https://github.com/apache/spark/commit/1d557fc24b761adee0b32c7df587c58d6980cf86). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4152] [SQL] Avoid data change in CTAS w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3013#issuecomment-61358398 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22678/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701722
--- Diff: network/shuffle/pom.xml ---
@@ -0,0 +1,102 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+org.apache.spark
+spark-parent
+1.2.0-SNAPSHOT
+../../pom.xml
+
+
+ org.apache.spark
+ spark-network-shuffle_2.10
+ jar
+ Spark Project Shuffle Streaming Service Code
+ http://spark.apache.org/
+
+network-shuffle
+
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+
+
+ org.slf4j
+ slf4j-api
+
+
+
+
+ com.google.guava
+ guava
+ provided
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+ test-jar
+ test
+
+
+ junit
+ junit
+ test
+
+
+ com.novocode
+ junit-interface
+ test
+
+
+ log4j
+ log4j
+ test
+
+
+ org.mockito
+ mockito-all
+ test
+
+
+ org.scalatest
+ scalatest_${scala.binary.version}
+ test
+
+
+
+
+
target/scala-${scala.binary.version}/classes
+
target/scala-${scala.binary.version}/test-classes
+
+
+org.scalatest
+scalatest-maven-plugin
--- End diff --
It does seem to work without, I've removed this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701725
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -151,6 +154,36 @@ public void operationComplete(ChannelFuture future)
throws Exception {
});
}
+ /**
+ * Synchronously sends an opaque message to the RpcHandler on the
server-side, waiting for up to
+ * a specified timeout for a response.
+ */
+ public byte[] sendRpcSync(byte[] message, long timeoutMs) {
+final SettableFuture result = SettableFuture.create();
+
+sendRpc(message, new RpcResponseCallback() {
+ @Override
+ public void onSuccess(byte[] response) {
+result.set(response);
+ }
+
+ @Override
+ public void onFailure(Throwable e) {
+result.setException(e);
+ }
+});
+
+try {
+ return result.get(timeoutMs, TimeUnit.MILLISECONDS);
+} catch (InterruptedException e) {
+ throw new RuntimeException(e);
--- End diff --
Neat!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4152] [SQL] Avoid data change in CTAS w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3013#issuecomment-61358397 [Test build #22678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22678/consoleFull) for PR 3013 at commit [`4085c67`](https://github.com/apache/spark/commit/4085c67fa3ecdbb81ac1a70c0ed3b7662d2c9741). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701715
--- Diff: network/shuffle/pom.xml ---
@@ -0,0 +1,102 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+org.apache.spark
+spark-parent
+1.2.0-SNAPSHOT
+../../pom.xml
+
+
+ org.apache.spark
+ spark-network-shuffle_2.10
+ jar
+ Spark Project Shuffle Streaming Service Code
+ http://spark.apache.org/
+
+network-shuffle
+
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+
+
+ org.slf4j
+ slf4j-api
+
+
+
+
+ com.google.guava
+ guava
+ provided
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+ test-jar
+ test
+
+
+ junit
+ junit
+ test
+
+
+ com.novocode
+ junit-interface
+ test
+
+
+ log4j
+ log4j
+ test
+
+
+ org.mockito
+ mockito-all
+ test
+
+
+ org.scalatest
+ scalatest_${scala.binary.version}
--- End diff --
This is necessary because we run all of our tests through scalatest,
including he junit ones. This is probably not ideal, as it means, for instance,
that maven cannot run our java unit tests, but it is the current state of
affairs in Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61358352 [Test build #22677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22677/consoleFull) for PR 2942 at commit [`b2e5b4a`](https://github.com/apache/spark/commit/b2e5b4a167e0e5835f3518d2b68e4063c3f9c955). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class StreamingKMeansModel(` * `class StreamingKMeans(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61358356 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22677/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701712
--- Diff: network/shuffle/pom.xml ---
@@ -0,0 +1,102 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+org.apache.spark
+spark-parent
+1.2.0-SNAPSHOT
+../../pom.xml
+
+
+ org.apache.spark
+ spark-network-shuffle_2.10
+ jar
+ Spark Project Shuffle Streaming Service Code
+ http://spark.apache.org/
+
+network-shuffle
+
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+
+
+ org.slf4j
+ slf4j-api
+
+
+
+
+ com.google.guava
+ guava
+ provided
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
--- End diff --
I'm not sure we expect network/common to be imported in modules other than
core and this network/shuffle. With the suggested replacement of
`${project.version}`, I think the version management is simple enough. I looked
in spark-parent's pom.xml, and we do not depend on any other submodules of
spark in this fashion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/3001#discussion_r19701698 --- Diff: network/shuffle/pom.xml --- @@ -0,0 +1,102 @@ + + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + 4.0.0 + +org.apache.spark +spark-parent +1.2.0-SNAPSHOT +../../pom.xml + + + org.apache.spark + spark-network-shuffle_2.10 --- End diff -- Yes, the module is scala-free. However, according to @pwendell, some of our code (such as prepending classes for testing) looks for the 2.10 suffix, and so he recommended having it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3161][MLLIB] Adding a node Id caching m...
Github user codedeft commented on the pull request: https://github.com/apache/spark/pull/2868#issuecomment-61358267 The conflict is caused by the GBoosting check-in. I'm taking a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3001#issuecomment-61358252 [Test build #22682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22682/consoleFull) for PR 3001 at commit [`3d62679`](https://github.com/apache/spark/commit/3d62679bc7bf74233d04fb7e134c4bb21b4bbec6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3247][SQL] An API for adding data sourc...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2475#issuecomment-61358186 The new API for sources looks good to me, thanks for making the changes. It will be easy to plug in a lot of neat data sources here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3034#issuecomment-61358163 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22676/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3034#issuecomment-61358162 [Test build #22676 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22676/consoleFull) for PR 3034 at commit [`42b2ed3`](https://github.com/apache/spark/commit/42b2ed33583f8278f1aa2df98b17fd38ade76957). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3247][SQL] An API for adding data sourc...
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2475#discussion_r19701683 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/filters.scala --- @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.sources + +abstract sealed class Filter --- End diff -- If you make it sealed, will this lead to compatibility issues across Spark versions (if someone has code compiled when there was only one subclass of this, but they link to some version of Spark with 2 of them)? At one point I thought sealed classes got their own numeric ID to avoid isInstanceOf checks, but I'm not sure that really happens. @heathermiller do you know how this works? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4163][Core][WebUI] Send the fetch failu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3032#issuecomment-61358019 [Test build #22681 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22681/consoleFull) for PR 3032 at commit [`0c07d1f`](https://github.com/apache/spark/commit/0c07d1fa8f472b88b65653b3471b5afba3ee3546). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4166][Core][WebUI] Display the executor...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3033#issuecomment-61357949 I updated to use `Utils.jsonOption` to support backward compatible. Also update #3032 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4017] show progress bar in console and ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3029#issuecomment-61357827 [Test build #503 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/503/consoleFull) for PR 3029 at commit [`e6bb189`](https://github.com/apache/spark/commit/e6bb1895c51bef6201296601725821d12d4deb8e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4166][Core][WebUI] Display the executor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3033#issuecomment-61357750 [Test build #22680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22680/consoleFull) for PR 3033 at commit [`ff4664c`](https://github.com/apache/spark/commit/ff4664ceb6b9e49fa282cfe4730967bc524921b0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3046#issuecomment-61357581 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22675/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4168][WebUI] web statges number should ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3035#issuecomment-61357578 [Test build #22679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22679/consoleFull) for PR 3035 at commit [`4ea8fd1`](https://github.com/apache/spark/commit/4ea8fd1238a6e8116ce369f3fbdac9539cf320de). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3046#issuecomment-61357580 [Test build #22675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22675/consoleFull) for PR 3046 at commit [`f1c90bb`](https://github.com/apache/spark/commit/f1c90bb2957f604522ee1d03f8be3d770c0d9e32). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4152] [SQL] Avoid data change in CTAS w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3013#issuecomment-61357494 [Test build #22678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22678/consoleFull) for PR 3013 at commit [`4085c67`](https://github.com/apache/spark/commit/4085c67fa3ecdbb81ac1a70c0ed3b7662d2c9741). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4168][WebUI] web statges number should ...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3035#discussion_r19701557
--- Diff:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
@@ -59,6 +59,11 @@ class JobProgressListener(conf: SparkConf) extends
SparkListener with Logging {
val failedStages = ListBuffer[StageInfo]()
val stageIdToData = new HashMap[(StageId, StageAttemptId), StageUIData]
val stageIdToInfo = new HashMap[StageId, StageInfo]
+
+ // Number of completed stages, not equal to completedStages.size
+ var completedStagesNum = 0
+ // number of failed stages, not equal to failedStagesNum.size
--- End diff --
Hi @andrewor14 , if stages are too large, may consume too much memory. So
there is a mechanism to limit the buffer maintaining the `completedStages` and
`failedStages`. And if the stage size is large than `retainedStages`, the
starting part of the completed stages will be removed. See the code here
([trimIfNecessary](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala#L121)).
So if the stage number is too large, `completedStages.size` is not the
actually the real completedStages. Currently, the threshold is 1000. So the
completed stages or failed stages showed on the WebUI will never higher than
1000, it will chages between 900~1000 if the stage number is more than 1000.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61356950 [Test build #22677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22677/consoleFull) for PR 2942 at commit [`b2e5b4a`](https://github.com/apache/spark/commit/b2e5b4a167e0e5835f3518d2b68e4063c3f9c955). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4017] show progress bar in console and ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3029#issuecomment-61356828 [Test build #503 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/503/consoleFull) for PR 3029 at commit [`e6bb189`](https://github.com/apache/spark/commit/e6bb1895c51bef6201296601725821d12d4deb8e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3034#issuecomment-61356809 [Test build #22676 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22676/consoleFull) for PR 3034 at commit [`42b2ed3`](https://github.com/apache/spark/commit/42b2ed33583f8278f1aa2df98b17fd38ade76957). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3886] [PySpark] simplify serializer, us...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2920#issuecomment-61356801 [Test build #502 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/502/consoleFull) for PR 2920 at commit [`1d557fc`](https://github.com/apache/spark/commit/1d557fc24b761adee0b32c7df587c58d6980cf86). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3034#issuecomment-61356792 It failed because of a Yarn compilation error that is now fixed. Let's try again. Jenkin, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user freeman-lab commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61356758 @mengxr great updates! LGMT. Just need to update the doc/examples in a couple places I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61356547 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22673/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61356545 [Test build #22673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22673/consoleFull) for PR 2942 at commit [`078617c`](https://github.com/apache/spark/commit/078617c167b5b2b698193cddfd567d60637ba906). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class StreamingKMeansModel(` * `class StreamingKMeans(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Sink2 and channel2 should be closed in Flume
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/3037#discussion_r19701371
--- Diff:
external/flume/src/main/scala/org/apache/spark/streaming/flume/FlumeUtils.scala
---
@@ -184,7 +184,7 @@ object FlumeUtils {
hostname: String,
port: Int
): JavaReceiverInputDStream[SparkFlumeEvent] = {
-createPollingStream(jssc, hostname, port,
StorageLevel.MEMORY_AND_DISK_SER_2)
+createPollingStream(jssc.ssc, hostname, port,
StorageLevel.MEMORY_AND_DISK_SER_2)
--- End diff --
I am talking about the change from using `jssc` to `jssc.ssc`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-61356068 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22671/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-61356064 [Test build #22671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22671/consoleFull) for PR 2701 at commit [`dedda56`](https://github.com/apache/spark/commit/dedda56fce0f50f7d7b4f2579e279306833d6c92). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AttributeReference(` * `case class StructField(` * `class MetadataBuilder ` * `class Metadata extends org.apache.spark.sql.catalyst.util.Metadata ` * `public class MetadataBuilder extends org.apache.spark.sql.catalyst.util.MetadataBuilder ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3034#issuecomment-61356048 @andrewor14 Is there any way to test a patch from my local machine? And check the test result, I don't know a specific problem about testing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3046#issuecomment-61356053 [Test build #22675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22675/consoleFull) for PR 3046 at commit [`f1c90bb`](https://github.com/apache/spark/commit/f1c90bb2957f604522ee1d03f8be3d770c0d9e32). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user sarutak closed the pull request at: https://github.com/apache/spark/pull/2647 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2647#issuecomment-61355965 Jenkins wouldn't pick this PR up so I opened another PR #3046 for this issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/3046 [SPARK-3787] Assembly jar name is wrong when we build with sbt omitting -Dhadoop.version This PR is another solution for When we build with sbt with profile for hadoop and without property for hadoop version like: sbt/sbt -Phadoop-2.2 assembly jar name is always used default version (1.0.4). When we build with maven with same condition for sbt, default version for each profile is used. For instance, if we build like: mvn -Phadoop-2.2 package jar name is used hadoop2.2.0 as a default version of hadoop-2.2. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sarutak/spark fix-assembly-jarname-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3046.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3046 commit eebbb7d03423efefcf1cdfb4bab8cbf7348f08a7 Author: Kousuke Saruta Date: 2014-10-03T22:13:47Z Fixed wrong jar name commit 5fc12597afe5964c7b9f688fd2919426b928b3ec Author: Kousuke Saruta Date: 2014-10-04T05:00:07Z Fixed typo. commit b2318eb227d59dbd61d2dd8a24592cdc2f64ac2b Author: Kousuke Saruta Date: 2014-10-06T14:16:50Z Merge branch 'master' of git://git.apache.org/spark into fix-assembly-jarname commit ad1f96ea36f7a4750d6fdaf3ab91239a20a7e6a1 Author: Kousuke Saruta Date: 2014-10-07T15:56:04Z Merge branch 'master' of git://git.apache.org/spark into fix-assembly-jarname commit c81806bda4744382d2657441404cbb1206c3aa8a Author: Kousuke Saruta Date: 2014-10-11T03:24:14Z Merge branch 'master' of git://git.apache.org/spark into fix-assembly-jarname commit af6b1005accd6310007992e8d96103b9a50f7de8 Author: Kousuke Saruta Date: 2014-10-31T21:23:53Z Merge branch 'master' of git://git.apache.org/spark into fix-assembly-jarname commit f1c90bb2957f604522ee1d03f8be3d770c0d9e32 Author: Kousuke Saruta Date: 2014-11-01T00:14:47Z Fixed SparkBuild.scala in order to read `hadoop.version` property from pom.xml --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4121] Set commons-math3 version based o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3023#issuecomment-61355850 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22672/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4121] Set commons-math3 version based o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3023#issuecomment-61355847 [Test build #22672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22672/consoleFull) for PR 3023 at commit [`580f6d9`](https://github.com/apache/spark/commit/580f6d98ec5839c657935bef005df23eddec3934). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701228
--- Diff: network/shuffle/pom.xml ---
@@ -0,0 +1,102 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+org.apache.spark
+spark-parent
+1.2.0-SNAPSHOT
+../../pom.xml
+
+
+ org.apache.spark
+ spark-network-shuffle_2.10
+ jar
+ Spark Project Shuffle Streaming Service Code
+ http://spark.apache.org/
+
+network-shuffle
+
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+
+
+ org.slf4j
+ slf4j-api
+
+
+
+
+ com.google.guava
+ guava
+ provided
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+ test-jar
+ test
+
+
+ junit
+ junit
+ test
+
+
+ com.novocode
+ junit-interface
+ test
+
+
+ log4j
+ log4j
+ test
+
+
+ org.mockito
+ mockito-all
+ test
+
+
+ org.scalatest
+ scalatest_${scala.binary.version}
+ test
+
+
+
+
+
target/scala-${scala.binary.version}/classes
+
target/scala-${scala.binary.version}/test-classes
+
+
+org.scalatest
+scalatest-maven-plugin
--- End diff --
Isn't this automatically inherited from the parent pom?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701221
--- Diff: network/shuffle/pom.xml ---
@@ -0,0 +1,102 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+org.apache.spark
+spark-parent
+1.2.0-SNAPSHOT
+../../pom.xml
+
+
+ org.apache.spark
+ spark-network-shuffle_2.10
+ jar
+ Spark Project Shuffle Streaming Service Code
+ http://spark.apache.org/
+
+network-shuffle
+
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+
+
+ org.slf4j
+ slf4j-api
+
+
+
+
+ com.google.guava
+ guava
+ provided
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+ test-jar
+ test
+
+
+ junit
+ junit
+ test
+
+
+ com.novocode
+ junit-interface
+ test
+
+
+ log4j
+ log4j
+ test
+
+
+ org.mockito
+ mockito-all
+ test
+
+
+ org.scalatest
+ scalatest_${scala.binary.version}
--- End diff --
Are you actually using scalatest in your code? If not this is probably
unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701209
--- Diff: network/shuffle/pom.xml ---
@@ -0,0 +1,102 @@
+
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+org.apache.spark
+spark-parent
+1.2.0-SNAPSHOT
+../../pom.xml
+
+
+ org.apache.spark
+ spark-network-shuffle_2.10
+ jar
+ Spark Project Shuffle Streaming Service Code
+ http://spark.apache.org/
+
+network-shuffle
+
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
+
+
+ org.slf4j
+ slf4j-api
+
+
+
+
+ com.google.guava
+ guava
+ provided
+
+
+
+
+ org.apache.spark
+ spark-network-common_2.10
+ 1.2.0-SNAPSHOT
--- End diff --
You can use `${project.version}` here (one less place to have to bump the
version on every release).
BTW, if this dependency is used in multiple places I'd suggest adding it +
version to the root pom's `dependencyManagement` section, and adding it here
(and in other modules that use it) without the version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3001#discussion_r19701192
--- Diff:
network/common/src/main/java/org/apache/spark/network/client/TransportClient.java
---
@@ -151,6 +154,36 @@ public void operationComplete(ChannelFuture future)
throws Exception {
});
}
+ /**
+ * Synchronously sends an opaque message to the RpcHandler on the
server-side, waiting for up to
+ * a specified timeout for a response.
+ */
+ public byte[] sendRpcSync(byte[] message, long timeoutMs) {
+final SettableFuture result = SettableFuture.create();
+
+sendRpc(message, new RpcResponseCallback() {
+ @Override
+ public void onSuccess(byte[] response) {
+result.set(response);
+ }
+
+ @Override
+ public void onFailure(Throwable e) {
+result.setException(e);
+ }
+});
+
+try {
+ return result.get(timeoutMs, TimeUnit.MILLISECONDS);
+} catch (InterruptedException e) {
+ throw new RuntimeException(e);
--- End diff --
You could use `com.google.commons.base.Throwables.propagate()` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4037][SQL] Removes the SessionState ins...
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/2887#discussion_r19701171
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -288,6 +296,14 @@ class HiveContext(sc: SparkContext) extends
SQLContext(sc) {
val cmd_1: String = cmd_trimmed.substring(tokens(0).length()).trim()
val proc: CommandProcessor =
HiveShim.getCommandProcessor(Array(tokens(0)), hiveconf)
+ // Makes sure the session represented by the `sessionState` field is
activated. This implies
+ // Spark SQL Hive support uses a single `SessionState` for all Hive
operations and breaks
+ // session isolation under multi-user scenarios (i.e.
HiveThriftServer2).
+ // TODO Fix session isolation
+ if (SessionState.get() != sessionState) {
--- End diff --
In which case this condition will be true?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3796] Create external service which can...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3001#discussion_r19701140 --- Diff: network/shuffle/pom.xml --- @@ -0,0 +1,102 @@ + + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + 4.0.0 + +org.apache.spark +spark-parent +1.2.0-SNAPSHOT +../../pom.xml + + + org.apache.spark + spark-network-shuffle_2.10 --- End diff -- Isn't this module supposed to be scala-free? Then the 2.10 suffix is probably unnecessary. (Pardon if I missed and there really is Scala code here.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Upgrading to roaring 0.4.5 (bug fix release)
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3044#issuecomment-61354515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22670/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Upgrading to roaring 0.4.5 (bug fix release)
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3044#issuecomment-61354511 [Test build #22670 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22670/consoleFull) for PR 3044 at commit [`54018c5`](https://github.com/apache/spark/commit/54018c54390944dc7b0017186eb5237126675cf9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2189][SQL] Adds dropTempTable API
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3039#issuecomment-61354498 [Test build #22674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22674/consoleFull) for PR 3039 at commit [`1948c14`](https://github.com/apache/spark/commit/1948c14455c97ec1c0b0f3ab3d96911f757bd157). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2189][SQL] Adds dropTempTable API
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3039#issuecomment-61354499 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22674/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3886] [PySpark] simplify serializer, us...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2920#issuecomment-61354455 [Test build #22668 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22668/consoleFull) for PR 2920 at commit [`1d557fc`](https://github.com/apache/spark/commit/1d557fc24b761adee0b32c7df587c58d6980cf86). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2189][SQL] Adds dropTempTable API
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3039#issuecomment-61354420 [Test build #22674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22674/consoleFull) for PR 3039 at commit [`1948c14`](https://github.com/apache/spark/commit/1948c14455c97ec1c0b0f3ab3d96911f757bd157). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3886] [PySpark] simplify serializer, us...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2920#issuecomment-61354458 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22668/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Streaming KMeans [MLLIB][SPARK-3254]
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2942#issuecomment-61354018 [Test build #22673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22673/consoleFull) for PR 2942 at commit [`078617c`](https://github.com/apache/spark/commit/078617c167b5b2b698193cddfd567d60637ba906). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-1547: Add Gradient Boosting to M...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2607 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3787] Assembly jar name is wrong when w...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2647#issuecomment-61353876 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-1547: Add Gradient Boosting to M...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2607#issuecomment-61353864 I've merged this into master. Thanks @manishamde for contributing and @codedeft and @jkbradley for review! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4017] show progress bar in console and ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3029#issuecomment-61353655 [Test build #22669 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22669/consoleFull) for PR 3029 at commit [`e6bb189`](https://github.com/apache/spark/commit/e6bb1895c51bef6201296601725821d12d4deb8e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4017] show progress bar in console and ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3029#issuecomment-61353658 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22669/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3936] Remove Bytecode Inspection for Jo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2815#issuecomment-61353442 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22667/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3936] Remove Bytecode Inspection for Jo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2815#issuecomment-61353438 [Test build #22667 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22667/consoleFull) for PR 2815 at commit [`4097ca7`](https://github.com/apache/spark/commit/4097ca7f967156399f9f398b4b1dca5a6b186efe). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/1270#issuecomment-61353330 @mengxr Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3984] [SPARK-3983] Improve UI task metr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2832#issuecomment-61352966 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22666/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3984] [SPARK-3983] Improve UI task metr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2832#issuecomment-61352961 [Test build #22666 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22666/consoleFull) for PR 2832 at commit [`335be4b`](https://github.com/apache/spark/commit/335be4bef9dbf724f7058161057acf1782c92377). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4057] Use -agentlib instead of -Xdebug ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2904#issuecomment-61352903 [Test build #22664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22664/consoleFull) for PR 2904 at commit [`39b5320`](https://github.com/apache/spark/commit/39b53202242df7a25e5970308440b5b02176a145). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4057] Use -agentlib instead of -Xdebug ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2904#issuecomment-61352907 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22664/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2189][SQL] Adds unregisterTempTable API
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3039#discussion_r19700967
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -267,6 +267,23 @@ class SQLContext(@transient val sparkContext:
SparkContext)
}
/**
+ * Unregisters the temporary table with the given table name in the
catalog. If the table has been
+ * cached/persisted before, it can be unpersisted if required.
+ *
+ * @param tableName the name of the table to be unregistered.
+ * @param unpersist whether to unpersist the table if it has been
cached/persisted before.
+ *
+ * @group userf
+ */
+ def unregisterTempTable(tableName: String, unpersist: Boolean = false):
Unit = {
--- End diff --
Agree with the DDL part.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-61352711 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22665/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-61352708 [Test build #22665 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22665/consoleFull) for PR 2520 at commit [`7acffda`](https://github.com/apache/spark/commit/7acffda93cdb985cf3672c33509559939e30f1dc). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3838][examples][mllib][python] Word2Vec...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2952 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3838][examples][mllib][python] Word2Vec...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2952#issuecomment-61352555 LGTM. Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
