[GitHub] spark pull request: [SPARK-10142][STREAMING] Made python checkpoin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8366#issuecomment-133629112 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10142][STREAMING] Made python checkpoin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8366#issuecomment-133629097 [Test build #41401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41401/console) for PR 8366 at commit [`3afa666`](https://github.com/apache/spark/commit/3afa666027a7a657166e77640a2448f5c8c90546). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-133629862 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133635527 thanks for the merge. I will not be available till Monday itself :( . If this is in a hurry, please feel free to fix it yourself. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10142][STREAMING] Made python checkpoin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8366#issuecomment-133629113 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41401/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-133630302 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-133630304 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-133630525 [Test build #41402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41402/consoleFull) for PR 7753 at commit [`f21a804`](https://github.com/apache/spark/commit/f21a8047e615c7dc17e5a823ebaabbbc84fffdb3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8354#issuecomment-133366793 [Test build #41360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41360/console) for PR 8354 at commit [`e1ef530`](https://github.com/apache/spark/commit/e1ef5306006f28a6eed72a512b894334ef6fbd3e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8354#issuecomment-133367307 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41360/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8354#issuecomment-133367304 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9401][SQL]Fully implement code generati...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8353#issuecomment-133370445 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9401][SQL]Fully implement code generati...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8353#issuecomment-133370447 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41361/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10154][SQL] remove the no-longer-necess...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8356#issuecomment-133370402 LGTM, please fix the MiMa failure by adding ```scala ProblemFilters.exclude[MissingClassProblem](org.apache.spark.sql.sources.CatalystScan) ``` to `project/MimaExcludes.scala`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9401][SQL]Fully implement code generati...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8353#issuecomment-133370369 [Test build #41361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41361/console) for PR 8353 at commit [`affd497`](https://github.com/apache/spark/commit/affd497299a6fab1c6454228a531f99078a2e7ba). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133371987 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41365/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10152] [SQL] Support Init script for hi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8355#issuecomment-133371929 [Test build #41359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41359/console) for PR 8355 at commit [`c5ac310`](https://github.com/apache/spark/commit/c5ac3106dadcc53b834c80c316325dbae0b37a2c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133371984 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10152] [SQL] Support Init script for hi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8355#issuecomment-133372040 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41359/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133371905 [Test build #41365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41365/console) for PR 8352 at commit [`969bbbd`](https://github.com/apache/spark/commit/969bbbdd5cba4abf1997a031d3efa6f57665ac73). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10152] [SQL] Support Init script for hi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8355#issuecomment-133372039 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8889] [Core]: Fix for OOM for graph cre...
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/7602#issuecomment-133373808 Thanks @srowen @andrewor14 for the merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10122][PySpark][Streaming] Fix getOffse...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8347#issuecomment-133449445 LGTM except one minor doc typo. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service r...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7943 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service r...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-133427799 committed to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9708] [MESOS] Spark should create local...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8358#issuecomment-133446766 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10155][SQL]Change SqlParser to object t...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/8357 [SPARK-10155][SQL]Change SqlParser to object to avoid memory leak Since `scala.util.parsing.combinator.Parsers` is thread-safe since Scala 2.10 (See [SI-4929](https://issues.scala-lang.org/browse/SI-4929)), we can change SqlParser to object to avoid memory leak. I didn't change other subclasses of `scala.util.parsing.combinator.Parsers` because there is only one instance in one SQLContext, which should not be an issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark sql-memory-leak Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8357.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8357 commit 4bc33e0440e844980f8119e19bba70e3656c856c Author: zsxwing zsxw...@gmail.com Date: 2015-08-21T10:55:28Z Change SqlParser to object to avoid memory leak --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10154][SQL] remove the no-longer-necess...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8356#issuecomment-133403523 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10155][SQL]Change SqlParser to object t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8357#issuecomment-133441976 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10155][SQL]Change SqlParser to object t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8357#issuecomment-133442004 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10122][PySpark][Streaming] Fix getOffse...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/8347#discussion_r37640572 --- Diff: python/pyspark/streaming/dstream.py --- @@ -610,7 +610,10 @@ def __init__(self, prev, func): self.is_checkpointed = False self._jdstream_val = None -if (isinstance(prev, TransformedDStream) and +# Using type() to avoid folding the functions and compacting the DStreams which is not +# not strictly a object of TransformedDStream. --- End diff -- nit: two `not`s --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10154][SQL] remove the no-longer-necess...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8356#issuecomment-133403710 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10155][SQL]Change SqlParser to object t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8357#issuecomment-133445626 [Test build #41369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41369/consoleFull) for PR 8357 at commit [`4bc33e0`](https://github.com/apache/spark/commit/4bc33e0440e844980f8119e19bba70e3656c856c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9708] [MESOS] Spark should create local...
GitHub user Zariel opened a pull request: https://github.com/apache/spark/pull/8358 [SPARK-9708] [MESOS] Spark should create local temporary directories in Mesos sandbox when launched with Mesos This is my own original work and I license this to the project under the project's open source license You can merge this pull request into a Git repository by running: $ git pull https://github.com/Zariel/spark mesos-local-dir Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8358.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8358 commit 30fe90bfcc71bd74ce7ab44d07900297cba43e42 Author: Chris Bannister chris.bannis...@swiftkey.net Date: 2015-08-21T14:19:52Z when running in mesos use the provided scratch space --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9613] [CORE] Ban use of JavaConversions...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8033#issuecomment-133448393 [Test build #1679 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1679/console) for PR 8033 at commit [`b4f58d5`](https://github.com/apache/spark/commit/b4f58d5ebb4afc673304250b9309724811a11b5f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8340#issuecomment-133460175 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133490920 @MechCoder Could you move the changes in SQL to another PR? I think SQL is fine with `@since` tag. Switching to `@Since` will cause regression in the generated Java API doc. Let's experiment with `@Since` annotation in MLlib first:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10154][SQL] remove the no-longer-necess...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8356#issuecomment-133408157 [Test build #41368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41368/consoleFull) for PR 8356 at commit [`6ab9de0`](https://github.com/apache/spark/commit/6ab9de09e37600d7dd2073ca6a078b9eab7cc3cf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9439] [yarn] External shuffle service r...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/7943#issuecomment-133426928 sounds like we missed 1.5 on this so we can put in master for 1.6 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10148][Streaming]Display active and ina...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8351#issuecomment-133481655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41371/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10148][Streaming]Display active and ina...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8351#issuecomment-133481653 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10155][SQL]Change SqlParser to object t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8357#issuecomment-133491665 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10155][SQL]Change SqlParser to object t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8357#issuecomment-133491668 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41369/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133494112 [Test build #41373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41373/consoleFull) for PR 8352 at commit [`d6dbd4f`](https://github.com/apache/spark/commit/d6dbd4f692d308b66f3903f1828d613957acf353). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8436][SQL] Inconsistent behavior when c...
Github user x1- commented on the pull request: https://github.com/apache/spark/pull/8339#issuecomment-133457237 @sarutak I tried *casting Timestamp to Long and Long to Timestamp* using Hive below. ```sql hive SELECT CAST(CAST(1.2 AS TIMESTAMP) AS DOUBLE); OK 1.2 Time taken: 3.637 seconds, Fetched: 1 row(s) hive SELECT CAST(CAST(1200 AS TIMESTAMP) AS INT); OK 1 ``` This mean that Hive has contradiction in converting Timestamp to Int, and then Int to Timestamp. Because of spark has different dialect, it is better way closing this PR, I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8436][SQL] Inconsistent behavior when c...
Github user x1- closed the pull request at: https://github.com/apache/spark/pull/8339 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8436][SQL] Inconsistent behavior when c...
Github user x1- commented on the pull request: https://github.com/apache/spark/pull/8339#issuecomment-133457361 So, now I close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10148][Streaming]Display active and ina...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8351#issuecomment-133463951 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10148][Streaming]Display active and ina...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8351#issuecomment-133463981 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10148][Streaming]Display active and ina...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8351#issuecomment-133466150 [Test build #41371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41371/consoleFull) for PR 8351 at commit [`59c56d4`](https://github.com/apache/spark/commit/59c56d48c2c8166bd7fe424d7adf7fbf4f60e2a2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: ENH: Add ability to specify s3 bootstrap scrip...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8359#issuecomment-133469979 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8340#issuecomment-133478816 [Test build #41370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41370/console) for PR 8340 at commit [`f0d1f6e`](https://github.com/apache/spark/commit/f0d1f6e9992afa2be6f9efc74abf155b9dccbcf2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133492487 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-133492452 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133492168 rebased --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-133492490 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: ENH: Add ability to specify s3 bootstrap scrip...
Github user mdagost commented on the pull request: https://github.com/apache/spark/pull/8359#issuecomment-133480514 Sure. Sorry that I forgot to open a Jira ticket first: https://issues.apache.org/jira/browse/SPARK-10157 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10155][SQL]Change SqlParser to object t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8357#issuecomment-133491439 [Test build #41369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41369/console) for PR 8357 at commit [`4bc33e0`](https://github.com/apache/spark/commit/4bc33e0440e844980f8119e19bba70e3656c856c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9864] [DOC] [MLlib] [SQL] Replace since...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8352#issuecomment-133492446 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/8340#discussion_r37644626
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverSchedulingPolicy.scala
---
@@ -144,27 +141,31 @@ private[streaming] class ReceiverSchedulingPolicy {
val scheduledExecutors = mutable.Set[String]()
scheduledExecutors ++= preferredLocation
-val executorWeights = receiverTrackingInfoMap.values.flatMap {
receiverTrackingInfo =
- receiverTrackingInfo.state match {
-case ReceiverState.INACTIVE = Nil
-case ReceiverState.SCHEDULED =
- val scheduledExecutors =
receiverTrackingInfo.scheduledExecutors.get
- // The probability that a scheduled receiver will run in an
executor is
- // 1.0 / scheduledLocations.size
- scheduledExecutors.map(location = location - (1.0 /
scheduledExecutors.size))
-case ReceiverState.ACTIVE =
Seq(receiverTrackingInfo.runningExecutor.get - 1.0)
- }
+val executorWeights = receiverTrackingInfoMap.filter(_._1 !=
receiverId).values.flatMap {
--- End diff --
Updated to use the new approach. And since we don't call rescheduleReceiver
when the receiver registers at the first time, it's not necessary to do
filtering.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10154][SQL] remove the no-longer-necess...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8356#issuecomment-133460607 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10154][SQL] remove the no-longer-necess...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8356#issuecomment-133460611 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41368/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10148][Streaming]Display active and ina...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8351#issuecomment-133463286 img width=749 alt=screen shot 2015-08-21 at 11 25 44 pm src=https://cloud.githubusercontent.com/assets/1000778/9412055/48cf6966-485c-11e5-9ff2-585e7392969e.png; --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/8340#discussion_r37644655
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverSchedulingPolicySuite.scala
---
@@ -127,4 +127,26 @@ class ReceiverSchedulingPolicySuite extends
SparkFunSuite {
assert(executors.isEmpty)
}
}
+
+ test(when scheduleReceivers return a balanced result, we should not
restart receivers) {
--- End diff --
Removed this test since we don't call rescheduleReceiver when the receiver
registers at the first time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9613] [CORE] Ban use of JavaConversions...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8033#issuecomment-133462879 @JoshRosen wanted to call your attention to this PR since it's ready for a look. It is complete and passes tests -- usually. It needs constant rebasing though, so wanted to get it reviewed in a passing state that just needs a minor rebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10154][SQL] remove the no-longer-necess...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8356#issuecomment-133460408 **[Test build #41368 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41368/console)** for PR 8356 at commit [`6ab9de0`](https://github.com/apache/spark/commit/6ab9de09e37600d7dd2073ca6a078b9eab7cc3cf) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8340#issuecomment-133460250 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: ENH: Add ability to specify s3 bootstrap scrip...
GitHub user mdagost opened a pull request: https://github.com/apache/spark/pull/8359 ENH: Add ability to specify s3 bootstrap script to spark-ec2 One of the nice features of using EMR to start spark clusters is that you can specify bootstrap actions to run on each node. This is useful for installing custom libraries, etc. However, you pay a per-node/per-hour premium for using EMR (https://aws.amazon.com/elasticmapreduce/pricing/). This pull request adds the `--bootstrap-script` flag to the `spark-ec2` script. This flag specifies an s3 path to a shell script. The script is downloaded and run on each node during the setup process. The flag can be specified multiple times for multiple bootstrap actions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mdagost/spark ec2_bootstrap Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8359 commit d51aec04dc3f700de09cc398fc69b96cb86061b9 Author: Michelangelo D'Agostino mdagost...@civisanalytics.com Date: 2015-08-21T15:43:10Z ENH: Add ability to specify s3 bootstrap script to spark-ec2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9104][SPARK-9105][SPARK-9106][SPARK-910...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7753#issuecomment-133494106 [Test build #41372 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41372/consoleFull) for PR 7753 at commit [`f21a804`](https://github.com/apache/spark/commit/f21a8047e615c7dc17e5a823ebaabbbc84fffdb3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8340#issuecomment-133462018 [Test build #41370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41370/consoleFull) for PR 8340 at commit [`f0d1f6e`](https://github.com/apache/spark/commit/f0d1f6e9992afa2be6f9efc74abf155b9dccbcf2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: ENH: Add ability to specify s3 bootstrap scrip...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8359#issuecomment-133478794 Please read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark to understand how to propose a change --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8340#issuecomment-133478999 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41370/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10137][Streaming]Avoid to restart recei...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8340#issuecomment-133478995 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10148][Streaming]Display active and ina...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8351#issuecomment-133481526 [Test build #41371 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41371/console) for PR 8351 at commit [`59c56d4`](https://github.com/apache/spark/commit/59c56d48c2c8166bd7fe424d7adf7fbf4f60e2a2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-133534880 [Test build #41379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41379/consoleFull) for PR 8349 at commit [`a8f2fe9`](https://github.com/apache/spark/commit/a8f2fe9a5b4bc4aaee1a66fac59a2694c2ee583c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5259][CORE] don't submit stage until it...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7699#issuecomment-133540222 @kayousterhout @markhamstra @pwendell @mateiz another one still waiting. Mostly got buy-in from Mark on https://github.com/apache/spark/pull/4055, but I'd still appreciate somebody else looking, especially since I was more involved in the code for this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...
Github user tarnfeld commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-133542132 :+1: This is awesome! Given it a little test locally and works a treat! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5259][CORE] don't submit stage until it...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7699#issuecomment-133541667 [Test build #41381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41381/consoleFull) for PR 7699 at commit [`51f3c47`](https://github.com/apache/spark/commit/51f3c47ab98a001a6649e51f58b4ee7bc8b7e721). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-133543920 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9741][SQL] Approximate Count Distinct u...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8362#issuecomment-133544919 This made my day. The approach is super cool. Couple suggestions: 1. Can we use HyperLogLogPlus? It's also in streamlib: https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/cardinality/HyperLogLogPlus.java 2. Can we write this in a way to make it more unit testable? Beyond this, would be cool to have count-min sketch too! (future work) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/8180#discussion_r37672673 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -47,15 +47,48 @@ import org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat * minimal schedule to run the job. It then submits stages as TaskSets to an underlying * TaskScheduler implementation that runs them on the cluster. * - * In addition to coming up with a DAG of stages, this class also determines the preferred + * Spark stages are created by breaking the RDD graph at shuffle boundaries. RDD operations with + * narrow dependencies, like map() and filter(), are pipelined together into one set of tasks + * in each stage, but operations with shuffle dependencies require multiple stages (one to write a + * set of map output files, and another to read those files after a barrier). In the end, every + * stage will have only shuffle dependencies on other stages, and may compute multiple operations + * inside it. The actual pipelining of these operations happens in the RDD.compute() functions of + * various RDDs (MappedRDD, FilteredRDD, etc). + * + * In addition to coming up with a DAG of stages, the DAGScheduler also determines the preferred * locations to run each task on, based on the current cache status, and passes these to the * low-level TaskScheduler. Furthermore, it handles failures due to shuffle output files being * lost, in which case old stages may need to be resubmitted. Failures *within* a stage that are * not caused by shuffle file loss are handled by the TaskScheduler, which will retry each task * a small number of times before cancelling the whole stage. * + * When looking through this code, there are several key concepts: + * + * - Jobs (represented by [[ActiveJob]]) are the top-level work items submitted to the scheduler. + *For example, when the user calls an action, like count(), a job will be submitted through + *submitJob. Each Job may require the execution of multiple stages to build intermediate data. + * + * - Stages ([[Stage]]) are sets of tasks that compute intermediate results in jobs, where each + *task computes the same function on partitions of the same RDD. Stages are separated at shuffle + *boundaries, which introduce a barrier (where we must wait for the previous stage to finish to + *fetch outputs). There are two types of stages: [[ResultStage]], for the final stage that + *executes an action, and [[ShuffleMapStage]], which writes map output files for a shuffle. + *Stages are often shared across multiple jobs, if these jobs reuse the same RDDs. --- End diff -- Its nice to see these expanded comments, but I think we really need to add a section on stage attempts. That is most probably the most confusing part of the dag schduler and where most bugs occur. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8180#discussion_r37673715
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -812,7 +971,7 @@ class DAGScheduler(
case s: ResultStage =
val job = s.resultOfJob.get
partitionsToCompute.map { id =
-val p = job.partitions(id)
+val p = s.partitions(id)
--- End diff --
I've wondered about this line many times in the past, but been too scared
to change it -- is there any reason why `job.partitions` will be different from
`resultStage.partitions`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK 5280] RDF Loader added + documentation
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4650#issuecomment-133550145 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10136] [SQL] A more robust fix for SPAR...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8361#issuecomment-133550172 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10136] [SQL] A more robust fix for SPAR...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8361#issuecomment-133550175 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41376/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8400] [ML] Added check for positive blo...
GitHub user BryanCutler opened a pull request: https://github.com/apache/spark/pull/8363 [SPARK-8400] [ML] Added check for positive block size Added check for positive block size with a note that -1 for auto-configuring is not supported You can merge this pull request into a Git repository by running: $ git pull https://github.com/BryanCutler/spark ml.ALS-neg-blocksize-8400-1.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8363.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8363 commit a231c96cc9a64010db9440d634f1c711ee510c70 Author: Bryan Cutler bjcut...@us.ibm.com Date: 2015-08-21T20:33:06Z [SPARK-8400] Added check for positive block size with a note that -1 auto-config is not supported --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8400] [ML] Added check for positive blo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8363#issuecomment-133556173 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10121] [SQL] When add jar, also explici...
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/8364 [SPARK-10121] [SQL] When add jar, also explicitly set the classloader of metadataHive's HiveConf inside metadataHive's state. https://issues.apache.org/jira/browse/SPARK-10121 You can merge this pull request into a Git repository by running: $ git pull https://github.com/yhuai/spark SPARK-10121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8364.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8364 commit 4aecbbaeb9da221a205a002739e1eca008b9756c Author: Yin Huai yh...@databricks.com Date: 2015-08-21T20:57:24Z Expose initialConf. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9741][SQL] Approximate Count Distinct u...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8362#issuecomment-133562454 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41377/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9741][SQL] Approximate Count Distinct u...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8362#issuecomment-133562302 [Test build #41377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41377/console) for PR 8362 at commit [`e178d9e`](https://github.com/apache/spark/commit/e178d9e942720c879dc947a5fd3e6593a351f04f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class HyperLogLog(child: Expression, relativeSD: Double = 0.05)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9851] Support submitting map stages ind...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8180#discussion_r37678288
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -746,6 +848,63 @@ class DAGScheduler(
submitWaitingStages()
}
+ private[scheduler] def handleMapStageSubmitted(jobId: Int,
+ dependency: ShuffleDependency[_, _, _],
+ callSite: CallSite,
+ listener: JobListener,
+ properties: Properties) {
+// Submitting this map stage might still require the creation of some
parent stages, so make
+// sure that happens.
+var finalStage: ShuffleMapStage = null
+try {
+ // New stage creation may throw an exception if, for example, jobs
are run on a
+ // HadoopRDD whose underlying HDFS files have been deleted.
+ finalStage = getShuffleMapStage(dependency, jobId)
+} catch {
+ case e: Exception =
+logWarning(Creating new stage failed due to exception - job: +
jobId, e)
+listener.jobFailed(e)
+return
+}
+if (finalStage != null) {
--- End diff --
can you add a comment on how `finalStage` can be `null`? would be nice to
also have this in `handleJobSubmitted` as well, actually
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133564829 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10004] [shuffle] Perform auth checks wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8218#issuecomment-133564844 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4413][SQL] Parquet support through data...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3269#discussion_r37668626
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -93,3 +98,18 @@ abstract class PrunedScan extends BaseRelation {
abstract class PrunedFilteredScan extends BaseRelation {
def buildScan(requiredColumns: Array[String], filters: Array[Filter]):
RDD[Row]
}
+
+/**
+ * ::Experimental::
+ * An interface for experimenting with a more direct connection to the
query planner. Compared to
+ * [[PrunedFilteredScan]], this operator receives the raw expressions from
the
+ * [[org.apache.spark.sql.catalyst.plans.logical.LogicalPlan]]. Unlike
the other APIs this
+ * interface is not designed to be binary compatible across releases and
thus should only be used
+ * for experimentation.
+ */
+@Experimental
+abstract class CatalystScan extends BaseRelation {
+ def buildScan(requiredColumns: Seq[Attribute], filters:
Seq[Expression]): RDD[Row]
+}
--- End diff --
Why remove it? Its labeled experimental, its not broken...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10130] [SQL] type coercion for IF shoul...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8331 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-133544483 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41382/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5882][GraphX] Add a test for GraphLoade...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4674#issuecomment-133544479 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10143] [SQL] Use parquet's block size (...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8346#issuecomment-133547106 Tested it on a cluster using ``` val count = sqlContext.table(store_sales).groupBy().count().queryExecution.executedPlan(3).execute().count ``` Basically, it reads 0 column of table `store_sales`. My table has 1824 parquet files with size from 80MB to 280MB (1 to 3 row group sizes). Without this patch, in a 16 worker cluster, the job had 5023 tasks and spent 102s. With this patch, the job had 2893 tasks and spent 64s. It is still not as good as using one mapper per file (1824 tasks and 42s), but it is much better than our master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10121] [SQL] [WIP] When add jar, also e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8364#issuecomment-133562976 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10121] [SQL] [WIP] When add jar, also e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8364#issuecomment-133562994 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
