[GitHub] spark pull request #16975: [SPARK-19522] Fix executor memory in local-cluste...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16975#discussion_r101887609
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -466,7 +466,7 @@ object SparkSubmit extends CommandLineUtils {
// Other options
OptionAssigner(args.executorCores, STANDALONE | YARN,
ALL_DEPLOY_MODES,
sysProp = "spark.executor.cores"),
- OptionAssigner(args.executorMemory, STANDALONE | MESOS | YARN,
ALL_DEPLOY_MODES,
+ OptionAssigner(args.executorMemory, ALL_CLUSTER_MGRS,
ALL_DEPLOY_MODES,
--- End diff --
also we're talking about a net addition of 7 LOC in `SparkContext.scala`,
about half of which are comments and warning logs. It's really not that much
code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16975: [SPARK-19522] Fix executor memory in local-cluste...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16975#discussion_r101887589
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -466,7 +466,7 @@ object SparkSubmit extends CommandLineUtils {
// Other options
OptionAssigner(args.executorCores, STANDALONE | YARN,
ALL_DEPLOY_MODES,
sysProp = "spark.executor.cores"),
- OptionAssigner(args.executorMemory, STANDALONE | MESOS | YARN,
ALL_DEPLOY_MODES,
+ OptionAssigner(args.executorMemory, ALL_CLUSTER_MGRS,
ALL_DEPLOY_MODES,
--- End diff --
The inconsistency is already inherent with the parameters in
`local-cluster[]`, so I'm not introducing it here with this change. I
personally think it's a really bad interface to force the user set executor
memory in two different places and require that these two values match.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73100/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16982 **[Test build #73100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73100/testReport)** for PR 16982 at commit [`bf904ea`](https://github.com/apache/spark/commit/bf904ea6fdfa614d8095f60ecb0ece8ec6c4571b). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16979: [SPARK-19617][SS]Fix the race condition when starting an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16979 **[Test build #73101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73101/testReport)** for PR 16979 at commit [`1d776c3`](https://github.com/apache/spark/commit/1d776c36f01563ea2bc99530738e882471fd1896). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16626 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73094/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16982 **[Test build #73100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73100/testReport)** for PR 16982 at commit [`bf904ea`](https://github.com/apache/spark/commit/bf904ea6fdfa614d8095f60ecb0ece8ec6c4571b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16626 **[Test build #73094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73094/testReport)** for PR 16626 at commit [`193c0c3`](https://github.com/apache/spark/commit/193c0c34a7ec55007fe93e397dace43223b32f58). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16981 **[Test build #73099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73099/testReport)** for PR 16981 at commit [`0488507`](https://github.com/apache/spark/commit/0488507f4c4b37a6c2e3d840af8ea0cf04ad2ac2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73097/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16982 **[Test build #73097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73097/testReport)** for PR 16982 at commit [`d9881d5`](https://github.com/apache/spark/commit/d9881d5cea3097b8da3d0d84f647c2a8bfd2e15f). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16395 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16395 **[Test build #73093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73093/testReport)** for PR 16395 at commit [`0b151bd`](https://github.com/apache/spark/commit/0b151bdeff40f3c6e5b5f730593d07971d5f731a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16395 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73093/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16982 **[Test build #73098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73098/testReport)** for PR 16982 at commit [`38f5382`](https://github.com/apache/spark/commit/38f538206f21f38064bd85572f210214ee3401a4). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73098/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16982 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16978 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16978 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73092/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16978 **[Test build #73092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73092/testReport)** for PR 16978 at commit [`664fa2b`](https://github.com/apache/spark/commit/664fa2b3b6337de4bbdb6aa7e780c8e6360bbeab). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16744 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73090/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16744 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16744 **[Test build #73090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73090/testReport)** for PR 16744 at commit [`67a4acb`](https://github.com/apache/spark/commit/67a4acb7b253ec558471d86bcbf3a1fa969d2229). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/16982 Wow - this is interesting ! cc @rxin @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16982 **[Test build #73098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73098/testReport)** for PR 16982 at commit [`38f5382`](https://github.com/apache/spark/commit/38f538206f21f38064bd85572f210214ee3401a4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16982: [SPARK-19654][SPARKR] Structured Streaming API for R
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16982 **[Test build #73097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73097/testReport)** for PR 16982 at commit [`d9881d5`](https://github.com/apache/spark/commit/d9881d5cea3097b8da3d0d84f647c2a8bfd2e15f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16982: [SPARK-19654][SPARKR] Structured Streaming API fo...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/16982#discussion_r101885260
--- Diff: R/pkg/R/DataFrame.R ---
@@ -133,9 +133,6 @@ setMethod("schema",
#'
#' Print the logical and physical Catalyst plans to the console for
debugging.
#'
-#' @param x a SparkDataFrame.
-#' @param extended Logical. If extended is FALSE, explain() only prints
the physical plan.
-#' @param ... further arguments to be passed to or from other methods.
--- End diff --
move this to generic.R for a shared doc block for DF and SQ
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16982: [SPARK-19654][SPARKR] Structured Streaming API fo...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/16982 [SPARK-19654][SPARKR] Structured Streaming API for R ## What changes were proposed in this pull request? Add "experimental" API for SS in R ## How was this patch tested? manual, unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rss Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16982.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16982 commit b2f2171e106c41c4542452a623ea4490ad022d01 Author: Felix Cheung Date: 2017-02-16T08:21:12Z first pass ss commit 2af5823caaf86ac1361541fe0111cbbdbc185e63 Author: Felix Cheung Date: 2017-02-17T04:24:31Z doc commit 1ac9c3c873afb5e8d57eb7cdfee43cfb53b416f0 Author: Felix Cheung Date: 2017-02-17T07:43:55Z working commit d9881d5cea3097b8da3d0d84f647c2a8bfd2e15f Author: Felix Cheung Date: 2017-02-18T01:10:31Z fix doc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16969: [SPARK-19639][SPARKR][Example]:Add spark.svmLinea...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16969 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16981 **[Test build #73096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73096/testReport)** for PR 16981 at commit [`ed87d9a`](https://github.com/apache/spark/commit/ed87d9a9ac98cfa0b68729b9fc0d50c497153874). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16981 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73096/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16981 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16969: [SPARK-19639][SPARKR][Example]:Add spark.svmLinear examp...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16969 merged to master. I figure you could update the programming guide after the other PR is merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16969: [SPARK-19639][SPARKR][Example]:Add spark.svmLinear examp...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/16969 I assume 2c3 is before you rebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16981 **[Test build #73096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73096/testReport)** for PR 16981 at commit [`ed87d9a`](https://github.com/apache/spark/commit/ed87d9a9ac98cfa0b68729b9fc0d50c497153874). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16979: [SPARK-19617][SS]Fix the race condition when starting an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16979 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73091/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16979: [SPARK-19617][SS]Fix the race condition when starting an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16979 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16979: [SPARK-19617][SS]Fix the race condition when starting an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16979 **[Test build #73091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73091/testReport)** for PR 16979 at commit [`7a0b199`](https://github.com/apache/spark/commit/7a0b199dc47a71001d44731b22a0addd1359d8ec). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16981 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16981 **[Test build #73095 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73095/testReport)** for PR 16981 at commit [`8df67ec`](https://github.com/apache/spark/commit/8df67ec51cccab0d312f1236dff0c5f7506d9668). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16981 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73095/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16981: [SPARK-19637][SQL] Add from_json/to_json in FunctionRegi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16981 **[Test build #73095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73095/testReport)** for PR 16981 at commit [`8df67ec`](https://github.com/apache/spark/commit/8df67ec51cccab0d312f1236dff0c5f7506d9668). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16981: [SPARK-19637][SQL] Add from_json/to_json in Funct...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/16981 [SPARK-19637][SQL] Add from_json/to_json in FunctionRegistry ## What changes were proposed in this pull request? This pr added entries in `FunctionRegistry` and supported `from_json`/`to_json` in SQL. ## How was this patch tested? Added tests in `JsonFunctionsSuite`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark SPARK-19637 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16981.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16981 commit 8df67ec51cccab0d312f1236dff0c5f7506d9668 Author: Takeshi Yamamuro Date: 2017-02-17T09:05:02Z Add from_json/to_json in FunctionRegistry --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14671: [SPARK-17091][SQL] ParquetFilters rewrite IN to O...
Github user a10y closed the pull request at: https://github.com/apache/spark/pull/14671 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16626: [SPARK-19261][SQL] Alter add columns for Hive serde and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16626 **[Test build #73094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73094/testReport)** for PR 16626 at commit [`193c0c3`](https://github.com/apache/spark/commit/193c0c34a7ec55007fe93e397dace43223b32f58). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16395: [SPARK-17075][SQL] implemented filter estimation
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16395 **[Test build #73093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73093/testReport)** for PR 16395 at commit [`0b151bd`](https://github.com/apache/spark/commit/0b151bdeff40f3c6e5b5f730593d07971d5f731a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16977: [SPARK-19651][CORE] ParallelCollectionRDD.collect should...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16977 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16977: [SPARK-19651][CORE] ParallelCollectionRDD.collect should...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16977 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73081/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16977: [SPARK-19651][CORE] ParallelCollectionRDD.collect should...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16977 **[Test build #73081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73081/testReport)** for PR 16977 at commit [`f46939d`](https://github.com/apache/spark/commit/f46939dd5bc82cd9931e9485e70cd47f5cc4892d). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16977: [SPARK-19651][CORE] ParallelCollectionRDD.collect should...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16977 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16977: [SPARK-19651][CORE] ParallelCollectionRDD.collect should...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16977 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73080/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16977: [SPARK-19651][CORE] ParallelCollectionRDD.collect should...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16977 **[Test build #73080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73080/testReport)** for PR 16977 at commit [`c7f4300`](https://github.com/apache/spark/commit/c7f4300f3eb764713be0f353a8bb0aedda638d69). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16976 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73089/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16976 **[Test build #73089 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73089/testReport)** for PR 16976 at commit [`373eec9`](https://github.com/apache/spark/commit/373eec9ca8ac0ebef0186ac3dc9b9089a8dbdf35). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16980: [SPARK-19617][SS]fix structured streaming restart bug
Github user gf53520 commented on the issue: https://github.com/apache/spark/pull/16980 test this please" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16980: [SPARK-19617][SS]fix structured streaming restart bug
Github user gf53520 commented on the issue: https://github.com/apache/spark/pull/16980 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16980: [SPARK-19617][SS]fix structured streaming restart bug
Github user gf53520 commented on the issue: https://github.com/apache/spark/pull/16980 cc @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16980: [SPARK-19617][SS]fix structured streaming restart bug
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16980 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16980: [SPARK-19617][SS]fix structured streaming restart...
GitHub user gf53520 opened a pull request: https://github.com/apache/spark/pull/16980 [SPARK-19617][SS]fix structured streaming restart bug ## What changes were proposed in this pull request? [SPARK-19617](https://issues.apache.org/jira/browse/SPARK-19645) When restart a streaming job, spark will recompute WAL offsets and generate the same hdfs delta file(latest delta file which generated before restart and named "currentBatchId.delta") ## How was this patch tested? manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/gf53520/spark SPARK-19645 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16980.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16980 commit 43fd9743b1a096784a13dde3116d921a70ec3a2b Author: guifeng Date: 2017-02-18T03:44:28Z fix structured streaming restart bug --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16790: [SPARK-19450] Replace askWithRetry with askSync.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16790 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16790: [SPARK-19450] Replace askWithRetry with askSync.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16790 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73084/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16790: [SPARK-19450] Replace askWithRetry with askSync.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16790 **[Test build #73084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73084/testReport)** for PR 16790 at commit [`b575c55`](https://github.com/apache/spark/commit/b575c55ed9901b4dcf5b3e977a55f135b02aa4b1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16895 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73087/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16895 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16895 **[Test build #73087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73087/testReport)** for PR 16895 at commit [`cdf53bf`](https://github.com/apache/spark/commit/cdf53bf517a2f9dd6bbe347455cfb1be1f15ca45). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16499#discussion_r101883526
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -1018,7 +1025,9 @@ private[spark] class BlockManager(
try {
replicate(blockId, bytesToReplicate, level, remoteClassTag)
} finally {
-bytesToReplicate.dispose()
+if (!level.useOffHeap) {
--- End diff --
> Allocating a direct byte buffer creates a java.nio.DirectByteBuffer,
which is in turn a subclass of java.nio.MappedByteBuffer. So calling dispose()
will dispose direct buffers, too.
yeah, right.
If `bytesToReplicate` comes from disk store, it could be a memory-mapped
byte buffer, doesn't this change may miss the change to dispose it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16895 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16895 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73086/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16895: [SPARK-15615][SQL] Add an API to load DataFrame from Dat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16895 **[Test build #73086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73086/testReport)** for PR 16895 at commit [`82561c0`](https://github.com/apache/spark/commit/82561c072c668e062e9d854074bb3ef50320dd5c). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class LSHParams(Params):` * `class LSHModel(JavaModel):` * `class BucketedRandomProjectionLSH(JavaEstimator, LSHParams, HasInputCol, HasOutputCol, HasSeed,` * `class BucketedRandomProjectionLSHModel(LSHModel, JavaMLReadable, JavaMLWritable):` * `class MinHashLSH(JavaEstimator, LSHParams, HasInputCol, HasOutputCol, HasSeed,` * `class MinHashLSHModel(LSHModel, JavaMLReadable, JavaMLWritable):` * `case class StreamingExplainCommand(` * `case class SaveIntoDataSourceCommand(` * `abstract class JsonDataSource[T] extends Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16499: [SPARK-17204][CORE] Fix replicated off heap stora...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/16499#discussion_r101883328
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -813,7 +813,14 @@ private[spark] class BlockManager(
false
}
} else {
- memoryStore.putBytes(blockId, size, level.memoryMode, () =>
bytes)
+ val memoryMode = level.memoryMode
+ memoryStore.putBytes(blockId, size, memoryMode, () => {
+if (memoryMode == MemoryMode.OFF_HEAP) {
--- End diff --
I mean if all `bytes` are copy before passing in, it is no need to do
another copy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16898: [SPARK-19563][SQL] avoid unnecessary sort in FileFormatW...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16898 Sorry, I am late. Will review it tonight. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16978 **[Test build #73092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73092/testReport)** for PR 16978 at commit [`664fa2b`](https://github.com/apache/spark/commit/664fa2b3b6337de4bbdb6aa7e780c8e6360bbeab). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16976 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73088/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16976 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehous...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16290#discussion_r101883075
--- Diff: R/pkg/R/sparkR.R ---
@@ -376,6 +377,12 @@ sparkR.session <- function(
overrideEnvs(sparkConfigMap, paramMap)
}
+ # NOTE(shivaram): Set default warehouse dir to tmpdir to meet CRAN
requirements
+ # See SPARK-18817 for more details
+ if (!exists("spark.sql.default.warehouse.dir", envir = sparkConfigMap)) {
--- End diff --
After rethinking it, we might not need to add an extra sql conf. We just
need to know whether the value of `spark.sql.warehouse.dir` is from the users
or the original default. If it is the default, R can simply change it.
Maybe it is a good to-have feature for users to know whether the SQLConf
value is from users or from the default. cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16978 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16976 **[Test build #73088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73088/testReport)** for PR 16976 at commit [`44c9465`](https://github.com/apache/spark/commit/44c946541f2b0bbeb00736f4b28db00a6169257f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16978 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73085/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16978 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16978: [SPARK-19652][UI] Do auth checks for REST API access.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16978 **[Test build #73085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73085/testReport)** for PR 16978 at commit [`664fa2b`](https://github.com/apache/spark/commit/664fa2b3b6337de4bbdb6aa7e780c8e6360bbeab). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15125: [SPARK-5484][GraphX] Periodically do checkpoint i...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/15125#discussion_r101882898
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala ---
@@ -122,27 +125,39 @@ object Pregel extends Logging {
require(maxIterations > 0, s"Maximum number of iterations must be
greater than 0," +
s" but got ${maxIterations}")
-var g = graph.mapVertices((vid, vdata) => vprog(vid, vdata,
initialMsg)).cache()
+val checkpointInterval = graph.vertices.sparkContext.getConf
+ .getInt("spark.graphx.pregel.checkpointInterval", 10)
--- End diff --
Do we need to document this config into GraphX related document?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16387 Thanks! @vanzin Yes we can always come back to fix things later if there is an issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16979: [SPARK-19617][SS]Fix the race condition when starting an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16979 **[Test build #73091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73091/testReport)** for PR 16979 at commit [`7a0b199`](https://github.com/apache/spark/commit/7a0b199dc47a71001d44731b22a0addd1359d8ec). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16979: [SPARK-19617][SS]Fix the race condition when star...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/16979#discussion_r101882783
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
---
@@ -63,8 +63,39 @@ class HDFSMetadataLog[T <: AnyRef :
ClassTag](sparkSession: SparkSession, path:
val metadataPath = new Path(path)
protected val fileManager = createFileManager()
- if (!fileManager.exists(metadataPath)) {
-fileManager.mkdirs(metadataPath)
+ runUninterruptiblyIfLocal {
+if (!fileManager.exists(metadataPath)) {
+ fileManager.mkdirs(metadataPath)
+}
+ }
+
+ private def runUninterruptiblyIfLocal[T](body: => T): T = {
+if (fileManager.isLocalFileSystem) {
+ Thread.currentThread match {
+case ut: UninterruptibleThread =>
+ // When using a local file system, some file system APIs like
"create" or "mkdirs" must be
--- End diff --
Fixed the comment. I added it in
https://github.com/apache/spark/commit/88c43f4fb5ea042a119819c11a5cdbe225095c54
but it was wrong. We don't need to use `runUninterruptibly ` to workaround
`HADOOP-14084`. The root cause is `HADOOP-10622`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16947: [SPARK-19617][SS]Fix the race condition when starting an...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16947 #16979 is the backport for branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16979: [SPARK-19617][SS]Fix the race condition when star...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/16979#discussion_r101882720 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala --- @@ -178,8 +178,9 @@ class StreamExecution( /** * The thread that runs the micro-batches of this stream. Note that this thread must be - * [[org.apache.spark.util.UninterruptibleThread]] to avoid swallowing `InterruptException` when - * using [[HDFSMetadataLog]]. See SPARK-19599 for more details. + * [[org.apache.spark.util.UninterruptibleThread]] to workaround KAFKA-1894: interrupting a + * running `KafkaConsumer` may cause endless loop, and HADOOP-10622: interrupting --- End diff -- This file is almost same as #16947 except this comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16744 **[Test build #73090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73090/testReport)** for PR 16744 at commit [`67a4acb`](https://github.com/apache/spark/commit/67a4acb7b253ec558471d86bcbf3a1fa969d2229). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16979: [SPARK-19617][SS]Fix the race condition when star...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/16979 [SPARK-19617][SS]Fix the race condition when starting and stopping a query quickly (branch-2.1) ## What changes were proposed in this pull request? Backport #16947 to branch 2.1. ## How was this patch tested? Jenkins You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK-19617-branch-2.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16979.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16979 commit 7a0b199dc47a71001d44731b22a0addd1359d8ec Author: Shixiong Zhu Date: 2017-02-16T00:59:57Z [SPARK-19617][SS]Fix the race condition when starting and stopping a query quickly (branch-2.1) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16947: [SPARK-19617][SS]Fix the race condition when star...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16947 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16947: [SPARK-19617][SS]Fix the race condition when starting an...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/16947 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16898: [SPARK-19563][SQL] avoid unnecessary sort in FileFormatW...
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/16898 @cloud-fan : LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16826: [SPARK-19540][SQL] Add ability to clone SparkSession whe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16826 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73083/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16826: [SPARK-19540][SQL] Add ability to clone SparkSession whe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16826 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16826: [SPARK-19540][SQL] Add ability to clone SparkSession whe...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16826 **[Test build #73083 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73083/testReport)** for PR 16826 at commit [`8ac778a`](https://github.com/apache/spark/commit/8ac778ab444f90eadd22d36b91889d81ef593d44). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/15435 cc @sethah @jkbradley Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16976: [SPARK-19610][SQL] Support parsing multiline CSV files
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16976 **[Test build #73089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73089/testReport)** for PR 16976 at commit [`373eec9`](https://github.com/apache/spark/commit/373eec9ca8ac0ebef0186ac3dc9b9089a8dbdf35). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
