[GitHub] spark pull request: [SPARK-5000][SQL] Alias support string literal...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3837#issuecomment-68336967 [Test build #24885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24885/consoleFull) for PR 3837 at commit [`6d9ec95`](https://github.com/apache/spark/commit/6d9ec95f972ae15ab088fcec9cc0ba019ef2b62a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5000][SQL] Alias support string literal...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3837#issuecomment-68336970 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24885/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4574][SQL] Adding support for defining ...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/3431#issuecomment-68336828 Hi @marmbrus, still working on this, tomorrow i will update this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4574][SQL] Adding support for defining ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3431#issuecomment-68336623 ping. any progress here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3821#issuecomment-68336490 [Test build #24887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24887/consoleFull) for PR 3821 at commit [`1bab9e4`](https://github.com/apache/spark/commit/1bab9e4b782e62485f01f4f650a54c5ccb86f2a1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4988][SQL] Fix: 'Create table ..as sele...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3821#issuecomment-68336405 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68335760 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24883/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68335758 [Test build #24883 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24883/consoleFull) for PR 3823 at commit [`55300bc`](https://github.com/apache/spark/commit/55300bcd566b53d0b9a4a579aa3074c7c3ac66b9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5002][SQL] Using ascending by default w...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3838#issuecomment-68335223 [Test build #24886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24886/consoleFull) for PR 3838 at commit [`114b64a`](https://github.com/apache/spark/commit/114b64a9b8dba469c44a455cb6f239ea1e8c0d2a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3820#issuecomment-68335236 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24884/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3820#issuecomment-68335232 [Test build #24884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24884/consoleFull) for PR 3820 at commit [`dc6eaba`](https://github.com/apache/spark/commit/dc6eaba7db957eb9038532c7c57282c040e870d4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5002][SQL] Using ascending by default w...
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/3838 [SPARK-5002][SQL] Using ascending by default when not specify order in order by spark sql does not support ```SELECT a, b FROM testData2 ORDER BY a desc, b```. You can merge this pull request into a Git repository by running: $ git pull https://github.com/scwf/spark orderby Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3838.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3838 commit 48145d3f3264860c5848e9ed4fba75dd0e413a34 Author: wangfei Date: 2014-12-30T06:56:55Z fix order, using asc by default commit 114b64a9b8dba469c44a455cb6f239ea1e8c0d2a Author: wangfei Date: 2014-12-30T06:59:00Z remove nouse methods --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4156 [MLLIB] EM algorithm for GMMs
Github user FlytxtRnD commented on the pull request: https://github.com/apache/spark/pull/3022#issuecomment-68335194 @tgaloppo Good Work @mengxr Thanks for giving us a chance to be a part of this contribution --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5000][SQL] Alias support string literal...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3837#issuecomment-68334080 [Test build #24885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24885/consoleFull) for PR 3837 at commit [`6d9ec95`](https://github.com/apache/spark/commit/6d9ec95f972ae15ab088fcec9cc0ba019ef2b62a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5000][SQL] Alias support string literal...
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/3837 [SPARK-5000][SQL] Alias support string literal in spark sql parser Alias support string literal in spark sql parser, such as select key , value as 'vvv' from tableA; You can merge this pull request into a Git repository by running: $ git pull https://github.com/scwf/spark as Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3837.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3837 commit 6d9ec95f972ae15ab088fcec9cc0ba019ef2b62a Author: wangfei Date: 2014-12-30T06:26:32Z support as string literal in spark sql parser --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3820#discussion_r22339701
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -84,7 +86,8 @@ private[parquet] class RowReadSupport extends
ReadSupport[Row] with Logging {
// TODO: Why it can be null?
if (schema == null) {
log.debug("falling back to Parquet read schema")
- schema = ParquetTypesConverter.convertToAttributes(parquetSchema,
false)
+ schema = ParquetTypesConverter.convertToAttributes(
+parquetSchema, new SQLContext(new SparkContext))
--- End diff --
The only thing used here inside this SQLContext is the
`isParquetBinaryAsString` and `isParquetINT96AsTimestamp`. I'll add a comment
here if necessary, to point this out clearly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3820#issuecomment-68333255 [Test build #24884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24884/consoleFull) for PR 3820 at commit [`dc6eaba`](https://github.com/apache/spark/commit/dc6eaba7db957eb9038532c7c57282c040e870d4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68332440 [Test build #24883 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24883/consoleFull) for PR 3823 at commit [`55300bc`](https://github.com/apache/spark/commit/55300bcd566b53d0b9a4a579aa3074c7c3ac66b9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4920][UI] add version on master and wor...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3769#issuecomment-68332339 This looks good to me, so I'll merge it. Thanks for the reminder. Before I merge this, though, I'm going to hotfix the package-level SPARK_VERSION into the backport branches, since I accidentally broke those builds the last time I merged a fix using org.apache.spark.SPARK_VERSION. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68331919 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24882/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68331913 [Test build #24882 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24882/consoleFull) for PR 3732 at commit [`d66b01c`](https://github.com/apache/spark/commit/d66b01cee0d5a69de303c9735c7796c5af7e5825). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `final class Date extends Ordered[Date] with Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68331877 [Test build #24881 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24881/consoleFull) for PR 3661 at commit [`52247f5`](https://github.com/apache/spark/commit/52247f5ff48f1fdf285daac20846c7587a30f340). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68331881 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24881/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2304] tera sort example program for shu...
Github user liuqiyun commented on the pull request: https://github.com/apache/spark/pull/1242#issuecomment-68331374 So how to save as the uncompressed binary representation in the GenSort.scala program? I want to compare it with Hadoop MR which also use the uncompressed binary representation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2686 Add Length and OctetLen support to ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-68331226 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24880/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2686 Add Length and OctetLen support to ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-68331223 QA results for PR 1586:- This patch FAILED unit tests.For more information see test ouptut:https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24880/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4908][SQL] Prevent multiple concurrent ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3834#issuecomment-68330655 According to my earlier investigation, this issue seems to be related to the asynchronous execution feature enabled by default in Hive 0.13.1 JDBC driver. I'm not pretty sure whether synchronizes `HiveContext.runHive` completely fixes this issue. But I'd vote to merge this first before we find the root cause, since doing so does no harm. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLlib]delete the "train" function
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3836#issuecomment-68329970 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLlib]delete the "train" function
GitHub user ljzzju opened a pull request: https://github.com/apache/spark/pull/3836 [MLlib]delete the "train" function To make the functions with the same in "object" effective, specially when using java reflection. As the "train" function defined in "class DecisionTree" will hide the functions with the same name in "object DecisionTree". JIRA[SPARK-4998] You can merge this pull request into a Git repository by running: $ git pull https://github.com/ljzzju/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3836.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3836 commit 4e13133b33d0f057764736d4e68a7026ec56eeac Author: Liu Jiongzhou Date: 2014-12-30T04:00:07Z [MLlib]delete the "train" function To make the functions with the same in "object" effective, specially when using java reflection. As the "train" function defined in "class DecisionTree" will hide the functions with the same name in "object DecisionTree". JIRA[SPARK-4998] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4963 [SQL] HiveTableScan return mutable ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3827#issuecomment-68329848 @yanbohappy Actually, we can move the `copy` call to [`execution.Sample.execute`] [1]. In this way, queries without sampling are not negatively effected. [1]: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala#L73 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4963 [SQL] HiveTableScan return mutable ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3827#issuecomment-68329556 Hey @yanbohappy, as I've commented in the JIRA, would you mind to do a micro benchmark using code in #758 to see whether this fix introduces noticeable performance regression? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4508] [SQL] build native date type to c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3732#issuecomment-68329500 [Test build #24882 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24882/consoleFull) for PR 3732 at commit [`d66b01c`](https://github.com/apache/spark/commit/d66b01cee0d5a69de303c9735c7796c5af7e5825). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4920][UI] add version on master and wor...
Github user liyezhang556520 commented on the pull request: https://github.com/apache/spark/pull/3769#issuecomment-68329234 @JoshRosen , is this PR ready to merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4908][SQL] Prevent multiple concurrent ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3834#issuecomment-68329208 [Test build #24879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24879/consoleFull) for PR 3834 at commit [`bf25300`](https://github.com/apache/spark/commit/bf25300ffbebdf7a313c6bc39e940ab75c357e67). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4908][SQL] Prevent multiple concurrent ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3834#issuecomment-68329213 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24879/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68328870 [Test build #24881 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24881/consoleFull) for PR 3661 at commit [`52247f5`](https://github.com/apache/spark/commit/52247f5ff48f1fdf285daac20846c7587a30f340). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68328777 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2686 Add Length and OctetLen support to ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1586#issuecomment-68328503 QA tests have started for PR 1586. This patch DID NOT merge cleanly! View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24880/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Changes to illustrate the principles of functi...
Github user yujunliang closed the pull request at: https://github.com/apache/spark/pull/3835 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Changes to illustrate the principles of functi...
Github user yujunliang commented on the pull request: https://github.com/apache/spark/pull/3835#issuecomment-68328295 Let me create another one instead. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68327501 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24877/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68327498 [Test build #24877 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24877/consoleFull) for PR 3823 at commit [`d8d3cb7`](https://github.com/apache/spark/commit/d8d3cb7972422361c538c3f2cc2c4126e1826fcd). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Changes to illustrate the principles of functi...
Github user yujunliang commented on the pull request: https://github.com/apache/spark/pull/3835#issuecomment-68327317 Please don't merge, it is just for illustration purpose. Thanks. ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Changes to illustrate the principles of functi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3835#issuecomment-68327211 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Changes to illustrate the principles of functi...
GitHub user yujunliang opened a pull request: https://github.com/apache/spark/pull/3835 Changes to illustrate the principles of functional programming. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yujunliang/spark functional-programming Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3835.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3835 commit 32d0ba79596ce58815d0e129b44966741d7419db Author: Yujun Liang Date: 2014-12-30T03:19:54Z Changes to illustrate the principles of functional programming. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4908][SQL] Prevent multiple concurrent ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3834#issuecomment-68326513 [Test build #24879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24879/consoleFull) for PR 3834 at commit [`bf25300`](https://github.com/apache/spark/commit/bf25300ffbebdf7a313c6bc39e940ab75c357e67). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4908][SQL] Prevent multiple concurrent ...
GitHub user marmbrus opened a pull request: https://github.com/apache/spark/pull/3834 [SPARK-4908][SQL] Prevent multiple concurrent hive native commands This is just a quick fix that locks when calling `runHive`. If we can find a way to avoid the error without a global lock that would be better. You can merge this pull request into a Git repository by running: $ git pull https://github.com/marmbrus/spark hiveConcurrency Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3834.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3834 commit bf25300ffbebdf7a313c6bc39e940ab75c357e67 Author: Michael Armbrust Date: 2014-12-30T03:09:20Z prevent multiple concurrent hive native commands --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68326338 [Test build #24876 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24876/consoleFull) for PR 3661 at commit [`52247f5`](https://github.com/apache/spark/commit/52247f5ff48f1fdf285daac20846c7587a30f340). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68326339 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24876/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4835] [WIP] Disable validateOutputSpecs...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3832#issuecomment-68326012 [Test build #24874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24874/consoleFull) for PR 3832 at commit [`762e473`](https://github.com/apache/spark/commit/762e473d3d2bd90110029006b06fb701825ecdde). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GaussianMixtureModel(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4921. TaskSetManager.dequeueTask returns...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/3816#issuecomment-68326024 yes, @mateiz was right, I would like to give more clues to facilitate your debugging 1, NO_PREF will not be adjusted by getAllowedLocalityLevel() method of TaskSetManager(https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L431). I made it to ensure that NO_PREF tasks can be scheduled ASAP instead of waiting for NODE_LOCAL 2, based on 1, when the resourceOffers() of TaskScheduleImpl (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L217) is called for another time **before the more local level expires** and ** after a NO_PREF task is scheduled **, we may see that, the locality level was bumped up to RACK_LOCAL, instead of going through PROCESS_LOCAL, NODE_LOCAL, etc. 3. in the JIRA discussion, Rui Li's understanding on that the if check is correct, because we need to return PROCESS_LOCAL and also don't want to reset currentLocalityIndex (so I didn't get the reason of the performance degrading you mentioned) 4. the logic of returning PROCESS_LOCAL for NO_PREF tasks actually has exited for a long while before my patch on TaskSetManager.scala...I just followed this idea and my own understanding on that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4835] [WIP] Disable validateOutputSpecs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3832#issuecomment-68326015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24874/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68325790 [Test build #24875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24875/consoleFull) for PR 3823 at commit [`07b9ebf`](https://github.com/apache/spark/commit/07b9ebf45cb9cf06cc3347aff829ae6d77562f87). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68325792 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24875/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user OopsOutOfMemory commented on a diff in the pull request: https://github.com/apache/spark/pull/3823#discussion_r22337458 --- Diff: bin/spark-submit --- @@ -42,7 +42,10 @@ while (($#)); do shift done -DEFAULT_PROPERTIES_FILE="$SPARK_HOME/conf/spark-defaults.conf" +if [ ! -d "$SPARK_CONF_DIR" ]; then + SPARK_CONF_DIR="$SPARK_HOME/conf" --- End diff -- I recommend to add `export` keyword to make the SPARK_CONF_DIR global :) ``` export SPARK_CONF_DIR="$SPARK_HOME/conf" ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2309][MLlib] Multinomial Logistic Regre...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3833#issuecomment-68325568 [Test build #24878 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24878/consoleFull) for PR 3833 at commit [`a97b03a`](https://github.com/apache/spark/commit/a97b03ad94c08b0f4be9e2b832c592dbfa32fd57). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2309][MLlib] Multinomial Logistic Regre...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3833#issuecomment-68325569 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24878/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2309][MLlib] Multinomial Logistic Regre...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3833#issuecomment-68325298 [Test build #24878 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24878/consoleFull) for PR 3833 at commit [`a97b03a`](https://github.com/apache/spark/commit/a97b03ad94c08b0f4be9e2b832c592dbfa32fd57). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2309][MLlib] Multinomial Logistic Regre...
GitHub user dbtsai opened a pull request: https://github.com/apache/spark/pull/3833 [SPARK-2309][MLlib] Multinomial Logistic Regression #1379 is automatically closed by asfgit, and github can not reopen it once it's closed, so this will be the new PR. Binary Logistic Regression can be extended to Multinomial Logistic Regression by running K-1 independent binary logistic regression models. The following formula is implemented. http://www.slideshare.net/dbtsai/2014-0620-mlor-36132297/25 You can merge this pull request into a Git repository by running: $ git pull https://github.com/AlpineNow/spark mlor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3833.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3833 commit a97b03ad94c08b0f4be9e2b832c592dbfa32fd57 Author: DB Tsai Date: 2014-12-17T23:53:10Z first commit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4501][Core] - Create build/mvn to autom...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/3707#issuecomment-68324363 @brennonyork Sorry, I tried many times, could not reproduce the issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68324325 [Test build #24877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24877/consoleFull) for PR 3823 at commit [`d8d3cb7`](https://github.com/apache/spark/commit/d8d3cb7972422361c538c3f2cc2c4126e1826fcd). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3779#issuecomment-68324255 [Test build #556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/556/consoleFull) for PR 3779 at commit [`32d363f`](https://github.com/apache/spark/commit/32d363f275c6dea225019a1cb23af4eeeb44781f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68323743 I deleted the commit about `isTerminated`. It will be sent in a different PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4813][Streaming] Fix the issue that Con...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3661#issuecomment-68323687 [Test build #24876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24876/consoleFull) for PR 3661 at commit [`52247f5`](https://github.com/apache/spark/commit/52247f5ff48f1fdf285daac20846c7587a30f340). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4417] New API: sample RDD to fixed numb...
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/3723#issuecomment-68322948 Mark and Matei - I hear you guys and understand what you're saying. Does it make sense to create new Jira to refactor the RDD interface to move the advanced sampling methods into a packages class? This would obviously involve deprecating the presently existing functions so I presume this wouldn't see the light of day for a while. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3823#issuecomment-68322885 [Test build #24875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24875/consoleFull) for PR 3823 at commit [`07b9ebf`](https://github.com/apache/spark/commit/07b9ebf45cb9cf06cc3347aff829ae6d77562f87). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4990][Deploy]to find default properties...
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3823#issuecomment-68322557
@OopsOutOfMemory Ok I got what you mean. After checking the logic in
`SparkSubmitArguments.scala` I do think that your solution is more reasonable.
Thanks.
> val sparkHomeConfig = env.get("SPARK_HOME").map(sparkHome =>
s"${sparkHome}${sep}conf")
val confDir = env.get("SPARK_CONF_DIR").orElse(sparkHomeConfig)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: spark-core - [SPARK-4787] - Stop sparkcontext ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3809#issuecomment-68322408 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24873/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: spark-core - [SPARK-4787] - Stop sparkcontext ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3809#issuecomment-68322406 [Test build #24873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24873/consoleFull) for PR 3809 at commit [`5661e01`](https://github.com/apache/spark/commit/5661e01c2b0aaf900b50fb2444db714f73021aa4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4989][CORE] avoid wrong eventlog conf c...
Github user liyezhang556520 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3824#discussion_r22336254
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -719,26 +719,28 @@ private[spark] class Master(
def rebuildSparkUI(app: ApplicationInfo): Boolean = {
val appName = app.desc.name
val notFoundBasePath = HistoryServer.UI_PATH_PREFIX + "/not-found"
-val eventLogFile = app.desc.eventLogDir
- .map { dir => EventLoggingListener.getLogPath(dir, app.id) }
- .getOrElse {
-// Event logging is not enabled for this application
-app.desc.appUiUrl = notFoundBasePath
-return false
-}
-val fs = Utils.getHadoopFileSystem(eventLogFile, hadoopConf)
+var eventLogFile: String = null
--- End diff --
Hi @JoshRosen , `eventLogFile` is also used in `catch` block in this file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4961] [CORE] Put HadoopRDD.getPartition...
Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/3794#discussion_r22336168
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -203,9 +204,27 @@ class HadoopRDD[K, V](
for (i <- 0 until inputSplits.size) {
array(i) = new HadoopPartition(id, i, inputSplits(i))
}
+logDebug("Get these partitions took %f s".format((System.nanoTime -
start) / 1e9))
array
}
+ @transient private var thesePartitions_ : Array[Partition] = {
+try {
+ getThesePartitions()
+} catch {
+ case e: Exception =>
+logDebug("Error initializing HadoopRDD's partitions", e)
+null
--- End diff --
> It seems like the fix in this patch is to force partitions to be
eagerly-computed in the driver thread that defines the RDD. This seems like a
good idea
How would this interact with the idea of @erikerlandson to defer partition
computation?
https://github.com/apache/spark/pull/3079
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4835] [WIP] Disable validateOutputSpecs...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3832#issuecomment-68322195 [Test build #24874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24874/consoleFull) for PR 3832 at commit [`762e473`](https://github.com/apache/spark/commit/762e473d3d2bd90110029006b06fb701825ecdde). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4835] [WIP] Disable validateOutputSpecs...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3832#issuecomment-68322201 The `[WIP]` tag in the title is because I just realized that we may be able to improve this based on some discussion at the JIRA ticket. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-68322155 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24872/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-68322153 [Test build #24872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24872/consoleFull) for PR 2833 at commit [`38c6342`](https://github.com/apache/spark/commit/38c63429e79fa1ed8c6d31004d14b2487b945939). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GaussianMixtureModel(` * `class Receiver(object):` * `>>> class MyReceiver(Receiver):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4835] Disable validateOutputSpecs for S...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/3832 [SPARK-4835] Disable validateOutputSpecs for Spark Streaming jobs This patch disables output spec. validation for jobs launched through Spark Streaming, since this interferes with checkpoint recovery. Hadoop OutputFormats have a `checkOutputSpecs` method which performs certain checks prior to writing output, such as checking whether the output directory already exists. SPARK-1100 added checks for FileOutputFormat, SPARK-1677 (#947) added a SparkConf configuration to disable these checks, and SPARK-2309 (#1088) extended these checks to run for all OutputFormats, not just FileOutputFormat. In Spark Streaming, we might have to re-process a batch during checkpoint recovery, so `save` actions may be called multiple times. In addition to `DStream`'s own save actions, users might use `transform` or `foreachRDD` and call the `RDD` and `PairRDD` save actions. When output spec. validation is enabled, the second calls to these actions will fail due to existing output. This patch automatically disables output spec. validation for jobs submitted by the Spark Streaming scheduler and introduces a `spark.streaming.hadoop.validateOutputSpecs` setting to re-enable the old behavior. This is done by using Scala's `DynamicVariable` to propagate the bypass setting without having to mutate SparkConf or introduce a global variable. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark SPARK-4835 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3832.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3832 commit 762e473d3d2bd90110029006b06fb701825ecdde Author: Josh Rosen Date: 2014-12-30T01:13:50Z [SPARK-4835] Disable validateOutputSpecs for Spark Streaming jobs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [YARN][SPARK-4929] Bug fix: fix the yarn-clien...
Github user SaintBacchus commented on the pull request:
https://github.com/apache/spark/pull/3771#issuecomment-68321575
what @tgravescs says is close to the scenario, but it happens during the
RM recover after broke down.
```scala
if (finalStatus == FinalApplicationStatus.SUCCEEDED ||
isLastAttempt) {
unregister(finalStatus, finalMsg)
cleanupStagingDir(fs)
}
```
In the code, it won't check the `isLastAttempt` if the `finalStatus` was
`FinalApplicationStatus.SUCCEEDED` .
When the RM recovering happens, it would not check the `isLastAttempt`
since the yarn-client had no chance to change the value of `finalStatus`. It's
going to the `unregister` and this application can't recover itself.
So the yarn-client can't support the RM HA now.(yarn-cluster is OK)
And dividing the `finalStatus` into two parts is an easy way to avoid this
problem and compatible with previous design.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4972][MLlib] Updated the scala doc for ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3808 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4972][MLlib] Updated the scala doc for ...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/3808#issuecomment-68321431 LGTM. Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-68321410 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24871/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-68321406 [Test build #24871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24871/consoleFull) for PR 2833 at commit [`3eb0d9f`](https://github.com/apache/spark/commit/3eb0d9fc37c8f72dca1ff42684e51d6fc55fc239). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class GaussianMixtureModel(` * `class Receiver(object):` * `>>> class MyReceiver(Receiver):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3779#issuecomment-68319961 [Test build #556 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/556/consoleFull) for PR 3779 at commit [`32d363f`](https://github.com/apache/spark/commit/32d363f275c6dea225019a1cb23af4eeeb44781f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4790][STREAMING] Fix ReceivedBlockTrack...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3726#issuecomment-68319258 [Test build #24869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24869/consoleFull) for PR 3726 at commit [`bbbacd1`](https://github.com/apache/spark/commit/bbbacd1a441e43ce46e49bea6c85c6d7834c5487). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4790][STREAMING] Fix ReceivedBlockTrack...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3726#issuecomment-68319259 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24869/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3779#issuecomment-68318551 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24870/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3779#issuecomment-68318545 [Test build #24870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24870/consoleFull) for PR 3779 at commit [`32d363f`](https://github.com/apache/spark/commit/32d363f275c6dea225019a1cb23af4eeeb44781f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: spark-core - [SPARK-4787] - Stop sparkcontext ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3809#issuecomment-68317909 [Test build #24873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24873/consoleFull) for PR 3809 at commit [`5661e01`](https://github.com/apache/spark/commit/5661e01c2b0aaf900b50fb2444db714f73021aa4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-68317573 [Test build #24872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24872/consoleFull) for PR 2833 at commit [`38c6342`](https://github.com/apache/spark/commit/38c63429e79fa1ed8c6d31004d14b2487b945939). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: spark-core - [SPARK-4787] - Stop sparkcontext ...
Github user tigerquoll commented on a diff in the pull request:
https://github.com/apache/spark/pull/3809#discussion_r22334486
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -329,8 +329,11 @@ class SparkContext(config: SparkConf) extends Logging
with ExecutorAllocationCli
try {
dagScheduler = new DAGScheduler(this)
} catch {
-case e: Exception => throw
- new SparkException("DAGScheduler cannot be initialized due to
%s".format(e.getMessage))
+case e: Exception => {
+ stop()
--- End diff --
Excellent idea Josh.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4925][SQL] Publish Spark SQL hive-thrif...
Github user alexliu68 commented on the pull request: https://github.com/apache/spark/pull/3766#issuecomment-68317258 I removed it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-68316677 [Test build #24871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24871/consoleFull) for PR 2833 at commit [`3eb0d9f`](https://github.com/apache/spark/commit/3eb0d9fc37c8f72dca1ff42684e51d6fc55fc239). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4882] Register PythonBroadcast with Kry...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3831#issuecomment-68315787 LGTM, thanks for fixing it! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4417] New API: sample RDD to fixed numb...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/3723#issuecomment-68315776 I agree with Mark about this. This method doesn't seem worth adding an API for by default, especially if it will be tricky to implement. For extracting small samples, takeSample already lets you specify an exact numbers, and for downsampling large RDDs, most users probably don't need an exact number (and wouldn't want to pay an extra pass over the data for it). This and other advanced sampling methods could make a good external package though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3955 part 2 [CORE] [HOTFIX] Different ve...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3829#issuecomment-68315395 @srowen Oops... that's my mistake that I didn't change artifact id correctly when I make a pull request again. Thanks for fixing it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added setMinCount to Word2Vec.scala
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3693 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added setMinCount to Word2Vec.scala
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/3693#issuecomment-68315410 LGTM (including the change to `norm`). Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4156 [MLLIB] EM algorithm for GMMs
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/3022#issuecomment-68315285 @tgaloppo I've merged this into master. Thanks for contributing GMM! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4156 [MLLIB] EM algorithm for GMMs
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3022 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4939] move to next locality when no pen...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3779#issuecomment-68314408 [Test build #24870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24870/consoleFull) for PR 3779 at commit [`32d363f`](https://github.com/apache/spark/commit/32d363f275c6dea225019a1cb23af4eeeb44781f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4790][STREAMING] Fix ReceivedBlockTrack...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3726#issuecomment-68314063 [Test build #24869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24869/consoleFull) for PR 3726 at commit [`bbbacd1`](https://github.com/apache/spark/commit/bbbacd1a441e43ce46e49bea6c85c6d7834c5487). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
