[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18507 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18159 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79078/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17758 **[Test build #79075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79075/testReport)** for PR 17758 at commit [`12159c4`](https://github.com/apache/spark/commit/12159c403955f54066ed8c532ed991f829edfc1f). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class PreprocessDDLCommands(sparkSession: SparkSession) extends Rule[LogicalPlan] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18174 **[Test build #79074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79074/testReport)** for PR 18174 at commit [`f6d895c`](https://github.com/apache/spark/commit/f6d895c944c514b7e51db19388ef00016671dddb). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18481 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79079/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18159 **[Test build #79078 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79078/testReport)** for PR 18159 at commit [`f575f0c`](https://github.com/apache/spark/commit/f575f0c999647d1f7fda22652ea480f7a0d07047). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait DataWritingCommand extends RunnableCommand ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18502 **[Test build #79076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79076/testReport)** for PR 18502 at commit [`f708dde`](https://github.com/apache/spark/commit/f708ddec38917867f9f13c7136ecef28c46af3a1). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18507 **[Test build #79077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79077/testReport)** for PR 18507 at commit [`9788b19`](https://github.com/apache/spark/commit/9788b19d06800cce243a79acc189c3424912f393). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18507 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79077/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18501 **[Test build #79080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79080/testReport)** for PR 18501 at commit [`8a1a64f`](https://github.com/apache/spark/commit/8a1a64f1d1c429709799c00087dabfb97f4ca8b7). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18501 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18481 **[Test build #79079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79079/testReport)** for PR 18481 at commit [`6d5adf0`](https://github.com/apache/spark/commit/6d5adf0f10338168943542e8f2177761d2140438). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18481 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18159 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18413 **[Test build #79073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79073/testReport)** for PR 18413 at commit [`da037c8`](https://github.com/apache/spark/commit/da037c810a8c121d7075b741478419ffb77202d8). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18502 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79076/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18501 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79080/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17758 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79075/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18174 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18413 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17758 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18413 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79073/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18174 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79074/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18502 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18507 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/18481 jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitl...
GitHub user jaceklaskowski opened a pull request: https://github.com/apache/spark/pull/18509 [SS][MINOR] Make EventTimeWatermarkExec explicitly UnaryExecNode ## What changes were proposed in this pull request? Making EventTimeWatermarkExec explicitly UnaryExecNode /cc @tdas @zsxwing ## How was this patch tested? Local build. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jaceklaskowski/spark EventTimeWatermarkExec-UnaryExecNode Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18509.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18509 commit 2cc9fdca381f50cb405bf5fbcaa7229652749a83 Author: Jacek Laskowski Date: 2017-07-03T07:07:12Z [SS][MINOR] Make EventTimeWatermarkExec explicitly UnaryExecNode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18509 **[Test build #79081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79081/testReport)** for PR 18509 at commit [`2cc9fdc`](https://github.com/apache/spark/commit/2cc9fdca381f50cb405bf5fbcaa7229652749a83). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18507 **[Test build #79082 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79082/testReport)** for PR 18507 at commit [`9788b19`](https://github.com/apache/spark/commit/9788b19d06800cce243a79acc189c3424912f393). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18481 **[Test build #79083 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79083/testReport)** for PR 18481 at commit [`6d5adf0`](https://github.com/apache/spark/commit/6d5adf0f10338168943542e8f2177761d2140438). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17227: [SPARK-19507][PySpark][SQL] Show field name in _verify_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17227 @dgingrich, I tried to address my comments at my best here - https://github.com/dgingrich/spark/pull/1. Could you review that change and merge it if it looks good to you so that the change is merged into this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17758 Who calls `DDLCommand.validate`? Do you assume a new analyzer rule like `DDLCommandAnalysis`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/18174 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18159 **[Test build #79084 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79084/testReport)** for PR 18159 at commit [`8380023`](https://github.com/apache/spark/commit/8380023ab96fa6febe19918da9284fe14aea1f53). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18491: [SPARK-21268][MLLIB] Move center calculations to a distr...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18491 Is there any real performance benefit here? Doesn't seem like it unless you have really large `k`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18159 **[Test build #79085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79085/testReport)** for PR 18159 at commit [`dfaa4f1`](https://github.com/apache/spark/commit/dfaa4f11f35e2615ce627f29b1a60ab27dfd5477). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18481 **[Test build #79083 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79083/testReport)** for PR 18481 at commit [`6d5adf0`](https://github.com/apache/spark/commit/6d5adf0f10338168943542e8f2177761d2140438). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18481 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18481 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79083/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/18510 [SPARK-21284][SQL] rename SessionCatalog.registerFunction parameter name ## What changes were proposed in this pull request? Looking at the code in `SessionCatalog.registerFunction`, the parameter `ignoreIfExists` has a wrong name. When `ignoreIfExists` is true, we will override the function if it already exists. So `overrideIfExists` should be the corrected name. ## How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark minor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18510.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18510 commit a518bdffb951c93418e2e4ea3fd4734255808ead Author: Wenchen Fan Date: 2017-07-03T07:52:58Z rename SessionCatalog.registerFunction parameter name --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18510 cc @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18510#discussion_r125228460 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -1104,10 +1104,10 @@ class SessionCatalog( */ def registerFunction( --- End diff -- @ouyangxiaochen , you can call this method in https://github.com/apache/spark/pull/17681/files#diff-d2a203f08c862bd762e6740c16e972f7R84 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18510 **[Test build #79086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79086/testReport)** for PR 18510 at commit [`a518bdf`](https://github.com/apache/spark/commit/a518bdffb951c93418e2e4ea3fd4734255808ead). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18510 **[Test build #79086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79086/testReport)** for PR 18510 at commit [`a518bdf`](https://github.com/apache/spark/commit/a518bdffb951c93418e2e4ea3fd4734255808ead). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18510 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18510 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79086/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17758 Actually after more thoughts, I think it's ok to do the validation in `RunnableCommand.run`, as it's still done in the driver side. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/18413 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18413 **[Test build #79087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79087/testReport)** for PR 18413 at commit [`da037c8`](https://github.com/apache/spark/commit/da037c810a8c121d7075b741478419ffb77202d8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18507 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18474: [SPARK-21235][TESTS] UTest should clear temp results whe...
Github user wangjiaochun commented on the issue: https://github.com/apache/spark/pull/18474 1. Test environment and test methodï¼IDEA project direct Run BlockManagerSuite.scala. 2. I test this case again use step through,find this case encryptionTest("on-disk storage") Runs a test twice, if SparkConf object with encryption off(false), the disk blocks will clear. if encryption is on(set ture), disk blocks not clearã --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18507: [SPARK-21283][core]FileOutputStream should be cre...
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/18507#discussion_r125237381 --- Diff: core/src/test/scala/org/apache/spark/util/FileAppenderSuite.scala --- @@ -52,10 +52,12 @@ class FileAppenderSuite extends SparkFunSuite with BeforeAndAfter with Logging { test("basic file appender") { val testString = (1 to 1000).mkString(", ") val inputStream = new ByteArrayInputStream(testString.getBytes(StandardCharsets.UTF_8)) +val header = "Add header" --- End diff -- Please add comment before this to explain the change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18511: [SPARK-21286][Test] Modified a unit test
GitHub user Geek-He opened a pull request: https://github.com/apache/spark/pull/18511 [SPARK-21286][Test] Modified a unit test ## What changes were proposed in this pull request? The old unit test not effect ## How was this patch tested? unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/Geek-He/spark dev_0703 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18511.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18511 commit 5c4279ddca49f36635658c8fbe8b37ecb31ecca4 Author: he.qiao Date: 2017-07-03T08:12:55Z ä¿®æ¹ä¸å¤åå æµè¯é误 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18510#discussion_r125238823 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala --- @@ -60,7 +60,7 @@ case class CreateFunctionCommand( // We first load resources and then put the builder in the function registry. // Please note that it is allowed to overwrite an existing temp function. catalog.loadFunctionResources(resources) - catalog.registerFunction(func, ignoreIfExists = false) + catalog.registerFunction(func, overrideIfExists = false) --- End diff -- This looks a bit weird. The above comment says `Please note that it is allowed to overwrite an existing temp function.`. But we don't override it if it exists. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18511: [SPARK-21286][Test] Modified a unit test
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18511 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17985: Add "full_outer" name to join types
Github user BartekH commented on the issue: https://github.com/apache/spark/pull/17985 It has failed again. It's not my fault, please retest it one more time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18484: [SPARK-21264][PYTHON] Call cross join path in joi...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18484#discussion_r125240048 --- Diff: python/pyspark/sql/tests.py --- @@ -2021,6 +2021,11 @@ def test_toDF_with_schema_string(self): self.assertEqual(df.schema.simpleString(), "struct") self.assertEqual(df.collect(), [Row(key=i) for i in range(100)]) +def test_join_without_on(self): +self.assertRaises( +AnalysisException, +lambda: self.spark.range(1).join(self.spark.range(1), how="inner").collect()) --- End diff -- Sure. Thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18484 **[Test build #79088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79088/testReport)** for PR 18484 at commit [`48a92d1`](https://github.com/apache/spark/commit/48a92d1166e9ab13ef8926885b12c5e513148115). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18499: [SPARK-21176][WEB UI] Use a single ProxyServlet to proxy...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18499 cc @gurvindersingh Could you look at this please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18308: [SPARK-21099][Spark Core] INFO Log Message Using Incorre...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18308 Is this the final modified code @ihazem , why do you check `hasCachedBlocks` both inside and outside of logInfo statement? Also the code is too long. Can you please at least do a round of self-review before pushing the changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18509 **[Test build #79081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79081/testReport)** for PR 18509 at commit [`2cc9fdc`](https://github.com/apache/spark/commit/2cc9fdca381f50cb405bf5fbcaa7229652749a83). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18509 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18509 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79081/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18484 **[Test build #79088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79088/testReport)** for PR 18484 at commit [`48a92d1`](https://github.com/apache/spark/commit/48a92d1166e9ab13ef8926885b12c5e513148115). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18484 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79088/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18512: [SPARK-20946][SQL] simplify the config setting lo...
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/18512 [SPARK-20946][SQL] simplify the config setting logic in SparkSession.getOrCreate ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dongjoon-hyun/spark SPARK-20946 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18512.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18512 commit fe7bdad1530dadefb0283fac597137a3db2538c1 Author: Wenchen Fan Date: 2017-06-01T05:17:05Z simplify the config setting logic in SparkSession.getOrCreate commit b84806b78c96564fb7f093b679979ef4cd733880 Author: Wenchen Fan Date: 2017-06-02T03:41:42Z address comments commit 705ccc007817bdcb9b87ef0141b16ae03e760ef6 Author: Dongjoon Hyun Date: 2017-07-03T09:30:31Z fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18484 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18512: [SPARK-20946][SQL] simplify the config setting logic in ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18512 **[Test build #79089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79089/testReport)** for PR 18512 at commit [`705ccc0`](https://github.com/apache/spark/commit/705ccc007817bdcb9b87ef0141b16ae03e760ef6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18441: [SPARK-21137][CORE] Spark reads many small files ...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18441#discussion_r125247939 --- Diff: core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala --- @@ -35,8 +36,12 @@ private[spark] class BinaryFileRDD[T]( extends NewHadoopRDD[String, T](sc, inputFormatClass, keyClass, valueClass, conf) { override def getPartitions: Array[Partition] = { -val inputFormat = inputFormatClass.newInstance val conf = getConf +// setMinPartitions below will call FileInputFormat.listStatus(), which can be quite slow when +// traversing a large number of directories and files. Parallelize it. +conf.setIfUnset(FileInputFormat.LIST_STATUS_NUM_THREADS, + Runtime.getRuntime.availableProcessors().toString) --- End diff -- +1. This code aggressively tries to use all of the cores on all of the Spark instances on a node. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18507 **[Test build #79090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79090/testReport)** for PR 18507 at commit [`139e16c`](https://github.com/apache/spark/commit/139e16c04f7532101c8fd44983fe42306c7bb5fc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user ouyangxiaochen commented on a diff in the pull request: https://github.com/apache/spark/pull/17681#discussion_r125250502 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -1056,6 +1056,27 @@ class SessionCatalog( } /** + * overwirte a metastore function in the database specified in `funcDefinition`.. + * If no database is specified, assume the function is in the current database. + */ + def alterFunction(funcDefinition: CatalogFunction): Unit = { +val db = formatDatabaseName(funcDefinition.identifier.database.getOrElse(getCurrentDatabase)) +requireDbExists(db) +val identifier = FunctionIdentifier(funcDefinition.identifier.funcName, Some(db)) +val newFuncDefinition = funcDefinition.copy(identifier = identifier) +if (functionExists(identifier)) { + if (functionRegistry.functionExists(identifier)) { +// If we have loaded this function into the FunctionRegistry, +// also drop it from there. +// For a permanent function, because we loaded it to the FunctionRegistry +// when it's first used, we also need to drop it from the FunctionRegistry. +functionRegistry.dropFunction(identifier) + } + externalCatalog.alterFunction(db, newFuncDefinition) +} --- End diff -- yes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18507: [SPARK-21283][core]FileOutputStream should be cre...
Github user 10110346 commented on a diff in the pull request: https://github.com/apache/spark/pull/18507#discussion_r125250432 --- Diff: core/src/test/scala/org/apache/spark/util/FileAppenderSuite.scala --- @@ -52,10 +52,12 @@ class FileAppenderSuite extends SparkFunSuite with BeforeAndAfter with Logging { test("basic file appender") { val testString = (1 to 1000).mkString(", ") val inputStream = new ByteArrayInputStream(testString.getBytes(StandardCharsets.UTF_8)) +val header = "Add header" --- End diff -- ok,thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18507 **[Test build #79091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79091/testReport)** for PR 18507 at commit [`a22951f`](https://github.com/apache/spark/commit/a22951f341901381480c04eae035e06bf9465bbe). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18513: [SPARK-13969][ML] Add FeatureHasher transformer
GitHub user MLnick opened a pull request: https://github.com/apache/spark/pull/18513 [SPARK-13969][ML] Add FeatureHasher transformer This PR adds a `FeatureHasher` transformer, modeled on [scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html) and [Vowpal wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki/Feature-Hashing-and-Extraction). The transformer operates on multiple input columns in one pass. Current behavior is: * for numerical columns, the values are assumed to be real values and the feature index is `hash(columnName)` while feature value is `feature_value` * for string columns, the values are assumed to be categorical and the feature index is `hash(column_name=feature_value)`, while feature value is `1.0` * For hash collisions, feature values will be summed * `null` (missing) values are ignored The following dataframe illustrates the basic semantics: ``` +---+--+-+-+--+-+ |int|double|float|stringNum|string|features | +---+--+-+-+--+-+ |3 |4.0 |5.0 |1|foo |(16,[0,8,11,12,15],[5.0,3.0,1.0,4.0,1.0])| |6 |7.0 |8.0 |2|bar |(16,[0,8,11,12,15],[8.0,6.0,1.0,7.0,1.0])| +---+--+-+-+--+-+ ``` ## How was this patch tested? New unit tests and manual experiments. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MLnick/spark FeatureHasher Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18513 commit 6ab19a963f35de29af0a6b7b1598d5add78f200a Author: Nick Pentreath Date: 2016-08-23T10:29:06Z initial WIP commit ebd2cbf3467f26121c602f7c77c2018253cbdf18 Author: Nick Pentreath Date: 2017-02-01T10:43:07Z Further work commit ba255bfda792d58aaded892e49c6cf48f0391159 Author: Nick Pentreath Date: 2017-06-22T10:52:12Z Clean up commit 0be1e6572110d7d550f69fd86d3dd4e96660fde6 Author: Nick Pentreath Date: 2017-06-22T10:52:37Z Add tests commit 2f3ea21e2e1835d7218e8c7bd096cc0787ed595c Author: Nick Pentreath Date: 2017-06-22T13:08:26Z Copy, save/load, clean up commit 7d678fbf5f88d377b79153212a3e0a2596039b17 Author: Nick Pentreath Date: 2017-06-26T12:38:02Z Move numFeatures to HasNumFeatures shared trait commit 60572776de80ebcf1782c3d7def749557c8bec61 Author: Nick Pentreath Date: 2017-07-03T07:18:25Z Update shared params from codegen run commit 9edb3bda8cbc4e00f05b91718249edf2750fc028 Author: Nick Pentreath Date: 2017-07-03T09:32:32Z Update tests. Null values ignored in feature hashing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18491: [SPARK-21268][MLLIB] Move center calculations to a distr...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18491 I doubt it makes much difference, yeah. It pushes a little computation off the driver, and computation that happens on each iteration. It seemed like a tiny win that didn't complicate things. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17227: [SPARK-19507][PySpark][SQL] Show field name in _verify_t...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17227 Wait ... this is possibly a hot path that requires row-per-operation ... @ueshin and @dgingrich, I think we should re-write this to avoid record-per-type dispatch ... For me, I wouldn't mind if we go and merge my approach, dgingrich#1 here for now. I will make a follow up right after it gets merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18512: [SPARK-20946][SQL] Simplify the config setting logic in ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18512 Hi, @cloud-fan and @gatorsmile . Could you review this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18513 **[Test build #79092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79092/testReport)** for PR 18513 at commit [`9edb3bd`](https://github.com/apache/spark/commit/9edb3bda8cbc4e00f05b91718249edf2750fc028). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18512: [SPARK-20946][SQL] Simplify the config setting lo...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/18512#discussion_r125252781 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -904,22 +906,14 @@ object SparkSession { // No active nor global default session. Create a new one. val sparkContext = userSuppliedContext.getOrElse { - // set app name if not given - val randomAppName = java.util.UUID.randomUUID().toString val sparkConf = new SparkConf() options.foreach { case (k, v) => sparkConf.set(k, v) } --- End diff -- This line is changed. `options` are copied into new SparkConf. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...
Github user ouyangxiaochen commented on a diff in the pull request: https://github.com/apache/spark/pull/17681#discussion_r125253313 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala --- @@ -590,6 +590,14 @@ class InMemoryCatalog( catalog(db).functions.remove(funcName) } + override protected def doAlterFunction(db: String, func: CatalogFunction): Unit = synchronized { +requireDbExists(db) +requireFunctionExists(db, func.identifier.funcName) +catalog(db).functions.remove(func.identifier.funcName) +requireFunctionNotExists(db, func.identifier.funcName) --- End diff -- Ah, there is really no need to do this. OK. remove it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18507 **[Test build #79082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79082/testReport)** for PR 18507 at commit [`9788b19`](https://github.com/apache/spark/commit/9788b19d06800cce243a79acc189c3424912f393). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18507 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79082/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18507 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18513 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79092/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18513 **[Test build #79092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79092/testReport)** for PR 18513 at commit [`9edb3bd`](https://github.com/apache/spark/commit/9edb3bda8cbc4e00f05b91718249edf2750fc028). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18513 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18513 **Note 1**: this is distinct from `HashingTF` which handles vectorizing text to term frequencies (analogous to [HashingVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html)). Thie feature hasher _could_ be extended to also handle `Seq[String]` input columns. But I feel it conflates concerns - e.g. `HashingTF` handles min term frequencies, binarization etc. However we could later add basic support for `Seq[String]` columns - this would handle raw text in a similar way to Vowpal Wabbit, i.e. it all gets hashed into one feature vector (can be combined with namespaces later). **Note 2**: some potential follow ups: * support specifying categorical columns explicitly. This would be to allow forcing some columns that are in numerical format to be treated as categorical. Strings would still be treated as categorical. * support using the sign of hashed value as sign of feature value, and then support `non_negative` param (see [scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html)) * support feature namespaces and feature interactions similar to [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki/Feature-interactions) (see [here](https://gist.github.com/luoq/b4c374b5cbabe3ae76ffacdac22750af) for an outline of the code used). This could provide an efficient and scalable form of `PolynomialExpansion`. cc @srowen @jkbradley @sethah @hhbyyh @yanboliang @BryanCutler @holdenk --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18513 I've moved `HashingTF` `numFeatures` param to `sharedParams` which results in the MiMa failure since it would now be marked `final`. Can't quite recall what we've done previously in this case - whether we accept that it breaks user code, but that in most cases users should not have really been extending or overriding these params. Or leave it as is. I'm ok with the latter - `numFeatures` is not really that necessary to be a shared param. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18491: [SPARK-21268][MLLIB] Move center calculations to ...
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/18491#discussion_r125256928 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala --- @@ -292,15 +292,16 @@ class KMeans private ( }.reduceByKey { case ((sum1, count1), (sum2, count2)) => axpy(1.0, sum2, sum1) (sum1, count1 + count2) + }.mapValues { case (sum, count) => +scal(1.0 / count, sum) +new VectorWithNorm(sum) }.collectAsMap() bcCenters.destroy(blocking = false) // Update the cluster centers and costs converged = true - totalContribs.foreach { case (j, (sum, count)) => -scal(1.0 / count, sum) -val newCenter = new VectorWithNorm(sum) + totalContribs.foreach { case (j, newCenter) => --- End diff -- perhaps the variable name should change from `totalContribs` to `newCenters`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18491: [SPARK-21268][MLLIB] Move center calculations to a distr...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18491 Fair enough. LGTM with small comment about possibly changing the variable name? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18502 Retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18460: [SPARK-21247][SQL] Allow case-insensitive type equality ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18460 Hi, @cloud-fan . Could you review this, too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18502 **[Test build #79093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79093/testReport)** for PR 18502 at commit [`f708dde`](https://github.com/apache/spark/commit/f708ddec38917867f9f13c7136ecef28c46af3a1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18159 **[Test build #79084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79084/testReport)** for PR 18159 at commit [`8380023`](https://github.com/apache/spark/commit/8380023ab96fa6febe19918da9284fe14aea1f53). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...
Github user ouyangxiaochen commented on a diff in the pull request: https://github.com/apache/spark/pull/18510#discussion_r125258324 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala --- @@ -1104,10 +1104,10 @@ class SessionCatalog( */ def registerFunction( --- End diff -- OK, I will do this change after your PR is merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18159 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18159 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79084/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18159 **[Test build #79085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79085/testReport)** for PR 18159 at commit [`dfaa4f1`](https://github.com/apache/spark/commit/dfaa4f11f35e2615ce627f29b1a60ab27dfd5477). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18159 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18491: [SPARK-21268][MLLIB] Move center calculations to a distr...
Github user dardelet commented on the issue: https://github.com/apache/spark/pull/18491 @MLnick Great ! Updated variable name --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org