[GitHub] spark pull request: [SPARK-8516][SQL]create table to specific data...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6695#issuecomment-109881898 [Test build #34409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34409/console) for PR 6695 at commit [`3d19ad9`](https://github.com/apache/spark/commit/3d19ad9ad5885fa5cf7da0c733dca9152fb15655). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8516][SQL]create table to specific data...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6695#issuecomment-109881928 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][minor] remove duplicated cases in `Decim...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6698#issuecomment-109881681 [Test build #34414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34414/consoleFull) for PR 6698 at commit [`413ad4a`](https://github.com/apache/spark/commit/413ad4a3c6f07bde96142fa47090b6e131b19b2e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][minor] remove duplicated cases in `Decim...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6698#issuecomment-109881633 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][minor] remove duplicated cases in `Decim...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6698#issuecomment-109881642 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8157] [SQL] should use expression.unapp...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/6697#issuecomment-109881591 If you try to write a regression test, you'll find this bug doesn't break anything... Actually we will handle this case in `WidenTypes` so it's safe to remove them. See https://github.com/apache/spark/pull/6698 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SQL] UDAF Interface Refactoring
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5542#issuecomment-109881409 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SQL] UDAF Interface Refactoring
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5542#issuecomment-109881408 [Test build #34413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34413/console) for PR 5542 at commit [`f0f907f`](https://github.com/apache/spark/commit/f0f907f4248f90d149986316e5197ef6c49576a9). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][minor] remove duplicated cases in `Decim...
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/6698 [SQL][minor] remove duplicated cases in `DecimalPrecision` We already have a rule to do type coercion for fixed decimal and unlimited decimal in `WidenTypes`, so we don't need to handle them in `DecimalPrecision`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6698.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6698 commit 413ad4a3c6f07bde96142fa47090b6e131b19b2e Author: Wenchen Fan Date: 2015-06-08T06:36:59Z remove duplicated cases --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SQL] UDAF Interface Refactoring
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5542#issuecomment-109881285 [Test build #34413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34413/consoleFull) for PR 5542 at commit [`f0f907f`](https://github.com/apache/spark/commit/f0f907f4248f90d149986316e5197ef6c49576a9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SQL] UDAF Interface Refactoring
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5542#issuecomment-109881231 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SQL] UDAF Interface Refactoring
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5542#issuecomment-109881214 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8139] [SQL] Updates docs and comments o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6683#issuecomment-109881069 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8139] [SQL] Updates docs and comments o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6683#issuecomment-109881061 [Test build #34408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34408/console) for PR 6683 at commit [`9c168e2`](https://github.com/apache/spark/commit/9c168e23a6058ea509df2a5b2bf733985ee0bd30). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8157] [SQL] should use expression.unapp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6697#issuecomment-109880840 [Test build #34412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34412/consoleFull) for PR 6697 at commit [`d56c97a`](https://github.com/apache/spark/commit/d56c97aab615b9ba7be0c99b36f27b90fad9687c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8157] [SQL] should use expression.unapp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6697#issuecomment-109880540 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8157] [SQL] should use expression.unapp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6697#issuecomment-109880579 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8157] [SQL] should use expression.unapp...
GitHub user adrian-wang opened a pull request: https://github.com/apache/spark/pull/6697 [SPARK-8157] [SQL] should use expression.unapply to match in HiveTypeCoercion fix bug for #6405 You can merge this pull request into a Git repository by running: $ git pull https://github.com/adrian-wang/spark typebug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6697.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6697 commit d56c97aab615b9ba7be0c99b36f27b90fad9687c Author: Daoyuan Wang Date: 2015-06-08T06:30:43Z fix bug for #6405 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109879801 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7939][SQL] Add conf to enable/disable p...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6503#issuecomment-109879586 [Test build #34411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34411/consoleFull) for PR 6503 at commit [`3e90470`](https://github.com/apache/spark/commit/3e90470ac7e9058e497e829d0cd58b1fc5299e26). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7939][SQL] Add conf to enable/disable p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6503#issuecomment-109879207 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7939][SQL] Add conf to enable/disable p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6503#issuecomment-109879193 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-8153 Add configuration for disabling par...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6696#issuecomment-109879190 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-8153 Add configuration for disabling par...
GitHub user navis opened a pull request: https://github.com/apache/spark/pull/6696 SPARK-8153 Add configuration for disabling partial aggregation in runtime Same thing with "hive.map.aggr.hash.min.reduction" in hive, which disables hash aggregation if it's not sufficiently decreasing the output size. Added two configuration - spark.sql.partial.aggregation.checkInterval - spark.sql.partial.aggregation.minReduction You can merge this pull request into a Git repository by running: $ git pull https://github.com/navis/spark SPARK-8153 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6696.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6696 commit 388ea7a5e4f5ee3636a63da33ce6b6a5b7f486f9 Author: navis.ryu Date: 2015-06-08T01:32:42Z SPARK-8153 Add configuration for disabling partial aggregation in runtime --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user cloud-fan closed the pull request at: https://github.com/apache/spark/pull/6574 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8118] [SQL] Mutes noisy Parquet log out...
Github user liancheng closed the pull request at: https://github.com/apache/spark/pull/6670 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8118] [SQL] Mutes noisy Parquet log out...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/6670#issuecomment-109878410 Closing this manually. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [DOC][TYPO] Fix typo in standalone deploy scri...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6691#issuecomment-109877933 Since this has been merged, @yijieshen can you close the pull request? I'm not sure why it is not closed automatically by github. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109876687 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6694 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8149][SQL] Break ExpressionEvaluationSu...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6693 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109876717 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109876168 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109876016 Please close this pull request also (since it's already been merged). You can see the up-to-date merges here: https://git-wip-us.apache.org/repos/asf?p=spark.git --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109875959 Can you submit a separate pull request? This patch has actually been merged already, but ASF's github sync is behind so it is not showing up here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8117] [SQL] Push codegen implementation...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6690#issuecomment-109875789 It's a problem of the test case that manifests from this. We shouldn't use a double value in the place where we intend to have a float value. Let's be more careful with that in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8148] Do not use FloatType in partition...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6692#issuecomment-109875697 If it is persisted in the metastore, then inference is no longer used, isn't it? And why would partition columns be stored in Parquet file metadata? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109875293 I've no idea why it's screwed up. Feel free to open a new one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7939][SQL] Add conf to enable/disable p...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/6503#issuecomment-109874676 This LGTM except for the default value of the added configuration. Also, would you mind to add this to the SQL programming guide? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRel...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6669#issuecomment-109871870 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRel...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6669#issuecomment-109871842 [Test build #34407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34407/console) for PR 6669 at commit [`fce089c`](https://github.com/apache/spark/commit/fce089c73d1ed5e1c5f7549e1de0dac104769157). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` logInfo(s"Using user defined output committer class $` * ` logInfo(s"Using output committer class $` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2205][SPARK-7871][SQL]Advoid redundancy...
Github user jeanlyn commented on the pull request: https://github.com/apache/spark/pull/6682#issuecomment-109871137 @yhuai .Yes,the full outer join cases shuffled the null key to the same reducer in spark-sql ,and the hive plan generated like: ```sql explain select a.value,b.value,c.value,d.value from a full outer join b on a.key = b.key full outer join c on a.key = c.key full outer join d on a.key = d.key STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: a Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Reduce Output Operator key expressions: key (type: string) sort order: + Map-reduce partition columns: key (type: string) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE value expressions: value (type: string) TableScan alias: b Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Reduce Output Operator key expressions: key (type: string) sort order: + Map-reduce partition columns: key (type: string) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE value expressions: value (type: string) TableScan alias: c Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Reduce Output Operator key expressions: key (type: string) sort order: + Map-reduce partition columns: key (type: string) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE value expressions: value (type: string) TableScan alias: d Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Reduce Output Operator key expressions: key (type: string) sort order: + Map-reduce partition columns: key (type: string) Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE value expressions: value (type: string) Reduce Operator Tree: Join Operator condition map: Outer Join 0 to 1 Outer Join 0 to 2 Outer Join 0 to 3 keys: 0 key (type: string) 1 key (type: string) 2 key (type: string) 3 key (type: string) outputColumnNames: _col1, _col6, _col11, _col16 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: _col1 (type: string), _col6 (type: string), _col11 (type: string), _col16 (type: string) outputColumnNames: _col0, _col1, _col2, _col3 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink ``` @chenghao-intel has a solution in #6413 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7939][SQL] Add conf to enable/disable p...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/6503#discussion_r31887868
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala ---
@@ -71,6 +71,9 @@ private[spark] object SQLConf {
// Whether to perform partition discovery when loading external data
sources. Default to true.
val PARTITION_DISCOVERY_ENABLED =
"spark.sql.sources.partitionDiscovery.enabled"
+ // Whether to perform partition column type inference. Default to false.
+ val PARTITION_COLUMN_TYPE_INFERENCE =
"spark.sql.sources.partitionColumnTypeInference.enabled"
--- End diff --
We probably want to set this to `true` to keep the default behavior
backwards compatible with Spark 1.3.x.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8148] Do not use FloatType in partition...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/6692#issuecomment-109867230 I'm worrying about skipping `FloatType` (and possibly `IntegerType`) might break existing user code because the partition column data types gets changed. Especially when the inferred schema gets persisted in places like Parquet file metadata and metastore. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8516][SQL]create table to specific data...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6695#issuecomment-109866924 [Test build #34409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34409/consoleFull) for PR 6695 at commit [`3d19ad9`](https://github.com/apache/spark/commit/3d19ad9ad5885fa5cf7da0c733dca9152fb15655). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8516][SQL]create table to specific data...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6695#issuecomment-109865817 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8516][SQL]create table to specific data...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6695#issuecomment-109865877 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8117] [SQL] Push codegen implementation...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/6690#issuecomment-109865812 @rxin The test failure was because this PR overrides `Literal.equals` which [uses `value.equals(o.value)`] [1]. A quick Scala REPL check: ``` scala> 1.5 == 1.5f res2: Boolean = true scala> 1.5.equals(1.5f) res3: Boolean = false ``` [1]: https://github.com/apache/spark/pull/6690/files#diff-f52e4a77ff9291d86359d609a9757781R85 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8516][SQL]create table to specific data...
GitHub user baishuo opened a pull request:
https://github.com/apache/spark/pull/6695
[SPARK-8516][SQL]create table to specific database by 'use dbname'
when i test the following code:
hiveContext.sql("""use testdb""")
val df = (1 to 3).map(i => (i, s"val_$i", i * 2)).toDF("a", "b", "c")
df.write
.format("parquet")
.mode(SaveMode.Overwrite)
.saveAsTable("ttt3")
hiveContext.sql("show TABLES in default")
found that the table ttt3 will be created under the database "default"
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/baishuo/spark SPARK-8516-use-database
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6695.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6695
commit 3d19ad9ad5885fa5cf7da0c733dca9152fb15655
Author: baishuo
Date: 2015-06-08T05:19:38Z
create table to specific database
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31887692
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -59,6 +59,32 @@ object HiveTypeCoercion {
}
/**
+ * Implicit promote the AtomicType to StringType if
+ * one of the data type in (dt1, dt2) is StringType, and the other is
not either
+ * BooleanType or BinaryType, the TightestCommonType should be StringType
+ * eg: 1. CaseWhenLike case when ... then dt1 else dt2 end
+ * 2. Coalesce(null, dt1, dt2)
+ */
+ private def promoteToStringType(t1: DataType, t2: DataType):
Option[DataType] = (t1, t2) match {
--- End diff --
Here we don't care about the parameter names(i.e. `t1` and `t2`). Using
functions, we can just write the partially applied function(i.e. `{case ...}`)
and don't need to add `(t1, t2) match` before it.
However, it's not a big deal, using method is OK, but we should make it
consistent with existing code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user dibbhatt commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109862250 @tdas @zsxwing , if this PR is screwed up , we can close this one and create a fresh one ? How to proceed further ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109862068 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109862060 [Test build #34406 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34406/console) for PR 6574 at commit [`d7b60c8`](https://github.com/apache/spark/commit/d7b60c897b3295c5fd96f06435552926ab289db9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class HasCachedBlocks(executorId: String) extends ToBlockManagerMaster` * `class KernelDensity(object):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8139] [SQL] Updates docs and comments o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6683#issuecomment-109857702 [Test build #34408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34408/consoleFull) for PR 6683 at commit [`9c168e2`](https://github.com/apache/spark/commit/9c168e23a6058ea509df2a5b2bf733985ee0bd30). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user dibbhatt commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109857699 hi @zsxwing ...No , I have made only commit for my changes . Not sure why it says 219 files changed. If you see the commits details , you can see only files related to this PR has changed. Just now I modified comments on a file to trigger the build once again. This 219 files changed came after I merged my repo from upstream/master to take your PR changes. And I merged only your changed with mine and committed those . Below are the my changes since the merge...and I have not committed all these 219 files :( https://github.com/dibbhatt/spark/commit/0892156a1f08e3092d924e57a77a871a4843b016 https://github.com/dibbhatt/spark/commit/c250fb5fb6a08d89b0aed05b5d3eec44b9991c01 https://github.com/apache/spark/commit/28225d58846e6c214fbc6f7e1ef09cda6b7cd94d --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8139] [SQL] Updates docs and comments o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6683#issuecomment-109857650 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8139] [SQL] Updates docs and comments o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6683#issuecomment-109857662 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109857333 [Test build #34403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34403/consoleFull) for PR 6614 at commit [`28225d5`](https://github.com/apache/spark/commit/28225d58846e6c214fbc6f7e1ef09cda6b7cd94d). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109857368 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7993] [SQL] Improved DataFrame.show() o...
Github user akhilthatipamula commented on the pull request: https://github.com/apache/spark/pull/6633#issuecomment-109855511 @rxin I have put both functios inline. Can you check this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109855357 [Test build #890 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/890/consoleFull) for PR 6694 at commit [`4e5056c`](https://github.com/apache/spark/commit/4e5056ccc2c668ec2c417876420215b876918de8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109855254 [Test build #34404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34404/console) for PR 6694 at commit [`4e5056c`](https://github.com/apache/spark/commit/4e5056ccc2c668ec2c417876420215b876918de8). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109855258 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31886263
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -39,6 +39,16 @@ class SQLQuerySuite extends QueryTest with
BeforeAndAfterAll with SQLTestUtils {
val sqlContext = TestSQLContext
import sqlContext.implicits._
+ test("SPARK-8010: promote numeric to string") {
+val df = Seq((1, 1)).toDF("key", "value")
+df.registerTempTable("src")
+val queryCaseWhen = sql("select case when true then 1.0 else '1' end
from src ")
+val queryCoalesce = sql("select coalesce(null, 1, '1') from src ")
--- End diff --
@yhuai, actually we often write querys by using udf in then value and else
value, like below:
`select case when boolean then split(city_code, ',')[0] else -99 end from
tablename `
Hive will implicit convert the case when expression value to a string type
since `split function` returns string type but `else value` is a integer.
Spark sql current will throw exceptions because the types of then value and
else value is not convertible.
Why we use `StringType` is because when do implicit conversion in
AtomicType, almost every type meets `StringType` will be converted to
`StringType` except `BinaryType` and `BooleanType`
You can refer the chart at the bottom of the page:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRel...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/6669#discussion_r31886183
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/commands.scala ---
@@ -350,6 +356,7 @@ private[sql] abstract class BaseWriterContainer(
}.getOrElse {
// If output committer class is not set, we will use the one
associated with the
// file output format.
+ logInfo(s"Using output committer class
${outputFormatClass.getCanonicalName}")
--- End diff --
Maybe we can get the instance of `outputCommitter` first and then log its
class?
```
val outputCommitter = ...
logInfo(...)
outputCommitter
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRel...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6669#issuecomment-109854037 [Test build #34407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34407/consoleFull) for PR 6669 at commit [`fce089c`](https://github.com/apache/spark/commit/fce089c73d1ed5e1c5f7549e1de0dac104769157). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRel...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/6669#discussion_r31886154
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/commands.scala ---
@@ -350,6 +356,7 @@ private[sql] abstract class BaseWriterContainer(
}.getOrElse {
// If output committer class is not set, we will use the one
associated with the
// file output format.
+ logInfo(s"Using output committer class
${outputFormatClass.getCanonicalName}")
--- End diff --
This one seems not right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRel...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6669#issuecomment-109853442 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8121] [SQL] Fixes InsertIntoHadoopFsRel...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6669#issuecomment-109853450 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31886041
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -59,6 +59,32 @@ object HiveTypeCoercion {
}
/**
+ * Implicit promote the AtomicType to StringType if
+ * one of the data type in (dt1, dt2) is StringType, and the other is
not either
+ * BooleanType or BinaryType, the TightestCommonType should be StringType
+ * eg: 1. CaseWhenLike case when ... then dt1 else dt2 end
+ * 2. Coalesce(null, dt1, dt2)
+ */
+ private def promoteToStringType(t1: DataType, t2: DataType):
Option[DataType] = (t1, t2) match {
--- End diff --
@cloud-fan I'm sorry, I don't quite understand when to use a function or
when to use a method. And what's the purpose here to use a function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31885861
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -39,6 +39,16 @@ class SQLQuerySuite extends QueryTest with
BeforeAndAfterAll with SQLTestUtils {
val sqlContext = TestSQLContext
import sqlContext.implicits._
+ test("SPARK-8010: promote numeric to string") {
+val df = Seq((1, 1)).toDF("key", "value")
+df.registerTempTable("src")
+val queryCaseWhen = sql("select case when true then 1.0 else '1' end
from src ")
+val queryCoalesce = sql("select coalesce(null, 1, '1') from src ")
--- End diff --
Looks like these two test queries are not really good examples. We can
determine the data type at compile time since `coalesce` and `case when ...` at
here basically represent a constant value. Actually, for these cases, why we
use `StringType` as the data type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31885774
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -59,6 +59,32 @@ object HiveTypeCoercion {
}
/**
+ * Implicit promote the AtomicType to StringType if
+ * one of the data type in (dt1, dt2) is StringType, and the other is
not either
+ * BooleanType or BinaryType, the TightestCommonType should be StringType
+ * eg: 1. CaseWhenLike case when ... then dt1 else dt2 end
+ * 2. Coalesce(null, dt1, dt2)
+ */
+ private def promoteToStringType(t1: DataType, t2: DataType):
Option[DataType] = (t1, t2) match {
--- End diff --
cc @rxin , [findTightestCommonTypeOfTwo]
(https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala#L44)
is a function and I think we should make them consistent...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-109849268 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109849269 @dibbhatt did you modify any commit by accident, or is it a GitHub bug? There are 219 files changed and it still contains conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-109849265 [Test build #34405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34405/console) for PR 6644 at commit [`2ed0614`](https://github.com/apache/spark/commit/2ed06148fc47b251fb06a3557174ba2fa03de667). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109848862 It's a mistake made in https://github.com/apache/spark/pull/6405, we should use `DecimalType.Expression` instead of `DecimalType.Fixed` to match expression. Sorry about it... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109848787 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109848817 [Test build #34406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34406/consoleFull) for PR 6574 at commit [`d7b60c8`](https://github.com/apache/spark/commit/d7b60c897b3295c5fd96f06435552926ab289db9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7952][SQL] use internal Decimal instead...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6574#issuecomment-109848780 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-109848646 [Test build #34405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34405/consoleFull) for PR 6644 at commit [`2ed0614`](https://github.com/apache/spark/commit/2ed06148fc47b251fb06a3557174ba2fa03de667). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-109848298 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6644#issuecomment-109848291 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8124] [SPARKR] [WIP] Created more examp...
Github user Emaasit commented on the pull request: https://github.com/apache/spark/pull/6668#issuecomment-109847221 @shivaram I wanted to provide two options for creating DataFrames. One where R users can convert their local dataframes into DataFrames and the second using the read.df(). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8100][UI]Make able to refer lost execut...
Github user suyanNone commented on a diff in the pull request:
https://github.com/apache/spark/pull/6644#discussion_r31885371
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
---
@@ -85,7 +88,7 @@ private[ui] class ExecutorsPage(
{if (logsExist) Logs else
Seq.empty}
- {if (threadDumpEnabled) Thread
Dump else Seq.empty}
--- End diff --
I not remove that actually, Because for the lost executor, it can't do
dump,
So I change that column head to "Status", for executor live, will show
dump, for executor lost, will show a "Removed" status.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109845092 [Test build #34404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34404/consoleFull) for PR 6694 at commit [`4e5056c`](https://github.com/apache/spark/commit/4e5056ccc2c668ec2c417876420215b876918de8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109845001 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109844953 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6694#issuecomment-109844957 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8154][SQL] Remove Term/Code type aliase...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/6694 [SPARK-8154][SQL] Remove Term/Code type aliases in code generation. From my perspective as a code reviewer, I find them more confusing than using String directly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-8154 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6694.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6694 commit f74be744d41586690e73ec57e5551c1fbabc1d6f Author: Reynold Xin Date: 2015-06-08T01:45:24Z [SPARK-8149][SQL] Break ExpressionEvaluationSuite down to multiple files Also moved a few files in expressions package around to match test suites. Author: Reynold Xin Closes #6693 from rxin/expr-refactoring and squashes the following commits: 857599f [Reynold Xin] Fixed style violation. c0eb74b [Reynold Xin] Fixed compilation. b3a40f8 [Reynold Xin] Refactored expression test suites. commit 4e5056ccc2c668ec2c417876420215b876918de8 Author: Reynold Xin Date: 2015-06-08T02:55:07Z [SPARK-8154][SQL] Remove Term/Code type aliases in code generation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109838329 [Test build #34403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34403/consoleFull) for PR 6614 at commit [`28225d5`](https://github.com/apache/spark/commit/28225d58846e6c214fbc6f7e1ef09cda6b7cd94d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109837948 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8080][STREAMING] Receiver.store with It...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6614#issuecomment-109838025 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8124] [SPARKR] [WIP] Created more examp...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6668#discussion_r31884319
--- Diff: examples/src/main/r/2-data-manipulation.R ---
@@ -0,0 +1,62 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+source("1-data.R")
--- End diff --
I'm not sure the `source` will work correctly given the way we encourage
users to run example programs. If you see the `Running examples` section at
http://people.apache.org/~pwendell/spark-releases/latest/index.html you can see
that we ask users to run example programs with something like
`./bin/spark-submit examples/src/main/r/data-manipulation.R`.
So I'd just recommend merging all the code into one file
`data-manipulation.R` and then take in the CSV file name as a command line
argument.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7160][SQL] Support converting DataFrame...
Github user rayortigas commented on the pull request: https://github.com/apache/spark/pull/5713#issuecomment-109826915 Updated the PR after #6222 was merged... 1. 6c36abf was rebasing on cafd505, so still non-invasive. 2. 3711a3e92de5972492e40621861908bfc54afc85 was an attempt to more tightly integrate with cafd505. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8124] [SPARKR] [WIP] Created more examp...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6668#discussion_r31884096
--- Diff: examples/src/main/r/2-data-manipulation.R ---
@@ -0,0 +1,62 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+source("1-data.R")
+
+
+# Install the magrittr pipeline operator
+install.packages("magrittr")
+library(magrittr)
+
+# Print the first 6 rows of the DataFrame
+showDF(flightsDF, numRows = 6) ## Or
+head(flightsDF)
+
+# Show the column names in the DataFrame
+columns(flightsDF)
+
+# Show the number of rows in the DataFrame
+count(flightsDF)
+
+# Show summary statistics for numeric colums
+Describe(flightsDF)
--- End diff --
This should be `describe` and not `Describe` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6551#issuecomment-109826467 @cloud-fan and @yhuai can you both take another look at this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31883982
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -59,6 +59,32 @@ object HiveTypeCoercion {
}
/**
+ * Implicit promote the AtomicType to StringType if
+ * one of the data type in (dt1, dt2) is StringType, and the other is
not either
+ * BooleanType or BinaryType, the TightestCommonType should be StringType
+ * eg: 1. CaseWhenLike case when ... then dt1 else dt2 end
+ * 2. Coalesce(null, dt1, dt2)
+ */
+ val promoteToStringTypeOfTwo: (DataType, DataType) => Option[DataType] =
{
--- End diff --
and make it private
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31883980
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -59,6 +59,32 @@ object HiveTypeCoercion {
}
/**
+ * Implicit promote the AtomicType to StringType if
+ * one of the data type in (dt1, dt2) is StringType, and the other is
not either
+ * BooleanType or BinaryType, the TightestCommonType should be StringType
+ * eg: 1. CaseWhenLike case when ... then dt1 else dt2 end
+ * 2. Coalesce(null, dt1, dt2)
+ */
+ val promoteToStringTypeOfTwo: (DataType, DataType) => Option[DataType] =
{
+case (t1: StringType, t2: AtomicType) if (t2 != BinaryType && t2 !=
BooleanType) =>
+Some(StringType)
+
+case (t1: AtomicType, t2: StringType) if (t1 != BinaryType && t1 !=
BooleanType) =>
+Some(StringType)
+
+case _ => None
+ }
+
+ private def findTightestCommonTypeAndTryPromoteToString(types:
Seq[DataType]) = {
--- End diff --
can you declare the return type here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8010][SQL]Promote types to StringType a...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6551#discussion_r31883966
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala
---
@@ -59,6 +59,32 @@ object HiveTypeCoercion {
}
/**
+ * Implicit promote the AtomicType to StringType if
+ * one of the data type in (dt1, dt2) is StringType, and the other is
not either
+ * BooleanType or BinaryType, the TightestCommonType should be StringType
+ * eg: 1. CaseWhenLike case when ... then dt1 else dt2 end
+ * 2. Coalesce(null, dt1, dt2)
+ */
+ val promoteToStringTypeOfTwo: (DataType, DataType) => Option[DataType] =
{
--- End diff --
why not make this a function instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7269] [SQL] [WIP] Refactor the class At...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6587#discussion_r31883893
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
---
@@ -68,7 +68,7 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] {
* generating case class `equals` methods
*/
def fastEquals(other: TreeNode[_]): Boolean = {
-this.eq(other) || this == other
+this.eq(other)// || this == other
--- End diff --
I thought that also before I made this change, but I don't think that the
strong reason we should stop this change.
For most of cases in the code, we returns the same references by
well-design the `rules` for a `TreeNode` object, and if the code still keep
creating the identical objects(`.equals` returns `true`) in its rule for every
iteration, even unnecessary, can this be considered as a bug of the user code?
I think it will be the responsibility for user code to decide whether
`TreeNode` object substitutions should be taken (via creating new instance), as
user code always knows when a object substitution needed, right? That's also
give more freedom for user code to define the `.equals()` in a semantic way for
`TreeNode` object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
