[GitHub] spark pull request #18771: [SPARK-21478][SQL] Avoid unpersisting related Dat...
Github user vinodkc closed the pull request at: https://github.com/apache/spark/pull/18771 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18771: [SPARK-21478][SQL] Avoid unpersisting related Dat...
Github user vinodkc commented on a diff in the pull request: https://github.com/apache/spark/pull/18771#discussion_r130234632 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala --- @@ -114,13 +114,13 @@ class CacheManager extends Logging { } /** - * Un-cache all the cache entries that refer to the given plan. + * Un-cache the cache entry that refers to the given plan. */ def uncacheQuery(spark: SparkSession, plan: LogicalPlan, blocking: Boolean): Unit = writeLock { val it = cachedData.iterator() while (it.hasNext) { val cd = it.next() - if (cd.plan.find(_.sameResult(plan)).isDefined) { + if (plan.sameResult(cd.plan)) { --- End diff -- @gatorsmile Thanks for the update. I'll close this PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18769 **[Test build #80050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80050/testReport)** for PR 18769 at commit [`cee6838`](https://github.com/apache/spark/commit/cee6838e6f188c1d751ef6d3bc2a191057f45e34). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18775: [SPARK-8577][SparkR] Eliminate needless synchronization ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18775 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18775: [SPARK-8577][SparkR] Eliminate needless synchroni...
GitHub user SereneAnt opened a pull request: https://github.com/apache/spark/pull/18775 [SPARK-8577][SparkR] Eliminate needless synchronization in java-R serialization ## What changes were proposed in this pull request? Remove surplus synchronized blocks. ## How was this patch tested? Unit tests run OK. You can merge this pull request into a Git repository by running: $ git pull https://github.com/SereneAnt/spark eliminate_unnecessary_synchronization_in_java-R_serialization Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18775.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18775 commit eafad7d0cdf73c9fab9a03f2898039de7b127bc7 Author: iurii.ant Date: 2017-07-30T05:04:06Z [SPARK-8577][SparkR] Eliminate needless synchronization in java-R serialization --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18772 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80049/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18772 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18772 **[Test build #80049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80049/testReport)** for PR 18772 at commit [`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18745: [SPARK-21544][DEPLOY] Tests jar of some module should no...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18745 cc @vanzin @srowen thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18772 **[Test build #80049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80049/testReport)** for PR 18772 at commit [`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18772 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18774: [MINOR][DOC]Replace numTasks with numPartitions in progr...
Github user remisharoon commented on the issue: https://github.com/apache/spark/pull/18774 @polarke : please refer to this Pull Request [ https://github.com/apache/spark/pull/18770 ] for best practices --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18774: [MINOR][DOC]Replace numTasks with numPartitions in progr...
Github user remisharoon commented on the issue: https://github.com/apache/spark/pull/18774 @polarke , can you please provide the details of how you tested this? please provide screenshots of the locally tested doc site --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17180 LGTM Wait for @JoshRosen for final sign off. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17849: [SPARK-10931][ML][PYSPARK] PySpark Models Copy Param Val...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17849 This looks pretty reasonable, sorry for the delay. If you have a chance to update this to master would be good to do. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18761 @gatorsmile OK. Usually the constructor expression parameters should also be in children. `RuntimeReplaceable` is a special case. I'd check if there are other expressions like it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17180 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80048/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17180 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17180 **[Test build #80048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80048/testReport)** for PR 17180 at commit [`2403c30`](https://github.com/apache/spark/commit/2403c30fc904bb6f4d1a1f8c79a9c513fc21e9f8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18772 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18772 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80047/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18772 **[Test build #80047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80047/testReport)** for PR 18772 at commit [`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18774: [MINOR][DOC]Replace numTasks with numPartitions in progr...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18774 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18774: [MINOR][DOC]Replace numTasks with numPartitions i...
GitHub user polarke opened a pull request: https://github.com/apache/spark/pull/18774 [MINOR][DOC]Replace numTasks with numPartitions in programming guide In programming guide, `numTasks` is used in several places as arguments of Transformations. However, in code, `numPartitions` is used. In this fix, I replace `numTasks` with `numPartitions` in programming guide for consistency. You can merge this pull request into a Git repository by running: $ git pull https://github.com/polarke/spark replace-numtasks-with-numpartitions-in-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18774.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18774 commit ec11300f4a2c38fdb7aca3774096a46076b58caf Author: Cheng Wang Date: 2017-07-29T20:13:48Z Replace numTasks with numPartitions in programming guide --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18773: [MINOR][DOC] Replace numTasks with numPartitions ...
Github user polarke closed the pull request at: https://github.com/apache/spark/pull/18773 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18771 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18771 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80046/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18771 **[Test build #80046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80046/testReport)** for PR 18771 at commit [`857b3dd`](https://github.com/apache/spark/commit/857b3dd5804355331d3bef4eb8136e6604232758). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18773: [MINOR][DOC] Replace numTasks with numPartitions ...
GitHub user polarke opened a pull request: https://github.com/apache/spark/pull/18773 [MINOR][DOC] Replace numTasks with numPartitions in programming guide In programming guide, `numTasks` is used in several places as arguments of Transformations. However, in code, `numPartitions` is used. In this fix, I replace `numTasks` with `numPartitions` in programming guide for consistency. You can merge this pull request into a Git repository by running: $ git pull https://github.com/polarke/spark rename-numtasks-to-numpartitions-in-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18773.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18773 commit 8a9f3da9aa20a5b3b61d22f74962510d65db9ce0 Author: Cheng Wang Date: 2017-07-29T19:43:58Z Rename numTasks to numPartitions in programming guide --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18773: [MINOR][DOC] Replace numTasks with numPartitions in prog...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18773 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17180 **[Test build #80048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80048/testReport)** for PR 17180 at commit [`2403c30`](https://github.com/apache/spark/commit/2403c30fc904bb6f4d1a1f8c79a9c513fc21e9f8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17180: [SPARK-19839][Core]release longArray in BytesToBytesMap
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17180 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18772 In `visitTableName` in `AstBuilder`, we already have a different analyzer path to resolve column aliases (https://github.com/maropu/spark/blob/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L685). We could also use `UnresolvedSubqueryColumnAliases` along with `visitAliasedQuery` and `visitAliasedRelation` though, I'm not sure it is worth fixing so. So, could you give me insight on this, too? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18771: [SPARK-21478][SQL] Avoid unpersisting related Dat...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18771#discussion_r130226083 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala --- @@ -114,13 +114,13 @@ class CacheManager extends Logging { } /** - * Un-cache all the cache entries that refer to the given plan. + * Un-cache the cache entry that refers to the given plan. */ def uncacheQuery(spark: SparkSession, plan: LogicalPlan, blocking: Boolean): Unit = writeLock { val it = cachedData.iterator() while (it.hasNext) { val cd = it.next() - if (cd.plan.find(_.sameResult(plan)).isDefined) { + if (plan.sameResult(cd.plan)) { --- End diff -- This is by design. This is for avoiding to get the incorrect results. See the original PR: https://github.com/apache/spark/pull/17097 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18772: [SPARK-20963][SQL] Support column aliases for join relat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18772 **[Test build #80047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80047/testReport)** for PR 18772 at commit [`6c93f85`](https://github.com/apache/spark/commit/6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18749 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18749 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80044/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18749 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80043/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18749 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18749 **[Test build #80044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80044/testReport)** for PR 18749 at commit [`f51ee74`](https://github.com/apache/spark/commit/f51ee74c77a476113cec164d3cc6bce9997e3519). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18749 **[Test build #80043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80043/testReport)** for PR 18749 at commit [`bf48875`](https://github.com/apache/spark/commit/bf48875057ca5978b88cacab455292b64daf1557). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18772: [SPARK-20963][SQL] Support column aliases for joi...
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/18772 [SPARK-20963][SQL] Support column aliases for join relations in FROM clause ## What changes were proposed in this pull request? This pr added parsing rules to support column aliases for join relations in FROM clause. This pr is a sub-task of #18079. ## How was this patch tested? Added tests in `AnalysisSuite`, `PlanParserSuite,` and `SQLQueryTestSuite`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/maropu/spark SPARK-20963-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18772.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18772 commit 6c93f859b7ef2cc7f7cb49ab6fbd0397b43fda84 Author: Takeshi Yamamuro Date: 2017-07-24T09:08:11Z Support aliases for join relations --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18768: [SPARK-21533][SQL] Print warning messages when ov...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18768#discussion_r130225918 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala --- @@ -111,6 +112,31 @@ private[hive] object HiveShim { } } + private def isSubClassOf(t: Type, parent: Class[_]): Boolean = t match { +case cls: Class[_] => parent.isAssignableFrom(cls) +case _ => false + } + + private def hasInheritanceOf[UDFType: ClassTag](funcName: String, clazz: Class[_]): Boolean = { +val clsTag = classTag[UDFType].runtimeClass +if (isSubClassOf(clazz, clsTag)) { + val funcClass = clazz.getMethod(funcName, classOf[MapredContext]) + funcClass.getDeclaringClass != clsTag +} else { + false +} + } + + def validateHiveUserDefinedFunction(udfClass: Class[_]): Unit = { +if (hasInheritanceOf[GenericUDF]("configure", udfClass) || --- End diff -- I checked it seemed this API had been implemented in v0.11.0 (https://github.com/apache/hive/blob/release-0.11.0/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java). This is [the commit](https://github.com/apache/hive/commit/a4705e51809e0f6c24cdc417c91600627cbe3482#diff-3111a4cd48c72a32f7713451d7b9ca1aR98) for this API. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18641 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18641 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80045/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18641 **[Test build #80045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80045/testReport)** for PR 18641 at commit [`e5431a0`](https://github.com/apache/spark/commit/e5431a02fda88c442146eb2297a04c2e36da65ad). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18771: [SPARK-21478][SQL] Avoid unpersisting related Datasets
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18771 **[Test build #80046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80046/testReport)** for PR 18771 at commit [`857b3dd`](https://github.com/apache/spark/commit/857b3dd5804355331d3bef4eb8136e6604232758). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18771: [SPARK-21478][SQL] Avoid unpersisting related Dat...
GitHub user vinodkc opened a pull request: https://github.com/apache/spark/pull/18771 [SPARK-21478][SQL] Avoid unpersisting related Datasets ## What changes were proposed in this pull request? While unpersisting a dataset, only unpersist and remove that datasets's plan from Cachemanager's cachedData. ## How was this patch tested? Added unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/vinodkc/spark br_SPARK-21478 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18771.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18771 commit e187aeb67a13493c6f5a9e540779a677d3502b04 Author: vinodkc Date: 2017-07-29T15:10:24Z Fixed unpersisting related DFs commit 857b3dd5804355331d3bef4eb8136e6604232758 Author: vinodkc Date: 2017-07-29T18:01:01Z Updated test cases and condition for unpersist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18645 Oh nice, it will help if that's not a required update along with everything else here as it requires a Jackson update. The Jackson update may be important eventually but nice if it can be separate. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18645: [SPARK-14280][BUILD][WIP] Update change-version.sh and p...
Github user xuwei-k commented on the issue: https://github.com/apache/spark/pull/18645 json4s 3.2.11 for Scala 2.12 released - https://oss.sonatype.org/content/repositories/releases/org/json4s/json4s-jackson_2.12/3.2.11 - https://github.com/json4s/json4s/releases/tag/v3.2.11_2.12 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18185: [SPARK-20962][SQL] Support subquery column aliase...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18185 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18185: [SPARK-20962][SQL] Support subquery column aliases in FR...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18185 LGTM. Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16818: [SPARK-19451][SQL][Core] Underlying integer overf...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16818 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18540 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18540 Thanks! Merging to master/2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18540 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18540: [SPARK-19451][SQL] rangeBetween method should acc...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18540#discussion_r130224334 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -1179,32 +1179,26 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging } /** - * Create or resolve a [[FrameBoundary]]. Simple math expressions are allowed for Value - * Preceding/Following boundaries. These expressions must be constant (foldable) and return an - * integer value. + * Create or resolve a frame boundary expressions. */ - override def visitFrameBound(ctx: FrameBoundContext): FrameBoundary = withOrigin(ctx) { -// We currently only allow foldable integers. -def value: Int = { + override def visitFrameBound(ctx: FrameBoundContext): Expression = withOrigin(ctx) { +def value: Expression = { val e = expression(ctx.expression) - validate(e.resolved && e.foldable && e.dataType == IntegerType, -"Frame bound value must be a constant integer.", -ctx) - e.eval().asInstanceOf[Int] + validate(e.resolved && e.foldable, "Frame bound value must be a literal.", ctx) + e } -// Create the FrameBoundary ctx.boundType.getType match { case SqlBaseParser.PRECEDING if ctx.UNBOUNDED != null => UnboundedPreceding case SqlBaseParser.PRECEDING => -ValuePreceding(value) +UnaryMinus(value) case SqlBaseParser.CURRENT => CurrentRow case SqlBaseParser.FOLLOWING if ctx.UNBOUNDED != null => UnboundedFollowing case SqlBaseParser.FOLLOWING => -ValueFollowing(value) +value --- End diff -- It sounds like we already allowed it in the previous release. Thus, we need to follow what we have now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18761: [SPARK-21555][SQL] RuntimeReplaceable should be c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18761 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18761 Thanks! Merging to master/2.2/2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18761 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18761 It looks much better now. Since `CleanupAliases` has such an issue, could you check whether the other expressions are facing the same issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp ...
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/18664 It looks like "SESSION_LOCAL_TIMEZONE" is not respected in most of the pyspark functionality. I think `df.collect()` and `df.toPandas` can be fixed to respect SESSION_LOCAL_TIMEZONE. `TimestampType().toInternal(dt)` is tricky because it doesn't have a reference to the SQLConf object, we should maybe deprecate this method? I think we can also create separate Jira to address these, this PR can fix just the Arrow path. @BryanCutler and @ueshin do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18749#discussion_r130223158 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java --- @@ -24,20 +24,50 @@ /** * ::DeveloperApi:: - + * * A function description type which can be recognized by FunctionRegistry, and will be used to * show the usage of the function in human language. * * `usage()` will be used for the function usage in brief way. - * `extended()` will be used for the function usage in verbose way, suppose - * an example will be provided. * - * And we can refer the function name by `_FUNC_`, in `usage` and `extended`, as it's + * These below are concatenated and used for the function usage in verbose way, suppose arguments, + * examples, note and since will be provided. + * + * `arguments()` describes arguments for the expression. This should follow the format as below: + * + * Arguments: + * * arg0 - ... + * + * * arg1 - ... + * + * + * `examples()` describes examples for the expression. This should follow the format as below: + * + * Examples: + * > SELECT ...; + * ... + * > SELECT ...; + * ... + * + * `note()` contains some notes for the expression optionally. + * + * `since()` contains version information for the expression. Version is specified by, + * for example, "2.2.0". + * + * We can refer the function name by `_FUNC_`, in `usage`, `arguments` and `examples`, as it's * registered in `FunctionRegistry`. + * + * Note that, if `extended()` is defined, `arguments()`, `examples()`, `note()` and `since()` will + * be ignored and `extended()` will be used for the extended description for backward + * compatibility. */ @DeveloperApi @Retention(RetentionPolicy.RUNTIME) public @interface ExpressionDescription { -String usage() default "_FUNC_ is undocumented"; -String extended() default "\nNo example/argument for _FUNC_.\n"; +String usage() default ""; +String extended() default ""; --- End diff -- In the last commit https://github.com/apache/spark/pull/18749/commits/bf48875057ca5978b88cacab455292b64daf1557, I tried to keep the original behaviour and compatibility (although I guess we are fine to break this as said in few comments above): For `ExpressionDescription` and `ExpressionInfo`, if `extended` is an empty string (the default value), it creates the `extended` made by `arguments`, `examples`, `note` and `since` and use this as the extended description. if `extended` is a non-empty string, it ignores the new `arguments`, `examples`, `note` and `since` but use this as the extended description, assuming this was set manually. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18749 **[Test build #80044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80044/testReport)** for PR 18749 at commit [`f51ee74`](https://github.com/apache/spark/commit/f51ee74c77a476113cec164d3cc6bce9997e3519). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit problem i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18641 **[Test build #80045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80045/testReport)** for PR 18641 at commit [`e5431a0`](https://github.com/apache/spark/commit/e5431a02fda88c442146eb2297a04c2e36da65ad). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18641: [SPARK-21413][SQL] Fix 64KB JVM bytecode limit pr...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18641#discussion_r130223022 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala --- @@ -270,37 +271,91 @@ case class CaseWhenCodegen( // } // } // } + +val isNull = ctx.freshName("caseWhenIsNull") +val value = ctx.freshName("caseWhenValue") +// Split these expressions only if they are created from a row object --- End diff -- good catch. done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18749 **[Test build #80043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80043/testReport)** for PR 18749 at commit [`bf48875`](https://github.com/apache/spark/commit/bf48875057ca5978b88cacab455292b64daf1557). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18749#discussion_r130222959 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java --- @@ -24,20 +24,50 @@ /** * ::DeveloperApi:: - + * * A function description type which can be recognized by FunctionRegistry, and will be used to * show the usage of the function in human language. * * `usage()` will be used for the function usage in brief way. - * `extended()` will be used for the function usage in verbose way, suppose - * an example will be provided. * - * And we can refer the function name by `_FUNC_`, in `usage` and `extended`, as it's + * These below are concatenated and used for the function usage in verbose way, suppose arguments, + * examples, note and since will be provided. + * + * `arguments()` describes arguments for the expression. This should follow the format as below: + * + * Arguments: + * * arg0 - ... + * + * * arg1 - ... + * + * + * `examples()` describes examples for the expression. This should follow the format as below: + * + * Examples: + * > SELECT ...; + * ... + * > SELECT ...; + * ... + * + * `note()` contains some notes for the expression optionally. + * + * `since()` contains version information for the expression. Version is specified by, + * for example, "2.2.0". + * + * We can refer the function name by `_FUNC_`, in `usage`, `arguments` and `examples`, as it's * registered in `FunctionRegistry`. + * + * Note that, if `extended()` is defined, `arguments()`, `examples()`, `note()` and `since()` will + * be ignored and `extended()` will be used for the extended description for backward + * compatibility. */ @DeveloperApi @Retention(RetentionPolicy.RUNTIME) public @interface ExpressionDescription { -String usage() default "_FUNC_ is undocumented"; -String extended() default "\nNo example/argument for _FUNC_.\n"; +String usage() default ""; +String extended() default ""; --- End diff -- I manually tested source compatibility and I guess it also won't break binary compatibility - https://stackoverflow.com/questions/21197255/is-adding-a-method-to-a-java-annotation-safe-for-backward-compatibility --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18750: Skip maven-compiler-plugin main and test compilations in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18750 **[Test build #3861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3861/testReport)** for PR 18750 at commit [`d9dfe3a`](https://github.com/apache/spark/commit/d9dfe3a147dd5530315572c523588d55c28120de). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18749#discussion_r13064 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionDescription.java --- @@ -29,15 +29,40 @@ * show the usage of the function in human language. * * `usage()` will be used for the function usage in brief way. - * `extended()` will be used for the function usage in verbose way, suppose - * an example will be provided. * - * And we can refer the function name by `_FUNC_`, in `usage` and `extended`, as it's + * These below are concatenated and used for the function usage in verbose way, suppose arguments, + * examples, note and since will be provided. + * + * `arguments()` describes arguments for the expression. This should follow the format as below: + * + * Arguments: + * * arg0 - ... + * + * * arg1 - ... + * + * + * `examples()` describes examples for the expression. This should follow the format as below: + * + * Examples: + * > SELECT ...; + * ... + * > SELECT ...; + * ... + * + * `note()` contains some notes for the expression optionally. + * + * `since()` contains version information for the expression. Version is specified by, + * for example, "2.2.0". + * + * We can refer the function name by `_FUNC_`, in `usage`, `arguments` and `examples`, as it's * registered in `FunctionRegistry`. */ @DeveloperApi --- End diff -- Will give a try. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18749: [SPARK-21485][FOLLOWUP][SQL][DOCS] Describes exam...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18749#discussion_r13001 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java --- @@ -26,6 +26,10 @@ private String name; private String extended; private String db; +private String arguments; --- End diff -- Yes, I think so for the former and sure will keep the original constructor. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18767: [SPARK-21572][Spark Shell] Add description on how to exi...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18767 I also don't think it is worth being printed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18315: [SPARK-21108] [ML] [WIP] convert LinearSVC to aggregator...
Github user hhbyyh commented on the issue: https://github.com/apache/spark/pull/18315 Thanks for the review. Updated to address the comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18750: Skip maven-compiler-plugin main and test compilations in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18750 **[Test build #3861 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3861/testReport)** for PR 18750 at commit [`d9dfe3a`](https://github.com/apache/spark/commit/d9dfe3a147dd5530315572c523588d55c28120de). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18750: Skip maven-compiler-plugin main and test compilations in...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18750 This will need a JIRA @gslowikowski , by the way --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18769 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18769 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80042/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18769 **[Test build #80042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80042/testReport)** for PR 18769 at commit [`4bdc9ad`](https://github.com/apache/spark/commit/4bdc9ad5506a6055b9bb0b82ad4cfb1af09708c0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18718: [SPARK-21357][DStreams] FileInputDStream not remo...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18718 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18770: [SPARK-21508][DOC] Fix example code provided in S...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18770 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18718: [SPARK-21357][DStreams] FileInputDStream not remove out ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18718 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18770: [SPARK-21508][DOC] Fix example code provided in Spark St...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18770 Merged to master/2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18770: [SPARK-21508][DOC] Fix example code provided in Spark St...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18770 **[Test build #3860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3860/testReport)** for PR 18770 at commit [`646699e`](https://github.com/apache/spark/commit/646699eca80969a82e82c6b7c7c100f910458266). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18770: [SPARK-21508][DOC] Fix example code provided in Spark St...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18770 **[Test build #3860 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3860/testReport)** for PR 18770 at commit [`646699e`](https://github.com/apache/spark/commit/646699eca80969a82e82c6b7c7c100f910458266). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18540 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18540 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80041/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18540 **[Test build #80041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80041/testReport)** for PR 18540 at commit [`38b04df`](https://github.com/apache/spark/commit/38b04df792caf298da63be565d1b46a4eae93d98). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18770: Error in example code provided in Spark Streaming Docume...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18770 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18770: Error in example code provided in Spark Streaming...
GitHub user remisharoon opened a pull request: https://github.com/apache/spark/pull/18770 Error in example code provided in Spark Streaming Documentation ## What changes were proposed in this pull request? JIRA ticket : SPARK-21508 correcting a mistake in example code provided in Spark Streaming Custom Receivers Documentation ## How was this patch tested? this documentation change is tested manually by jekyll build , running below commands ``` jekyll build jekyll serve --watch ``` screen-shots provided below ![screenshot1](https://user-images.githubusercontent.com/8828470/28744636-a6de1ac6-7482-11e7-843b-ff84b5855ec0.png) ![screenshot2](https://user-images.githubusercontent.com/8828470/28744637-a6def496-7482-11e7-9512-7f4bbe027c6a.png) You can merge this pull request into a Git repository by running: $ git pull https://github.com/remisharoon/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18770.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18770 commit 646699eca80969a82e82c6b7c7c100f910458266 Author: Remis Haroon Date: 2017-07-25T14:34:20Z correcting an error in example code provided in Spark Streaming Custom Receivers Documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18767: [SPARK-21572][Spark Shell] Add description on how to exi...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18767 Iâm not convinced this would be useful, could you close this please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18768 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80040/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18768 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18768: [SPARK-21533][SQL] Print warning messages when override ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18768 **[Test build #80040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80040/testReport)** for PR 18768 at commit [`2d586a8`](https://github.com/apache/spark/commit/2d586a80e6f67e3adac17a8c1b5d09fbbf4252d5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partiti...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18769 **[Test build #80042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80042/testReport)** for PR 18769 at commit [`4bdc9ad`](https://github.com/apache/spark/commit/4bdc9ad5506a6055b9bb0b82ad4cfb1af09708c0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18769: [SPARK-21574][SQL] Fix set hive.exec.max.dynamic....
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/18769 [SPARK-21574][SQL] Fix set hive.exec.max.dynamic.partitions lose effect. ## What changes were proposed in this pull request? How to reproduce: ```scala val data = (0 until 1001).map(i => (i, i)) spark.createDataFrame(data).toDF("c1", "c2").createOrReplaceTempView("sourceTable") spark.sql("create table targetTable(c1 int) PARTITIONED BY(c2 int)") spark.sql("set hive.exec.dynamic.partition.mode=nonstrict") spark.sql("set hive.exec.max.dynamic.partitions=1001") spark.sql("insert overwrite table targetTable partition(c2) select * from sourceTable") ``` This PR fix this issue. ## How was this patch tested? unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/wangyum/spark SPARK-21574 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18769.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18769 commit 4bdc9ad5506a6055b9bb0b82ad4cfb1af09708c0 Author: Yuming Wang Date: 2017-07-29T10:48:02Z Fix set hive.exec.max.dynamic.partitions lose effect. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18761 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18761 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80039/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18761: [SPARK-21555][SQL] RuntimeReplaceable should be compared...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18761 **[Test build #80039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80039/testReport)** for PR 18761 at commit [`82d54ed`](https://github.com/apache/spark/commit/82d54ed769e75d2a8b74b8fb87d49904e3d405d9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18768: [SPARK-21533][SQL] Print warning messages when ov...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18768#discussion_r130217731 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala --- @@ -111,6 +112,31 @@ private[hive] object HiveShim { } } + private def isSubClassOf(t: Type, parent: Class[_]): Boolean = t match { +case cls: Class[_] => parent.isAssignableFrom(cls) +case _ => false + } + + private def hasInheritanceOf[UDFType: ClassTag](funcName: String, clazz: Class[_]): Boolean = { +val clsTag = classTag[UDFType].runtimeClass +if (isSubClassOf(clazz, clsTag)) { + val funcClass = clazz.getMethod(funcName, classOf[MapredContext]) + funcClass.getDeclaringClass != clsTag +} else { + false +} + } + + def validateHiveUserDefinedFunction(udfClass: Class[_]): Unit = { +if (hasInheritanceOf[GenericUDF]("configure", udfClass) || --- End diff -- When `GenericUDF` API has `configure` method? Seems `GenericUDF` at 0.10.0 has no such method? https://hive.apache.org/javadocs/r0.10.0/api/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18540: [SPARK-19451][SQL] rangeBetween method should accept Lon...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18540 **[Test build #80041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80041/testReport)** for PR 18540 at commit [`38b04df`](https://github.com/apache/spark/commit/38b04df792caf298da63be565d1b46a4eae93d98). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org