[GitHub] [spark] SparkQA commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
SparkQA commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880444654 **[Test build #141058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141058/testReport)** for PR 31522 at commit [`b92f0e4`](https://github.com

[GitHub] [spark] dongjoon-hyun commented on pull request #33355: [SPARK-36150][INFRA][TESTS] Disable MiMa for Scala 2.13 artifacts

2021-07-14 Thread GitBox
dongjoon-hyun commented on pull request #33355: URL: https://github.com/apache/spark/pull/33355#issuecomment-880444203 Could you review this please, @HyukjinKwon ? This is required to run `dev/run-tests.py` with Scala-2.13 because it runs MiMa always for SBT build. ``` # backwa

[GitHub] [spark] SparkQA commented on pull request #33355: [SPARK-36150][INFRA][TESTS] Disable MiMa for Scala 2.13 artifacts

2021-07-14 Thread GitBox
SparkQA commented on pull request #33355: URL: https://github.com/apache/spark/pull/33355#issuecomment-880441656 **[Test build #141057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141057/testReport)** for PR 33355 at commit [`dc5607b`](https://github.com

[GitHub] [spark] dongjoon-hyun opened a new pull request #33355: [SPARK-36150][INFRA][TESTS] Disable MiMa for Scala 2.13 artifacts

2021-07-14 Thread GitBox
dongjoon-hyun opened a new pull request #33355: URL: https://github.com/apache/spark/pull/33355 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### H

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880437328 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141056/ -

[GitHub] [spark] SparkQA removed a comment on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
SparkQA removed a comment on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880436106 **[Test build #141056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141056/testReport)** for PR 31522 at commit [`e69e279`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880437328 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141056/ -- This

[GitHub] [spark] SparkQA commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
SparkQA commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880437311 **[Test build #141056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141056/testReport)** for PR 31522 at commit [`e69e279`](https://github.co

[GitHub] [spark] HeartSaVioR commented on a change in pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
HeartSaVioR commented on a change in pull request #33081: URL: https://github.com/apache/spark/pull/33081#discussion_r670174985 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ## @@ -335,12 +339,29 @@ abstract class SparkStrategies ex

[GitHub] [spark] SparkQA commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
SparkQA commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880436106 **[Test build #141056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141056/testReport)** for PR 31522 at commit [`e69e279`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33353: URL: https://github.com/apache/spark/pull/33353#issuecomment-880413323 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

[GitHub] [spark] HeartSaVioR commented on a change in pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
HeartSaVioR commented on a change in pull request #33081: URL: https://github.com/apache/spark/pull/33081#discussion_r670174122 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -1610,6 +1610,26 @@ object SQLConf { .checkValue(v

[GitHub] [spark] SparkQA commented on pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
SparkQA commented on pull request #33353: URL: https://github.com/apache/spark/pull/33353#issuecomment-880435387 **[Test build #141055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141055/testReport)** for PR 33353 at commit [`af6ee8a`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #33354: [SPARK-36037][TESTS][FOLLOWUP] Avoid wrong test results on daylight saving time

2021-07-14 Thread GitBox
SparkQA commented on pull request #33354: URL: https://github.com/apache/spark/pull/33354#issuecomment-880435385 **[Test build #141054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141054/testReport)** for PR 33354 at commit [`867b2f7`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880434824 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141042/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880434825 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141048/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33343: URL: https://github.com/apache/spark/pull/33343#issuecomment-880434827 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45567/

[GitHub] [spark] AmplabJenkins commented on pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33343: URL: https://github.com/apache/spark/pull/33343#issuecomment-880434827 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45567/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880434825 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141048/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880434824 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141042/ -- This

[GitHub] [spark] SparkQA commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
SparkQA commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880434532 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45568/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA removed a comment on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
SparkQA removed a comment on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880320901 **[Test build #141042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141042/testReport)** for PR 33081 at commit [`eef55e8`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
SparkQA commented on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880430443 **[Test build #141042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141042/testReport)** for PR 33081 at commit [`eef55e8`](https://github.co

[GitHub] [spark] dominikgehl commented on a change in pull request #33345: [PYTHON] clarify documentation for dayofweek

2021-07-14 Thread GitBox
dominikgehl commented on a change in pull request #33345: URL: https://github.com/apache/spark/pull/33345#discussion_r670165603 ## File path: python/pyspark/sql/functions.py ## @@ -1779,7 +1779,7 @@ def month(col): def dayofweek(col): """ -Extract the day of the wee

[GitHub] [spark] Peng-Lei edited a comment on pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
Peng-Lei edited a comment on pull request #33353: URL: https://github.com/apache/spark/pull/33353#issuecomment-880426373 > @Peng-Lei > If the output is deterministic, should we remove `orderBy` in the following test code? > https://github.com/apache/spark/blob/416a7fd49002e207dd188bd

[GitHub] [spark] Peng-Lei commented on pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
Peng-Lei commented on pull request #33353: URL: https://github.com/apache/spark/pull/33353#issuecomment-880426373 > @Peng-Lei > If the output is deterministic, should we remove `orderBy` in the following test code? > https://github.com/apache/spark/blob/416a7fd49002e207dd188bd1547b8b

[GitHub] [spark] SparkQA commented on pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
SparkQA commented on pull request #33343: URL: https://github.com/apache/spark/pull/33343#issuecomment-880424377 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45567/ -- This is an automated message from the A

[GitHub] [spark] SparkQA removed a comment on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
SparkQA removed a comment on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880365729 **[Test build #141048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141048/testReport)** for PR 33349 at commit [`53a8151`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
SparkQA commented on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880423361 **[Test build #141048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141048/testReport)** for PR 33349 at commit [`53a8151`](https://github.co

[GitHub] [spark] gengliangwang commented on a change in pull request #33346: [SPARK-36037][SQL][FOLLOWUP] Fix flaky test for datetime function localtimestamp

2021-07-14 Thread GitBox
gengliangwang commented on a change in pull request #33346: URL: https://github.com/apache/spark/pull/33346#discussion_r670160612 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala ## @@ -98,10 +98,10 @@ class DateExp

[GitHub] [spark] gengliangwang opened a new pull request #33354: [SPARK-36037][TESTS][FOLLOWUP] Avoid wrong test results on daylight saving time

2021-07-14 Thread GitBox
gengliangwang opened a new pull request #33354: URL: https://github.com/apache/spark/pull/33354 ### What changes were proposed in this pull request? Only use the zone ids that has no daylight saving for testing `localtimestamp` ### Why are the changes needed? ht

[GitHub] [spark] sarutak commented on pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
sarutak commented on pull request #33353: URL: https://github.com/apache/spark/pull/33353#issuecomment-880420918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

[GitHub] [spark] EnricoMi commented on pull request #31905: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

2021-07-14 Thread GitBox
EnricoMi commented on pull request #31905: URL: https://github.com/apache/spark/pull/31905#issuecomment-880420616 The tests succeeded: https://github.com/G-Research/spark/runs/3071443396?check_suite_focus=true -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HeartSaVioR commented on a change in pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
HeartSaVioR commented on a change in pull request #33081: URL: https://github.com/apache/spark/pull/33081#discussion_r670157882 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3933,6 +3938,83 @@ object TimeWindowing extend

[GitHub] [spark] Peng-Lei commented on pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
Peng-Lei commented on pull request #33353: URL: https://github.com/apache/spark/pull/33353#issuecomment-880415750 @sarutak FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[GitHub] [spark] Peng-Lei edited a comment on pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
Peng-Lei edited a comment on pull request #33343: URL: https://github.com/apache/spark/pull/33343#issuecomment-880414872 @sarutak Thank you very much for finding and fixing the bug I introduced. I think we also can keep the output deterministic for `SHOW TBLPROPERTIES` in [#33353](https://

[GitHub] [spark] Peng-Lei commented on pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
Peng-Lei commented on pull request #33343: URL: https://github.com/apache/spark/pull/33343#issuecomment-880414872 @sarutak Thank you very much for finding and fixing the bug I introduced. I think we also can keep the output deterministic for `SHOW TBLPROPERTIES` in [#33353](https://github.

[GitHub] [spark] HeartSaVioR commented on a change in pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
HeartSaVioR commented on a change in pull request #33081: URL: https://github.com/apache/spark/pull/33081#discussion_r670151257 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3933,6 +3938,83 @@ object TimeWindowing extend

[GitHub] [spark] AmplabJenkins commented on pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33353: URL: https://github.com/apache/spark/pull/33353#issuecomment-880413323 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[GitHub] [spark] Peng-Lei opened a new pull request #33353: [SPARK-29519][SQL][FOLLOWUP] Keep output is deterministic for show tblproperties

2021-07-14 Thread GitBox
Peng-Lei opened a new pull request #33353: URL: https://github.com/apache/spark/pull/33353 ### What changes were proposed in this pull request? Keep the output order is deterministic for `SHOW TBLPROPERTIES` ### Why are the changes needed? [#33343](https://github.com/apache/spar

[GitHub] [spark] SparkQA removed a comment on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-14 Thread GitBox
SparkQA removed a comment on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-880367325 **[Test build #141051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141051/testReport)** for PR 32401 at commit [`caaf76d`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-880410933 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141051/ -

[GitHub] [spark] viirya commented on a change in pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
viirya commented on a change in pull request #33081: URL: https://github.com/apache/spark/pull/33081#discussion_r670139067 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SessionWindow.scala ## @@ -0,0 +1,103 @@ +/* + * Licensed to the Apach

[GitHub] [spark] AmplabJenkins commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-880410933 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141051/ -- This

[GitHub] [spark] HyukjinKwon commented on a change in pull request #24595: [SPARK-20774][SPARK-27036][SQL] Cancel the running broadcast execution on BroadcastTimeout

2021-07-14 Thread GitBox
HyukjinKwon commented on a change in pull request #24595: URL: https://github.com/apache/spark/pull/24595#discussion_r670147801 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala ## @@ -67,68 +70,74 @@ case class Broadcast

[GitHub] [spark] SparkQA commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-14 Thread GitBox
SparkQA commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-880410333 **[Test build #141051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141051/testReport)** for PR 32401 at commit [`caaf76d`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
SparkQA commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880410294 **[Test build #141053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141053/testReport)** for PR 33352 at commit [`a5386f0`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880408620 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141047/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33341: [WIP][SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-880408615 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45564/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880408617 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45565/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880408614 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45563/

[GitHub] [spark] AmplabJenkins commented on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880408617 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45565/ -- T

[GitHub] [spark] SparkQA commented on pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
SparkQA commented on pull request #33343: URL: https://github.com/apache/spark/pull/33343#issuecomment-880408720 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45567/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #33341: [WIP][SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-880408615 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45564/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880408614 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45563/ -- T

[GitHub] [spark] AmplabJenkins commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880408620 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141047/ -- This

[GitHub] [spark] Shockang commented on a change in pull request #24595: [SPARK-20774][SPARK-27036][SQL] Cancel the running broadcast execution on BroadcastTimeout

2021-07-14 Thread GitBox
Shockang commented on a change in pull request #24595: URL: https://github.com/apache/spark/pull/24595#discussion_r670141248 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala ## @@ -67,68 +70,74 @@ case class BroadcastExc

[GitHub] [spark] SparkQA commented on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
SparkQA commented on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880403811 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45565/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
SparkQA commented on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880402912 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45563/ -- This is an automated message from the A

[GitHub] [spark] SparkQA commented on pull request #33341: [WIP][SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

2021-07-14 Thread GitBox
SparkQA commented on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-880393625 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45564/ -- This is an automated message from the A

[GitHub] [spark] MaxGekk commented on a change in pull request #33346: [SPARK-36037][SQL][FOLLOWUP] Fix flaky test for datetime function localtimestamp

2021-07-14 Thread GitBox
MaxGekk commented on a change in pull request #33346: URL: https://github.com/apache/spark/pull/33346#discussion_r670135296 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala ## @@ -98,10 +98,10 @@ class DateExpressio

[GitHub] [spark] SparkQA removed a comment on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
SparkQA removed a comment on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880346048 **[Test build #141047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141047/testReport)** for PR 33352 at commit [`0cce896`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
SparkQA commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880391396 **[Test build #141047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141047/testReport)** for PR 33352 at commit [`0cce896`](https://github.co

[GitHub] [spark] Ngone51 commented on pull request #33340: [SPARK-32915][SHUFFLE][FOLLOW-UP] Rename classes in shuffle RPC used for block push operations

2021-07-14 Thread GitBox
Ngone51 commented on pull request #33340: URL: https://github.com/apache/spark/pull/33340#issuecomment-880389549 I haven't taken a deep look into code but just wondering according to the PR description. I know some code was added in 3.1 but push-based wasn't usable at that time. So, do we

[GitHub] [spark] xuanyuanking commented on a change in pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
xuanyuanking commented on a change in pull request #33081: URL: https://github.com/apache/spark/pull/33081#discussion_r670116061 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3933,6 +3938,83 @@ object TimeWindowing exten

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-880387381 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45566/

[GitHub] [spark] AmplabJenkins commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-880387381 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45566/ -- T

[GitHub] [spark] SparkQA commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-14 Thread GitBox
SparkQA commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-880387361 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45566/ -- This

[GitHub] [spark] SparkQA commented on pull request #33081: [SPARK-34893][SS] Support session window natively

2021-07-14 Thread GitBox
SparkQA commented on pull request #33081: URL: https://github.com/apache/spark/pull/33081#issuecomment-880386680 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45565/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
SparkQA commented on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880385702 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45563/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33284: [SPARK-36063][SQL] Optimize OneRowRelation subqueries

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33284: URL: https://github.com/apache/spark/pull/33284#issuecomment-880383415 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141044/ -

[GitHub] [spark] sunchao commented on a change in pull request #33348: [SPARK-36128][SQL] Apply spark.sql.hive.metastorePartitionPruning for non-Hive tables that uses Hive metastore for partition mana

2021-07-14 Thread GitBox
sunchao commented on a change in pull request #33348: URL: https://github.com/apache/spark/pull/33348#discussion_r670126962 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -979,9 +979,7 @@ object SQLConf { val HIVE_METASTORE_PARTI

[GitHub] [spark] sunchao commented on a change in pull request #33348: [SPARK-36128][SQL] Apply spark.sql.hive.metastorePartitionPruning for non-Hive tables that uses Hive metastore for partition mana

2021-07-14 Thread GitBox
sunchao commented on a change in pull request #33348: URL: https://github.com/apache/spark/pull/33348#discussion_r670126640 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -979,9 +979,7 @@ object SQLConf { val HIVE_METASTORE_PARTI

[GitHub] [spark] SparkQA commented on pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
SparkQA commented on pull request #33343: URL: https://github.com/apache/spark/pull/33343#issuecomment-880384357 **[Test build #141052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141052/testReport)** for PR 33343 at commit [`4d6d401`](https://github.com

[GitHub] [spark] tooptoop4 commented on pull request #33332: [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker

2021-07-14 Thread GitBox
tooptoop4 commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-880384246 > can you keep the Pr description template? https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE @HyukjinKwon done -- This is an automated messag

[GitHub] [spark] sunchao commented on a change in pull request #33348: [SPARK-36128][SQL] Apply spark.sql.hive.metastorePartitionPruning for non-Hive tables that uses Hive metastore for partition mana

2021-07-14 Thread GitBox
sunchao commented on a change in pull request #33348: URL: https://github.com/apache/spark/pull/33348#discussion_r670125162 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala ## @@ -94,6 +93,19 @@ class CatalogFileIndex(

[GitHub] [spark] AmplabJenkins commented on pull request #33284: [SPARK-36063][SQL] Optimize OneRowRelation subqueries

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33284: URL: https://github.com/apache/spark/pull/33284#issuecomment-880383415 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141044/ -- This

[GitHub] [spark] SparkQA commented on pull request #33341: [WIP][SPARK-36091][SQL] Support TimestampNTZ type in expression TimeWindow

2021-07-14 Thread GitBox
SparkQA commented on pull request #33341: URL: https://github.com/apache/spark/pull/33341#issuecomment-880382416 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45564/ -- This is an automated message from the Apache

[GitHub] [spark] sarutak commented on a change in pull request #33343: [SPARK-33898][SQL][FOLLOWUP] Fix the behavior of `SHOW CREATE TABLE` to output deterministic results

2021-07-14 Thread GitBox
sarutak commented on a change in pull request #33343: URL: https://github.com/apache/spark/pull/33343#discussion_r670123615 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -2004,12 +2005,16 @@ class DataSourceV2SQLSuite

[GitHub] [spark] HyukjinKwon commented on pull request #33332: [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker

2021-07-14 Thread GitBox
HyukjinKwon commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-880382004 can you keep the Pr description template? https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE -- This is an automated message from the Apache Git Ser

[GitHub] [spark] SparkQA removed a comment on pull request #33284: [SPARK-36063][SQL] Optimize OneRowRelation subqueries

2021-07-14 Thread GitBox
SparkQA removed a comment on pull request #33284: URL: https://github.com/apache/spark/pull/33284#issuecomment-880343235 **[Test build #141044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141044/testReport)** for PR 33284 at commit [`73afab1`](https://gi

[GitHub] [spark] SparkQA commented on pull request #33284: [SPARK-36063][SQL] Optimize OneRowRelation subqueries

2021-07-14 Thread GitBox
SparkQA commented on pull request #33284: URL: https://github.com/apache/spark/pull/33284#issuecomment-880380490 **[Test build #141044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141044/testReport)** for PR 33284 at commit [`73afab1`](https://github.co

[GitHub] [spark] dongjoon-hyun commented on pull request #33325: [SPARK-36076][SQL][3.0] ArrayIndexOutOfBounds in Cast string to timestamp

2021-07-14 Thread GitBox
dongjoon-hyun commented on pull request #33325: URL: https://github.com/apache/spark/pull/33325#issuecomment-880378651 Merged to branch-3.0. Thank you, @dgd-contributor and @HyukjinKwon . -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun closed pull request #33325: [SPARK-36076][SQL][3.0] ArrayIndexOutOfBounds in Cast string to timestamp

2021-07-14 Thread GitBox
dongjoon-hyun closed pull request #33325: URL: https://github.com/apache/spark/pull/33325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880376871 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45561/

[GitHub] [spark] SparkQA commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
SparkQA commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880376860 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45561/ -- This is an automated message from the A

[GitHub] [spark] AmplabJenkins commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880376871 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45561/ -- T

[GitHub] [spark] tooptoop4 commented on pull request #33332: [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker

2021-07-14 Thread GitBox
tooptoop4 commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-880376623 @HyukjinKwon done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880374897 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45562/

[GitHub] [spark] SparkQA commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
SparkQA commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880374882 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45562/ -- This is an automated message from the A

[GitHub] [spark] AmplabJenkins commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-880374897 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45562/ -- T

[GitHub] [spark] dongjoon-hyun commented on pull request #33346: [SPARK-36037][SQL][FOLLOWUP] Fix flaky test for datetime function localtimestamp

2021-07-14 Thread GitBox
dongjoon-hyun commented on pull request #33346: URL: https://github.com/apache/spark/pull/33346#issuecomment-880374142 Thank you, @gengliangwang and all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [spark] cloud-fan commented on a change in pull request #33286: [SPARK-36079][SQL] Null-based filter estimate should always be in the range [0, 1]

2021-07-14 Thread GitBox
cloud-fan commented on a change in pull request #33286: URL: https://github.com/apache/spark/pull/33286#discussion_r670115469 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/FilterEstimationSuite.scala ## @@ -822,6 +822,41 @@ class Filte

[GitHub] [spark] dongjoon-hyun closed pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
dongjoon-hyun closed pull request #33349: URL: https://github.com/apache/spark/pull/33349 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-

[GitHub] [spark] dongjoon-hyun commented on pull request #33349: [SPARK-36139][INFRA][TESTS] Remove Python 3.6 from `pyspark` GitHub Action job

2021-07-14 Thread GitBox
dongjoon-hyun commented on pull request #33349: URL: https://github.com/apache/spark/pull/33349#issuecomment-880373352 Thank you, @HyukjinKwon . All PySpark tests passed in GitHub Action. Merged to master. -- This is an automated message from the Apache Git Service. To respond to the m

[GitHub] [spark] cfmcgrady commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`

2021-07-14 Thread GitBox
cfmcgrady commented on a change in pull request #33212: URL: https://github.com/apache/spark/pull/33212#discussion_r670113970 ## File path: docs/sql-migration-guide.md ## @@ -22,6 +22,10 @@ license: | * Table of contents {:toc} +## Upgrading from Spark SQL 3.2 to 3.3 + + -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
AmplabJenkins removed a comment on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880370943 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45560/

[GitHub] [spark] AmplabJenkins commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
AmplabJenkins commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880370943 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45560/ -- T

[GitHub] [spark] SparkQA commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

2021-07-14 Thread GitBox
SparkQA commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-880370796 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45560/ -- This is an automated message from the A

[GitHub] [spark] Ngone51 commented on a change in pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state

2021-07-14 Thread GitBox
Ngone51 commented on a change in pull request #33078: URL: https://github.com/apache/spark/pull/33078#discussion_r670110360 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java ## @@ -821,20 +914,132 @@ public vo

  1   2   3   4   5   6   7   8   >