[GitHub] [spark] xuanyuanking commented on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

2021-07-06 Thread GitBox
xuanyuanking commented on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874760929 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] cloud-fan commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
cloud-fan commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874757377 @ulysses-you since 3.1 does not enable AQE by default, I didn't backport this PR. If you think it's necessary, feel free to open a backport PR, probably with a modified test

[GitHub] [spark] SparkQA removed a comment on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874586655 **[Test build #140695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140695/testReport)** for PR 33220 at commit

[GitHub] [spark] cloud-fan commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
cloud-fan commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874752841 thanks for the review, merging to master/3.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] SparkQA commented on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

2021-07-06 Thread GitBox
SparkQA commented on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874752463 **[Test build #140695 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140695/testReport)** for PR 33220 at commit

[GitHub] [spark] SparkQA commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits

2021-07-06 Thread GitBox
SparkQA commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-874752278 **[Test build #140704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140704/testReport)** for PR 33231 at commit

[GitHub] [spark] cloud-fan closed pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
cloud-fan closed pull request #33222: URL: https://github.com/apache/spark/pull/33222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] AngersZhuuuu commented on pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits

2021-07-06 Thread GitBox
AngersZh commented on pull request #33231: URL: https://github.com/apache/spark/pull/33231#issuecomment-874751941 FYI @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] tgravescs commented on a change in pull request #33140: [SPARK-35881][SQL] Add support for columnar execution of final query stage in AdaptiveSparkPlanExec

2021-07-06 Thread GitBox
tgravescs commented on a change in pull request #33140: URL: https://github.com/apache/spark/pull/33140#discussion_r664545861 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala ## @@ -110,9 +110,12 @@ class SparkSessionExtensions { type

[GitHub] [spark] AngersZhuuuu opened a new pull request #33231: [SPARK-36021][SQL] Parse interval literals should support more than 2 digits

2021-07-06 Thread GitBox
AngersZh opened a new pull request #33231: URL: https://github.com/apache/spark/pull/33231 ### What changes were proposed in this pull request? For case ``` spark-sql> select interval '123456:12' minute to second; Error in query: requirement failed: Interval string must

[GitHub] [spark] SparkQA commented on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core

2021-07-06 Thread GitBox
SparkQA commented on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-874739404 **[Test build #140703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140703/testReport)** for PR 33216 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874736309 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140694/

[GitHub] [spark] AmplabJenkins commented on pull request #33230: [SPARK-36011][SQL][3.1] Disallow altering permanent views based on temporary views or UDFs

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33230: URL: https://github.com/apache/spark/pull/33230#issuecomment-874737431 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874736312 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45213/

[GitHub] [spark] AmplabJenkins commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874736312 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45213/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874736309 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140694/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874554144 **[Test build #140694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140694/testReport)** for PR 32959 at commit

[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874723506 **[Test build #140694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140694/testReport)** for PR 32959 at commit

[GitHub] [spark] jerqi opened a new pull request #33230: [SPARK-36011][SQL][3.1] Disallow altering permanent views based on temporary views or UDFs

2021-07-06 Thread GitBox
jerqi opened a new pull request #33230: URL: https://github.com/apache/spark/pull/33230 backport #33204 to 3.1 ### What changes were proposed in this pull request? PR #15764 disabled creating permanent views based on temporary views or UDFs. But AlterViewCommand didn't block

[GitHub] [spark] srowen commented on pull request #32973: [SPARK-36009][GRAPHX] Add missing GraphX classes to registerKryoClasses util method

2021-07-06 Thread GitBox
srowen commented on pull request #32973: URL: https://github.com/apache/spark/pull/32973#issuecomment-874715411 Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] srowen closed pull request #32973: [SPARK-36009][GRAPHX] Add missing GraphX classes to registerKryoClasses util method

2021-07-06 Thread GitBox
srowen closed pull request #32973: URL: https://github.com/apache/spark/pull/32973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] srowen commented on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core

2021-07-06 Thread GitBox
srowen commented on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-874712348 Why are these improvements? there's no detail here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] srowen closed pull request #33228: Update README.md

2021-07-06 Thread GitBox
srowen closed pull request #33228: URL: https://github.com/apache/spark/pull/33228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] SparkQA commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
SparkQA commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874710309 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45213/ -- This is an automated message from the

[GitHub] [spark] gengliangwang closed pull request #33229: [SPARK-36025][SQL][TESTS] Reduce the run time of DateExpressionsSuite

2021-07-06 Thread GitBox
gengliangwang closed pull request #33229: URL: https://github.com/apache/spark/pull/33229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] gengliangwang commented on pull request #33229: [SPARK-36025][SQL][TESTS] Reduce the run time of DateExpressionsSuite

2021-07-06 Thread GitBox
gengliangwang commented on pull request #33229: URL: https://github.com/apache/spark/pull/33229#issuecomment-874709125 Merging to master and branch-3.2(in case of future code conflicts in DateExpressionsSuite since we are working on new interval and timestmap types) -- This is an

[GitHub] [spark] srowen closed pull request #33199: [SPARK-36004][INFRA] Update MiMa and audit API changes

2021-07-06 Thread GitBox
srowen closed pull request #33199: URL: https://github.com/apache/spark/pull/33199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] srowen commented on pull request #33199: [SPARK-36004][INFRA] Update MiMa and audit API changes

2021-07-06 Thread GitBox
srowen commented on pull request #33199: URL: https://github.com/apache/spark/pull/33199#issuecomment-874709007 Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-874697442 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140691/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33229: [SPARK-36025][SQL][TESTS] Reduce the run time of DateExpressionsSuite

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33229: URL: https://github.com/apache/spark/pull/33229#issuecomment-874697445 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45210/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33227: [SPARK-35972][SQL][3.1] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33227: URL: https://github.com/apache/spark/pull/33227#issuecomment-874697438 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140692/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33226: [SPARK-35999][SQL] Make from_csv/to_csv to handle day-time intervals properly

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33226: URL: https://github.com/apache/spark/pull/33226#issuecomment-874697440 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140686/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33182: [SPARK-35984][SQL] Config to force applying shuffled hash join

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33182: URL: https://github.com/apache/spark/pull/33182#issuecomment-874697441 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45212/

[GitHub] [spark] AmplabJenkins commented on pull request #33226: [SPARK-35999][SQL] Make from_csv/to_csv to handle day-time intervals properly

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33226: URL: https://github.com/apache/spark/pull/33226#issuecomment-874697440 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140686/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33182: [SPARK-35984][SQL] Config to force applying shuffled hash join

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33182: URL: https://github.com/apache/spark/pull/33182#issuecomment-874697441 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45212/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-874697442 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140691/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33227: [SPARK-35972][SQL][3.1] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33227: URL: https://github.com/apache/spark/pull/33227#issuecomment-874697438 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140692/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33229: [SPARK-36025][SQL][TESTS] Reduce the run time of DateExpressionsSuite

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33229: URL: https://github.com/apache/spark/pull/33229#issuecomment-874697445 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45210/ --

[GitHub] [spark] AngersZhuuuu commented on pull request #33227: [SPARK-35972][SQL][3.1] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
AngersZh commented on pull request #33227: URL: https://github.com/apache/spark/pull/33227#issuecomment-874697241 @viirya gen_alias can't have a stable value. just remove the ut? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

2021-07-06 Thread GitBox
SparkQA commented on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-874690662 **[Test build #140691 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140691/testReport)** for PR 33214 at commit

[GitHub] [spark] linar-jether commented on a change in pull request #29719: [SPARK-32846][SQL][PYTHON] Support createDataFrame from an RDD of pd.DataFrames

2021-07-06 Thread GitBox
linar-jether commented on a change in pull request #29719: URL: https://github.com/apache/spark/pull/29719#discussion_r664480865 ## File path: python/pyspark/sql/pandas/conversion.py ## @@ -297,8 +297,11 @@ class SparkConversionMixin(object): """ Min-in for the

[GitHub] [spark] SparkQA commented on pull request #33227: [SPARK-35972][SQL][3.1] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
SparkQA commented on pull request #33227: URL: https://github.com/apache/spark/pull/33227#issuecomment-874676341 **[Test build #140692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140692/testReport)** for PR 33227 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33227: [SPARK-35972][SQL][3.1] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33227: URL: https://github.com/apache/spark/pull/33227#issuecomment-874548375 **[Test build #140692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140692/testReport)** for PR 33227 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33214: [SPARK-35929][PYTHON] Support to infer nested dict as a struct when creating a DataFrame

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33214: URL: https://github.com/apache/spark/pull/33214#issuecomment-874516035 **[Test build #140691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140691/testReport)** for PR 33214 at commit

[GitHub] [spark] SparkQA commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
SparkQA commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874690238 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45213/ -- This is an automated message from the Apache

[GitHub] [spark] Peng-Lei commented on a change in pull request #33175: [SPARK-35973][SQL] DataSourceV2: Support SHOW CATALOGS

2021-07-06 Thread GitBox
Peng-Lei commented on a change in pull request #33175: URL: https://github.com/apache/spark/pull/33175#discussion_r664475325 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -662,6 +662,14 @@ case class

[GitHub] [spark] SparkQA removed a comment on pull request #33226: [SPARK-35999][SQL] Make from_csv/to_csv to handle day-time intervals properly

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33226: URL: https://github.com/apache/spark/pull/33226#issuecomment-87450 **[Test build #140686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140686/testReport)** for PR 33226 at commit

[GitHub] [spark] SparkQA commented on pull request #33182: [SPARK-35984][SQL] Config to force applying shuffled hash join

2021-07-06 Thread GitBox
SparkQA commented on pull request #33182: URL: https://github.com/apache/spark/pull/33182#issuecomment-874682415 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45212/ -- This is an automated message from the

[GitHub] [spark] Peng-Lei commented on a change in pull request #33175: [SPARK-35973][SQL] DataSourceV2: Support SHOW CATALOGS

2021-07-06 Thread GitBox
Peng-Lei commented on a change in pull request #33175: URL: https://github.com/apache/spark/pull/33175#discussion_r664475325 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -662,6 +662,14 @@ case class

[GitHub] [spark] srowen commented on pull request #33101: [SPARK-35907][CORE] Instead of File#mkdirs, Files#createDirectories is expected.

2021-07-06 Thread GitBox
srowen commented on pull request #33101: URL: https://github.com/apache/spark/pull/33101#issuecomment-874252406 I wouldn't worry about the test cases, but if there are places in the source code that look like they probably should leverage the Utils code to make a dir, in order to get all

[GitHub] [spark] cloud-fan commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
cloud-fan commented on a change in pull request #32959: URL: https://github.com/apache/spark/pull/32959#discussion_r664235831 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ## @@ -224,12 +224,12 @@ object DateTimeUtils { *

[GitHub] [spark] cloud-fan commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types

2021-07-06 Thread GitBox
cloud-fan commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-874465537 @sarutak any progress? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA removed a comment on pull request #33217: [SPARK-35735][SPARK-35768][SQL] Refactor code about parse string to DT/YM

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33217: URL: https://github.com/apache/spark/pull/33217#issuecomment-874018329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] cloud-fan commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
cloud-fan commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874297441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874464717 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] cloud-fan commented on a change in pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
cloud-fan commented on a change in pull request #33222: URL: https://github.com/apache/spark/pull/33222#discussion_r664228339 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala ## @@ -48,7 +48,7 @@ object RemoveRedundantProjects

[GitHub] [spark] SparkQA commented on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
SparkQA commented on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874391662 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33183: URL: https://github.com/apache/spark/pull/33183#issuecomment-874422409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] viirya commented on a change in pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
viirya commented on a change in pull request #33183: URL: https://github.com/apache/spark/pull/33183#discussion_r664101976 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala ## @@ -738,6 +738,31 @@ class

[GitHub] [spark] sunpe commented on pull request #33154: [SPARK-35949][CORE]Add `keep-spark-context-alive` arg for to prevent closing spark context after invoking main for some case

2021-07-06 Thread GitBox
sunpe commented on pull request #33154: URL: https://github.com/apache/spark/pull/33154#issuecomment-874422054 Hello @kotlovs . I added a arg called `keep-spark-context-alive` to set whether should keep spark context alive after invoke main method. -- This is an automated message from

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33228: Update README.md

2021-07-06 Thread GitBox
HyukjinKwon commented on a change in pull request #33228: URL: https://github.com/apache/spark/pull/33228#discussion_r664358710 ## File path: README.md ## @@ -21,6 +21,8 @@ You can find the latest Spark documentation, including a programming guide, on the [project web

[GitHub] [spark] SparkQA removed a comment on pull request #33225: [SPARK-35983][SQL] Allow from_json/to_json for map types where value types are day-time intervals

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33225: URL: https://github.com/apache/spark/pull/33225#issuecomment-874483254 **[Test build #140682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140682/testReport)** for PR 33225 at commit

[GitHub] [spark] HeartSaVioR closed pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

2021-07-06 Thread GitBox
HeartSaVioR closed pull request #32934: URL: https://github.com/apache/spark/pull/32934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] cloud-fan commented on a change in pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

2021-07-06 Thread GitBox
cloud-fan commented on a change in pull request #32401: URL: https://github.com/apache/spark/pull/32401#discussion_r664290337 ## File path: core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ## @@ -133,6 +145,23 @@ this.peakMemoryUsedBytes =

[GitHub] [spark] SparkQA removed a comment on pull request #33187: [SPARK-35988][SS] The implementation for RocksDBStateStoreProvider

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33187: URL: https://github.com/apache/spark/pull/33187#issuecomment-874490690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA removed a comment on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874304157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] gengliangwang closed pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

2021-07-06 Thread GitBox
gengliangwang closed pull request #33224: URL: https://github.com/apache/spark/pull/33224 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`

2021-07-06 Thread GitBox
HyukjinKwon commented on a change in pull request #33212: URL: https://github.com/apache/spark/pull/33212#discussion_r664430269 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala ## @@ -404,6 +407,19 @@ class JacksonParser(

[GitHub] [spark] ulysses-you commented on a change in pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
ulysses-you commented on a change in pull request #33222: URL: https://github.com/apache/spark/pull/33222#discussion_r664216184 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantProjects.scala ## @@ -48,7 +48,7 @@ object

[GitHub] [spark] gengliangwang commented on pull request #33215: [SPARK-35979][SQL] Return different timestamp literals based on the default timestamp type

2021-07-06 Thread GitBox
gengliangwang commented on pull request #33215: URL: https://github.com/apache/spark/pull/33215#issuecomment-874236323 Merging to master/branch-3.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] sarutak commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types

2021-07-06 Thread GitBox
sarutak commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-874467563 @cloud-fan This is in progress. SPARK-35983 blocks this to fix the issue pointed out [here](https://github.com/apache/spark/pull/32949#discussion_r659884896) but I'll open

[GitHub] [spark] xuanyuanking commented on a change in pull request #33187: [SPARK-35988][SS] The implementation for RocksDBStateStoreProvider

2021-07-06 Thread GitBox
xuanyuanking commented on a change in pull request #33187: URL: https://github.com/apache/spark/pull/33187#discussion_r664264022 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala ## @@ -0,0 +1,329 @@ +/* + *

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33226: [SPARK-35999][SQL] Make from_csv/to_csv to handle day-time intervals properly

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33226: URL: https://github.com/apache/spark/pull/33226#issuecomment-874586150 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45196/

[GitHub] [spark] MaxGekk closed pull request #33221: [SPARK-35977][SQL] Support non-reserved keyword TIMESTAMP_NTZ

2021-07-06 Thread GitBox
MaxGekk closed pull request #33221: URL: https://github.com/apache/spark/pull/33221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] cloud-fan commented on a change in pull request #33183: [SPARK-35972][SQL] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
cloud-fan commented on a change in pull request #33183: URL: https://github.com/apache/spark/pull/33183#discussion_r664062979 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala ## @@ -738,6 +738,31 @@ class

[GitHub] [spark] SparkQA commented on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

2021-07-06 Thread GitBox
SparkQA commented on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874405151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] sarutak commented on pull request #33225: [SPARK-35983][SQL] Allow from_json/to_json for map types where value types are day-time intervals

2021-07-06 Thread GitBox
sarutak commented on pull request #33225: URL: https://github.com/apache/spark/pull/33225#issuecomment-874485337 cc: @MaxGekk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] cloud-fan commented on a change in pull request #33191: [SPARK-35985][SQL] push partitionFilters for empty readDataSchema

2021-07-06 Thread GitBox
cloud-fan commented on a change in pull request #33191: URL: https://github.com/apache/spark/pull/33191#discussion_r664352501 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala ## @@ -120,7 +120,7 @@ private[sql]

[GitHub] [spark] Shockang commented on pull request #33101: [SPARK-35907][CORE] Instead of File#mkdirs, Files#createDirectories is expected.

2021-07-06 Thread GitBox
Shockang commented on pull request #33101: URL: https://github.com/apache/spark/pull/33101#issuecomment-874215443 > Looks fine. The tests may be overkill but OK :) > Are there any other instance of .mkdirs() out there that should use the utility method, while we're here? Thank

[GitHub] [spark] AmplabJenkins commented on pull request #33222: [SPARK-36020][SQL] Check logical link in remove redundant projects

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33222: URL: https://github.com/apache/spark/pull/33222#issuecomment-874306368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] cloud-fan commented on a change in pull request #33204: [SPARK-36011][SQL] Disallow altering permanent views based on temporary views or UDFs

2021-07-06 Thread GitBox
cloud-fan commented on a change in pull request #33204: URL: https://github.com/apache/spark/pull/33204#discussion_r664207077 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala ## @@ -465,4 +465,35 @@ class PersistedViewTestSuite

[GitHub] [spark] HyukjinKwon commented on pull request #33228: Update README.md

2021-07-06 Thread GitBox
HyukjinKwon commented on pull request #33228: URL: https://github.com/apache/spark/pull/33228#issuecomment-874577924 @drakelam, please read also https://spark.apache.org/contributing.html -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33224: [SPARK-35978][SQL] Support non-reserved keyword TIMESTAMP_LTZ

2021-07-06 Thread GitBox
AmplabJenkins removed a comment on pull request #33224: URL: https://github.com/apache/spark/pull/33224#issuecomment-874462583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins commented on pull request #33227: [SPARK-35972][SQL][3.1] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33227: URL: https://github.com/apache/spark/pull/33227#issuecomment-874600241 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45203/ --

[GitHub] [spark] xuanyuanking commented on a change in pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

2021-07-06 Thread GitBox
xuanyuanking commented on a change in pull request #33220: URL: https://github.com/apache/spark/pull/33220#discussion_r664360253 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala ## @@ -337,12 +338,12 @@ class RocksDBSuite

[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-874440343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA commented on pull request #33187: [WIP][SPARK-35988][SS] The implementation for RocksDBStateStoreProvider

2021-07-06 Thread GitBox
SparkQA commented on pull request #33187: URL: https://github.com/apache/spark/pull/33187#issuecomment-874490690 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] gengliangwang commented on a change in pull request #33229: [SPARK-36025][SQL][TESTS] Reduce the run time of DateExpressionsSuite

2021-07-06 Thread GitBox
gengliangwang commented on a change in pull request #33229: URL: https://github.com/apache/spark/pull/33229#discussion_r664400203 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeTestUtils.scala ## @@ -54,6 +55,13 @@ object

[GitHub] [spark] imback82 commented on a change in pull request #33213: [SPARK-34302][SQL][FOLLOWUP] More code cleanup

2021-07-06 Thread GitBox
imback82 commented on a change in pull request #33213: URL: https://github.com/apache/spark/pull/33213#discussion_r664064796 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalogSuite.scala ## @@ -321,7 +321,7 @@ class

[GitHub] [spark] linhongliu-db commented on a change in pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range

2021-07-06 Thread GitBox
linhongliu-db commented on a change in pull request #32959: URL: https://github.com/apache/spark/pull/32959#discussion_r664343346 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala ## @@ -224,12 +224,12 @@ object DateTimeUtils {

[GitHub] [spark] AmplabJenkins commented on pull request #33225: [SPARK-35983][SQL] Allow from_json/to_json for map types where value types are day-time intervals

2021-07-06 Thread GitBox
AmplabJenkins commented on pull request #33225: URL: https://github.com/apache/spark/pull/33225#issuecomment-874545801 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] cloud-fan commented on pull request #33226: [SPARK-35999][SQL] Make from_csv/to_csv to handle day-time intervals properly

2021-07-06 Thread GitBox
cloud-fan commented on pull request #33226: URL: https://github.com/apache/spark/pull/33226#issuecomment-874614117 thanks, merging to master/3.2! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] viirya commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

2021-07-06 Thread GitBox
viirya commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874406115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] MaxGekk closed pull request #33225: [SPARK-35983][SQL] Allow from_json/to_json for map types where value types are day-time intervals

2021-07-06 Thread GitBox
MaxGekk closed pull request #33225: URL: https://github.com/apache/spark/pull/33225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] AngersZhuuuu commented on pull request #33227: [SPARK-35972][SQL][3.0] When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-06 Thread GitBox
AngersZh commented on pull request #33227: URL: https://github.com/apache/spark/pull/33227#issuecomment-874528233 FYI @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA removed a comment on pull request #33221: [SPARK-35977][SQL] Support non-reserved keyword TIMESTAMP_NTZ

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33221: URL: https://github.com/apache/spark/pull/33221#issuecomment-874264660 **[Test build #140669 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140669/testReport)** for PR 33221 at commit

[GitHub] [spark] HeartSaVioR commented on pull request #32934: [SPARK-35788][SS] Metrics support for RocksDB instance

2021-07-06 Thread GitBox
HeartSaVioR commented on pull request #32934: URL: https://github.com/apache/spark/pull/32934#issuecomment-874408509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA commented on pull request #33229: [SPARK-36025][SQL][TESTS] Reduce the run time of DateExpressionsSuite

2021-07-06 Thread GitBox
SparkQA commented on pull request #33229: URL: https://github.com/apache/spark/pull/33229#issuecomment-874626885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA commented on pull request #33220: [WIP][SPARK-35993][TESTS] Fix flaky tests for RocksDBSuite

2021-07-06 Thread GitBox
SparkQA commented on pull request #33220: URL: https://github.com/apache/spark/pull/33220#issuecomment-874217616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] SparkQA removed a comment on pull request #33223: [SPARK-35684][INFRA][PYTHON] Bump up mypy version in GitHub Actions

2021-07-06 Thread GitBox
SparkQA removed a comment on pull request #33223: URL: https://github.com/apache/spark/pull/33223#issuecomment-874405151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

<    1   2   3   4   5   6   7   8   9   10   >