[GitHub] [spark] sadikovi commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
sadikovi commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r752882377 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala ## @@ -188,7 +193,13 @@ class

[GitHub] [spark] sadikovi commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader

2021-11-18 Thread GitBox
sadikovi commented on pull request #34659: URL: https://github.com/apache/spark/pull/34659#issuecomment-973781793 ACK. I will review it tomorrow or next week if that's okay . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] sadikovi commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
sadikovi commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r752884179 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala ## @@ -188,7 +193,13 @@ class

[GitHub] [spark] HyukjinKwon closed pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
HyukjinKwon closed pull request #34661: URL: https://github.com/apache/spark/pull/34661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
HyukjinKwon commented on pull request #34661: URL: https://github.com/apache/spark/pull/34661#issuecomment-973780848 Tests passed (https://github.com/HyukjinKwon/spark/runs/4260376896?check_suite_focus=true). Merged to master. -- This is an automated message from the Apache Git

[GitHub] [spark] sadikovi commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
sadikovi commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r752883351 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala ## @@ -121,6 +122,10 @@ class

[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973774051 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145430/ -- This

[GitHub] [spark] LuciferYang commented on pull request #34648: [SPARK-37282][TESTS][FOLLOWUP] Extract `Utils.isAppleSilicon` for reuse in UTs

2021-11-18 Thread GitBox
LuciferYang commented on pull request #34648: URL: https://github.com/apache/spark/pull/34648#issuecomment-973773944 @dongjoon-hyun d98e2cb rename `isAppleSilicon` to `isMacOnAppleSilicon` -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] HyukjinKwon commented on pull request #34663: [SPARK-37385][SQL][TESTS] Add tests for TimestampNTZ and TimestampLTZ for Parquet data source

2021-11-18 Thread GitBox
HyukjinKwon commented on pull request #34663: URL: https://github.com/apache/spark/pull/34663#issuecomment-973774496 cc @MaxGekk too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #34648: [SPARK-37282][TESTS][FOLLOWUP] Extract `Utils.isMacOnAppleSilicon` for reuse in UTs

2021-11-18 Thread GitBox
SparkQA commented on pull request #34648: URL: https://github.com/apache/spark/pull/34648#issuecomment-973774298 **[Test build #145438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145438/testReport)** for PR 34648 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973774051 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145430/

[GitHub] [spark] SparkQA removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973680863 **[Test build #145430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145430/testReport)** for PR 34326 at commit

[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
SparkQA commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973773382 **[Test build #145430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145430/testReport)** for PR 34326 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973745305 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145431/

[GitHub] [spark] SparkQA commented on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
SparkQA commented on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973772014 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49904/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
SparkQA commented on pull request #34661: URL: https://github.com/apache/spark/pull/34661#issuecomment-973770113 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49906/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
HyukjinKwon commented on pull request #34661: URL: https://github.com/apache/spark/pull/34661#issuecomment-973770130 Thanks @Ngone51 !! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973769698 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49903/

[GitHub] [spark] SparkQA commented on pull request #34662: [SPARK-37386][SQL] Simplify OptimizeSkewedJoin to not run the cost evaluator

2021-11-18 Thread GitBox
SparkQA commented on pull request #34662: URL: https://github.com/apache/spark/pull/34662#issuecomment-973770064 **[Test build #145437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145437/testReport)** for PR 34662 at commit

[GitHub] [spark] SparkQA commented on pull request #34663: [SPARK-37385][SQL][TESTS] Add tests for TimestampNTZ and TimestampLTZ for Parquet data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34663: URL: https://github.com/apache/spark/pull/34663#issuecomment-973769988 **[Test build #145436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145436/testReport)** for PR 34663 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973769698 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49903/ --

[GitHub] [spark] SparkQA commented on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973769012 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49907/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-973768497 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49905/ -- This is an automated message from the Apache

[GitHub] [spark] cloud-fan commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
cloud-fan commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r752870734 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala ## @@ -121,6 +122,10 @@ class

[GitHub] [spark] cloud-fan commented on a change in pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
cloud-fan commented on a change in pull request #34596: URL: https://github.com/apache/spark/pull/34596#discussion_r752870526 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala ## @@ -188,7 +193,13 @@ class

[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
SparkQA commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973762355 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49903/ -- This is an automated message from the

[GitHub] [spark] sadikovi commented on pull request #34663: [SPARK-37385][SQL][TESTS] Add tests for TimestampNTZ and TimestampLTZ for Parquet data source

2021-11-18 Thread GitBox
sadikovi commented on pull request #34663: URL: https://github.com/apache/spark/pull/34663#issuecomment-973758306 cc @gengliangwang @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] sadikovi opened a new pull request #34663: [SPARK-37385][SQL][TESTS] Add tests for TimestampNTZ and TimestampLTZ for Parquet data source

2021-11-18 Thread GitBox
sadikovi opened a new pull request #34663: URL: https://github.com/apache/spark/pull/34663 ### What changes were proposed in this pull request? The PR updates/adds more tests for TimestampNTZ and TimestampLTZ types support in Parquet data source to make sure most of the

[GitHub] [spark] c21 commented on pull request #34643: [SPARK-37370][SQL] Add SQL configs to control newly added join code-gen in 3.3

2021-11-18 Thread GitBox
c21 commented on pull request #34643: URL: https://github.com/apache/spark/pull/34643#issuecomment-973753257 Thank you @cloud-fan for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun commented on pull request #34630: [SPARK-37224][SS][FOLLOWUP] Add benchmark on basic state store operations

2021-11-18 Thread GitBox
dongjoon-hyun commented on pull request #34630: URL: https://github.com/apache/spark/pull/34630#issuecomment-973749532 cc @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] cloud-fan commented on pull request #34662: [SPARK-37386][SQL] Simplify OptimizeSkewedJoin to not run the cost evaluator

2021-11-18 Thread GitBox
cloud-fan commented on pull request #34662: URL: https://github.com/apache/spark/pull/34662#issuecomment-973747732 @ulysses-you @viirya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] cloud-fan opened a new pull request #34662: [SPARK-37386][SQL] Simplify OptimizeSkewedJoin to not run the cost evaluator

2021-11-18 Thread GitBox
cloud-fan opened a new pull request #34662: URL: https://github.com/apache/spark/pull/34662 ### What changes were proposed in this pull request? The only overhead `OptimizeSkewedJoin` may introduce is extra shuffles. We don't need to run the cost evaluator, which can be

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34646: [SPARK-37372][K8S] Remove redundant Pod label edition

2021-11-18 Thread GitBox
dongjoon-hyun commented on a change in pull request #34646: URL: https://github.com/apache/spark/pull/34646#discussion_r752862952 ## File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala ## @@ -34,7

[GitHub] [spark] SparkQA removed a comment on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973729486 **[Test build #145431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145431/testReport)** for PR 34660 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973745305 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145431/ -- This

[GitHub] [spark] SparkQA commented on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
SparkQA commented on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973744744 **[Test build #145431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145431/testReport)** for PR 34660 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #34620: [SPARK-37209][YARN][TESTS] Fix `YarnShuffleIntegrationSuite` releated UTs when using `hadoop-3.2` profile without `assembly/target/scal

2021-11-18 Thread GitBox
dongjoon-hyun commented on pull request #34620: URL: https://github.com/apache/spark/pull/34620#issuecomment-973743709 cc @sunchao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] sarutak commented on pull request #34649: [SPARK-35672][FOLLOWUP][TESTS] Add an exclusion rule to MimaExcludes.scala for Scala 2.13.

2021-11-18 Thread GitBox
sarutak commented on pull request #34649: URL: https://github.com/apache/spark/pull/34649#issuecomment-973743453 > BTW, when I rebased to the master, it seems that there is another issue. Oh, O.K, I'll check it. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] HyukjinKwon commented on pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
HyukjinKwon commented on pull request #34661: URL: https://github.com/apache/spark/pull/34661#issuecomment-973743092 Thanks @dongjoon-hyun ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon closed pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
HyukjinKwon closed pull request #34660: URL: https://github.com/apache/spark/pull/34660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
HyukjinKwon commented on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973741273 Tests got passed. Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] dongjoon-hyun commented on pull request #34649: [SPARK-35672][FOLLOWUP][TESTS] Add an exclusion rule to MimaExcludes.scala for Scala 2.13.

2021-11-18 Thread GitBox
dongjoon-hyun commented on pull request #34649: URL: https://github.com/apache/spark/pull/34649#issuecomment-973741055 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun closed pull request #34649: [SPARK-35672][FOLLOWUP][TESTS] Add an exclusion rule to MimaExcludes.scala for Scala 2.13.

2021-11-18 Thread GitBox
dongjoon-hyun closed pull request #34649: URL: https://github.com/apache/spark/pull/34649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34659: URL: https://github.com/apache/spark/pull/34659#issuecomment-973736175 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49902/

[GitHub] [spark] AmplabJenkins commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34659: URL: https://github.com/apache/spark/pull/34659#issuecomment-973736175 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49902/ --

[GitHub] [spark] SparkQA commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader

2021-11-18 Thread GitBox
SparkQA commented on pull request #34659: URL: https://github.com/apache/spark/pull/34659#issuecomment-973736109 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49902/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973732334 **[Test build #145435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145435/testReport)** for PR 34638 at commit

[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-973730980 **[Test build #145434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145434/testReport)** for PR 34596 at commit

[GitHub] [spark] SparkQA commented on pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
SparkQA commented on pull request #34661: URL: https://github.com/apache/spark/pull/34661#issuecomment-973730846 **[Test build #145433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145433/testReport)** for PR 34661 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
HyukjinKwon commented on pull request #34661: URL: https://github.com/apache/spark/pull/34661#issuecomment-973730572 cc @Ngone51, mind taking a quick look please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] HyukjinKwon opened a new pull request #34661: [SPARK-37384][CORE][TESTS] Increase timeout for job termination in SchedulerIntegrationSuite

2021-11-18 Thread GitBox
HyukjinKwon opened a new pull request #34661: URL: https://github.com/apache/spark/pull/34661 ### What changes were proposed in this pull request? This PR proposes to increase timeout for job termination in SchedulerIntegrationSuite to deflake the tests. ### Why are the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973729568 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145428/

[GitHub] [spark] cloud-fan commented on pull request #34645: [SPARK-37371][SQL] UnionExec should support columnar if all children support columnar

2021-11-18 Thread GitBox
cloud-fan commented on pull request #34645: URL: https://github.com/apache/spark/pull/34645#issuecomment-973729700 late LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] cloud-fan commented on pull request #34643: [SPARK-37370][SQL] Add SQL configs to control newly added join code-gen in 3.3

2021-11-18 Thread GitBox
cloud-fan commented on pull request #34643: URL: https://github.com/apache/spark/pull/34643#issuecomment-973729566 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA removed a comment on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973653649 **[Test build #145428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145428/testReport)** for PR 34638 at commit

[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-973729506 **[Test build #145432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145432/testReport)** for PR 34596 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973729568 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145428/ -- This

[GitHub] [spark] SparkQA commented on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
SparkQA commented on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973729486 **[Test build #145431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145431/testReport)** for PR 34660 at commit

[GitHub] [spark] SparkQA commented on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973729433 **[Test build #145428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145428/testReport)** for PR 34638 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34637: [WIP] Spark-37349 add SQL Rest API parsing logic

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34637: URL: https://github.com/apache/spark/pull/34637#issuecomment-973728747 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145421/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34645: [SPARK-37371][SQL] UnionExec should support columnar if all children support columnar

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34645: URL: https://github.com/apache/spark/pull/34645#issuecomment-973728750 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145412/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34653: [SPARK-37374][PYTHON] Fix StatCounter to use mergeStats when merging with self

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34653: URL: https://github.com/apache/spark/pull/34653#issuecomment-973728753 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145426/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973728749 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49900/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-973728748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34648: [SPARK-37282][TESTS][FOLLOWUP] Extract `Utils.isAppleSilicon` for reuse in UTs

2021-11-18 Thread GitBox
dongjoon-hyun commented on a change in pull request #34648: URL: https://github.com/apache/spark/pull/34648#discussion_r752857194 ## File path: core/src/main/scala/org/apache/spark/util/Utils.scala ## @@ -1961,6 +1961,11 @@ private[spark] object Utils extends Logging { */

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34647: [SPARK-36180][SQL] Support TimestampNTZ type in Hive

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34647: URL: https://github.com/apache/spark/pull/34647#issuecomment-973728751 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49901/

[GitHub] [spark] AmplabJenkins commented on pull request #34645: [SPARK-37371][SQL] UnionExec should support columnar if all children support columnar

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34645: URL: https://github.com/apache/spark/pull/34645#issuecomment-973728750 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145412/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34647: [SPARK-36180][SQL] Support TimestampNTZ type in Hive

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34647: URL: https://github.com/apache/spark/pull/34647#issuecomment-973728751 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49901/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34637: [WIP] Spark-37349 add SQL Rest API parsing logic

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34637: URL: https://github.com/apache/spark/pull/34637#issuecomment-973728747 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145421/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973728749 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49900/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-973728752 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] AmplabJenkins commented on pull request #34653: [SPARK-37374][PYTHON] Fix StatCounter to use mergeStats when merging with self

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34653: URL: https://github.com/apache/spark/pull/34653#issuecomment-973728753 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145426/ -- This

[GitHub] [spark] cloud-fan closed pull request #34643: [SPARK-37370][SQL] Add SQL configs to control newly added join code-gen in 3.3

2021-11-18 Thread GitBox
cloud-fan closed pull request #34643: URL: https://github.com/apache/spark/pull/34643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] dongjoon-hyun commented on pull request #34645: [SPARK-37371][SQL] UnionExec should support columnar if all children support columnar

2021-11-18 Thread GitBox
dongjoon-hyun commented on pull request #34645: URL: https://github.com/apache/spark/pull/34645#issuecomment-973728436 +1, LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon commented on a change in pull request #34401: [SPARK-30537][PYTHON], Fix toPandas wrong dtypes when applied on empty DF when Arrow enabled

2021-11-18 Thread GitBox
HyukjinKwon commented on a change in pull request #34401: URL: https://github.com/apache/spark/pull/34401#discussion_r752856092 ## File path: python/pyspark/sql/pandas/conversion.py ## @@ -171,7 +171,18 @@ def toPandas(self) -> "PandasDataFrameLike":

[GitHub] [spark] SparkQA commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer

2021-11-18 Thread GitBox
SparkQA commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973726375 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49903/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #34647: [SPARK-36180][SQL] Support TimestampNTZ type in Hive

2021-11-18 Thread GitBox
SparkQA commented on pull request #34647: URL: https://github.com/apache/spark/pull/34647#issuecomment-973726082 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49901/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34638: [SPARK-37360][SQL] Support TimestampNTZ in JSON data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34638: URL: https://github.com/apache/spark/pull/34638#issuecomment-973725928 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49900/ -- This is an automated message from the

[GitHub] [spark] cloud-fan commented on pull request #34659: [SPARK-34863][SQL] Support complex types for Parquet vectorized reader

2021-11-18 Thread GitBox
cloud-fan commented on pull request #34659: URL: https://github.com/apache/spark/pull/34659#issuecomment-973722372 cc @sadikovi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA removed a comment on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-973648801 **[Test build #145424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145424/testReport)** for PR 34656 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34653: [SPARK-37374][PYTHON] Fix StatCounter to use mergeStats when merging with self

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34653: URL: https://github.com/apache/spark/pull/34653#issuecomment-973650510 **[Test build #145426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145426/testReport)** for PR 34653 at commit

[GitHub] [spark] SparkQA commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-18 Thread GitBox
SparkQA commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-973719481 **[Test build #145424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145424/testReport)** for PR 34656 at commit

[GitHub] [spark] SparkQA commented on pull request #34653: [SPARK-37374][PYTHON] Fix StatCounter to use mergeStats when merging with self

2021-11-18 Thread GitBox
SparkQA commented on pull request #34653: URL: https://github.com/apache/spark/pull/34653#issuecomment-973719217 **[Test build #145426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145426/testReport)** for PR 34653 at commit

[GitHub] [spark] ulysses-you edited a comment on pull request #34602: [SPARK-37328][SQL] Fix bug that OptimizeSkewedJoin may not work after it was moved from queryStageOptimizerRules to queryStagePrep

2021-11-18 Thread GitBox
ulysses-you edited a comment on pull request #34602: URL: https://github.com/apache/spark/pull/34602#issuecomment-972458746 > And it will not only work in cases where just 2 tables join, many complex combination need to be considered, such as multiple table joins in same stage. Why

[GitHub] [spark] HyukjinKwon commented on pull request #34660: [SPARK-37038][SQL][TESTS][FOLLOWUP] Fix flaky test by loosening the number of results for TABLESAMPLE

2021-11-18 Thread GitBox
HyukjinKwon commented on pull request #34660: URL: https://github.com/apache/spark/pull/34660#issuecomment-973716925 Thanks @huaxingao ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] huaxingao opened a new pull request #34660: [SPARK-37038][SQL][FOLLOWUP] FIx flaky test

2021-11-18 Thread GitBox
huaxingao opened a new pull request #34660: URL: https://github.com/apache/spark/pull/34660 ### What changes were proposed in this pull request? Fix expected number of rows in sample test ### Why are the changes needed? TABLESAMPLE is not repeatable. Shouldn't use a

[GitHub] [spark] SparkQA removed a comment on pull request #34637: [WIP] Spark-37349 add SQL Rest API parsing logic

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34637: URL: https://github.com/apache/spark/pull/34637#issuecomment-973607825 **[Test build #145421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145421/testReport)** for PR 34637 at commit

[GitHub] [spark] SparkQA commented on pull request #34637: [WIP] Spark-37349 add SQL Rest API parsing logic

2021-11-18 Thread GitBox
SparkQA commented on pull request #34637: URL: https://github.com/apache/spark/pull/34637#issuecomment-973715463 **[Test build #145421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145421/testReport)** for PR 34637 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #34645: [SPARK-37371][SQL] UnionExec should support columnar if all children support columnar

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34645: URL: https://github.com/apache/spark/pull/34645#issuecomment-973335316 **[Test build #145412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145412/testReport)** for PR 34645 at commit

[GitHub] [spark] SparkQA commented on pull request #34645: [SPARK-37371][SQL] UnionExec should support columnar if all children support columnar

2021-11-18 Thread GitBox
SparkQA commented on pull request #34645: URL: https://github.com/apache/spark/pull/34645#issuecomment-973713860 **[Test build #145412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145412/testReport)** for PR 34645 at commit

[GitHub] [spark] SparkQA commented on pull request #34656: [SPARK-37376][SQL] Introduce a new DataSource V2 interface HasPartitionKey

2021-11-18 Thread GitBox
SparkQA commented on pull request #34656: URL: https://github.com/apache/spark/pull/34656#issuecomment-973712265 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49897/ -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
SparkQA removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-973648808 **[Test build #145425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145425/testReport)** for PR 34596 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34657: [WIP] Support TimedeltaIndex in pandas API on Spark

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-973711033 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49896/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-973711136 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145425/

[GitHub] [spark] AmplabJenkins commented on pull request #34657: [WIP] Support TimedeltaIndex in pandas API on Spark

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-973711033 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49896/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-973711136 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145425/ -- This

[GitHub] [spark] SparkQA commented on pull request #34596: [SPARK-37326][SQL] Support TimestampNTZ in CSV data source

2021-11-18 Thread GitBox
SparkQA commented on pull request #34596: URL: https://github.com/apache/spark/pull/34596#issuecomment-973711007 **[Test build #145425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145425/testReport)** for PR 34596 at commit

[GitHub] [spark] SparkQA commented on pull request #34657: [WIP] Support TimedeltaIndex in pandas API on Spark

2021-11-18 Thread GitBox
SparkQA commented on pull request #34657: URL: https://github.com/apache/spark/pull/34657#issuecomment-973711019 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49896/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34655: [SPARK-37380][PYTHON] Miscellaneous Python lint infra cleanup

2021-11-18 Thread GitBox
AmplabJenkins removed a comment on pull request #34655: URL: https://github.com/apache/spark/pull/34655#issuecomment-973709594 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145417/

[GitHub] [spark] AmplabJenkins commented on pull request #34655: [SPARK-37380][PYTHON] Miscellaneous Python lint infra cleanup

2021-11-18 Thread GitBox
AmplabJenkins commented on pull request #34655: URL: https://github.com/apache/spark/pull/34655#issuecomment-973709594 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145417/ -- This

<    1   2   3   4   5   6   7   8   >