[GitHub] [spark] mridulm commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

2021-04-20 Thread GitBox
mridulm commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r617221199 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -28,16 +28,23 @@ import org.apache.spark.internal.config import

[GitHub] [spark] SparkQA removed a comment on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-823784240 **[Test build #137715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137715/testReport)** for PR 31437 at commit

[GitHub] [spark] SparkQA commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case

2021-04-20 Thread GitBox
SparkQA commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-823795031 **[Test build #137715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137715/testReport)** for PR 31437 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #31993: URL: https://github.com/apache/spark/pull/31993#issuecomment-823701348 **[Test build #137707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137707/testReport)** for PR 31993 at commit

[GitHub] [spark] wangyum commented on a change in pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
wangyum commented on a change in pull request #31993: URL: https://github.com/apache/spark/pull/31993#discussion_r617212769 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala ## @@ -21,9 +21,12 @@ import

[GitHub] [spark] wangyum commented on a change in pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
wangyum commented on a change in pull request #31993: URL: https://github.com/apache/spark/pull/31993#discussion_r617212439 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala ## @@ -40,7 +40,7 @@ import

[GitHub] [spark] SparkQA commented on pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
SparkQA commented on pull request #31993: URL: https://github.com/apache/spark/pull/31993#issuecomment-823788856 **[Test build #137707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137707/testReport)** for PR 31993 at commit

[GitHub] [spark] gengliangwang closed pull request #32258: [SPARK-35120][INFRA][FOLLOW-UP] Try catch an error to show the correct guidance

2021-04-20 Thread GitBox
gengliangwang closed pull request #32258: URL: https://github.com/apache/spark/pull/32258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] gengliangwang commented on pull request #32258: [SPARK-35120][INFRA][FOLLOW-UP] Try catch an error to show the correct guidance

2021-04-20 Thread GitBox
gengliangwang commented on pull request #32258: URL: https://github.com/apache/spark/pull/32258#issuecomment-823788078 Thanks, merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] SparkQA removed a comment on pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #32242: URL: https://github.com/apache/spark/pull/32242#issuecomment-823701221 **[Test build #137705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137705/testReport)** for PR 32242 at commit

[GitHub] [spark] SparkQA commented on pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

2021-04-20 Thread GitBox
SparkQA commented on pull request #32242: URL: https://github.com/apache/spark/pull/32242#issuecomment-823785651 **[Test build #137705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137705/testReport)** for PR 32242 at commit

[GitHub] [spark] SparkQA commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case

2021-04-20 Thread GitBox
SparkQA commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-823784240 **[Test build #137715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137715/testReport)** for PR 31437 at commit

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case

2021-04-20 Thread GitBox
AngersZh edited a comment on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-823763714 RM clean preempted container's info, then code in `ClientRMService` throw `ApplicationAttemptNotFoundException` ``` RMAppAttempt appAttempt =

[GitHub] [spark] MaxGekk commented on pull request #32252: [SPARK-35094][SQL]Spark from_json(JsonToStruct) function return wrong value in permissive mode

2021-04-20 Thread GitBox
MaxGekk commented on pull request #32252: URL: https://github.com/apache/spark/pull/32252#issuecomment-823781929 @HyukjinKwon Could you help to review this PR, please. It is related to https://github.com/apache/spark/pull/30031, I guess. -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823778063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32256: [SPARK-31225][SQL][2.4] Override sql method of OuterReference

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32256: URL: https://github.com/apache/spark/pull/32256#issuecomment-823778066 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42240/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-823778062 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42242/

[GitHub] [spark] AmplabJenkins commented on pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-823778062 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42242/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32256: [SPARK-31225][SQL][2.4] Override sql method of OuterReference

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32256: URL: https://github.com/apache/spark/pull/32256#issuecomment-823778066 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42240/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823778063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

2021-04-20 Thread GitBox
SparkQA commented on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-823776534 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823775190 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42241/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823773327 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42241/ -- This is an automated message from the Apache

[GitHub] [spark] MaxGekk closed pull request #32259: [SPARK-35113][SQL] Support ANSI intervals in the Hash expression

2021-04-20 Thread GitBox
MaxGekk closed pull request #32259: URL: https://github.com/apache/spark/pull/32259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] MaxGekk commented on pull request #32259: [SPARK-35113][SQL] Support ANSI intervals in the Hash expression

2021-04-20 Thread GitBox
MaxGekk commented on pull request #32259: URL: https://github.com/apache/spark/pull/32259#issuecomment-823771731 +1, LGTM. Merging to master. Thank you, @AngersZh . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] SparkQA removed a comment on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823760479 **[Test build #137713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137713/testReport)** for PR 32245 at commit

[GitHub] [spark] SparkQA commented on pull request #32256: [SPARK-31225][SQL][2.4] Override sql method of OuterReference

2021-04-20 Thread GitBox
SparkQA commented on pull request #32256: URL: https://github.com/apache/spark/pull/32256#issuecomment-823767899 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42240/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #32256: [SPARK-31225][SQL][2.4] Override sql method of OuterReference

2021-04-20 Thread GitBox
SparkQA commented on pull request #32256: URL: https://github.com/apache/spark/pull/32256#issuecomment-823767331 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42240/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823766582 **[Test build #137713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137713/testReport)** for PR 32245 at commit

[GitHub] [spark] AngersZhuuuu commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case

2021-04-20 Thread GitBox
AngersZh commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-823763714 RM clean preempted container's info, then code in `ClientRMService` throw `ApplicationAttemptNotFoundException` ``` RMAppAttempt appAttempt =

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823760909 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42238/

[GitHub] [spark] AmplabJenkins commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823760909 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42238/ --

[GitHub] [spark] SparkQA commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823760887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

2021-04-20 Thread GitBox
SparkQA commented on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-823760548 **[Test build #137714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137714/testReport)** for PR 32198 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823760238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #32256: [SPARK-31225][SQL][2.4] Override sql method of OuterReference

2021-04-20 Thread GitBox
SparkQA commented on pull request #32256: URL: https://github.com/apache/spark/pull/32256#issuecomment-823760453 **[Test build #137712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137712/testReport)** for PR 32256 at commit

[GitHub] [spark] SparkQA commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823760479 **[Test build #137713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137713/testReport)** for PR 32245 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823760238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823759465 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] c21 commented on pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

2021-04-20 Thread GitBox
c21 commented on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-823754660 @cloud-fan - updated the PR to keep single and concurrent writers implementation separately. The PR is ready for review again, thanks. -- This is an automated message from the

[GitHub] [spark] SparkQA removed a comment on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823746426 **[Test build #137711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137711/testReport)** for PR 32264 at commit

[GitHub] [spark] SparkQA commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823749761 **[Test build #137711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137711/testReport)** for PR 32264 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823745554 **[Test build #137710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137710/testReport)** for PR 32264 at commit

[GitHub] [spark] SparkQA commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823748951 **[Test build #137710 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137710/testReport)** for PR 32264 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823747962 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137704/

[GitHub] [spark] AmplabJenkins commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823747962 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137704/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32259: [SPARK-35113][SQL] Support ANSI intervals in the Hash expression

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32259: URL: https://github.com/apache/spark/pull/32259#issuecomment-823747175 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42237/

[GitHub] [spark] SparkQA removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823682380 **[Test build #137704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137704/testReport)** for PR 32248 at commit

[GitHub] [spark] SparkQA commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
SparkQA commented on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823747344 **[Test build #137704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137704/testReport)** for PR 32248 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #32259: [SPARK-35113][SQL] Support ANSI intervals in the Hash expression

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32259: URL: https://github.com/apache/spark/pull/32259#issuecomment-823747175 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42237/ --

[GitHub] [spark] SparkQA commented on pull request #32259: [SPARK-35113][SQL] Support ANSI intervals in the Hash expression

2021-04-20 Thread GitBox
SparkQA commented on pull request #32259: URL: https://github.com/apache/spark/pull/32259#issuecomment-823747153 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AngersZhuuuu commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
AngersZh commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823746928 I am not sure if we need to put this page in which menu page -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31010: [SPARK-33976][SQL][DOCS] Add a SQL doc page for a TRANSFORM clause

2021-04-20 Thread GitBox
AngersZh commented on a change in pull request #31010: URL: https://github.com/apache/spark/pull/31010#discussion_r617171323 ## File path: docs/sql-ref-syntax-qry-select-transform.md ## @@ -0,0 +1,235 @@ +--- +layout: global +title: TRANSFORM +displayTitle: TRANSFORM

[GitHub] [spark] SparkQA commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823746426 **[Test build #137711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137711/testReport)** for PR 32264 at commit

[GitHub] [spark] SparkQA commented on pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
SparkQA commented on pull request #32264: URL: https://github.com/apache/spark/pull/32264#issuecomment-823745554 **[Test build #137710 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137710/testReport)** for PR 32264 at commit

[GitHub] [spark] AngersZhuuuu opened a new pull request #32264: [SPARK-35159][SQL][DOCS] Extract hive format doc

2021-04-20 Thread GitBox
AngersZh opened a new pull request #32264: URL: https://github.com/apache/spark/pull/32264 ### What changes were proposed in this pull request? Extract common doc about hive format for `sql-ref-syntax-ddl-create-table-hiveformat.md` and `sql-ref-syntax-qry-select-transform.md` to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823742541 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42236/

[GitHub] [spark] AmplabJenkins commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823742541 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42236/ --

[GitHub] [spark] hddong commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
hddong commented on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823742124 @HyukjinKwon : I think there some different with `-S`. 1. `-S` cannot change log to `INFO`(or other level) 2. with log4j we can change `spark-sql` init log level. 3.

[GitHub] [spark] SparkQA commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823738562 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42236/ --

[GitHub] [spark] LuciferYang commented on pull request #32232: [SPARK-35135][CORE] Extract a class implementation of `WritablePartitionedIterator` to remove duplicate code

2021-04-20 Thread GitBox
LuciferYang commented on pull request #32232: URL: https://github.com/apache/spark/pull/32232#issuecomment-823734643 @Ngone51 Do you have time to review this pr ? thx ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] LuciferYang commented on a change in pull request #32261: [SPARK-35151][BUILD] Add compile arg to suppress `symbol literal is deprecated` compilation warnings in Scala 2.13

2021-04-20 Thread GitBox
LuciferYang commented on a change in pull request #32261: URL: https://github.com/apache/spark/pull/32261#discussion_r617159158 ## File path: pom.xml ## @@ -3369,6 +3369,7 @@ -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s

[GitHub] [spark] LuciferYang commented on a change in pull request #32261: [SPARK-35151][BUILD] Add compile arg to suppress `symbol literal is deprecated` compilation warnings in Scala 2.13

2021-04-20 Thread GitBox
LuciferYang commented on a change in pull request #32261: URL: https://github.com/apache/spark/pull/32261#discussion_r617159158 ## File path: pom.xml ## @@ -3369,6 +3369,7 @@ -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s

[GitHub] [spark] shardulm94 commented on pull request #31591: [SPARK-34472][YARN] Ship ivySettings file to driver in cluster mode

2021-04-20 Thread GitBox
shardulm94 commented on pull request #31591: URL: https://github.com/apache/spark/pull/31591#issuecomment-823726696 Thanks @tgravescs and @xkrogen for the reviews on this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823725038 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137708/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823725038 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137708/

[GitHub] [spark] SparkQA removed a comment on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823720317 **[Test build #137708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137708/testReport)** for PR 32245 at commit

[GitHub] [spark] SparkQA commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823724892 **[Test build #137708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137708/testReport)** for PR 32245 at commit

[GitHub] [spark] LuciferYang commented on pull request #32227: [SPARK-35132][BUILD][CORE] Upgrade netty-all to 4.1.63.Final

2021-04-20 Thread GitBox
LuciferYang commented on pull request #32227: URL: https://github.com/apache/spark/pull/32227#issuecomment-823724746 thx @HyukjinKwon @dongjoon-hyun @srowen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] SparkQA commented on pull request #32259: [SPARK-35113][SQL] Support ANSI intervals in the Hash expression

2021-04-20 Thread GitBox
SparkQA commented on pull request #32259: URL: https://github.com/apache/spark/pull/32259#issuecomment-823724338 **[Test build #137709 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137709/testReport)** for PR 32259 at commit

[GitHub] [spark] AngersZhuuuu commented on pull request #32259: [SPARK-35113][SQL] Support ANSI intervals in the Hash expression

2021-04-20 Thread GitBox
AngersZh commented on pull request #32259: URL: https://github.com/apache/spark/pull/32259#issuecomment-823724205 > Could you add comments about ANSI types like > >

[GitHub] [spark] sandeep-katta edited a comment on pull request #32194: [SPARK-35096][SQL] SchemaPruning should adhere spark.sql.caseSensitive config

2021-04-20 Thread GitBox
sandeep-katta edited a comment on pull request #32194: URL: https://github.com/apache/spark/pull/32194#issuecomment-823261848 @dongjoon-hyun @viirya , can this PR be merged ?. If not, I am happy to address review comments if there are any -- This is an automated message from the Apache

[GitHub] [spark] wangyum commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
wangyum commented on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823721949 Yes. We already have `-S` mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823066628 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] SparkQA commented on pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
SparkQA commented on pull request #32245: URL: https://github.com/apache/spark/pull/32245#issuecomment-823720317 **[Test build #137708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137708/testReport)** for PR 32245 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #31993: URL: https://github.com/apache/spark/pull/31993#issuecomment-823719172 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42235/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32242: URL: https://github.com/apache/spark/pull/32242#issuecomment-823719171 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42233/ --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-823719173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #31993: URL: https://github.com/apache/spark/pull/31993#issuecomment-823719172 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42235/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

2021-04-20 Thread GitBox
AmplabJenkins removed a comment on pull request #32242: URL: https://github.com/apache/spark/pull/32242#issuecomment-823719171 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42233/

[GitHub] [spark] AmplabJenkins commented on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

2021-04-20 Thread GitBox
AmplabJenkins commented on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-823719173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] HyukjinKwon commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
HyukjinKwon commented on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823717673 cc @wangyum FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql

2021-04-20 Thread GitBox
HyukjinKwon removed a comment on pull request #32248: URL: https://github.com/apache/spark/pull/32248#issuecomment-823681401 ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA commented on pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
SparkQA commented on pull request #31993: URL: https://github.com/apache/spark/pull/31993#issuecomment-823717509 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42235/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

2021-04-20 Thread GitBox
SparkQA commented on pull request #32242: URL: https://github.com/apache/spark/pull/32242#issuecomment-823717076 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42233/ --

[GitHub] [spark] SparkQA commented on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

2021-04-20 Thread GitBox
SparkQA commented on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-823716835 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] SparkQA commented on pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
SparkQA commented on pull request #31993: URL: https://github.com/apache/spark/pull/31993#issuecomment-823715879 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42235/ -- This is an automated message from the Apache

[GitHub] [spark] Yikun edited a comment on pull request #32246: [SPARK-35143][ML] Simplify Python for loop code

2021-04-20 Thread GitBox
Yikun edited a comment on pull request #32246: URL: https://github.com/apache/spark/pull/32246#issuecomment-823713170 > @ChenDou2021 ok,I know,thanks And much thanks your first step in Spark community, feel free to open the next PR if you find some other improvements. -- This is

[GitHub] [spark] Yikun commented on pull request #32246: [SPARK-35143][ML] Simplify Python for loop code

2021-04-20 Thread GitBox
Yikun commented on pull request #32246: URL: https://github.com/apache/spark/pull/32246#issuecomment-823713170 > @ChenDou2021 ok,I know,thanks And much thanks your first step in Spark community, feel free to open the next PR if you find to other improvements. -- This is an

[GitHub] [spark] SparkQA removed a comment on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

2021-04-20 Thread GitBox
SparkQA removed a comment on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-823701287 **[Test build #137706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137706/testReport)** for PR 32177 at commit

[GitHub] [spark] SparkQA commented on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

2021-04-20 Thread GitBox
SparkQA commented on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-823710538 **[Test build #137706 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137706/testReport)** for PR 32177 at commit

[GitHub] [spark] ChenDou2021 commented on pull request #32246: [SPARK-35143][ML] Simplify Python for loop code

2021-04-20 Thread GitBox
ChenDou2021 commented on pull request #32246: URL: https://github.com/apache/spark/pull/32246#issuecomment-823709828 ok,I know,thanks --原始邮件-- 发件人: "Yikun ***@***.***; 发送时间: 2021年4月21日(星期三) 上午9:30 收件人: ***@***.***; 抄送:

[GitHub] [spark] Yikun commented on pull request #32246: [SPARK-35143][ML] Simplify Python for loop code

2021-04-20 Thread GitBox
Yikun commented on pull request #32246: URL: https://github.com/apache/spark/pull/32246#issuecomment-823709270 @ChenDou2021 Hi, thanks for the contributions, as mentioned from @HyukjinKwon , the changes in your patch are all coding style preference (also, there were some pros and cons

[GitHub] [spark] ulysses-you commented on pull request #31653: [SPARK-33832][SQL] v2. move OptimzieSkewedJoin to query stage preparation

2021-04-20 Thread GitBox
ulysses-you commented on pull request #31653: URL: https://github.com/apache/spark/pull/31653#issuecomment-823707302 We found the same issue about failed to optimize skewed join due to the extra shuffle. Before submit a ticket, I just found this PR and

[GitHub] [spark] c21 commented on a change in pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

2021-04-20 Thread GitBox
c21 commented on a change in pull request #32198: URL: https://github.com/apache/spark/pull/32198#discussion_r617137444 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala ## @@ -247,34 +290,168 @@ class

[GitHub] [spark] c21 commented on a change in pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

2021-04-20 Thread GitBox
c21 commented on a change in pull request #32198: URL: https://github.com/apache/spark/pull/32198#discussion_r617137223 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala ## @@ -247,34 +290,168 @@ class

[GitHub] [spark] harupy commented on a change in pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
harupy commented on a change in pull request #32245: URL: https://github.com/apache/spark/pull/32245#discussion_r617135442 ## File path: python/pyspark/ml/tests/test_algorithms.py ## @@ -115,6 +115,7 @@ def test_output_columns(self): model = ovr.fit(df)

[GitHub] [spark] WeichenXu123 commented on a change in pull request #32245: [SPARK-35142][ML] Fix incorrect return type for `rawPredictionUDF` in `OneVsRestModel`

2021-04-20 Thread GitBox
WeichenXu123 commented on a change in pull request #32245: URL: https://github.com/apache/spark/pull/32245#discussion_r617131801 ## File path: python/pyspark/ml/tests/test_algorithms.py ## @@ -115,6 +115,7 @@ def test_output_columns(self): model = ovr.fit(df)

[GitHub] [spark] SparkQA commented on pull request #31993: [SPARK-34897][SQL] Support reconcile schemas based on index after nested column pruning

2021-04-20 Thread GitBox
SparkQA commented on pull request #31993: URL: https://github.com/apache/spark/pull/31993#issuecomment-823701348 **[Test build #137707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137707/testReport)** for PR 31993 at commit

[GitHub] [spark] SparkQA commented on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

2021-04-20 Thread GitBox
SparkQA commented on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-823701287 **[Test build #137706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137706/testReport)** for PR 32177 at commit

  1   2   3   4   5   6   >