[GitHub] [spark] yaooqinn commented on a change in pull request #30045: [SPARK-32991][SQL] Use conf in shared state as the original configuraion for RESET

2020-10-16 Thread GitBox
yaooqinn commented on a change in pull request #30045: URL: https://github.com/apache/spark/pull/30045#discussion_r506352466 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ## @@ -55,10 +55,11 @@ private[sql] class SharedState( Shared

[GitHub] [spark] yaooqinn commented on a change in pull request #30045: [SPARK-32991][SQL] Use conf in shared state as the original configuraion for RESET

2020-10-16 Thread GitBox
yaooqinn commented on a change in pull request #30045: URL: https://github.com/apache/spark/pull/30045#discussion_r506354101 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ## @@ -55,10 +55,11 @@ private[sql] class SharedState( Shared

[GitHub] [spark] SparkQA commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
SparkQA commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710010756 **[Test build #129885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129885/testReport)** for PR 30026 at commit [`5769222`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
SparkQA removed a comment on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-709899037 **[Test build #129885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129885/testReport)** for PR 30026 at commit [`5769222`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710011549 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710011549 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
SparkQA commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-710011513 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34499/ -

[GitHub] [spark] dossett commented on pull request #29231: [SPARK-32436][CORE] Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal

2020-10-16 Thread GitBox
dossett commented on pull request #29231: URL: https://github.com/apache/spark/pull/29231#issuecomment-710012954 @dongjoon-hyun Will this be included in a Spark 3.0.x release or is the plan to wait for a fix on the scala side? I ran into this very issue today, so just wondering. Thank yo

[GitHub] [spark] MaxGekk opened a new pull request #30067: [SPARK-33169][SQL][TESTS] Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-16 Thread GitBox
MaxGekk opened a new pull request #30067: URL: https://github.com/apache/spark/pull/30067 ### What changes were proposed in this pull request? 1. Add the common trait `CommonFileDataSourceSuite` with tests that can be executed for all built-in file-based datasources. 2. Add a test `Co

[GitHub] [spark] SparkQA commented on pull request #29982: [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql

2020-10-16 Thread GitBox
SparkQA commented on pull request #29982: URL: https://github.com/apache/spark/pull/29982#issuecomment-710014809 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34498/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29982: [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29982: URL: https://github.com/apache/spark/pull/29982#issuecomment-710014833 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins commented on pull request #29982: [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #29982: URL: https://github.com/apache/spark/pull/29982#issuecomment-710014833 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] MaxGekk commented on pull request #29976: [SPARK-33094][SQL] Make ORC format propagate Hadoop config from DS options to underlying HDFS file system

2020-10-16 Thread GitBox
MaxGekk commented on pull request #29976: URL: https://github.com/apache/spark/pull/29976#issuecomment-710015169 Here is the PR https://github.com/apache/spark/pull/30067 with common test. This is an automated message from th

[GitHub] [spark] MaxGekk commented on pull request #30067: [SPARK-33169][SQL][TESTS] Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-16 Thread GitBox
MaxGekk commented on pull request #30067: URL: https://github.com/apache/spark/pull/30067#issuecomment-710015456 @cloud-fan @HyukjinKwon Please, take a look at it when you have time. This is an automated message from the Apac

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29982: [SPARK-33100][SQL] Ignore a semicolon inside a bracketed comment in spark-sql

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29982: URL: https://github.com/apache/spark/pull/29982#issuecomment-710014843 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8

[GitHub] [spark] SparkQA commented on pull request #30067: [SPARK-33169][SQL][TESTS] Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-16 Thread GitBox
SparkQA commented on pull request #30067: URL: https://github.com/apache/spark/pull/30067#issuecomment-710017515 **[Test build #129896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129896/testReport)** for PR 30067 at commit [`dc7142a`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
SparkQA commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-710019726 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34499/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-710019753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-710019753 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710024768 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34500/ -

[GitHub] [spark] srowen commented on pull request #29231: [SPARK-32436][CORE] Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal

2020-10-16 Thread GitBox
srowen commented on pull request #29231: URL: https://github.com/apache/spark/pull/29231#issuecomment-710028325 This only seems to affect scala 2.13, regardless, and only 3.1.x supports scala 2.13, so no there isn't a need to put it in 3.0.x. The workaround doesn't require a scala fix if a

[GitHub] [spark] srowen commented on pull request #30034: [SPARK-33111][ML][Follow-Up] aft transform optimization - predictQuantiles

2020-10-16 Thread GitBox
srowen commented on pull request #30034: URL: https://github.com/apache/spark/pull/30034#issuecomment-710029484 It seems OK ; out of curiosity does this help much? This is an automated message from the Apache Git Service. To

[GitHub] [spark] dossett commented on pull request #29231: [SPARK-32436][CORE] Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal

2020-10-16 Thread GitBox
dossett commented on pull request #29231: URL: https://github.com/apache/spark/pull/29231#issuecomment-710031660 Thank you @srowen, the environment I saw this on was running spark 3.0.1 and scala 2.12.12. If I can reproduce it today I can share a stack trace and other details if that would

[GitHub] [spark] AmplabJenkins commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710036785 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710036771 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34500/ ---

[GitHub] [spark] SparkQA commented on pull request #30067: [SPARK-33169][SQL][TESTS] Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-16 Thread GitBox
SparkQA commented on pull request #30067: URL: https://github.com/apache/spark/pull/30067#issuecomment-710037274 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34501/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710036785 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA removed a comment on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710006244 **[Test build #129895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129895/testReport)** for PR 30018 at commit [`ef64abf`](https://gi

[GitHub] [spark] SparkQA commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710038924 **[Test build #129895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129895/testReport)** for PR 30018 at commit [`ef64abf`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710039315 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710039315 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710039325 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129

[GitHub] [spark] SparkQA commented on pull request #30056: [WIP][SPARK-33160][SQL] Allow saving/loading INT96 in parquet w/o rebasing

2020-10-16 Thread GitBox
SparkQA commented on pull request #30056: URL: https://github.com/apache/spark/pull/30056#issuecomment-710045243 **[Test build #129897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129897/testReport)** for PR 30056 at commit [`fb67c68`](https://github.com

[GitHub] [spark] tgravescs commented on pull request #29977: [SPARK-33088][CORE] Enhance ExecutorPlugin API to include callbacks on task start and end events

2020-10-16 Thread GitBox
tgravescs commented on pull request #29977: URL: https://github.com/apache/spark/pull/29977#issuecomment-710047148 I added @fsamuel-bs as a contributor in lira and assigned it to him. thanks.. This is an automated message fro

[GitHub] [spark] tgravescs edited a comment on pull request #29977: [SPARK-33088][CORE] Enhance ExecutorPlugin API to include callbacks on task start and end events

2020-10-16 Thread GitBox
tgravescs edited a comment on pull request #29977: URL: https://github.com/apache/spark/pull/29977#issuecomment-710047148 I added @fsamuel-bs as a contributor in jira and assigned it to him. thanks.. This is an automated mess

[GitHub] [spark] AmplabJenkins commented on pull request #30067: [SPARK-33169][SQL][TESTS] Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30067: URL: https://github.com/apache/spark/pull/30067#issuecomment-710048301 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30067: [SPARK-33169][SQL][TESTS] Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-16 Thread GitBox
SparkQA commented on pull request #30067: URL: https://github.com/apache/spark/pull/30067#issuecomment-710048233 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34501/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30067: [SPARK-33169][SQL][TESTS] Check propagation of datasource options to underlying file system for built-in file-based datasources

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30067: URL: https://github.com/apache/spark/pull/30067#issuecomment-710048301 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30023: [SPARK-33126][SQL] Simplify offset window function(Remove direction field)

2020-10-16 Thread GitBox
SparkQA commented on pull request #30023: URL: https://github.com/apache/spark/pull/30023#issuecomment-710049061 **[Test build #129889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129889/testReport)** for PR 30023 at commit [`07b3a92`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30023: [SPARK-33126][SQL] Simplify offset window function(Remove direction field)

2020-10-16 Thread GitBox
SparkQA removed a comment on pull request #30023: URL: https://github.com/apache/spark/pull/30023#issuecomment-709929939 **[Test build #129889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129889/testReport)** for PR 30023 at commit [`07b3a92`](https://gi

[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature

2020-10-16 Thread GitBox
tgravescs commented on pull request #29906: URL: https://github.com/apache/spark/pull/29906#issuecomment-710049939 @srowen @mridulm see the discussion back : https://github.com/apache/spark/pull/29906#discussion_r499125900 I'm fine either way. More input would be great so let me kno

[GitHub] [spark] tgravescs edited a comment on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature

2020-10-16 Thread GitBox
tgravescs edited a comment on pull request #29906: URL: https://github.com/apache/spark/pull/29906#issuecomment-710049939 @srowen @mridulm see the discussion back : https://github.com/apache/spark/pull/29906#discussion_r499125900 I'm fine either way. More input would be great so let

[GitHub] [spark] AmplabJenkins commented on pull request #30023: [SPARK-33126][SQL] Simplify offset window function(Remove direction field)

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30023: URL: https://github.com/apache/spark/pull/30023#issuecomment-710050202 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30023: [SPARK-33126][SQL] Simplify offset window function(Remove direction field)

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30023: URL: https://github.com/apache/spark/pull/30023#issuecomment-710050202 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] tgravescs commented on pull request #29906: [SPARK-32037][CORE] Rename blacklisting feature

2020-10-16 Thread GitBox
tgravescs commented on pull request #29906: URL: https://github.com/apache/spark/pull/29906#issuecomment-710051232 Before the comment I think to that is what I had -just deprecated, so it would be fairly easy to revert to it. Also cc @Ngone51 since he was part of that discussion ---

[GitHub] [spark] tanelk commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
tanelk commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710054486 > **[Test build #129895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129895/testReport)** for PR 30018 at commit [`ef64abf`](https://github.c

[GitHub] [spark] LuciferYang commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
LuciferYang commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710056295 Address 73e2ea6 reorganize the benchmark file This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
SparkQA commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710060401 **[Test build #129898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129898/testReport)** for PR 30026 at commit [`6d80788`](https://github.com

[GitHub] [spark] LuciferYang commented on a change in pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
LuciferYang commented on a change in pull request #30026: URL: https://github.com/apache/spark/pull/30026#discussion_r506446756 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala ## @@ -76,7 +79,7 @@ class BasicWriteTa

[GitHub] [spark] LuciferYang commented on a change in pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
LuciferYang commented on a change in pull request #30026: URL: https://github.com/apache/spark/pull/30026#discussion_r506448618 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/InsertTableWithDynamicPartitionsBenchmark.scala ## @@ -0,0 +1,138 @@ +

[GitHub] [spark] dossett commented on pull request #29231: [SPARK-32436][CORE] Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal

2020-10-16 Thread GitBox
dossett commented on pull request #29231: URL: https://github.com/apache/spark/pull/29231#issuecomment-710061815 Running on GCP's dataproc 2.0: ``` aniskodedossett@dossett-delta-w-0:~$ spark-sql --version Welcome to __ / __/__ ___ _/ /__

[GitHub] [spark] LuciferYang commented on a change in pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
LuciferYang commented on a change in pull request #30026: URL: https://github.com/apache/spark/pull/30026#discussion_r506448136 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/InsertTableWithDynamicPartitionsBenchmark.scala ## @@ -0,0 +1,138 @@ +

[GitHub] [spark] SparkQA commented on pull request #30056: [WIP][SPARK-33160][SQL] Allow saving/loading INT96 in parquet w/o rebasing

2020-10-16 Thread GitBox
SparkQA commented on pull request #30056: URL: https://github.com/apache/spark/pull/30056#issuecomment-710069516 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34502/ -

[GitHub] [spark] cloud-fan commented on a change in pull request #29587: [SPARK-32376][SQL] Make unionByName null-filling behavior work with struct columns

2020-10-16 Thread GitBox
cloud-fan commented on a change in pull request #29587: URL: https://github.com/apache/spark/pull/29587#discussion_r506473939 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala ## @@ -17,29 +17,188 @@ package org.apache.spark

[GitHub] [spark] akiyamaneko edited a comment on pull request #30035: [SPARK-33145] [WEBUI] Fix when `Succeeded Jobs` has many child url elements,they will extend over the edge of the page.

2020-10-16 Thread GitBox
akiyamaneko edited a comment on pull request #30035: URL: https://github.com/apache/spark/pull/30035#issuecomment-709997845 @gengliangwang I have attached the screenshot, or you can visit this: [fix-screenshot](https://user-images.githubusercontent.com/52202080/95977319-50734600-0e4b-11eb

[GitHub] [spark] SparkQA commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710075408 **[Test build #129899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129899/testReport)** for PR 30018 at commit [`4bf08bb`](https://github.com

[GitHub] [spark] cloud-fan commented on a change in pull request #29587: [SPARK-32376][SQL] Make unionByName null-filling behavior work with struct columns

2020-10-16 Thread GitBox
cloud-fan commented on a change in pull request #29587: URL: https://github.com/apache/spark/pull/29587#discussion_r506477673 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2067,6 +2067,13 @@ class Dataset[T] private[sql]( * // +++

[GitHub] [spark] SparkQA commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
SparkQA commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710079144 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34503/ -

[GitHub] [spark] ulysses-you commented on pull request #30029: [SPARK-33131][SQL] Fix grouping sets with having clause can not resolve qualified col name

2020-10-16 Thread GitBox
ulysses-you commented on pull request #30029: URL: https://github.com/apache/spark/pull/30029#issuecomment-710081710 @cloud-fan sure will do it tomorrow. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] cloud-fan commented on a change in pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
cloud-fan commented on a change in pull request #30026: URL: https://github.com/apache/spark/pull/30026#discussion_r506487800 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/InsertTableWithDynamicPartitionsBenchmark.scala ## @@ -0,0 +1,111 @@ +/*

[GitHub] [spark] SparkQA commented on pull request #30056: [WIP][SPARK-33160][SQL] Allow saving/loading INT96 in parquet w/o rebasing

2020-10-16 Thread GitBox
SparkQA commented on pull request #30056: URL: https://github.com/apache/spark/pull/30056#issuecomment-710083092 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34502/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #30056: [WIP][SPARK-33160][SQL] Allow saving/loading INT96 in parquet w/o rebasing

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30056: URL: https://github.com/apache/spark/pull/30056#issuecomment-710083115 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] cloud-fan commented on a change in pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
cloud-fan commented on a change in pull request #30026: URL: https://github.com/apache/spark/pull/30026#discussion_r506488151 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/InsertTableWithDynamicPartitionsBenchmark.scala ## @@ -0,0 +1,111 @@ +/*

[GitHub] [spark] cloud-fan commented on a change in pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
cloud-fan commented on a change in pull request #30026: URL: https://github.com/apache/spark/pull/30026#discussion_r506489291 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/BasicWriteJobStatsTrackerMetricSuite.scala ## @@ -0,0 +1,59 @@ +/* + *

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30056: [WIP][SPARK-33160][SQL] Allow saving/loading INT96 in parquet w/o rebasing

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30056: URL: https://github.com/apache/spark/pull/30056#issuecomment-710083115 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on a change in pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
cloud-fan commented on a change in pull request #30026: URL: https://github.com/apache/spark/pull/30026#discussion_r506491449 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/BasicWriteJobStatsTrackerMetricSuite.scala ## @@ -0,0 +1,59 @@ +/* + *

[GitHub] [spark] SparkQA commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
SparkQA commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710086101 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34504/ -

[GitHub] [spark] AmplabJenkins commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710088277 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
SparkQA commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710088262 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34503/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710088277 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] zsxwing commented on a change in pull request #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file

2020-10-16 Thread GitBox
zsxwing commented on a change in pull request #26312: URL: https://github.com/apache/spark/pull/26312#discussion_r506501666 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala ## @@ -281,6 +281,10 @@ object FileFormatWriter e

[GitHub] [spark] zsxwing commented on a change in pull request #26312: [SPARK-29649][SQL] Stop task set if FileAlreadyExistsException was thrown when writing to output file

2020-10-16 Thread GitBox
zsxwing commented on a change in pull request #26312: URL: https://github.com/apache/spark/pull/26312#discussion_r506501666 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala ## @@ -281,6 +281,10 @@ object FileFormatWriter e

[GitHub] [spark] AmplabJenkins commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710096702 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
SparkQA commented on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710096679 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34504/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30026: [SPARK-32978][SQL] Make sure the number of dynamic part metric is correct

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30026: URL: https://github.com/apache/spark/pull/30026#issuecomment-710096702 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710099229 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34505/ -

[GitHub] [spark] SparkQA commented on pull request #29891: [SPARK-30796][SQL] Add parameter position for REGEXP_REPLACE

2020-10-16 Thread GitBox
SparkQA commented on pull request #29891: URL: https://github.com/apache/spark/pull/29891#issuecomment-710102016 **[Test build #129891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129891/testReport)** for PR 29891 at commit [`5ea2cbb`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29891: [SPARK-30796][SQL] Add parameter position for REGEXP_REPLACE

2020-10-16 Thread GitBox
SparkQA removed a comment on pull request #29891: URL: https://github.com/apache/spark/pull/29891#issuecomment-709966078 **[Test build #129891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129891/testReport)** for PR 29891 at commit [`5ea2cbb`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29891: [SPARK-30796][SQL] Add parameter position for REGEXP_REPLACE

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #29891: URL: https://github.com/apache/spark/pull/29891#issuecomment-710103442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29891: [SPARK-30796][SQL] Add parameter position for REGEXP_REPLACE

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #29891: URL: https://github.com/apache/spark/pull/29891#issuecomment-710103442 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] gengliangwang commented on pull request #30035: [SPARK-33145] [WEBUI] Fix when `Succeeded Jobs` has many child url elements,they will extend over the edge of the page.

2020-10-16 Thread GitBox
gengliangwang commented on pull request #30035: URL: https://github.com/apache/spark/pull/30035#issuecomment-710105019 Thanks, merging to master This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [spark] gengliangwang closed pull request #30035: [SPARK-33145] [WEBUI] Fix when `Succeeded Jobs` has many child url elements,they will extend over the edge of the page.

2020-10-16 Thread GitBox
gengliangwang closed pull request #30035: URL: https://github.com/apache/spark/pull/30035 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710109991 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34505/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710110026 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30035: [SPARK-33145] [WEBUI] Fix when `Succeeded Jobs` has many child url elements,they will extend over the edge of the page.

2020-10-16 Thread GitBox
SparkQA commented on pull request #30035: URL: https://github.com/apache/spark/pull/30035#issuecomment-710110134 **[Test build #129892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129892/testReport)** for PR 30035 at commit [`db61218`](https://github.co

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710110026 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #30035: [SPARK-33145] [WEBUI] Fix when `Succeeded Jobs` has many child url elements,they will extend over the edge of the page.

2020-10-16 Thread GitBox
SparkQA removed a comment on pull request #30035: URL: https://github.com/apache/spark/pull/30035#issuecomment-709980869 **[Test build #129892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129892/testReport)** for PR 30035 at commit [`db61218`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710110045 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8

[GitHub] [spark] AmplabJenkins commented on pull request #30035: [SPARK-33145] [WEBUI] Fix when `Succeeded Jobs` has many child url elements,they will extend over the edge of the page.

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #30035: URL: https://github.com/apache/spark/pull/30035#issuecomment-710111632 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30035: [SPARK-33145] [WEBUI] Fix when `Succeeded Jobs` has many child url elements,they will extend over the edge of the page.

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #30035: URL: https://github.com/apache/spark/pull/30035#issuecomment-710111632 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] srowen commented on pull request #29231: [SPARK-32436][CORE] Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal

2020-10-16 Thread GitBox
srowen commented on pull request #29231: URL: https://github.com/apache/spark/pull/29231#issuecomment-710113429 That is strange - it doesn't seem to happen in Spark unit tests in 2.12. But Spark is on 2.12.10. From the links above, it seems like it could be an issue in 2.12.12. Therefore @

[GitHub] [spark] SparkQA commented on pull request #30045: [SPARK-32991][SQL] Use conf in shared state as the original configuraion for RESET

2020-10-16 Thread GitBox
SparkQA commented on pull request #30045: URL: https://github.com/apache/spark/pull/30045#issuecomment-710114143 **[Test build #129900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129900/testReport)** for PR 30045 at commit [`91c2e91`](https://github.com

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

2020-10-16 Thread GitBox
xkrogen commented on a change in pull request #29874: URL: https://github.com/apache/spark/pull/29874#discussion_r506547330 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ## @@ -61,7 +61,8 @@ private[hive] object IsolatedClien

[GitHub] [spark] AngersZhuuuu commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory

2020-10-16 Thread GitBox
AngersZh commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710117413 > The failure looks relevant. Could you check it with `hadoop-2.7`? > > ``` > - SPARK-25993 CREATE EXTERNAL TABLE with subdirectories *** FAILED *** > Expected

[GitHub] [spark] SparkQA commented on pull request #30027: [SPARK-32069][CORE][SQL][test-maven][test-hadoop2.7] Improve error message on reading unexpected directory

2020-10-16 Thread GitBox
SparkQA commented on pull request #30027: URL: https://github.com/apache/spark/pull/30027#issuecomment-710119210 **[Test build #129901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129901/testReport)** for PR 30027 at commit [`d5947e1`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
SparkQA commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-710124561 **[Test build #129894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129894/testReport)** for PR 2 at commit [`f657ff0`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
SparkQA removed a comment on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-709988169 **[Test build #129894 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129894/testReport)** for PR 2 at commit [`f657ff0`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
AmplabJenkins commented on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-710125763 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29999: [SPARK-33045][SQL] Support build-in function like_all and fix StackOverflowError issue.

2020-10-16 Thread GitBox
AmplabJenkins removed a comment on pull request #2: URL: https://github.com/apache/spark/pull/2#issuecomment-710125763 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

2020-10-16 Thread GitBox
SparkQA commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-710139858 **[Test build #129899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129899/testReport)** for PR 30018 at commit [`4bf08bb`](https://github.co

<    1   2   3   4   5   6   7   8   >