[GitHub] [spark] HyukjinKwon closed pull request #31939: [SPARK-34832][SQL][TEST] Set EXECUTOR_ALLOW_SPARK_CONTEXT to true to ensure ExternalAppendOnlyUnsafeRowArrayBenchmark run successfully

2021-03-23 Thread GitBox
HyukjinKwon closed pull request #31939: URL: https://github.com/apache/spark/pull/31939 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] baohe-zhang commented on a change in pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
baohe-zhang commented on a change in pull request #31945: URL: https://github.com/apache/spark/pull/31945#discussion_r600193366 ## File path: core/src/test/scala/org/apache/spark/executor/ProcfsMetricsGetterSuite.scala ## @@ -38,4 +40,26 @@ class ProcfsMetricsGetterSuite

[GitHub] [spark] baohe-zhang commented on a change in pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
baohe-zhang commented on a change in pull request #31945: URL: https://github.com/apache/spark/pull/31945#discussion_r600192481 ## File path: core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ## @@ -199,7 +201,7 @@ private[spark] class

[GitHub] [spark] SparkQA removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
SparkQA removed a comment on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-805454967 **[Test build #136432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136432/testReport)** for PR 31517 at commit

[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
SparkQA commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-805519253 **[Test build #136432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136432/testReport)** for PR 31517 at commit

[GitHub] [spark] LuciferYang commented on a change in pull request #31939: [SPARK-34832][SQL][TEST] Set EXECUTOR_ALLOW_SPARK_CONTEXT to true to ensure ExternalAppendOnlyUnsafeRowArrayBenchmark run suc

2021-03-23 Thread GitBox
LuciferYang commented on a change in pull request #31939: URL: https://github.com/apache/spark/pull/31939#discussion_r600189025 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArrayBenchmark.scala ## @@ -47,6 +47,7 @@ object

[GitHub] [spark] xuanyuanking commented on pull request #31944: [SPARK-34297][SQL] Expose source metrics API and Add Kafka metrics to report delay.

2021-03-23 Thread GitBox
xuanyuanking commented on pull request #31944: URL: https://github.com/apache/spark/pull/31944#issuecomment-805517034 Agree with @viirya on creating another JIRA. Let's link these two tickets together. IMO, we should have the metrics both in UI and progress reporter. We can do the

[GitHub] [spark] SparkQA commented on pull request #31946: [WIP][SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
SparkQA commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805516703 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41017/ -- This is an automated message from the

[GitHub] [spark] cloud-fan commented on pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

2021-03-23 Thread GitBox
cloud-fan commented on pull request #31854: URL: https://github.com/apache/spark/pull/31854#issuecomment-805513969 thanks, merging to master/3.1/3.0! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] cloud-fan closed pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

2021-03-23 Thread GitBox
cloud-fan closed pull request #31854: URL: https://github.com/apache/spark/pull/31854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] cloud-fan commented on a change in pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

2021-03-23 Thread GitBox
cloud-fan commented on a change in pull request #31854: URL: https://github.com/apache/spark/pull/31854#discussion_r600185552 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala ## @@ -2738,6 +2738,31 @@ class DataFrameSuite extends QueryTest

[GitHub] [spark] SparkQA commented on pull request #31935: [WIP][SPARK-34789][TEST] Introduce Jetty based construct for integration tests where HTTP server is used

2021-03-23 Thread GitBox
SparkQA commented on pull request #31935: URL: https://github.com/apache/spark/pull/31935#issuecomment-805510429 **[Test build #136436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136436/testReport)** for PR 31935 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31936: [SPARK-34828][YARN] Make shuffle service name configurable on client side and allow for classpath-based config override on

2021-03-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #31936: URL: https://github.com/apache/spark/pull/31936#discussion_r600183889 ## File path: docs/running-on-yarn.md ## @@ -761,8 +761,27 @@ The following extra configuration options are available when the shuffle service

[GitHub] [spark] attilapiros commented on pull request #31935: [WIP][SPARK-34789][TEST] Introduce Jetty based construct for integration tests where HTTP server is used

2021-03-23 Thread GitBox
attilapiros commented on pull request #31935: URL: https://github.com/apache/spark/pull/31935#issuecomment-805509142 jenkins retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] attilapiros commented on pull request #31935: [WIP][SPARK-34789][TEST] Introduce Jetty based construct for integration tests where HTTP server is used

2021-03-23 Thread GitBox
attilapiros commented on pull request #31935: URL: https://github.com/apache/spark/pull/31935#issuecomment-805508938 > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41004/ ``` FileNotFoundException means that the credentials Jenkins is using is probably

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-805503802 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41015/

[GitHub] [spark] SparkQA removed a comment on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
SparkQA removed a comment on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-805474689 **[Test build #136434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136434/testReport)** for PR 31859 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-805503804 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #30684: [SPARK-33720][K8S] Support submit to k8s only with token

2021-03-23 Thread GitBox
dongjoon-hyun edited a comment on pull request #30684: URL: https://github.com/apache/spark/pull/30684#issuecomment-805477566 Sorry for the delay, @hddong . I'll review right now again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #31947: [SPARK-34847][SQL] Simplify ResolveAggregateFunctions

2021-03-23 Thread GitBox
SparkQA commented on pull request #31947: URL: https://github.com/apache/spark/pull/31947#issuecomment-805504328 **[Test build #136435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136435/testReport)** for PR 31947 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-805503805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-805503802 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41015/ --

[GitHub] [spark] dongjoon-hyun commented on pull request #30684: [SPARK-33720][K8S] Support submit to k8s only with token

2021-03-23 Thread GitBox
dongjoon-hyun commented on pull request #30684: URL: https://github.com/apache/spark/pull/30684#issuecomment-805501425 cc @attilapiros , too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun closed pull request #30684: [SPARK-33720][K8S] Support submit to k8s only with token

2021-03-23 Thread GitBox
dongjoon-hyun closed pull request #30684: URL: https://github.com/apache/spark/pull/30684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] SparkQA commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
SparkQA commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-805500343 **[Test build #136434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136434/testReport)** for PR 31859 at commit

[GitHub] [spark] SparkQA commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
SparkQA commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-805499048 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41018/ -- This is an automated message from the

[GitHub] [spark] cloud-fan opened a new pull request #31947: [SPARK-34847][SQL] Simplify ResolveAggregateFunctions

2021-03-23 Thread GitBox
cloud-fan opened a new pull request #31947: URL: https://github.com/apache/spark/pull/31947 ### What changes were proposed in this pull request? The current `ResolveAggregateFunctions` is very complicated. It recursively calls the entire analyzer, and has duplicated code for

[GitHub] [spark] maropu commented on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
maropu commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805496327 yea, the current one is not acceptable, so I'll look for another way. https://github.com/apache/spark/pull/31946#discussion_r600145150 -- This is an automated message from the

[GitHub] [spark] Ngone51 commented on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
Ngone51 commented on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805496338 Ok, I get it now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
SparkQA commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-805495626 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41018/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
SparkQA commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805495426 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41017/ -- This is an automated message from the Apache

[GitHub] [spark] otterc edited a comment on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
otterc edited a comment on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805493922 > > * If stream2Duplicate encounters a failure now, it has isWriting on and so can reset currentMapId of partition_0_2. This again gives a chance to another stream say

[GitHub] [spark] otterc edited a comment on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
otterc edited a comment on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805061280 > hmm..I try to understand how those 2 scenarios cause the merged block corrupted. > > 1. Do you mean called `StreamCallback.onFailure()` for 2 times cause the

[GitHub] [spark] otterc commented on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
otterc commented on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805493922 > > * If stream2Duplicate encounters a failure now, it has isWriting on and so can reset currentMapId of partition_0_2. This again gives a chance to another stream say stream3

[GitHub] [spark] SparkQA commented on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
SparkQA commented on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-805493787 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41015/ -- This is an automated message from the

[GitHub] [spark] maropu commented on a change in pull request #31913: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function

2021-03-23 Thread GitBox
maropu commented on a change in pull request #31913: URL: https://github.com/apache/spark/pull/31913#discussion_r600167101 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -4140,6 +4140,20 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] otterc edited a comment on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
otterc edited a comment on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805488580 > To confirm, for the example 2, I think the first 2 steps are not necessary, right? @Ngone51 I think the first 2 steps are necessary because in this edge case

[GitHub] [spark] Ngone51 commented on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
Ngone51 commented on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805492402 > * If stream2Duplicate encounters a failure now, it has isWriting on and so can reset currentMapId of partition_0_2. This again gives a chance to another stream say stream3 to

[GitHub] [spark] cloud-fan commented on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
cloud-fan commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805490152 If we always sort the result, how can we test the sort behavior? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] otterc edited a comment on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
otterc edited a comment on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805488580 > To confirm, for the example 2, I think the first 2 steps are not necessary, right? @Ngone51 I think the first 2 steps are necessary because in this edge case

[GitHub] [spark] otterc commented on pull request #31934: [SPARK-34840][SHUFFLE] Fixes cases of corruption in merged shuffle …

2021-03-23 Thread GitBox
otterc commented on pull request #31934: URL: https://github.com/apache/spark/pull/31934#issuecomment-805488580 > To confirm, for the example 2, I think the first 2 steps are not necessary, right? @Ngone51 I think the first 2 steps are necessary because in this edge case this can

[GitHub] [spark] sarutak commented on a change in pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

2021-03-23 Thread GitBox
sarutak commented on a change in pull request #31854: URL: https://github.com/apache/spark/pull/31854#discussion_r600162755 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala ## @@ -2738,6 +2738,31 @@ class DataFrameSuite extends QueryTest

[GitHub] [spark] sarutak commented on a change in pull request #31854: [SPARK-34763][SQL] col(), $"" and df("name") should handle quoted column names properly

2021-03-23 Thread GitBox
sarutak commented on a change in pull request #31854: URL: https://github.com/apache/spark/pull/31854#discussion_r600162755 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala ## @@ -2738,6 +2738,31 @@ class DataFrameSuite extends QueryTest

[GitHub] [spark] dongjoon-hyun commented on pull request #30684: [SPARK-33720][K8S] Support submit to k8s only with token

2021-03-23 Thread GitBox
dongjoon-hyun commented on pull request #30684: URL: https://github.com/apache/spark/pull/30684#issuecomment-805477566 Sorry for the detail, @hddong . I'll review right now again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
SparkQA commented on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-805476978 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41015/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
SparkQA commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-805474689 **[Test build #136434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136434/testReport)** for PR 31859 at commit

[GitHub] [spark] SparkQA commented on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
SparkQA commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805474618 **[Test build #136433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136433/testReport)** for PR 31946 at commit

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
dongjoon-hyun edited a comment on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805467058 This seems controversial, but I'll not be against with your proposal. cc @gatorsmile , @cloud-fan -- This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-805473352 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41014/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31940: [SPARK-34833][SQL] Apply right-padding correctly for correlated subqueries

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31940: URL: https://github.com/apache/spark/pull/31940#issuecomment-805473351 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136427/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-805473353 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41016/

[GitHub] [spark] AmplabJenkins commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-805473353 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41016/ --

[GitHub] [spark] AmplabJenkins commented on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-805473352 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41014/ --

[GitHub] [spark] AmplabJenkins commented on pull request #31940: [SPARK-34833][SQL] Apply right-padding correctly for correlated subqueries

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31940: URL: https://github.com/apache/spark/pull/31940#issuecomment-805473351 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136427/ -- This

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #31945: URL: https://github.com/apache/spark/pull/31945#discussion_r600151187 ## File path: core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ## @@ -199,7 +201,7 @@ private[spark] class

[GitHub] [spark] SparkQA commented on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

2021-03-23 Thread GitBox
SparkQA commented on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-805468953 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41014/ -- This is an automated message from the

[GitHub] [spark] yaooqinn edited a comment on pull request #31940: [SPARK-34833][SQL] Apply right-padding correctly for correlated subqueries

2021-03-23 Thread GitBox
yaooqinn edited a comment on pull request #31940: URL: https://github.com/apache/spark/pull/31940#issuecomment-805417257 > The PR LGTM. > > But I have a quick question, a bit unrelated to this PR but related to fixed-length string columns. Is this expected that we have in Spark:

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #31945: URL: https://github.com/apache/spark/pull/31945#discussion_r600145269 ## File path: core/src/test/scala/org/apache/spark/executor/ProcfsMetricsGetterSuite.scala ## @@ -38,4 +40,26 @@ class ProcfsMetricsGetterSuite

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #31945: URL: https://github.com/apache/spark/pull/31945#discussion_r600145124 ## File path: core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala ## @@ -210,11 +212,16 @@ private[spark] class

[GitHub] [spark] maropu commented on a change in pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
maropu commented on a change in pull request #31946: URL: https://github.com/apache/spark/pull/31946#discussion_r600145150 ## File path: sql/core/src/test/resources/sql-tests/results/group-analytics.sql.out ## @@ -101,29 +101,29 @@ SELECT course, year, SUM(earnings) FROM

[GitHub] [spark] SparkQA removed a comment on pull request #31940: [SPARK-34833][SQL] Apply right-padding correctly for correlated subqueries

2021-03-23 Thread GitBox
SparkQA removed a comment on pull request #31940: URL: https://github.com/apache/spark/pull/31940#issuecomment-805344866 **[Test build #136427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136427/testReport)** for PR 31940 at commit

[GitHub] [spark] SparkQA commented on pull request #31940: [SPARK-34833][SQL] Apply right-padding correctly for correlated subqueries

2021-03-23 Thread GitBox
SparkQA commented on pull request #31940: URL: https://github.com/apache/spark/pull/31940#issuecomment-805467463 **[Test build #136427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136427/testReport)** for PR 31940 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
dongjoon-hyun commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805467058 This seems to be controversial, but I'll not be against with your proposal. cc @gatorsmile , @cloud-fan -- This is an automated message from the Apache Git

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #31946: URL: https://github.com/apache/spark/pull/31946#discussion_r600143853 ## File path: sql/core/src/test/resources/sql-tests/results/group-analytics.sql.out ## @@ -244,8 +244,8 @@ HAVING GROUPING(year) = 1 AND

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
dongjoon-hyun commented on a change in pull request #31946: URL: https://github.com/apache/spark/pull/31946#discussion_r600143217 ## File path: sql/core/src/test/resources/sql-tests/results/group-analytics.sql.out ## @@ -101,29 +101,29 @@ SELECT course, year, SUM(earnings)

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31939: [SPARK-34832][SQL][TEST] Set EXECUTOR_ALLOW_SPARK_CONTEXT to true to ensure ExternalAppendOnlyUnsafeRowArrayBenchmark run suc

2021-03-23 Thread GitBox
HyukjinKwon commented on a change in pull request #31939: URL: https://github.com/apache/spark/pull/31939#discussion_r600142995 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/ExternalAppendOnlyUnsafeRowArrayBenchmark.scala ## @@ -47,6 +47,7 @@ object

[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
SparkQA commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-805466009 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41016/ --

[GitHub] [spark] maropu commented on a change in pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
maropu commented on a change in pull request #31946: URL: https://github.com/apache/spark/pull/31946#discussion_r600142098 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ## @@ -405,7 +404,7 @@ class SQLQueryTestSuite extends QueryTest with

[GitHub] [spark] maropu commented on a change in pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
maropu commented on a change in pull request #31946: URL: https://github.com/apache/spark/pull/31946#discussion_r600140456 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ## @@ -59,8 +58,7 @@ import org.apache.spark.util.Utils * * To

[GitHub] [spark] ben-manes commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
ben-manes commented on a change in pull request #31517: URL: https://github.com/apache/spark/pull/31517#discussion_r600139517 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubExprEvaluationRuntimeSuite.scala ## @@ -45,6 +45,8 @@ class

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
HyukjinKwon commented on a change in pull request #31946: URL: https://github.com/apache/spark/pull/31946#discussion_r600139442 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ## @@ -59,8 +58,7 @@ import org.apache.spark.util.Utils * * To

[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
LuciferYang commented on a change in pull request #31517: URL: https://github.com/apache/spark/pull/31517#discussion_r600136193 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SubExprEvaluationRuntimeSuite.scala ## @@ -45,6 +45,8 @@ class

[GitHub] [spark] huaxingao commented on a change in pull request #31899: [SPARK-34525][DOCS] Update Spark Create Table DDL to reflect alternative key value notation

2021-03-23 Thread GitBox
huaxingao commented on a change in pull request #31899: URL: https://github.com/apache/spark/pull/31899#discussion_r600134154 ## File path: docs/sql-ref-syntax-ddl-create-table-datasource.md ## @@ -29,14 +29,14 @@ The `CREATE TABLE` statement defines a new table using a Data

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31791: [SPARK-34678][SQL] Add table function registry

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31791: URL: https://github.com/apache/spark/pull/31791#issuecomment-805453776 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136423/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-798871084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31945: URL: https://github.com/apache/spark/pull/31945#issuecomment-805453777 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41013/

[GitHub] [spark] SparkQA removed a comment on pull request #31937: [SPARK-10816][SS] Support session window natively

2021-03-23 Thread GitBox
SparkQA removed a comment on pull request #31937: URL: https://github.com/apache/spark/pull/31937#issuecomment-805308046 **[Test build #136422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136422/testReport)** for PR 31937 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31937: [SPARK-10816][SS] Support session window natively

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31937: URL: https://github.com/apache/spark/pull/31937#issuecomment-805456080 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136422/

[GitHub] [spark] SparkQA removed a comment on pull request #31791: [SPARK-34678][SQL] Add table function registry

2021-03-23 Thread GitBox
SparkQA removed a comment on pull request #31791: URL: https://github.com/apache/spark/pull/31791#issuecomment-805308158 **[Test build #136423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136423/testReport)** for PR 31791 at commit

[GitHub] [spark] ben-manes commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
ben-manes commented on a change in pull request #31517: URL: https://github.com/apache/spark/pull/31517#discussion_r600131955 ## File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ## @@ -223,7 +224,10 @@ private[history] class

[GitHub] [spark] AmplabJenkins commented on pull request #31937: [SPARK-10816][SS] Support session window natively

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31937: URL: https://github.com/apache/spark/pull/31937#issuecomment-805456080 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136422/ -- This

[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
LuciferYang commented on a change in pull request #31517: URL: https://github.com/apache/spark/pull/31517#discussion_r600131352 ## File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ## @@ -223,7 +224,10 @@ private[history] class

[GitHub] [spark] SparkQA commented on pull request #31937: [SPARK-10816][SS] Support session window natively

2021-03-23 Thread GitBox
SparkQA commented on pull request #31937: URL: https://github.com/apache/spark/pull/31937#issuecomment-805455411 **[Test build #136422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136422/testReport)** for PR 31937 at commit

[GitHub] [spark] SparkQA commented on pull request #31517: [WIP][SPARK-34309][BUILD][CORE][SQL] [K8S]Use Caffeine instead of Guava Cache

2021-03-23 Thread GitBox
SparkQA commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-805454967 **[Test build #136432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136432/testReport)** for PR 31517 at commit

[GitHub] [spark] SparkQA commented on pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
SparkQA commented on pull request #31830: URL: https://github.com/apache/spark/pull/31830#issuecomment-805454834 **[Test build #136431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136431/testReport)** for PR 31830 at commit

[GitHub] [spark] SparkQA commented on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
SparkQA commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805454389 Kubernetes integration test unable to build dist. exiting with code: 141 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41012/ --

[GitHub] [spark] AmplabJenkins commented on pull request #31791: [SPARK-34678][SQL] Add table function registry

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31791: URL: https://github.com/apache/spark/pull/31791#issuecomment-805453776 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136423/ -- This

[GitHub] [spark] SparkQA commented on pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
SparkQA commented on pull request #31945: URL: https://github.com/apache/spark/pull/31945#issuecomment-805453756 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41013/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31945: URL: https://github.com/apache/spark/pull/31945#issuecomment-805453777 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41013/ --

[GitHub] [spark] gengliangwang commented on a change in pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

2021-03-23 Thread GitBox
gengliangwang commented on a change in pull request #31859: URL: https://github.com/apache/spark/pull/31859#discussion_r600129509 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercion.scala ## @@ -191,9 +194,35 @@ object

[GitHub] [spark] SparkQA commented on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

2021-03-23 Thread GitBox
SparkQA commented on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-805453365 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41014/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #31791: [SPARK-34678][SQL] Add table function registry

2021-03-23 Thread GitBox
SparkQA commented on pull request #31791: URL: https://github.com/apache/spark/pull/31791#issuecomment-805452922 **[Test build #136423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136423/testReport)** for PR 31791 at commit

[GitHub] [spark] ulysses-you commented on a change in pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
ulysses-you commented on a change in pull request #31830: URL: https://github.com/apache/spark/pull/31830#discussion_r600125011 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala ## @@ -46,6 +46,7 @@ case class

[GitHub] [spark] ulysses-you commented on a change in pull request #31830: [SPARK-34735][SQL][UI] Add modified configs for SQL execution in UI

2021-03-23 Thread GitBox
ulysses-you commented on a change in pull request #31830: URL: https://github.com/apache/spark/pull/31830#discussion_r600124482 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLJsonProtocolSuite.scala ## @@ -28,28 +29,46 @@ import

[GitHub] [spark] HeartSaVioR commented on a change in pull request #31638: [SPARK-34526][SS] Add a flag to skip checking file sink format and handle glob path

2021-03-23 Thread GitBox
HeartSaVioR commented on a change in pull request #31638: URL: https://github.com/apache/spark/pull/31638#discussion_r600114628 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala ## @@ -40,17 +41,35 @@ object FileStreamSink

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-805433664 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136430/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805433661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
AmplabJenkins removed a comment on pull request #31945: URL: https://github.com/apache/spark/pull/31945#issuecomment-805433662 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136429/

[GitHub] [spark] AmplabJenkins commented on pull request #31946: [SPARK-34846][SQL][TEST] Sort output rows in SQLQueryTestSuite even if a plan has a Sort node

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31946: URL: https://github.com/apache/spark/pull/31946#issuecomment-805433661 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #31945: [SPARK-34845][CORE] ProcfsMetricsGetter shouldn't return partial procfs metrics

2021-03-23 Thread GitBox
AmplabJenkins commented on pull request #31945: URL: https://github.com/apache/spark/pull/31945#issuecomment-805433662 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136429/ -- This

  1   2   3   4   5   6   7   >