[GitHub] [spark] AmplabJenkins removed a comment on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

AmplabJenkins removed a comment on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660598280 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

AmplabJenkins commented on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660598280 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29155: Fix link to Google Cloud Storage connector docs.

AmplabJenkins removed a comment on pull request #29155: URL: https://github.com/apache/spark/pull/29155#issuecomment-660597463 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA commented on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

SparkQA commented on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660598146 **[Test build #126124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126124/testReport)** for PR 27331 at commit [`3093c35`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #29155: Fix link to Google Cloud Storage connector docs.

AmplabJenkins commented on pull request #29155: URL: https://github.com/apache/spark/pull/29155#issuecomment-660598097 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins commented on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

AmplabJenkins commented on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660597489 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

SparkQA removed a comment on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660597396 **[Test build #126123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126123/testReport)** for PR 27331 at commit [`90ddbcc`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

AmplabJenkins removed a comment on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660597489 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

SparkQA commented on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660597488 **[Test build #126123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126123/testReport)** for PR 27331 at commit [`90ddbcc`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #29155: Fix link to Google Cloud Storage connector docs.

AmplabJenkins commented on pull request #29155: URL: https://github.com/apache/spark/pull/29155#issuecomment-660597463 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] medb opened a new pull request #29155: Fix link to Google Cloud Storage connector docs.

medb opened a new pull request #29155: URL: https://github.com/apache/spark/pull/29155 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] SparkQA commented on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

SparkQA commented on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660597396 **[Test build #126123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126123/testReport)** for PR 27331 at commit [`90ddbcc`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

AmplabJenkins removed a comment on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660596055 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

AmplabJenkins commented on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660596055 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #27331: [SPARK-29157][SQL][PYSPARK] Add DataFrameWriterV2 to Python API

SparkQA commented on pull request #27331: URL: https://github.com/apache/spark/pull/27331#issuecomment-660595940 **[Test build #126122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126122/testReport)** for PR 27331 at commit [`c8fe7e7`](https://github.com

[GitHub] [spark] kiszk commented on pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

kiszk commented on pull request #29067: URL: https://github.com/apache/spark/pull/29067#issuecomment-660593980 Can we add additional API for saying whether a given type is cached or not to `CachedBatchSerializer`? It depends on the implementation of `CachedBatchSerializer`. This addresses

[GitHub] [spark] kiszk commented on a change in pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

kiszk commented on a change in pull request #29067: URL: https://github.com/apache/spark/pull/29067#discussion_r456862990 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/InMemoryColumnarQuerySuite.scala ## @@ -42,12 +49,13 @@ class InMemoryColumna

[GitHub] [spark] kiszk commented on a change in pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

kiszk commented on a change in pull request #29067: URL: https://github.com/apache/spark/pull/29067#discussion_r456862806 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala ## @@ -130,34 +97,32 @@ case class InMemoryTableS

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

AmplabJenkins removed a comment on pull request #29079: URL: https://github.com/apache/spark/pull/29079#issuecomment-660590396 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

AmplabJenkins commented on pull request #29079: URL: https://github.com/apache/spark/pull/29079#issuecomment-660590396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

SparkQA commented on pull request #29079: URL: https://github.com/apache/spark/pull/29079#issuecomment-660590273 **[Test build #126121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126121/testReport)** for PR 29079 at commit [`4c65c7f`](https://github.com

[GitHub] [spark] c21 commented on pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

c21 commented on pull request #29079: URL: https://github.com/apache/spark/pull/29079#issuecomment-660590187 Addressed all comments and rebased to latest master. Thanks. cc @maropu, @cloud-fan and @viirya. This is an automat

[GitHub] [spark] kiszk commented on a change in pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

kiszk commented on a change in pull request #29067: URL: https://github.com/apache/spark/pull/29067#discussion_r456862232 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala ## @@ -85,69 +208,55 @@ case class CachedRDDBuilder(

[GitHub] [spark] c21 commented on a change in pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

c21 commented on a change in pull request #29079: URL: https://github.com/apache/spark/pull/29079#discussion_r456862050 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2651,12 +2651,13 @@ object SQLConf { .booleanConf

[GitHub] [spark] c21 commented on a change in pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

c21 commented on a change in pull request #29079: URL: https://github.com/apache/spark/pull/29079#discussion_r456862018 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2659,12 +2660,24 @@ object SQLConf { buildConf("spark.sql.

[GitHub] [spark] AmplabJenkins commented on pull request #29154: [MINOR][DOCS] add link for Debugging your Application in running-on-yarn.html#launching-spark-on-yarn

AmplabJenkins commented on pull request #29154: URL: https://github.com/apache/spark/pull/29154#issuecomment-660588814 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29154: [MINOR][DOCS] add link for Debugging your Application in running-on-yarn.html#launching-spark-on-yarn

AmplabJenkins removed a comment on pull request #29154: URL: https://github.com/apache/spark/pull/29154#issuecomment-660588714 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins commented on pull request #29154: [MINOR][DOCS] add link for Debugging your Application in running-on-yarn.html#launching-spark-on-yarn

AmplabJenkins commented on pull request #29154: URL: https://github.com/apache/spark/pull/29154#issuecomment-660588714 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-660588232 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-660588232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] brandonJY opened a new pull request #29154: [MINOR][DOCS] add link for Debugging your Application in running-on-yarn.html#launching-spark-on-yarn

brandonJY opened a new pull request #29154: URL: https://github.com/apache/spark/pull/29154 ### What changes were proposed in this pull request? add link for Debugging your Application in `running-on-yarn.html#launching-spark-on-yar` ### Why are the changes needed? Cur

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-660588126 **[Test build #126120 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126120/testReport)** for PR 29085 at commit [`4ef4d76`](https://github.com

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456860293 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -87,17 +170,59 @@ trait BaseScriptTran

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456860128 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,47 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS S

[GitHub] [spark] holdenk commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-660582133 The python packaging tests are failing on Jenkins post upgrade and this passes all of the GH actions so unless there is any more discussion I intend to merge this tomorrow. --

[GitHub] [spark] AngersZhuuuu commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

AngersZh commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-660580865 > > Some some can't run without hive > > Which kind of queries? DataType such as CalenderIntervalType, https://github.com/apache/spark/pull/29085#discussion_r4

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

AmplabJenkins removed a comment on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660579657 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

AmplabJenkins removed a comment on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660579655 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

SparkQA removed a comment on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660553870 **[Test build #126116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126116/testReport)** for PR 29138 at commit [`43d4f18`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

AmplabJenkins commented on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660579655 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

SparkQA commented on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660579492 **[Test build #126116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126116/testReport)** for PR 29138 at commit [`43d4f18`](https://github.co

[GitHub] [spark] huaxingao commented on pull request #29056: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs

huaxingao commented on pull request #29056: URL: https://github.com/apache/spark/pull/29056#issuecomment-660577750 @GuoPhilipse Sorry I have more nitpicks on grammar. I know this is very tedious. I went through all these when I did the doc PRs. Thanks a lot for doing this! I have no mor

[GitHub] [spark] HyukjinKwon closed pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions

HyukjinKwon closed pull request #29133: URL: https://github.com/apache/spark/pull/29133 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions

HyukjinKwon commented on pull request #29133: URL: https://github.com/apache/spark/pull/29133#issuecomment-660577416 I filed new JIRA for test reporter (SPARK-32357).. let's see how it goes. Merged to master. This is

[GitHub] [spark] baohe-zhang commented on a change in pull request #29149: [SPARK-32350][CORE] Add batch-write on LevelDB to improve performance of HybridStore

baohe-zhang commented on a change in pull request #29149: URL: https://github.com/apache/spark/pull/29149#discussion_r456851608 ## File path: common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java ## @@ -171,6 +172,58 @@ public void write(Object value) throws

[GitHub] [spark] huaxingao commented on a change in pull request #29056: [SPARK-31753][SQL][DOCS] Add missing keywords in the SQL docs

huaxingao commented on a change in pull request #29056: URL: https://github.com/apache/spark/pull/29056#discussion_r456849627 ## File path: docs/sql-ref-syntax-qry-select-lateral-view.md ## @@ -0,0 +1,130 @@ +--- +layout: global +title: LATERAL VIEW Clause +displayTitle: LATERA

[GitHub] [spark] HyukjinKwon commented on pull request #29143: [SPARK-32344][SQL] Unevaluable expr is set to FIRST/LAST ignoreNullsExpr in distinct aggregates

HyukjinKwon commented on pull request #29143: URL: https://github.com/apache/spark/pull/29143#issuecomment-660573442 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HyukjinKwon closed pull request #29143: [SPARK-32344][SQL] Unevaluable expr is set to FIRST/LAST ignoreNullsExpr in distinct aggregates

HyukjinKwon closed pull request #29143: URL: https://github.com/apache/spark/pull/29143 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29143: [SPARK-32344][SQL] Unevaluable expr is set to FIRST/LAST ignoreNullsExpr in distinct aggregates

HyukjinKwon commented on a change in pull request #29143: URL: https://github.com/apache/spark/pull/29143#discussion_r456848709 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala ## @@ -120,3 +120,11 @@ case class First(ch

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660573211 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660573211 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

SparkQA removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660569876 **[Test build #126118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126118/testReport)** for PR 29117 at commit [`d7974a4`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660573178 **[Test build #126118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126118/testReport)** for PR 29117 at commit [`d7974a4`](https://github.co

[GitHub] [spark] HyukjinKwon commented on pull request #29151: [SPARK-29802][BUILD] Use python3 in build scripts

HyukjinKwon commented on pull request #29151: URL: https://github.com/apache/spark/pull/29151#issuecomment-660572682 With the test results from Jenkins and GitHub Actions, seems all tests pass. This is an automated message fr

[GitHub] [spark] HyukjinKwon closed pull request #29151: [SPARK-29802][BUILD] Use python3 in build scripts

HyukjinKwon closed pull request #29151: URL: https://github.com/apache/spark/pull/29151 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on pull request #29151: [SPARK-29802][BUILD] Use python3 in build scripts

HyukjinKwon commented on pull request #29151: URL: https://github.com/apache/spark/pull/29151#issuecomment-660572614 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660572338 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660572337 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

SparkQA removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660570649 **[Test build #126119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126119/testReport)** for PR 29117 at commit [`d7974a4`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660572332 **[Test build #126119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126119/testReport)** for PR 29117 at commit [`d7974a4`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660572337 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon commented on pull request #29151: [SPARK-29909][BUILD] Use python3 in build scripts

HyukjinKwon commented on pull request #29151: URL: https://github.com/apache/spark/pull/29151#issuecomment-660572238 @srowen, I am investigating it in #29117. I will be able to fix soon I guess. This is an automated message f

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

HyukjinKwon commented on a change in pull request #29067: URL: https://github.com/apache/spark/pull/29067#discussion_r456847406 ## File path: sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala ## @@ -0,0 +1,279 @@ +/* + * Licensed to the Apache S

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

HyukjinKwon commented on a change in pull request #29067: URL: https://github.com/apache/spark/pull/29067#discussion_r456847406 ## File path: sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala ## @@ -0,0 +1,279 @@ +/* + * Licensed to the Apache S

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456847553 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,47 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS SELECT

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r456847499 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,47 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t1 AS SELECT

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

HyukjinKwon commented on a change in pull request #29067: URL: https://github.com/apache/spark/pull/29067#discussion_r456847406 ## File path: sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala ## @@ -0,0 +1,279 @@ +/* + * Licensed to the Apache S

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29067: [SPARK-32274][SQL] Make SQL cache serialization pluggable

HyukjinKwon commented on a change in pull request #29067: URL: https://github.com/apache/spark/pull/29067#discussion_r456847406 ## File path: sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala ## @@ -0,0 +1,279 @@ +/* + * Licensed to the Apache S

[GitHub] [spark] maropu commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

maropu commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-660571530 > Some some can't run without hive Which kind of queries? > For doc, I prefer to update after all these done. Ok? Yea, it looks okay to me. -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660570773 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660570773 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] viirya commented on pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions

viirya commented on pull request #29133: URL: https://github.com/apache/spark/pull/29133#issuecomment-660570738 Looks okay to me. We can keep improving this. This is an automated message from the Apache Git Service. To respon

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660570649 **[Test build #126119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126119/testReport)** for PR 29117 at commit [`d7974a4`](https://github.com

[GitHub] [spark] HyukjinKwon commented on pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions

HyukjinKwon commented on pull request #29133: URL: https://github.com/apache/spark/pull/29133#issuecomment-660570181 I am okay to try this out for now until we have a way to report the test results but I'll leave it to @dongjoon-hyun and @viirya. --

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins removed a comment on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660569987 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29117: [WIP] Debug flaky pip installation test failure

AmplabJenkins commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660569987 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon commented on pull request #29117: [WIP] Debug flaky pip installation test failure

HyukjinKwon commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660569991 retest this please This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [spark] SparkQA commented on pull request #29117: [WIP] Debug flaky pip installation test failure

SparkQA commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660569876 **[Test build #126118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126118/testReport)** for PR 29117 at commit [`d7974a4`](https://github.com

[GitHub] [spark] HyukjinKwon commented on pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions

HyukjinKwon commented on pull request #29133: URL: https://github.com/apache/spark/pull/29133#issuecomment-660569838 @dongjoon-hyun, I am debugging the flakiness at https://github.com/apache/spark/pull/29117 to make a complete fix. I think it was caused by my recent fixes and Jenkins envir

[GitHub] [spark] HyukjinKwon commented on pull request #29117: [WIP] Debug flaky pip installation test failure

HyukjinKwon commented on pull request #29117: URL: https://github.com/apache/spark/pull/29117#issuecomment-660569710 retest this please This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29145: [WIP][SPARK-32346][SQL] Support filters pushdown in Avro datasource

AmplabJenkins removed a comment on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-660566442 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29145: [WIP][SPARK-32346][SQL] Support filters pushdown in Avro datasource

AmplabJenkins commented on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-660566442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29145: [WIP][SPARK-32346][SQL] Support filters pushdown in Avro datasource

SparkQA commented on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-660566210 **[Test build #126115 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126115/testReport)** for PR 29145 at commit [`072eab0`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29145: [WIP][SPARK-32346][SQL] Support filters pushdown in Avro datasource

SparkQA removed a comment on pull request #29145: URL: https://github.com/apache/spark/pull/29145#issuecomment-660536420 **[Test build #126115 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126115/testReport)** for PR 29145 at commit [`072eab0`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

AmplabJenkins removed a comment on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660564857 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

AmplabJenkins removed a comment on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660564855 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA commented on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

SparkQA commented on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660564831 **[Test build #126117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126117/testReport)** for PR 29153 at commit [`86e0579`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

AmplabJenkins commented on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660564855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

SparkQA removed a comment on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660561046 **[Test build #126117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126117/testReport)** for PR 29153 at commit [`86e0579`](https://gi

[GitHub] [spark] frankyin-factual commented on pull request #28898: [SPARK-32059][SQL] Allow nested schema pruning thru window/sort/filter plans

frankyin-factual commented on pull request #28898: URL: https://github.com/apache/spark/pull/28898#issuecomment-660563330 @maropu @viirya @dongjoon-hyun friendly bump This is an automated message from the Apache Git Service.

[GitHub] [spark] AmplabJenkins commented on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

AmplabJenkins commented on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660561180 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

AmplabJenkins removed a comment on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660561180 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

SparkQA commented on pull request #29153: URL: https://github.com/apache/spark/pull/29153#issuecomment-660561046 **[Test build #126117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126117/testReport)** for PR 29153 at commit [`86e0579`](https://github.com

[GitHub] [spark] huaxingao opened a new pull request #29153: [SPARK-32310][ML][PySpark][WIP] ML params default value parity in feature and tuning

huaxingao opened a new pull request #29153: URL: https://github.com/apache/spark/pull/29153 ### What changes were proposed in this pull request? set params default values in trait Params for feature and tuning in both Scala and Python. ### Why are the changes needed?

[GitHub] [spark] viirya commented on a change in pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

viirya commented on a change in pull request #29079: URL: https://github.com/apache/spark/pull/29079#discussion_r456839808 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2651,12 +2651,13 @@ object SQLConf { .booleanConf

[GitHub] [spark] viirya commented on a change in pull request #29079: [SPARK-32286][SQL] Coalesce bucketed table for shuffled hash join if applicable

viirya commented on a change in pull request #29079: URL: https://github.com/apache/spark/pull/29079#discussion_r456839329 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2659,12 +2660,24 @@ object SQLConf { buildConf("spark.s

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

AmplabJenkins removed a comment on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660553983 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

AmplabJenkins commented on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660553983 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

SparkQA commented on pull request #29138: URL: https://github.com/apache/spark/pull/29138#issuecomment-660553870 **[Test build #126116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126116/testReport)** for PR 29138 at commit [`43d4f18`](https://github.com

[GitHub] [spark] nvander1 commented on a change in pull request #29138: [SPARK-32338] [SQL] Overload slice to accept Column for start and length

nvander1 commented on a change in pull request #29138: URL: https://github.com/apache/spark/pull/29138#discussion_r456835780 ## File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ## @@ -3406,6 +3406,21 @@ object functions { Slice(x.expr, Literal(start)

  1   2   3   4   >