[GitHub] [spark] c21 edited a comment on pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

c21 edited a comment on pull request #31892: URL: https://github.com/apache/spark/pull/31892#issuecomment-802578765 > How about branch-3.0 and branch-2.4? They have this issue, too? @maropu - Based on my knowledge I don't think so. The shortcut for `HashJoin` (not calling parent's

[GitHub] [spark] c21 commented on pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

c21 commented on pull request #31892: URL: https://github.com/apache/spark/pull/31892#issuecomment-802578765 > How about branch-3.0 and branch-2.4? They have this issue, too? Based on my knowledge I don't think so. The shortcut for `HashJoin` (not calling parent's doConsume()) was

[GitHub] [spark] maropu commented on a change in pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

maropu commented on a change in pull request #31892: URL: https://github.com/apache/spark/pull/31892#discussion_r597423584 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -4097,6 +4097,25 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] AmplabJenkins commented on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

AmplabJenkins commented on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802577270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

AmplabJenkins commented on pull request #31892: URL: https://github.com/apache/spark/pull/31892#issuecomment-802577267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

AmplabJenkins commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-802577269 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136235/ -- This

[GitHub] [spark] cloud-fan commented on a change in pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

cloud-fan commented on a change in pull request #31892: URL: https://github.com/apache/spark/pull/31892#discussion_r597422339 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -4097,6 +4097,25 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] yaooqinn commented on pull request #31891: SPARK-34800 use fine-grained lock in SessionCatalog.tableExists

yaooqinn commented on pull request #31891: URL: https://github.com/apache/spark/pull/31891#issuecomment-802574469 please fix the format of the PR title, like `[SPARK-34800][SQL] U...` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on a change in pull request #31884: [SPARK-34793][SQL] Prohibit saving of day-time and year-month intervals

cloud-fan commented on a change in pull request #31884: URL: https://github.com/apache/spark/pull/31884#discussion_r597419979 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -579,6 +573,14 @@ case class DataSource(

[GitHub] [spark] yaooqinn commented on pull request #31891: SPARK-34800 use fine-grained lock in SessionCatalog.tableExists

yaooqinn commented on pull request #31891: URL: https://github.com/apache/spark/pull/31891#issuecomment-802573590 ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] maropu edited a comment on pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

maropu edited a comment on pull request #31892: URL: https://github.com/apache/spark/pull/31892#issuecomment-802573189 How about `branch-3.0` and `branch-2.4`? They have this issue, too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maropu commented on pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

maropu commented on pull request #31892: URL: https://github.com/apache/spark/pull/31892#issuecomment-802573189 How about `branch-3.0` and `branch-2.4`? They have the issue, too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] maropu commented on a change in pull request #31874: [SPARK-34708][SQL] Code-gen for left semi/anti broadcast nested loop join (build right side)

maropu commented on a change in pull request #31874: URL: https://github.com/apache/spark/pull/31874#discussion_r597418024 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala ## @@ -452,4 +457,50 @@ case class

[GitHub] [spark] maropu commented on a change in pull request #31874: [SPARK-34708][SQL] Code-gen for left semi/anti broadcast nested loop join (build right side)

maropu commented on a change in pull request #31874: URL: https://github.com/apache/spark/pull/31874#discussion_r597418024 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala ## @@ -452,4 +457,50 @@ case class

[GitHub] [spark] cloud-fan closed pull request #31881: [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

cloud-fan closed pull request #31881: URL: https://github.com/apache/spark/pull/31881 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] yaooqinn commented on a change in pull request #31884: [SPARK-34793][SQL] Prohibit saving of day-time and year-month intervals

yaooqinn commented on a change in pull request #31884: URL: https://github.com/apache/spark/pull/31884#discussion_r597418087 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -579,6 +573,14 @@ case class DataSource(

[GitHub] [spark] maropu commented on a change in pull request #31874: [SPARK-34708][SQL] Code-gen for left semi/anti broadcast nested loop join (build right side)

maropu commented on a change in pull request #31874: URL: https://github.com/apache/spark/pull/31874#discussion_r597418024 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala ## @@ -452,4 +457,50 @@ case class

[GitHub] [spark] LuciferYang edited a comment on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

LuciferYang edited a comment on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-802570617 > I don't really know enough to evaluate this. It looks reasonable and tests pass. Are there any compatibility concerns, or possible changes to behavior you can

[GitHub] [spark] LuciferYang commented on pull request #31776: [SPARK-34661][SQL] Clean up `OriginalType` and `DecimalMetadata ` usage in Parquet related code

LuciferYang commented on pull request #31776: URL: https://github.com/apache/spark/pull/31776#issuecomment-802570617 > I don't really know enough to evaluate this. It looks reasonable and tests pass. Are there any compatibility concerns, or possible changes to behavior you can think of?

[GitHub] [spark] warrenzhu25 commented on a change in pull request #31869: [SPARK-34777][UI] StagePage input size records not show when records greater than zero

warrenzhu25 commented on a change in pull request #31869: URL: https://github.com/apache/spark/pull/31869#discussion_r597417040 ## File path: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ## @@ -786,9 +786,13 @@ private[spark] object ApiHelper {

[GitHub] [spark] SparkQA commented on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

SparkQA commented on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802569619 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40816/ -- This is an automated message from the

[GitHub] [spark] wangyum commented on pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

wangyum commented on pull request #31864: URL: https://github.com/apache/spark/pull/31864#issuecomment-802569383 Merged to master, branch-3.1 and branch-3.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] MaxGekk commented on a change in pull request #31884: [SPARK-34793][SQL] Prohibit saving of day-time and year-month intervals

MaxGekk commented on a change in pull request #31884: URL: https://github.com/apache/spark/pull/31884#discussion_r597415662 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -579,6 +573,14 @@ case class DataSource(

[GitHub] [spark] cloud-fan commented on pull request #31881: [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

cloud-fan commented on pull request #31881: URL: https://github.com/apache/spark/pull/31881#issuecomment-802566306 the scala 2.13 failure is unrelated, thanks, merging to 3.1/3.0! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

AmplabJenkins removed a comment on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-802561775 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40817/

[GitHub] [spark] AmplabJenkins commented on pull request #30018: [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

AmplabJenkins commented on pull request #30018: URL: https://github.com/apache/spark/pull/30018#issuecomment-802561775 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40817/ --

[GitHub] [spark] ulysses-you commented on pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

ulysses-you commented on pull request #31864: URL: https://github.com/apache/spark/pull/31864#issuecomment-802561610 thanks all ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

AmplabJenkins removed a comment on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802531728 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] c21 commented on pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

c21 commented on pull request #31892: URL: https://github.com/apache/spark/pull/31892#issuecomment-802559650 cc @cloud-fan, @maropu and @HyukjinKwon to take a look if you have time, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] c21 opened a new pull request #31892: [SPARK-34796][SQL] Initialize counter variable for LIMIT code-gen in doProduce()

c21 opened a new pull request #31892: URL: https://github.com/apache/spark/pull/31892 ### What changes were proposed in this pull request? This PR is to fix the LIMIT code-gen bug in https://issues.apache.org/jira/browse/SPARK-34796, where the counter variable from

[GitHub] [spark] SparkQA commented on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

SparkQA commented on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802559214 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40816/ -- This is an automated message from the Apache

[GitHub] [spark] wangyum closed pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

wangyum closed pull request #31864: URL: https://github.com/apache/spark/pull/31864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] wangyum commented on pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

wangyum commented on pull request #31864: URL: https://github.com/apache/spark/pull/31864#issuecomment-802553916 Yes it is. It is better to backport it to branch-3.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] sarutak commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

sarutak commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-802542418 This GA failure seems to be what is fixed at c5cadfe. If you rebase to `master`, the failure will go away. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] ulysses-you commented on pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

ulysses-you commented on pull request #31864: URL: https://github.com/apache/spark/pull/31864#issuecomment-802541943 `RebaseDateTime` is created in branch-3.0, so it would be better to backport 3.0. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

AmplabJenkins removed a comment on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802538422 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136234/

[GitHub] [spark] AmplabJenkins commented on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

AmplabJenkins commented on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802538422 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136234/ -- This

[GitHub] [spark] mridulm commented on a change in pull request #31881: [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

mridulm commented on a change in pull request #31881: URL: https://github.com/apache/spark/pull/31881#discussion_r597394971 ## File path: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala ## @@ -76,7 +76,11 @@ trait QueryExecutionListener { class

[GitHub] [spark] mridulm commented on a change in pull request #31876: [WIP][SPARK-XXXX][API][CORE] Abstract Location in MapStatus to enable support for custom storage

mridulm commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r597391662 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -216,7 +223,10 @@ private[spark] class HighlyCompressedMapStatus

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

AmplabJenkins removed a comment on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-801561161 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136184/

[GitHub] [spark] cloud-fan closed pull request #31855: [SPARK-34761][SQL] Support add/subtract of a day-time interval to/from a timestamp

cloud-fan closed pull request #31855: URL: https://github.com/apache/spark/pull/31855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] cloud-fan commented on pull request #31855: [SPARK-34761][SQL] Support add/subtract of a day-time interval to/from a timestamp

cloud-fan commented on pull request #31855: URL: https://github.com/apache/spark/pull/31855#issuecomment-802534666 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] Ngone51 commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

Ngone51 commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-802534480 Yea, I think so. The Jenkins infra is really unhealthy these days. For the GA failure, the community has just fixed one failure related to Scala 2.13 at:

[GitHub] [spark] cloud-fan commented on a change in pull request #31884: [SPARK-34793][SQL] Prohibit saving of day-time and year-month intervals

cloud-fan commented on a change in pull request #31884: URL: https://github.com/apache/spark/pull/31884#discussion_r597391929 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -579,6 +573,14 @@ case class DataSource(

[GitHub] [spark] cloud-fan commented on a change in pull request #31884: [SPARK-34793][SQL] Prohibit saving of day-time and year-month intervals

cloud-fan commented on a change in pull request #31884: URL: https://github.com/apache/spark/pull/31884#discussion_r597391806 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -510,10 +510,7 @@ case class DataSource(

[GitHub] [spark] cloud-fan commented on pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

cloud-fan commented on pull request #31864: URL: https://github.com/apache/spark/pull/31864#issuecomment-802533026 LGTM. How far shall we backport this? 3.0? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] cloud-fan commented on a change in pull request #31874: [SPARK-34708][SQL] Code-gen for left semi/anti broadcast nested loop join (build right side)

cloud-fan commented on a change in pull request #31874: URL: https://github.com/apache/spark/pull/31874#discussion_r597390501 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala ## @@ -452,4 +457,50 @@ case class

[GitHub] [spark] AmplabJenkins commented on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

AmplabJenkins commented on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802531728 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins commented on pull request #31891: SPARK-34800 use fine-grained lock in SessionCatalog.tableExists

AmplabJenkins commented on pull request #31891: URL: https://github.com/apache/spark/pull/31891#issuecomment-802531710 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible

AmplabJenkins removed a comment on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-802531250 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136233/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31877: [SPARK-34783][K8S] Support remote template files

AmplabJenkins removed a comment on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802531248 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40811/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31649: [SPARK-34542][BUILD] Upgrade Parquet to 1.12.0

AmplabJenkins removed a comment on pull request #31649: URL: https://github.com/apache/spark/pull/31649#issuecomment-802531249 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136232/

[GitHub] [spark] AmplabJenkins commented on pull request #31649: [SPARK-34542][BUILD] Upgrade Parquet to 1.12.0

AmplabJenkins commented on pull request #31649: URL: https://github.com/apache/spark/pull/31649#issuecomment-802531249 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136232/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible

AmplabJenkins commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-802531250 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136233/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

AmplabJenkins commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802531248 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40811/ --

[GitHub] [spark] viirya closed pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

viirya closed pull request #28715: URL: https://github.com/apache/spark/pull/28715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [spark] viirya commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

viirya commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-802530100 Oh, the same Scala 2.13 SBT build error. I think it's because this doesn't sync up with latest master. It is unrelated so I'm going to merge this to master. Thanks

[GitHub] [spark] wangyum commented on pull request #31890: [SPARK-34798][SQL][TESTS] Fix incorrect join condition

wangyum commented on pull request #31890: URL: https://github.com/apache/spark/pull/31890#issuecomment-802524235 ok to test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] cloud-fan commented on pull request #31886: [WIP][SPARK-34795][SQL][TEST] Adds a new job in GitHub Actions to check the output of TPC-DS queries

cloud-fan commented on pull request #31886: URL: https://github.com/apache/spark/pull/31886#issuecomment-802524020 +1 to have this check! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] woyumen4597 opened a new pull request #31891: SPARK-34800 use fine-grained lock in SessionCatalog.tableExists

woyumen4597 opened a new pull request #31891: URL: https://github.com/apache/spark/pull/31891 ### What changes were proposed in this pull request? Use fine-grained lock in SessionCatalog.tableExists, in order to lock currentDB variable rather than lock `tableExists` method which will

[GitHub] [spark] opensky142857 opened a new pull request #31890: SPARK-34798 fix incorrect join condition

opensky142857 opened a new pull request #31890: URL: https://github.com/apache/spark/pull/31890 ### What changes were proposed in this pull request? join condition 'a.attr == 'c.attr check the reference of these 2 objects which will always returns false. we need to use

[GitHub] [spark] RussellSpitzer commented on a change in pull request #31887: [WIP][SPARK-34794][SQL] fix nested transform issue

RussellSpitzer commented on a change in pull request #31887: URL: https://github.com/apache/spark/pull/31887#discussion_r597384076 ## File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ## @@ -3799,23 +3801,26 @@ object functions {

[GitHub] [spark] SparkQA commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

SparkQA commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802520119 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40811/ -- This is an automated message from the

[GitHub] [spark] dmsolow commented on pull request #31887: [WIP][SPARK-34794][SQL] fix nested transform issue

dmsolow commented on pull request #31887: URL: https://github.com/apache/spark/pull/31887#issuecomment-802518964 @nvander1 test added -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible

AmplabJenkins removed a comment on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-802511142 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40815/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31649: [SPARK-34542][BUILD] Upgrade Parquet to 1.12.0

AmplabJenkins removed a comment on pull request #31649: URL: https://github.com/apache/spark/pull/31649#issuecomment-802511097 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40814/

[GitHub] [spark] AmplabJenkins commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible

AmplabJenkins commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-802511142 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40815/ --

[GitHub] [spark] Ngone51 commented on a change in pull request #31881: [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

Ngone51 commented on a change in pull request #31881: URL: https://github.com/apache/spark/pull/31881#discussion_r597378055 ## File path: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala ## @@ -76,7 +76,11 @@ trait QueryExecutionListener { class

[GitHub] [spark] ekoifman commented on pull request #31653: [SPARK-33832][SQL] v2. move OptimzieSkewedJoin to query stage preparation

ekoifman commented on pull request #31653: URL: https://github.com/apache/spark/pull/31653#issuecomment-802510977 I wanted to have a call because there was disconnect about how the patch is structured. If you now agree that `OptimizeSkewedJoin` does run as part of query stage preparation

[GitHub] [spark] AmplabJenkins commented on pull request #31649: [SPARK-34542][BUILD] Upgrade Parquet to 1.12.0

AmplabJenkins commented on pull request #31649: URL: https://github.com/apache/spark/pull/31649#issuecomment-802511097 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40814/ --

[GitHub] [spark] gengliangwang commented on a change in pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

gengliangwang commented on a change in pull request #31859: URL: https://github.com/apache/spark/pull/31859#discussion_r597377870 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnsiTypeCoercionSuite.scala ## @@ -377,10 +368,26 @@ class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

AmplabJenkins removed a comment on pull request #31880: URL: https://github.com/apache/spark/pull/31880#issuecomment-802254061 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136218/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

AmplabJenkins removed a comment on pull request #31864: URL: https://github.com/apache/spark/pull/31864#issuecomment-802502145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31877: [SPARK-34783][K8S] Support remote template files

AmplabJenkins removed a comment on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-801628249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] jsoref commented on pull request #30679: [SPARK-33717][LAUNCHER] deprecate spark.launcher.childConectionTimeout

jsoref commented on pull request #30679: URL: https://github.com/apache/spark/pull/30679#issuecomment-802508765 @HyukjinKwon: I filed https://issues.apache.org/jira/browse/SPARK-33717... @srowen is there something I need to do to push this along? -- This is an automated message

[GitHub] [spark] jsoref commented on pull request #30678: [MINOR][SQL] Spelling: filters - PushedFilers

jsoref commented on pull request #30678: URL: https://github.com/apache/spark/pull/30678#issuecomment-802507534 @srowen / @bersprockets: anything I need to do to push this along? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] kyoty commented on a change in pull request #31882: [WIP][SPARK-34787] Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

kyoty commented on a change in pull request #31882: URL: https://github.com/apache/spark/pull/31882#discussion_r597375793 ## File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ## @@ -156,18 +156,19 @@ private[history] class ApplicationCache(

[GitHub] [spark] SparkQA commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

SparkQA commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802506643 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40811/ -- This is an automated message from the Apache

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31881: [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

HyukjinKwon commented on a change in pull request #31881: URL: https://github.com/apache/spark/pull/31881#discussion_r597374876 ## File path: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala ## @@ -76,7 +76,11 @@ trait QueryExecutionListener {

[GitHub] [spark] jsoref commented on pull request #30678: [MINOR][SQL] Spelling: filters - PushedFilers

jsoref commented on pull request #30678: URL: https://github.com/apache/spark/pull/30678#issuecomment-802505156 Thanks bot -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] github-actions[bot] commented on pull request #30679: [SPARK-33717][LAUNCHER] deprecate spark.launcher.childConectionTimeout

github-actions[bot] commented on pull request #30679: URL: https://github.com/apache/spark/pull/30679#issuecomment-802504169 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] github-actions[bot] commented on pull request #30678: [MINOR][SQL] Spelling: filters - PushedFilers

github-actions[bot] commented on pull request #30678: URL: https://github.com/apache/spark/pull/30678#issuecomment-802504176 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] github-actions[bot] commented on pull request #29994: [DONOTMERGE][WHITESPACE] workflow exercise

github-actions[bot] commented on pull request #29994: URL: https://github.com/apache/spark/pull/29994#issuecomment-802504186 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] AmplabJenkins commented on pull request #31889: [SPARK-34797][ML] Refactor Logistic Aggregator - support virtual centering

AmplabJenkins commented on pull request #31889: URL: https://github.com/apache/spark/pull/31889#issuecomment-802502147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #31864: [SPARK-34772][SQL] RebaseDateTime loadRebaseRecords should use Spark classloader instead of context

AmplabJenkins commented on pull request #31864: URL: https://github.com/apache/spark/pull/31864#issuecomment-802502145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

AmplabJenkins commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802502144 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136229/ -- This

[GitHub] [spark] srowen commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

srowen commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r597372248 ## File path: dev/change-scala-version.sh ## @@ -60,14 +60,13 @@ BASEDIR=$(dirname $0)/.. find "$BASEDIR" -name 'pom.xml' -not -path '*target*' -print \

[GitHub] [spark] mridulm commented on a change in pull request #31881: [SPARK-34087][3.1][SQL] Fix memory leak of ExecutionListenerBus

mridulm commented on a change in pull request #31881: URL: https://github.com/apache/spark/pull/31881#discussion_r597370114 ## File path: sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala ## @@ -76,7 +76,11 @@ trait QueryExecutionListener { class

[GitHub] [spark] sarutak commented on a change in pull request #31880: [SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-* with better solution

sarutak commented on a change in pull request #31880: URL: https://github.com/apache/spark/pull/31880#discussion_r597369358 ## File path: dev/change-scala-version.sh ## @@ -60,14 +60,13 @@ BASEDIR=$(dirname $0)/.. find "$BASEDIR" -name 'pom.xml' -not -path '*target*' -print \

[GitHub] [spark] dongjoon-hyun commented on pull request #31877: [SPARK-34783][K8S] Support remote template files

dongjoon-hyun commented on pull request #31877: URL: https://github.com/apache/spark/pull/31877#issuecomment-802493670 Thank you for review and approval, @HyukjinKwon and @attilapiros . -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on pull request #31811: [SPARK-34719][SQL][3.1] Correctly resolve the view query with duplicated column names

cloud-fan commented on pull request #31811: URL: https://github.com/apache/spark/pull/31811#issuecomment-802492163 I'll open PRs for master/3.0/2.4 shortly, thanks for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31882: [WIP][SPARK-34787] Option variable in Spark historyServer log should be displayed as actual value instead of Some(XX)

dongjoon-hyun commented on a change in pull request #31882: URL: https://github.com/apache/spark/pull/31882#discussion_r597366145 ## File path: core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala ## @@ -156,18 +156,19 @@ private[history] class

[GitHub] [spark] zhengruifeng commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible

zhengruifeng commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-802468006 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] zhengruifeng commented on pull request #31889: [SPARK-34797][ML] Refactor Logistic Aggregator

zhengruifeng commented on pull request #31889: URL: https://github.com/apache/spark/pull/31889#issuecomment-802465203 friendly ping @srowen @WeichenXu123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31886: [WIP][SPARK-34795][SQL][TEST] Adds a new job in GitHub Actions to check the output of TPC-DS queries

AmplabJenkins removed a comment on pull request #31886: URL: https://github.com/apache/spark/pull/31886#issuecomment-802424031 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40809/

[GitHub] [spark] zhengruifeng edited a comment on pull request #31889: [SPARK-34797][ML] Refactor Logistic Aggregator

zhengruifeng edited a comment on pull request #31889: URL: https://github.com/apache/spark/pull/31889#issuecomment-802462693 I also test using the two new agg in `LogisticRegressionSuite`, and all test passed except "multinomial logistic regression with intercept with L1 regularization":

[GitHub] [spark] zhengruifeng commented on pull request #31889: [SPARK-34797][ML] Refactor Logistic Aggregator

zhengruifeng commented on pull request #31889: URL: https://github.com/apache/spark/pull/31889#issuecomment-802462693 I also test using the two new agg in `LogisticRegressionSuite`, and all test passed except "multinomial logistic regression with intercept with L1 regularization":

[GitHub] [spark] maropu commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

maropu commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-802461832 You've already opened a new PR to address the @cloud-fan comment? https://github.com/apache/spark/pull/31598#discussion_r586155367 The direction looks reasonable to me, too.

[GitHub] [spark] zhengruifeng opened a new pull request #31889: [SPARK-34797][ML] Refactor Logistic Aggregator

zhengruifeng opened a new pull request #31889: URL: https://github.com/apache/spark/pull/31889 ### What changes were proposed in this pull request? 1, add `BinaryLogisticBlockAggregator` and `MultinomialLogisticBlockAggregator` and related testsuites; 2, impl `virtual centering`

[GitHub] [spark] dongjoon-hyun commented on pull request #31811: [SPARK-34719][SQL][3.1] Correctly resolve the view query with duplicated column names

dongjoon-hyun commented on pull request #31811: URL: https://github.com/apache/spark/pull/31811#issuecomment-802459156 Thank you, @cloud-fan and all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

  1   2   3   4   5   >