[GitHub] [spark] olaky commented on pull request #36753: [SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries

2022-06-07 Thread GitBox
olaky commented on PR #36753: URL: https://github.com/apache/spark/pull/36753#issuecomment-1148258363 I cherry-picked 583a9c75bbb35387169d4f0cf763ef566d899954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] olaky commented on pull request #36762: [SPARK-39259][SQL][TEST][FOLLOWUP] Fix Scala 2.13 `ClassCastException` in `ComputeCurrentTimeSuite`

2022-06-07 Thread GitBox
olaky commented on PR #36762: URL: https://github.com/apache/spark/pull/36762#issuecomment-1148259858 @dongjoon-hyun thanks a lot for picking this up for me! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AngersZhuuuu commented on pull request #35612: [SPARK-38289][SQL] Refactor SQL CLI exit code to make it more clear

2022-06-07 Thread GitBox
AngersZh commented on PR #35612: URL: https://github.com/apache/spark/pull/35612#issuecomment-1148234463 Looks like the latest failed test not related to this pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] shiyuhang0 commented on pull request #21990: [SPARK-25003][PYSPARK] Use SessionExtensions in Pyspark

2022-06-07 Thread GitBox
shiyuhang0 commented on PR #21990: URL: https://github.com/apache/spark/pull/21990#issuecomment-1148248826 Why not port it to Spark < 3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] beliefer commented on a diff in pull request #36776: [SPARK-38997][SPARK-39037][SQL][FOLLOWUP] `PushableColumnWithoutNestedColumn` need be translated to predicate too

2022-06-07 Thread GitBox
beliefer commented on code in PR #36776: URL: https://github.com/apache/spark/pull/36776#discussion_r890808786 ## sql/core/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala: ## @@ -55,8 +55,13 @@ class V2ExpressionBuilder( } else {

[GitHub] [spark] LuciferYang commented on pull request #36732: [SPARK-39345][CORE][SQL][DSTREAM][ML][MESOS][SS] Replace `filter(!condition)` with `filterNot(condition)`

2022-06-07 Thread GitBox
LuciferYang commented on PR #36732: URL: https://github.com/apache/spark/pull/36732#issuecomment-1148250861 > No, we wouldn't backport this, that's more change. Does this offer any benefit? I'm not sure it's more readable even. If the readability is not improved, let me close this

[GitHub] [spark] zhengruifeng commented on pull request #35250: [SPARK-37961][SQL] Override maxRows/maxRowsPerPartition for some logical operators

2022-06-07 Thread GitBox
zhengruifeng commented on PR #35250: URL: https://github.com/apache/spark/pull/35250#issuecomment-1148255084 @cloud-fan Sure, Let me update this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] huaxingao commented on a diff in pull request #36776: [SPARK-38997][SPARK-39037][SQL][FOLLOWUP] `PushableColumnWithoutNestedColumn` need be translated to predicate too

2022-06-07 Thread GitBox
huaxingao commented on code in PR #36776: URL: https://github.com/apache/spark/pull/36776#discussion_r890797844 ## sql/core/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala: ## @@ -55,8 +55,13 @@ class V2ExpressionBuilder( } else {

[GitHub] [spark] ArvinZheng commented on a diff in pull request #35484: [SPARK-38181][SS][DOCS] Update comments in KafkaDataConsumer.scala

2022-06-07 Thread GitBox
ArvinZheng commented on code in PR #35484: URL: https://github.com/apache/spark/pull/35484#discussion_r890843318 ## connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala: ## @@ -298,9 +296,10 @@ private[kafka010] class

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r890949860 ## core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala: ## @@ -65,7 +66,70 @@ private[spark] class ApplicationInfo( appSource = new

[GitHub] [spark] MaxGekk commented on pull request #36703: [SPARK-39321][SQL] Refactor TryCast to use RuntimeReplaceable

2022-06-07 Thread GitBox
MaxGekk commented on PR #36703: URL: https://github.com/apache/spark/pull/36703#issuecomment-1148502630 @cloud-fan Could you resolve conflicts, please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] LuciferYang closed pull request #36732: [SPARK-39345][CORE][SQL][DSTREAM][ML][MESOS][SS] Replace `filter(!condition)` with `filterNot(condition)`

2022-06-07 Thread GitBox
LuciferYang closed pull request #36732: [SPARK-39345][CORE][SQL][DSTREAM][ML][MESOS][SS] Replace `filter(!condition)` with `filterNot(condition)` URL: https://github.com/apache/spark/pull/36732 -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r890938856 ## core/src/main/scala/org/apache/spark/deploy/ExecutorDescription.scala: ## @@ -25,10 +25,13 @@ package org.apache.spark.deploy private[deploy] class

[GitHub] [spark] wangyum commented on pull request #36784: [SPARK-39396][SQL] Fix LDAP login exception 'error code 49 - invalid credentials'

2022-06-07 Thread GitBox
wangyum commented on PR #36784: URL: https://github.com/apache/spark/pull/36784#issuecomment-1148459572 cc @yaooqinn @pan3793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] AngersZhuuuu commented on pull request #35612: [SPARK-38289][SQL] Refactor SQL CLI exit code to make it more clear

2022-06-07 Thread GitBox
AngersZh commented on PR #35612: URL: https://github.com/apache/spark/pull/35612#issuecomment-1148499208 > @AngersZh can you retrigger the tests? GA passed now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gengliangwang closed pull request #36174: [SPARK-34659][UI] Fix wrong application ID when reverse proxy URL contains "proxy" or "history"

2022-06-07 Thread GitBox
gengliangwang closed pull request #36174: [SPARK-34659][UI] Fix wrong application ID when reverse proxy URL contains "proxy" or "history" URL: https://github.com/apache/spark/pull/36174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] gengliangwang closed pull request #34970: [DO NOT MERGE] investigate test failures if we test ANSI mode in github actions

2022-06-07 Thread GitBox
gengliangwang closed pull request #34970: [DO NOT MERGE] investigate test failures if we test ANSI mode in github actions URL: https://github.com/apache/spark/pull/34970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] AngersZhuuuu opened a new pull request, #36786: [SPARK-39400][SQL] spark-sql should remove hive resource dir in all case

2022-06-07 Thread GitBox
AngersZh opened a new pull request, #36786: URL: https://github.com/apache/spark/pull/36786 ### What changes were proposed in this pull request? In current code, when we use `spark-sql` `-e` , `-f` or use `ctrl + c` to close `spark-sql` session, will remain hive session resource dir

[GitHub] [spark] Ngone51 commented on pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on PR #36716: URL: https://github.com/apache/spark/pull/36716#issuecomment-1148335486 cc @tgravescs for review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r890912848 ## core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala: ## @@ -19,23 +19,28 @@ package org.apache.spark.deploy import java.net.URI -import

[GitHub] [spark] MaxGekk commented on pull request #36753: [SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries

2022-06-07 Thread GitBox
MaxGekk commented on PR #36753: URL: https://github.com/apache/spark/pull/36753#issuecomment-1148505223 I guess the failure is not related to PR's changes: ``` [info] - check simplified (tpcds-v1.4/q4) *** FAILED *** (945 milliseconds) [info] Plans did not match: ``` -- This

[GitHub] [spark] cloud-fan commented on a diff in pull request #36745: [SPARK-39359][SQL] Restrict DEFAULT columns to allowlist of supported data source types

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36745: URL: https://github.com/apache/spark/pull/36745#discussion_r890902366 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -2881,6 +2881,15 @@ object SQLConf { .booleanConf

[GitHub] [spark] MaxGekk commented on pull request #36753: [SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries

2022-06-07 Thread GitBox
MaxGekk commented on PR #36753: URL: https://github.com/apache/spark/pull/36753#issuecomment-1148509294 +1, LGTM. Merging to 3.2 and trying to merge to 3.1/3.0. Thank you, @olaky and @JoshRosen @dongjoon-hyun for review. -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] Borjianamin98 commented on pull request #36781: [SPARK-39393][SQL] Parquet data source only supports push-down predicate filters for non-repeated primitive types

2022-06-07 Thread GitBox
Borjianamin98 commented on PR #36781: URL: https://github.com/apache/spark/pull/36781#issuecomment-1148284070 > @Borjianamin98 Could you please add a test? I agree. I added a test for this. This is my first experience participating in the Spark project and I hope I did well. :) --

[GitHub] [spark] cloud-fan commented on a diff in pull request #35612: [SPARK-38289][SQL] Refactor SQL CLI exit code to make it more clear

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #35612: URL: https://github.com/apache/spark/pull/35612#discussion_r890851508 ## sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/HiveServer2.java: ## @@ -259,7 +260,7 @@ static class HelpOptionExecutor implements

[GitHub] [spark] ulysses-you opened a new pull request, #36785: [SPARK-39397][SQL] Relax AliasAwareOutputExpression to support alias with expression

2022-06-07 Thread GitBox
ulysses-you opened a new pull request, #36785: URL: https://github.com/apache/spark/pull/36785 ### What changes were proposed in this pull request? Change AliasAwareOutputExpression to using expression rather than attribute to track if we can nomalize. So the aliased

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r890982027 ## core/src/main/scala/org/apache/spark/deploy/master/Master.scala: ## @@ -725,26 +729,38 @@ private[deploy] class Master( */ private def

[GitHub] [spark] MaxGekk commented on pull request #36780: [SPARK-39392][SQL] Refine ANSI error messages for try_* function hints

2022-06-07 Thread GitBox
MaxGekk commented on PR #36780: URL: https://github.com/apache/spark/pull/36780#issuecomment-1148499879 +1, LGTM. Merging to master. Thank you, @vli-databricks and @gengliangwang @HyukjinKwon for review. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] cloud-fan commented on pull request #35612: [SPARK-38289][SQL] Refactor SQL CLI exit code to make it more clear

2022-06-07 Thread GitBox
cloud-fan commented on PR #35612: URL: https://github.com/apache/spark/pull/35612#issuecomment-1148292841 @AngersZh can you retrigger the tests? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] Resol1992 commented on pull request #30317: [SPARK-33409][SQL] And task killed check in BroadcastNestedLoopJoin to interrupt it after the job is killed

2022-06-07 Thread GitBox
Resol1992 commented on PR #30317: URL: https://github.com/apache/spark/pull/30317#issuecomment-1148301891 hi, @constzhou Recently, the same issue aslo occurs to me, could I talk with you about this issue? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HyukjinKwon commented on pull request #36784: [SPARK-39396][SQL] Fix LDAP login exception 'error code 49 - invalid credentials'

2022-06-07 Thread GitBox
HyukjinKwon commented on PR #36784: URL: https://github.com/apache/spark/pull/36784#issuecomment-1148308085 cc @wangyum FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] cloud-fan commented on pull request #36662: [SPARK-39286][DOC] Update documentation for the decode function

2022-06-07 Thread GitBox
cloud-fan commented on PR #36662: URL: https://github.com/apache/spark/pull/36662#issuecomment-1148337126 thanks, merging to master/3.3! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] cloud-fan closed pull request #36662: [SPARK-39286][DOC] Update documentation for the decode function

2022-06-07 Thread GitBox
cloud-fan closed pull request #36662: [SPARK-39286][DOC] Update documentation for the decode function URL: https://github.com/apache/spark/pull/36662 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r890973724 ## core/src/main/scala/org/apache/spark/deploy/master/ExecutorResourceDescription.scala: ## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

[GitHub] [spark] chenzhx commented on a diff in pull request #36663: [SPARK-38899][SQL]DS V2 supports push down datetime functions

2022-06-07 Thread GitBox
chenzhx commented on code in PR #36663: URL: https://github.com/apache/spark/pull/36663#discussion_r890253947 ## sql/core/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala: ## @@ -259,6 +259,55 @@ class V2ExpressionBuilder( } else {

[GitHub] [spark] MaxGekk closed pull request #36780: [SPARK-39392][SQL] Refine ANSI error messages for try_* function hints

2022-06-07 Thread GitBox
MaxGekk closed pull request #36780: [SPARK-39392][SQL] Refine ANSI error messages for try_* function hints URL: https://github.com/apache/spark/pull/36780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] MaxGekk commented on pull request #36780: [SPARK-39392][SQL] Refine ANSI error messages for try_* function hints

2022-06-07 Thread GitBox
MaxGekk commented on PR #36780: URL: https://github.com/apache/spark/pull/36780#issuecomment-1148501765 @vli-databricks Could you backport the changes to branch-3.3, please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] gengliangwang commented on a diff in pull request #36745: [SPARK-39359][SQL] Restrict DEFAULT columns to allowlist of supported data source types

2022-06-07 Thread GitBox
gengliangwang commented on code in PR #36745: URL: https://github.com/apache/spark/pull/36745#discussion_r891067965 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -2881,6 +2881,15 @@ object SQLConf { .booleanConf

[GitHub] [spark] MaxGekk commented on pull request #36753: [SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries

2022-06-07 Thread GitBox
MaxGekk commented on PR #36753: URL: https://github.com/apache/spark/pull/36753#issuecomment-1148512960 @olaky The changes cause conflicts in branch-3.1. Could you PRs w/ backports to 3.1 and 3.0, please. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AngersZhuuuu commented on pull request #36786: [SPARK-39400][SQL] spark-sql should remove hive resource dir in all case

2022-06-07 Thread GitBox
AngersZh commented on PR #36786: URL: https://github.com/apache/spark/pull/36786#issuecomment-1148527000 ping @cloud-fan @wangyum -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] olaky commented on pull request #36386: [SPARK-38918][SQL][3.2] Nested column pruning should filter out attributes that do not belong to the current relation

2022-06-07 Thread GitBox
olaky commented on PR #36386: URL: https://github.com/apache/spark/pull/36386#issuecomment-1148609532 I am facing the same issues here: https://github.com/apache/spark/pull/36753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r891166646 ## core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala: ## @@ -299,9 +300,10 @@ private[spark] class StandaloneAppClient( * * @return

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891293373 ## core/src/main/resources/error/error-classes.json: ## @@ -333,7 +332,7 @@ }, "SECOND_FUNCTION_ARGUMENT_NOT_INTEGER" : { "message" : [ - "The second

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891312375 ## core/src/test/scala/org/apache/spark/SparkFunSuite.scala: ## @@ -264,6 +264,87 @@ abstract class SparkFunSuite } } + /** + * Checks an exception with

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891312743 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891311964 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] tgravescs commented on pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
tgravescs commented on PR #36716: URL: https://github.com/apache/spark/pull/36716#issuecomment-1148763057 > The feature is enabled when dynamic allocation enabled in standalone cluster. So last time I checked dynamic allocation in standalone mode had issues. Have this been

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891318626 ## core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala: ## @@ -863,6 +872,29 @@ private[spark] class TaskSchedulerImpl( executorUpdates) }

[GitHub] [spark] wangyum commented on pull request #36787: [SPARK-39387][BUILD][FOLLOWUP] Upgrade hive-storage-api to 2.7.3

2022-06-07 Thread GitBox
wangyum commented on PR #36787: URL: https://github.com/apache/spark/pull/36787#issuecomment-1148767712 The 2.7.2 will throw runtime exception: ``` 22:38:20.734 ERROR org.apache.spark.util.Utils: Aborting task java.lang.RuntimeException: Overflow of newLength.

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891322239 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] LuciferYang commented on pull request #36694: [MINOR][BUILD] Remove redundant maven `` definition

2022-06-07 Thread GitBox
LuciferYang commented on PR #36694: URL: https://github.com/apache/spark/pull/36694#issuecomment-1148767278 > Is it redundant because of the parent POM? Yes > yeah maybe but I don't think it hurts anything and it's 2 lines so just leave these as they areļ¼Ÿ

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891337273 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1108,45 +1164,48 @@ private[spark] class TaskSetManager( //

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891337356 ## sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala: ## @@ -97,8 +98,18 @@ class CatalogImpl(sparkSession: SparkSession) extends Catalog {

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891336456 ## sql/core/src/main/scala/org/apache/spark/sql/catalog/interface.scala: ## @@ -64,15 +65,34 @@ class Database( @Stable class Table( val name: String, -

[GitHub] [spark] srielau commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
srielau commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891349846 ## core/src/main/resources/error/error-classes.json: ## @@ -333,7 +332,7 @@ }, "SECOND_FUNCTION_ARGUMENT_NOT_INTEGER" : { "message" : [ - "The second

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891346881 ## sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala: ## @@ -553,4 +570,100 @@ class CatalogSuite extends SharedSparkSession with AnalysisTest

[GitHub] [spark] gengliangwang commented on a diff in pull request #36703: [SPARK-39321][SQL] Refactor TryCast to use RuntimeReplaceable

2022-06-07 Thread GitBox
gengliangwang commented on code in PR #36703: URL: https://github.com/apache/spark/pull/36703#discussion_r891136404 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -1792,15 +1792,16 @@ class AstBuilder extends

[GitHub] [spark] HeartSaVioR closed pull request #35484: [SPARK-38181][SS][DOCS] Update comments in KafkaDataConsumer.scala

2022-06-07 Thread GitBox
HeartSaVioR closed pull request #35484: [SPARK-38181][SS][DOCS] Update comments in KafkaDataConsumer.scala URL: https://github.com/apache/spark/pull/35484 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #36784: [SPARK-39396][SQL] Fix LDAP login exception 'error code 49 - invalid credentials'

2022-06-07 Thread GitBox
AmplabJenkins commented on PR #36784: URL: https://github.com/apache/spark/pull/36784#issuecomment-1148680767 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #36778: [SPARK-39383][SQL] Support DEFAULT columns in ALTER TABLE ALTER COLUMNS to V2 data sources

2022-06-07 Thread GitBox
AmplabJenkins commented on PR #36778: URL: https://github.com/apache/spark/pull/36778#issuecomment-1148680929 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891307188 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891314795 ## core/src/test/scala/org/apache/spark/SparkFunSuite.scala: ## @@ -264,6 +264,87 @@ abstract class SparkFunSuite } } + /** + * Checks an exception with

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891314239 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891340370 ## sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala: ## @@ -117,14 +128,44 @@ class CatalogImpl(sparkSession: SparkSession) extends Catalog {

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891341003 ## sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala: ## @@ -117,14 +128,44 @@ class CatalogImpl(sparkSession: SparkSession) extends Catalog {

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891344191 ## sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala: ## @@ -290,7 +304,8 @@ class CatalogSuite extends SharedSparkSession with AnalysisTest {

[GitHub] [spark] cloud-fan closed pull request #35612: [SPARK-38289][SQL] Refactor SQL CLI exit code to make it more clear

2022-06-07 Thread GitBox
cloud-fan closed pull request #35612: [SPARK-38289][SQL] Refactor SQL CLI exit code to make it more clear URL: https://github.com/apache/spark/pull/35612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r891129088 ## core/src/test/scala/org/apache/spark/deploy/JsonProtocolSuite.scala: ## @@ -107,11 +107,11 @@ object JsonConstants { |{"id":"id","starttime":3,"name":"name",

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r891178775 ## core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala: ## @@ -530,6 +535,87 @@ class MasterSuite extends SparkFunSuite

[GitHub] [spark] gengliangwang commented on pull request #36745: [SPARK-39359][SQL] Restrict DEFAULT columns to allowlist of supported data source types

2022-06-07 Thread GitBox
gengliangwang commented on PR #36745: URL: https://github.com/apache/spark/pull/36745#issuecomment-1148625244 I am merging this one to master now. We can have a new DS API for this later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] MaxGekk closed pull request #36753: [SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries

2022-06-07 Thread GitBox
MaxGekk closed pull request #36753: [SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries URL: https://github.com/apache/spark/pull/36753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] srowen commented on a diff in pull request #36499: [SPARK-38846][SQL] Add explicit data mapping between Teradata Numeric Type and Spark DecimalType

2022-06-07 Thread GitBox
srowen commented on code in PR #36499: URL: https://github.com/apache/spark/pull/36499#discussion_r891287048 ## sql/core/src/main/scala/org/apache/spark/sql/jdbc/TeradataDialect.scala: ## @@ -96,4 +97,29 @@ private case object TeradataDialect extends JdbcDialect { override

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891309492 ## core/src/test/scala/org/apache/spark/SparkFunSuite.scala: ## @@ -264,6 +264,87 @@ abstract class SparkFunSuite } } + /** + * Checks an exception with

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891324885 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] srowen commented on a diff in pull request #36737: [SPARK-39347] [SS] Generate wrong time window when (timestamp-startTime) % slideDurationā€¦

2022-06-07 Thread GitBox
srowen commented on code in PR #36737: URL: https://github.com/apache/spark/pull/36737#discussion_r891330435 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -3963,8 +3966,10 @@ object TimeWindowing extends Rule[LogicalPlan] {

[GitHub] [spark] ulysses-you commented on pull request #36785: [SPARK-39397][SQL] Relax AliasAwareOutputExpression to support alias with expression

2022-06-07 Thread GitBox
ulysses-you commented on PR #36785: URL: https://github.com/apache/spark/pull/36785#issuecomment-1148578665 cc @cloud-fan @prakharjain09 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #36704: [SPARK-39346][SQL] Convert asserts/illegal state exception to internal errors on each phase

2022-06-07 Thread GitBox
HeartSaVioR commented on code in PR #36704: URL: https://github.com/apache/spark/pull/36704#discussion_r891139790 ## connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala: ## @@ -666,9 +667,10 @@ abstract class

[GitHub] [spark] HeartSaVioR commented on pull request #35484: [SPARK-38181][SS][DOCS] Update comments in KafkaDataConsumer.scala

2022-06-07 Thread GitBox
HeartSaVioR commented on PR #35484: URL: https://github.com/apache/spark/pull/35484#issuecomment-1148596916 Thanks! Merging to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] olaky commented on pull request #36386: [SPARK-38918][SQL][3.2] Nested column pruning should filter out attributes that do not belong to the current relation

2022-06-07 Thread GitBox
olaky commented on PR #36386: URL: https://github.com/apache/spark/pull/36386#issuecomment-1148616709 So the only change in the plan I can see that makes the test fail is that the last plan node has a source filename in it now, for example `Scan parquet default.web_site

[GitHub] [spark] AmplabJenkins commented on pull request #36781: [SPARK-39393][SQL] Parquet data source only supports push-down predicate filters for non-repeated primitive types

2022-06-07 Thread GitBox
AmplabJenkins commented on PR #36781: URL: https://github.com/apache/spark/pull/36781#issuecomment-1148680852 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] cloud-fan commented on a diff in pull request #36703: [SPARK-39321][SQL] Refactor TryCast to use RuntimeReplaceable

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36703: URL: https://github.com/apache/spark/pull/36703#discussion_r891256881 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala: ## @@ -1792,15 +1792,16 @@ class AstBuilder extends

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891292604 ## core/src/main/resources/error/error-classes.json: ## @@ -157,8 +157,7 @@ "See more details in SPARK-31404. You can set the SQL config or",

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891290994 ## core/src/main/java/org/apache/spark/SparkThrowable.java: ## @@ -36,6 +36,10 @@ public interface SparkThrowable { // If null, error class is not set String

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891304223 ## core/src/main/scala/org/apache/spark/ErrorInfo.scala: ## @@ -73,18 +73,20 @@ private[spark] object SparkThrowableHelper { def getMessage( errorClass:

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891320687 ## sql/catalyst/src/main/scala/org/apache/spark/sql/AnalysisException.scala: ## @@ -36,13 +36,31 @@ class AnalysisException protected[sql] ( @transient val plan:

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891339983 ## sql/core/src/main/scala/org/apache/spark/sql/catalog/interface.scala: ## @@ -55,7 +55,8 @@ class Database( * A table in Spark, as returned by the `listTables`

[GitHub] [spark] cloud-fan commented on pull request #35612: [SPARK-38289][SQL] Refactor SQL CLI exit code to make it more clear

2022-06-07 Thread GitBox
cloud-fan commented on PR #35612: URL: https://github.com/apache/spark/pull/35612#issuecomment-1148543123 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] gengliangwang closed pull request #36745: [SPARK-39359][SQL] Restrict DEFAULT columns to allowlist of supported data source types

2022-06-07 Thread GitBox
gengliangwang closed pull request #36745: [SPARK-39359][SQL] Restrict DEFAULT columns to allowlist of supported data source types URL: https://github.com/apache/spark/pull/36745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] olaky commented on pull request #36753: [SPARK-39259][SQL][3.2] Evaluate timestamps consistently in subqueries

2022-06-07 Thread GitBox
olaky commented on PR #36753: URL: https://github.com/apache/spark/pull/36753#issuecomment-1148633785 Merging is blocked because of a test failure that also surfaces in https://github.com/apache/spark/pull/36386 -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] cloud-fan commented on a diff in pull request #36704: [SPARK-39346][SQL] Convert asserts/illegal state exception to internal errors on each phase

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36704: URL: https://github.com/apache/spark/pull/36704#discussion_r891224222 ## connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala: ## @@ -666,9 +667,10 @@ abstract class

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891297431 ## core/src/main/java/org/apache/spark/memory/SparkOutOfMemoryError.java: ## @@ -39,11 +39,17 @@ public SparkOutOfMemoryError(OutOfMemoryError e) { }

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891301726 ## core/src/main/scala/org/apache/spark/ErrorInfo.scala: ## @@ -73,18 +73,20 @@ private[spark] object SparkThrowableHelper { def getMessage( errorClass:

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891302240 ## core/src/main/scala/org/apache/spark/ErrorInfo.scala: ## @@ -98,6 +100,29 @@ private[spark] object SparkThrowableHelper { s"[$displayClass]

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891301726 ## core/src/main/scala/org/apache/spark/ErrorInfo.scala: ## @@ -73,18 +73,20 @@ private[spark] object SparkThrowableHelper { def getMessage( errorClass:

[GitHub] [spark] cloud-fan commented on a diff in pull request #36693: [SPARK-39349] Add a centralized CheckError method for QA of error path

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36693: URL: https://github.com/apache/spark/pull/36693#discussion_r891302240 ## core/src/main/scala/org/apache/spark/ErrorInfo.scala: ## @@ -98,6 +100,29 @@ private[spark] object SparkThrowableHelper { s"[$displayClass]

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891316334 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -800,6 +814,10 @@ private[spark] class TaskSetManager(

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891334793 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -2185,6 +2185,11 @@ object QueryCompilationErrors extends

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891334508 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] cloud-fan commented on a diff in pull request #36586: [SPARK-39236][SQL] Make CreateTable and ListTables be compatible with 3 layer namespace

2022-06-07 Thread GitBox
cloud-fan commented on code in PR #36586: URL: https://github.com/apache/spark/pull/36586#discussion_r891335856 ## sql/core/src/main/scala/org/apache/spark/sql/catalog/interface.scala: ## @@ -64,15 +65,34 @@ class Database( @Stable class Table( val name: String, -

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-07 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r891335474 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1217,6 +1260,71 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

  1   2   3   >