[GitHub] [spark] cloud-fan commented on a diff in pull request #36934: [SPARK-39537][SQL] Ensure that the expressions order returned by `InferFiltersFromConstraints`

2022-06-20 Thread GitBox
cloud-fan commented on code in PR #36934: URL: https://github.com/apache/spark/pull/36934#discussion_r902185170 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala: ## @@ -1272,8 +1272,11 @@ object InferFiltersFromConstraints extends

[GitHub] [spark] lsm1 opened a new pull request, #36937: [SPARK-39539][SQL] millisToMicros overflow

2022-06-20 Thread GitBox
lsm1 opened a new pull request, #36937: URL: https://github.com/apache/spark/pull/36937 ### What changes were proposed in this pull request? ### Why are the changes needed? When convert the millis to micro value, it will raise Overflow Error ```scala

[GitHub] [spark] ulysses-you opened a new pull request, #36936: [SPARK-39503][SQL] Add session catalog name for v1 database table and function

2022-06-20 Thread GitBox
ulysses-you opened a new pull request, #36936: URL: https://github.com/apache/spark/pull/36936 ### What changes were proposed in this pull request? - Add session catalog name in identifiers, then all identifiers will be 3 part name - Make `CatalogImpl#refreshTable` support

[GitHub] [spark] zhengruifeng commented on pull request #36926: [SPARK-39533][ML] Remove scoreLabelsWeight in BinaryClassificationMetrics

2022-06-20 Thread GitBox
zhengruifeng commented on PR #36926: URL: https://github.com/apache/spark/pull/36926#issuecomment-1161256434 since it is only exposed in the `.mllib` side, and it is not listed in the [official documents

[GitHub] [spark] Ngone51 commented on pull request #36512: [SPARK-39152][CORE] Deregistering disk persisted local blocks in case of IO related errors

2022-06-20 Thread GitBox
Ngone51 commented on PR #36512: URL: https://github.com/apache/spark/pull/36512#issuecomment-1161238375 Thanks, merged to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] Ngone51 closed pull request #36512: [SPARK-39152][CORE] Deregistering disk persisted local blocks in case of IO related errors

2022-06-20 Thread GitBox
Ngone51 closed pull request #36512: [SPARK-39152][CORE] Deregistering disk persisted local blocks in case of IO related errors URL: https://github.com/apache/spark/pull/36512 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] Ngone51 commented on a diff in pull request #36716: [SPARK-39062][CORE] Add stage level resource scheduling support for standalone cluster

2022-06-20 Thread GitBox
Ngone51 commented on code in PR #36716: URL: https://github.com/apache/spark/pull/36716#discussion_r902140262 ## docs/job-scheduling.md: ## @@ -83,6 +83,10 @@ This feature is disabled by default and available on all coarse-grained cluster [Mesos coarse-grained

[GitHub] [spark] srowen commented on pull request #36926: [SPARK-39533][ML] Remove scoreLabelsWeight in BinaryClassificationMetrics

2022-06-20 Thread GitBox
srowen commented on PR #36926: URL: https://github.com/apache/spark/pull/36926#issuecomment-1161221959 Right, it was still public in the binary code. If there's an easy way to retain source compatibility, that's great, otherwise OK -- This is an automated message from the Apache Git

[GitHub] [spark] srowen commented on pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
srowen commented on PR #36925: URL: https://github.com/apache/spark/pull/36925#issuecomment-1161222543 Other things were removed, like the override of union(). May be fine, just now seems to be worth another look -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] zhengruifeng commented on pull request #36926: [SPARK-39533][ML] Remove scoreLabelsWeight in BinaryClassificationMetrics

2022-06-20 Thread GitBox
zhengruifeng commented on PR #36926: URL: https://github.com/apache/spark/pull/36926#issuecomment-1161214871 @srowen hmm, I just think it is an internal variable, should not be exposed to end users, like other priavte variables in `MulticlassMetrics` and so on. However, it had been public

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-20 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r902126766 ## core/src/main/scala/org/apache/spark/internal/config/package.scala: ## @@ -2073,6 +2073,37 @@ package object config { .timeConf(TimeUnit.MILLISECONDS)

[GitHub] [spark] Ngone51 commented on a diff in pull request #36162: [SPARK-32170][CORE] Improve the speculation through the stage task metrics.

2022-06-20 Thread GitBox
Ngone51 commented on code in PR #36162: URL: https://github.com/apache/spark/pull/36162#discussion_r902100661 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1218,6 +1248,70 @@ private[spark] class TaskSetManager( def executorAdded(): Unit = {

[GitHub] [spark] zhengruifeng commented on pull request #36927: [SPARK-39534][PS] Series.argmax only needs single pass

2022-06-20 Thread GitBox
zhengruifeng commented on PR #36927: URL: https://github.com/apache/spark/pull/36927#issuecomment-1161195094 thanks @HyukjinKwon for reviewing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] cloud-fan closed pull request #36641: [SPARK-39263][SQL] Make GetTable, TableExists and DatabaseExists be compatible with 3 layer namespace

2022-06-20 Thread GitBox
cloud-fan closed pull request #36641: [SPARK-39263][SQL] Make GetTable, TableExists and DatabaseExists be compatible with 3 layer namespace URL: https://github.com/apache/spark/pull/36641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] cloud-fan commented on pull request #36641: [SPARK-39263][SQL] Make GetTable, TableExists and DatabaseExists be compatible with 3 layer namespace

2022-06-20 Thread GitBox
cloud-fan commented on PR #36641: URL: https://github.com/apache/spark/pull/36641#issuecomment-1161188261 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon opened a new pull request, #36935: [MINOR][INFRA] Remove branch and types in GitHub Actions

2022-06-20 Thread GitBox
HyukjinKwon opened a new pull request, #36935: URL: https://github.com/apache/spark/pull/36935 ### What changes were proposed in this pull request? After SPARK-39521, we don't need such information anymore because they are all classified under Actions as below:

[GitHub] [spark] LuciferYang commented on pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
LuciferYang commented on PR #36925: URL: https://github.com/apache/spark/pull/36925#issuecomment-1161176292 > Merged to master. It looks like this wasn't a problem in 3.3.x; this change removed the overrides: > > https://issues.apache.org/jira/browse/SPARK-38836 #36121 > > I

[GitHub] [spark] beliefer opened a new pull request, #36934: [SPARK-39537][SQL] Ensure that the expressions order returned by `InferFiltersFromConstraints`

2022-06-20 Thread GitBox
beliefer opened a new pull request, #36934: URL: https://github.com/apache/spark/pull/36934 ### What changes were proposed in this pull request? Currently, `InferFiltersFromConstraints` construct a new `Filter` which condition without the same order in scala 2.12 and 2.13 It seems

[GitHub] [spark] HyukjinKwon commented on pull request #36930: [SPARK-39538][SQL] Convert CaseInsensitiveStringMap#logger to static

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36930: URL: https://github.com/apache/spark/pull/36930#issuecomment-1161163389  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HyukjinKwon commented on pull request #36930: [SPARK-39538][SQL] Convert CaseInsensitiveStringMap#logger to static

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36930: URL: https://github.com/apache/spark/pull/36930#issuecomment-1161151658 Pushed another fix just now. Mind trying agian? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] HyukjinKwon closed pull request #36931: [DO-NOT-MERGE] Test PR

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36931: [DO-NOT-MERGE] Test PR URL: https://github.com/apache/spark/pull/36931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [spark] HyukjinKwon closed pull request #36933: [SPARK-39521][INFRA][FOLLOW-UP] Update step name to "Run / Check changes" to detect the workflow

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36933: [SPARK-39521][INFRA][FOLLOW-UP] Update step name to "Run / Check changes" to detect the workflow URL: https://github.com/apache/spark/pull/36933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on pull request #36933: [SPARK-39521][INFRA][FOLLOW-UP] Update step name to "Run / Check changes" to detect the workflow

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36933: URL: https://github.com/apache/spark/pull/36933#issuecomment-1161147277 Merged to master. To recover the build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] HyukjinKwon opened a new pull request, #36933: [SPARK-39521][INFRA][FOLLOW-UP] Update step name to "Run / Check changes" to detect the workflow

2022-06-20 Thread GitBox
HyukjinKwon opened a new pull request, #36933: URL: https://github.com/apache/spark/pull/36933 ### What changes were proposed in this pull request? This PR changes the step name that's renamed in previous PR. ### Why are the changes needed? To recover the build. ###

[GitHub] [spark] HyukjinKwon closed pull request #36931: [DO-NOT-MERGE] Test PR

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36931: [DO-NOT-MERGE] Test PR URL: https://github.com/apache/spark/pull/36931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [spark] zinking commented on a diff in pull request #36918: [SQL][SPARK-39528] Use V2 Filter in SupportsRuntimeFiltering

2022-06-20 Thread GitBox
zinking commented on code in PR #36918: URL: https://github.com/apache/spark/pull/36918#discussion_r902107022 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala: ## @@ -55,16 +57,27 @@ case class BatchScanExec( @transient private

[GitHub] [spark] pan3793 commented on pull request #36930: [SPARK-39538][SQL] Convert CaseInsensitiveStringMap#logger to static

2022-06-20 Thread GitBox
pan3793 commented on PR #36930: URL: https://github.com/apache/spark/pull/36930#issuecomment-1161120159 Thanks @HyukjinKwon, let me try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon closed pull request #36931: [DO-NOT-MERGE] Test PR

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36931: [DO-NOT-MERGE] Test PR URL: https://github.com/apache/spark/pull/36931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [spark] HyukjinKwon commented on pull request #36930: [SPARK-39538][SQL] Convert CaseInsensitiveStringMap#logger to static

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36930: URL: https://github.com/apache/spark/pull/36930#issuecomment-1161117081 Just fixed. mind syncing again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon commented on pull request #36932: [SPARK-39521][INFRA][FOLLOW-UP] Fix notify workload to detect the main build, and readme link

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36932: URL: https://github.com/apache/spark/pull/36932#issuecomment-1161116603 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon closed pull request #36932: [SPARK-39521][INFRA][FOLLOW-UP] Fix notify workload to detect the main build, and readme link

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36932: [SPARK-39521][INFRA][FOLLOW-UP] Fix notify workload to detect the main build, and readme link URL: https://github.com/apache/spark/pull/36932 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon opened a new pull request, #36932: [SPARK-39521][INFRA][FOLLOW-UP] Fix notify workload to detect the main build, and readme link

2022-06-20 Thread GitBox
HyukjinKwon opened a new pull request, #36932: URL: https://github.com/apache/spark/pull/36932 ### What changes were proposed in this pull request? This PR fixes the notify_test_workflow.yml to detect build_main.yml that is for PR builds. In addition, this PR fixes the link of

[GitHub] [spark] HyukjinKwon commented on pull request #36932: [SPARK-39521][INFRA][FOLLOW-UP] Fix notify workload to detect the main build, and readme link

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36932: URL: https://github.com/apache/spark/pull/36932#issuecomment-1161116097 I am merging this to recover ASAP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] pan3793 commented on pull request #36930: [SPARK-39538][SQL] Convert CaseInsensitiveStringMap#logger to static

2022-06-20 Thread GitBox
pan3793 commented on PR #36930: URL: https://github.com/apache/spark/pull/36930#issuecomment-1161102852 I think my master branch has been synced. https://github.com/pan3793/spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on pull request #36930: [SPARK-39538][SQL] Convert CaseInsensitiveStringMap#logger to static

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36930: URL: https://github.com/apache/spark/pull/36930#issuecomment-1161099595 @pan3793 mind rebasing your master branch in your forked repo, to the latest master branch in Apache Spark? -- This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] pan3793 opened a new pull request, #36930: [SPARK-39538][SQL] Convert CaseInsensitiveStringMap#logger to static

2022-06-20 Thread GitBox
pan3793 opened a new pull request, #36930: URL: https://github.com/apache/spark/pull/36930 ### What changes were proposed in this pull request? Convert `CaseInsensitiveStringMap#logger` to static, it's safe since the `org.slf4j.Logger` is thread-safe. ### Why are the

[GitHub] [spark] chenjunbiao001 commented on pull request #35721: [SPARK-38405][SQL] Supports customization of the entire optimizer and planner

2022-06-20 Thread GitBox
chenjunbiao001 commented on PR #35721: URL: https://github.com/apache/spark/pull/35721#issuecomment-1160981399 这是来自QQ邮箱的假期自动回复邮件。   您好,邮件已收到,感谢您的来信! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] github-actions[bot] commented on pull request #35721: [SPARK-38405][SQL] Supports customization of the entire optimizer and planner

2022-06-20 Thread GitBox
github-actions[bot] commented on PR #35721: URL: https://github.com/apache/spark/pull/35721#issuecomment-1160981158 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.

[GitHub] [spark] dongjoon-hyun commented on pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
dongjoon-hyun commented on PR #36922: URL: https://github.com/apache/spark/pull/36922#issuecomment-1160972770 +1, LGTM! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36787: [SPARK-39387][FOLLOWUP][TESTS] Add a test case for HIVE-25190

2022-06-20 Thread GitBox
HyukjinKwon commented on code in PR #36787: URL: https://github.com/apache/spark/pull/36787#discussion_r902056954 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala: ## @@ -832,6 +832,18 @@ abstract class OrcQuerySuite extends

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36883: [SPARK-39061][SQL] Set nullable correctly for `Inline` output attributes

2022-06-20 Thread GitBox
HyukjinKwon commented on code in PR #36883: URL: https://github.com/apache/spark/pull/36883#discussion_r902056893 ## sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala: ## @@ -389,6 +389,31 @@ class GeneratorFunctionSuite extends QueryTest with

[GitHub] [spark] HyukjinKwon closed pull request #36927: [SPARK-39534][PS] Series.argmax only needs single pass

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36927: [SPARK-39534][PS] Series.argmax only needs single pass URL: https://github.com/apache/spark/pull/36927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon commented on pull request #36927: [SPARK-39534][PS] Series.argmax only needs single pass

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36927: URL: https://github.com/apache/spark/pull/36927#issuecomment-1160966557 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon commented on pull request #36928: [SPARK-39529][INFRA] Combine two configure jobs, centralize logic

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36928: URL: https://github.com/apache/spark/pull/36928#issuecomment-1160964597 and .. seems like actually what this PR does is similar with https://github.com/apache/spark/pull/36922 in fact :-). -- This is an automated message from the Apache Git Service. To

[GitHub] [spark] HyukjinKwon commented on pull request #36928: [SPARK-39529][CI] Combine two configure jobs, centralize logic

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36928: URL: https://github.com/apache/spark/pull/36928#issuecomment-1160963905 Thanks for taking a look, @EnricoMi . Mind rebasing this please? I think it conflicted with https://github.com/apache/spark/pull/36922 -- This is an automated message from the

[GitHub] [spark] dependabot[bot] commented on pull request #36929: Bump mysql-connector-java from 8.0.27 to 8.0.28

2022-06-20 Thread GitBox
dependabot[bot] commented on PR #36929: URL: https://github.com/apache/spark/pull/36929#issuecomment-1160961481 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version,

[GitHub] [spark] HyukjinKwon closed pull request #36929: Bump mysql-connector-java from 8.0.27 to 8.0.28

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36929: Bump mysql-connector-java from 8.0.27 to 8.0.28 URL: https://github.com/apache/spark/pull/36929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] HyukjinKwon closed pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
HyukjinKwon closed pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow URL: https://github.com/apache/spark/pull/36922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] HyukjinKwon commented on pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36922: URL: https://github.com/apache/spark/pull/36922#issuecomment-1160960656 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] uchiiii commented on a diff in pull request #36920: [SPARK-39446][MLLIB][FOLLOWUP] Modify constructor of RankingMetrics class

2022-06-20 Thread GitBox
uch commented on code in PR #36920: URL: https://github.com/apache/spark/pull/36920#discussion_r902047346 ## mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala: ## @@ -154,54 +151,68 @@ class RankingMetrics[T: ClassTag] @Since("3.4.0") (

[GitHub] [spark] uchiiii commented on a diff in pull request #36920: [SPARK-39446][MLLIB][FOLLOWUP] Modify constructor of RankingMetrics class

2022-06-20 Thread GitBox
uch commented on code in PR #36920: URL: https://github.com/apache/spark/pull/36920#discussion_r902047346 ## mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala: ## @@ -154,54 +151,68 @@ class RankingMetrics[T: ClassTag] @Since("3.4.0") (

[GitHub] [spark] dependabot[bot] opened a new pull request, #36929: Bump mysql-connector-java from 8.0.27 to 8.0.28

2022-06-20 Thread GitBox
dependabot[bot] opened a new pull request, #36929: URL: https://github.com/apache/spark/pull/36929 Bumps [mysql-connector-java](https://github.com/mysql/mysql-connector-j) from 8.0.27 to 8.0.28. Changelog Sourced from

[GitHub] [spark] uchiiii commented on a diff in pull request #36920: [SPARK-39446][MLLIB][FOLLOWUP] Modify constructor of RankingMetrics class

2022-06-20 Thread GitBox
uch commented on code in PR #36920: URL: https://github.com/apache/spark/pull/36920#discussion_r902047346 ## mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala: ## @@ -154,54 +151,68 @@ class RankingMetrics[T: ClassTag] @Since("3.4.0") (

[GitHub] [spark] srowen commented on pull request #36499: [SPARK-38846][SQL] Add explicit data mapping between Teradata Numeric Type and Spark DecimalType

2022-06-20 Thread GitBox
srowen commented on PR #36499: URL: https://github.com/apache/spark/pull/36499#issuecomment-1160937525 Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [spark] srowen closed pull request #36499: [SPARK-38846][SQL] Add explicit data mapping between Teradata Numeric Type and Spark DecimalType

2022-06-20 Thread GitBox
srowen closed pull request #36499: [SPARK-38846][SQL] Add explicit data mapping between Teradata Numeric Type and Spark DecimalType URL: https://github.com/apache/spark/pull/36499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] HyukjinKwon commented on pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36922: URL: https://github.com/apache/spark/pull/36922#issuecomment-1160922013 Guys I'll merge this soon and monitor today (it's morning my time) so I can follow up and fix some more if there are something going wrong -- This is an automated message from the

[GitHub] [spark] srowen commented on pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
srowen commented on PR #36925: URL: https://github.com/apache/spark/pull/36925#issuecomment-1160811940 Merged to master. It looks like this wasn't a problem in 3.3.x; this change removed the overrides: https://issues.apache.org/jira/browse/SPARK-38836

[GitHub] [spark] srowen closed pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
srowen closed pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13 URL: https://github.com/apache/spark/pull/36925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] gengliangwang closed pull request #36896: [SPARK-39497][SQL] Improve the analysis exception of missing map key column

2022-06-20 Thread GitBox
gengliangwang closed pull request #36896: [SPARK-39497][SQL] Improve the analysis exception of missing map key column URL: https://github.com/apache/spark/pull/36896 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] gengliangwang commented on pull request #36896: [SPARK-39497][SQL] Improve the analysis exception of missing map key column

2022-06-20 Thread GitBox
gengliangwang commented on PR #36896: URL: https://github.com/apache/spark/pull/36896#issuecomment-1160696817 Merging to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] bersprockets commented on a diff in pull request #36883: [SPARK-39061][SQL] Set nullable correctly for `Inline` output attributes

2022-06-20 Thread GitBox
bersprockets commented on code in PR #36883: URL: https://github.com/apache/spark/pull/36883#discussion_r901871005 ## sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala: ## @@ -389,6 +389,31 @@ class GeneratorFunctionSuite extends QueryTest with

[GitHub] [spark] Eugene-Mark commented on a diff in pull request #36499: [SPARK-38846][SQL] Add explicit data mapping between Teradata Numeric Type and Spark DecimalType

2022-06-20 Thread GitBox
Eugene-Mark commented on code in PR #36499: URL: https://github.com/apache/spark/pull/36499#discussion_r901863572 ## docs/sql-migration-guide.md: ## @@ -22,6 +22,11 @@ license: | * Table of contents {:toc} +## Upgrading from Spark SQL 3.3 to 3.4 + Review Comment:

[GitHub] [spark] srowen commented on a diff in pull request #36499: [SPARK-38846][SQL] Add explicit data mapping between Teradata Numeric Type and Spark DecimalType

2022-06-20 Thread GitBox
srowen commented on code in PR #36499: URL: https://github.com/apache/spark/pull/36499#discussion_r901825209 ## docs/sql-migration-guide.md: ## @@ -22,6 +22,11 @@ license: | * Table of contents {:toc} +## Upgrading from Spark SQL 3.3 to 3.4 + Review Comment: Is this

[GitHub] [spark] dongjoon-hyun commented on pull request #36923: [SPARK-39530][SS][TESTS] Fix `KafkaTestUtils` to support IPv6

2022-06-20 Thread GitBox
dongjoon-hyun commented on PR #36923: URL: https://github.com/apache/spark/pull/36923#issuecomment-1160597253 Thank you, @wangyum ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] Eugene-Mark commented on pull request #36499: [SPARK-38846][SQL] Add explicit data mapping between Teradata Numeric Type and Spark DecimalType

2022-06-20 Thread GitBox
Eugene-Mark commented on PR #36499: URL: https://github.com/apache/spark/pull/36499#issuecomment-1160577823 Document updated, thanks for previous valuable comments! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] cxzl25 commented on a diff in pull request #36787: [SPARK-39387][FOLLOWUP][TESTS] Add a test case for HIVE-25190

2022-06-20 Thread GitBox
cxzl25 commented on code in PR #36787: URL: https://github.com/apache/spark/pull/36787#discussion_r901783862 ## sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala: ## @@ -832,6 +832,18 @@ abstract class OrcQuerySuite extends OrcQueryTest

[GitHub] [spark] EnricoMi commented on pull request #36888: [CI] Check if tests are to be run for build and pyspark matrix jobs

2022-06-20 Thread GitBox
EnricoMi commented on PR #36888: URL: https://github.com/apache/spark/pull/36888#issuecomment-1160553251 @HyukjinKwon here is an improved version: https://github.com/G-Research/spark/pull/9/files It is based on https://github.com/apache/spark/pull/36928. -- This is an automated

[GitHub] [spark] xiuzhu9527 commented on a diff in pull request #36784: [SPARK-39396][SQL] Fix LDAP login exception 'error code 49 - invalid credentials'

2022-06-20 Thread GitBox
xiuzhu9527 commented on code in PR #36784: URL: https://github.com/apache/spark/pull/36784#discussion_r901761793 ## sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java: ## @@ -30,21 +35,19 @@ public class

[GitHub] [spark] AmplabJenkins commented on pull request #36920: [SPARK-39446][MLLIB][FOLLOWUP] Modify constructor of RankingMetrics class

2022-06-20 Thread GitBox
AmplabJenkins commented on PR #36920: URL: https://github.com/apache/spark/pull/36920#issuecomment-1160520565 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] wangyum commented on pull request #36842: [SPARK-39445][SQL] Remove the window if windowExpressions is empty in column pruning

2022-06-20 Thread GitBox
wangyum commented on PR #36842: URL: https://github.com/apache/spark/pull/36842#issuecomment-1160511334 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] wangyum closed pull request #36842: [SPARK-39445][SQL] Remove the window if windowExpressions is empty in column pruning

2022-06-20 Thread GitBox
wangyum closed pull request #36842: [SPARK-39445][SQL] Remove the window if windowExpressions is empty in column pruning URL: https://github.com/apache/spark/pull/36842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] LuciferYang commented on a diff in pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
LuciferYang commented on code in PR #36925: URL: https://github.com/apache/spark/pull/36925#discussion_r901720849 ## sql/catalyst/src/main/scala-2.13/org/apache/spark/sql/catalyst/expressions/ExpressionSet.scala: ## @@ -132,6 +132,12 @@ class ExpressionSet protected(

[GitHub] [spark] wangyum commented on a diff in pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
wangyum commented on code in PR #36925: URL: https://github.com/apache/spark/pull/36925#discussion_r901709796 ## sql/catalyst/src/main/scala-2.13/org/apache/spark/sql/catalyst/expressions/ExpressionSet.scala: ## @@ -132,6 +132,12 @@ class ExpressionSet protected( newSet

[GitHub] [spark] wangyum commented on pull request #36923: [SPARK-39530][SS][TESTS] Fix `KafkaTestUtils` to support IPv6

2022-06-20 Thread GitBox
wangyum commented on PR #36923: URL: https://github.com/apache/spark/pull/36923#issuecomment-1160494791 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] wangyum closed pull request #36923: [SPARK-39530][SS][TESTS] Fix `KafkaTestUtils` to support IPv6

2022-06-20 Thread GitBox
wangyum closed pull request #36923: [SPARK-39530][SS][TESTS] Fix `KafkaTestUtils` to support IPv6 URL: https://github.com/apache/spark/pull/36923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] EnricoMi commented on pull request #36928: [SPARK-39529][CI] Combine two configure jobs, centralize logic

2022-06-20 Thread GitBox
EnricoMi commented on PR #36928: URL: https://github.com/apache/spark/pull/36928#issuecomment-1160466102 @HyukjinKwon here is some simplification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] EnricoMi opened a new pull request, #36928: [SPARK-39529][CI] Combine two configure jobs, centralize logic

2022-06-20 Thread GitBox
EnricoMi opened a new pull request, #36928: URL: https://github.com/apache/spark/pull/36928 ### What changes were proposed in this pull request? CI jobs `configure-jobs` and `precondition` can be combined into a single job, skip logic can be moved into one place. ### Why are the

[GitHub] [spark] zhengruifeng opened a new pull request, #36927: [SPARK-39534][PS] Series.argmax only needs single pass

2022-06-20 Thread GitBox
zhengruifeng opened a new pull request, #36927: URL: https://github.com/apache/spark/pull/36927 ### What changes were proposed in this pull request? compute `Series.argmax ` with one pass ### Why are the changes needed? existing implemation of `Series.argmax` needs two pass

[GitHub] [spark] LuciferYang commented on pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
LuciferYang commented on PR #36925: URL: https://github.com/apache/spark/pull/36925#issuecomment-1160395973 run ``` dev/change-scala-version.sh 2.13 mvn clean install -DskipTests -pl sql/catalyst -am -Pscala-2.13 mvn test -pl sql/catalyst -Pscala-2.13 ``` All test

[GitHub] [spark] zhengruifeng opened a new pull request, #36926: [SPARK-39533][ML] Remove scoreLabelsWeight in BinaryClassificationMetrics

2022-06-20 Thread GitBox
zhengruifeng opened a new pull request, #36926: URL: https://github.com/apache/spark/pull/36926 ### What changes were proposed in this pull request? Remove `scoreLabelsWeight` in `BinaryClassificationMetrics` ### Why are the changes needed? `scoreLabelsWeight` in

[GitHub] [spark] pan3793 commented on a diff in pull request #36784: [SPARK-39396][SQL] Fix LDAP login exception 'error code 49 - invalid credentials'

2022-06-20 Thread GitBox
pan3793 commented on code in PR #36784: URL: https://github.com/apache/spark/pull/36784#discussion_r901603588 ## sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/LdapAuthenticationProviderImpl.java: ## @@ -30,21 +35,19 @@ public class

[GitHub] [spark] zhengruifeng commented on a diff in pull request #36920: [SPARK-39446][MLLIB][FOLLOWUP] Modify constructor of RankingMetrics class

2022-06-20 Thread GitBox
zhengruifeng commented on code in PR #36920: URL: https://github.com/apache/spark/pull/36920#discussion_r901586489 ## mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala: ## @@ -70,9 +62,14 @@ class RankingMetrics[T: ClassTag] @Since("3.4.0") (

[GitHub] [spark] LuciferYang commented on pull request #36925: [SPARK-39520][SQL] Override `--` method for `ExpressionSet` in Scala 2.13

2022-06-20 Thread GitBox
LuciferYang commented on PR #36925: URL: https://github.com/apache/spark/pull/36925#issuecomment-1160357491 cc @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [spark] LuciferYang opened a new pull request, #36925: [SPARK-39520][SQL] Override `--` method for Scala 2.13 `ExpressionSet`

2022-06-20 Thread GitBox
LuciferYang opened a new pull request, #36925: URL: https://github.com/apache/spark/pull/36925 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was

[GitHub] [spark] uchiiii commented on a diff in pull request #36920: [SPARK-39446][MLLIB][FOLLOWUP] Modify constructor of RankingMetrics class

2022-06-20 Thread GitBox
uch commented on code in PR #36920: URL: https://github.com/apache/spark/pull/36920#discussion_r901533321 ## mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala: ## @@ -70,9 +62,14 @@ class RankingMetrics[T: ClassTag] @Since("3.4.0") (

[GitHub] [spark] EnricoMi commented on a diff in pull request #36924: [SPARK-39532][INFRA] Move checkout and sync steps into re-usable action

2022-06-20 Thread GitBox
EnricoMi commented on code in PR #36924: URL: https://github.com/apache/spark/pull/36924#discussion_r901524495 ## .github/actions/sync-fork-with-upstream/action.yml: ## @@ -0,0 +1,19 @@ +name: 'Sync sources' +author: 'Apache Spark' +description: 'A composite GitHub Action that

[GitHub] [spark] HyukjinKwon commented on pull request #36924: [SPARK-39532][INFRA] Move checkout and sync steps into re-usable action

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36924: URL: https://github.com/apache/spark/pull/36924#issuecomment-1160286264 uhoh, right. thanks for explanation though :-). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] EnricoMi closed pull request #36924: [SPARK-39532][INFRA] Move checkout and sync steps into re-usable action

2022-06-20 Thread GitBox
EnricoMi closed pull request #36924: [SPARK-39532][INFRA] Move checkout and sync steps into re-usable action URL: https://github.com/apache/spark/pull/36924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] EnricoMi commented on pull request #36924: [SPARK-39532][INFRA] Move checkout and sync steps into re-usable action

2022-06-20 Thread GitBox
EnricoMi commented on PR #36924: URL: https://github.com/apache/spark/pull/36924#issuecomment-1160285430 This is not applicable to Apache Spark CI as it first checksout Spark repo, then the fork repo, while this action (until merged into upstream repo) only becomes available after the sync

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36924: [SPARK-39532][INFRA] Move checkout and sync steps into re-usable action

2022-06-20 Thread GitBox
HyukjinKwon commented on code in PR #36924: URL: https://github.com/apache/spark/pull/36924#discussion_r901519239 ## .github/actions/sync-fork-with-upstream/action.yml: ## @@ -0,0 +1,19 @@ +name: 'Sync sources' +author: 'Apache Spark' +description: 'A composite GitHub Action

[GitHub] [spark] AmplabJenkins commented on pull request #36924: [SPARK-39532][INFRA] Move checkout and sync steps into re-usable action

2022-06-20 Thread GitBox
AmplabJenkins commented on PR #36924: URL: https://github.com/apache/spark/pull/36924#issuecomment-1160283324 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] EnricoMi commented on pull request #36924: [CI] Move checkout and sync steps into re-usable composite action

2022-06-20 Thread GitBox
EnricoMi commented on PR #36924: URL: https://github.com/apache/spark/pull/36924#issuecomment-1160250439 @HyukjinKwon here is some first simplification of the `build-and-test` workflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] EnricoMi opened a new pull request, #36924: [CI] Move checkout and sync steps into re-usable composite action

2022-06-20 Thread GitBox
EnricoMi opened a new pull request, #36924: URL: https://github.com/apache/spark/pull/36924 ### What changes were proposed in this pull request? Steps "Checkout Spark repository" and "Sync the current branch with the latest in Apache Spark" are defined 9 times in `build-and-test.yml`.

[GitHub] [spark] HyukjinKwon commented on pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36922: URL: https://github.com/apache/spark/pull/36922#issuecomment-1160206599 Seems working fine as expected. We can merge this once https://github.com/HyukjinKwon/spark/actions/runs/2527981545 gets passed. After this PR, developers should fetch

[GitHub] [spark] HyukjinKwon commented on pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36922: URL: https://github.com/apache/spark/pull/36922#issuecomment-1160198370 Seems working fine as expected. We can merge this once https://github.com/HyukjinKwon/spark/actions/runs/2527942990 gets passed. After this PR, developers should fetch

[GitHub] [spark] HyukjinKwon commented on pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
HyukjinKwon commented on PR #36922: URL: https://github.com/apache/spark/pull/36922#issuecomment-1160193622 Seems working fine as expected. We can merge this once https://github.com/HyukjinKwon/spark/actions/runs/2527898372 gets passed. After this PR, developers should fetch

[GitHub] [spark] dongjoon-hyun commented on pull request #36923: [SPARK-39530][SS][TESTS] Fix `KafkaTestUtils` to support IPv6

2022-06-20 Thread GitBox
dongjoon-hyun commented on PR #36923: URL: https://github.com/apache/spark/pull/36923#issuecomment-1160175814 Thank you, @HyukjinKwon ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun commented on pull request #36922: [SPARK-39521][INFRA] Separate scheduled jobs to each workflow

2022-06-20 Thread GitBox
dongjoon-hyun commented on PR #36922: URL: https://github.com/apache/spark/pull/36922#issuecomment-1160152206 Thank you for this refactoring. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] LuciferYang commented on pull request #36876: [SPARK-39464][CORE][TESTS][FOLLOWUP] Use Utils.localHostNameForURI instead of Utils.localCanonicalHostName in tests

2022-06-20 Thread GitBox
LuciferYang commented on PR #36876: URL: https://github.com/apache/spark/pull/36876#issuecomment-1160150926 thanks @dongjoon-hyun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] dongjoon-hyun closed pull request #36876: [SPARK-39464][CORE][TESTS][FOLLOWUP] Use Utils.localHostNameForURI instead of Utils.localCanonicalHostName in tests

2022-06-20 Thread GitBox
dongjoon-hyun closed pull request #36876: [SPARK-39464][CORE][TESTS][FOLLOWUP] Use Utils.localHostNameForURI instead of Utils.localCanonicalHostName in tests URL: https://github.com/apache/spark/pull/36876 -- This is an automated message from the Apache Git Service. To respond to the

  1   2   >