[GitHub] [spark] dongjoon-hyun commented on pull request #31962: [SPARK-34869][K8S][TEST] Extend "EXTRA LOGS FOR THE FAILED TEST" section of k8s integration test log with the describe pods output

dongjoon-hyun commented on pull request #31962: URL: https://github.com/apache/spark/pull/31962#issuecomment-808922744 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [spark] dongjoon-hyun closed pull request #31962: [SPARK-34869][K8S][TEST] Extend "EXTRA LOGS FOR THE FAILED TEST" section of k8s integration test log with the describe pods output

dongjoon-hyun closed pull request #31962: URL: https://github.com/apache/spark/pull/31962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] dongjoon-hyun closed pull request #31969: [SPARK-32855][SQL][FOLLOWUP] Fix code format in SQLConf and comment in PartitionPruning

dongjoon-hyun closed pull request #31969: URL: https://github.com/apache/spark/pull/31969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] SparkQA commented on pull request #31967: [SPAKR-34819][SQL]MapType supports orderable semantics

SparkQA commented on pull request #31967: URL: https://github.com/apache/spark/pull/31967#issuecomment-808923852 **[Test build #136614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136614/testReport)** for PR 31967 at commit [`58bb3cc`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31967: [SPAKR-34819][SQL]MapType supports orderable semantics

SparkQA removed a comment on pull request #31967: URL: https://github.com/apache/spark/pull/31967#issuecomment-808890727 **[Test build #136614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136614/testReport)** for PR 31967 at commit [`58bb3cc`](https://gi

[GitHub] [spark] dongjoon-hyun closed pull request #31955: [SPARK-34829][SQL] Fix higher order function results

dongjoon-hyun closed pull request #31955: URL: https://github.com/apache/spark/pull/31955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602906304 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -528,4 +530,65 @@ class HiveScriptTra

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602906474 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -528,4 +530,65 @@ class HiveScriptTra

[GitHub] [spark] AmplabJenkins commented on pull request #31967: [SPAKR-34819][SQL]MapType supports orderable semantics

AmplabJenkins commented on pull request #31967: URL: https://github.com/apache/spark/pull/31967#issuecomment-808928971 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136614/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31967: [SPAKR-34819][SQL]MapType supports orderable semantics

AmplabJenkins removed a comment on pull request #31967: URL: https://github.com/apache/spark/pull/31967#issuecomment-808928971 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136614/ -

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602907906 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -528,4 +530,65 @@ class HiveScriptTra

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602908226 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -528,4 +530,65 @@ class HiveScriptTra

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602907906 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -528,4 +530,65 @@ class HiveScriptTra

[GitHub] [spark] srowen commented on a change in pull request #31827: [SPARK-34492][DOCS] Add "CSV Files" page for Data Source documents.

srowen commented on a change in pull request #31827: URL: https://github.com/apache/spark/pull/31827#discussion_r602909491 ## File path: docs/sql-data-sources-csv.md ## @@ -0,0 +1,54 @@ +--- +layout: global +title: CSV Files +displayTitle: CSV Files +license: | + Licensed to t

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602909924 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ## @@ -346,6 +347,20 @@ private[hive] trait HiveInspectors {

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602910050 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ## @@ -512,6 +527,13 @@ private[hive] trait HiveInspectors {

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602910113 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ## @@ -512,6 +527,13 @@ private[hive] trait HiveInspectors {

[GitHub] [spark] srowen commented on pull request #31965: [SPARK-34843][SQL] Calculate more precise partition stride in JDBCRelation

srowen commented on pull request #31965: URL: https://github.com/apache/spark/pull/31965#issuecomment-808933065 Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [spark] srowen closed pull request #31965: [SPARK-34843][SQL] Calculate more precise partition stride in JDBCRelation

srowen closed pull request #31965: URL: https://github.com/apache/spark/pull/31965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please

[GitHub] [spark] MaxGekk commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

MaxGekk commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602910875 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ## @@ -1024,6 +1102,24 @@ private[hive] trait HiveInspectors {

[GitHub] [spark] SparkQA commented on pull request #31980: [SPARK-34807][SQL] Transpose Window nodes with Project between them

SparkQA commented on pull request #31980: URL: https://github.com/apache/spark/pull/31980#issuecomment-808942755 **[Test build #136616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136616/testReport)** for PR 31980 at commit [`8f15d15`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31980: [SPARK-34807][SQL] Transpose Window nodes with Project between them

SparkQA removed a comment on pull request #31980: URL: https://github.com/apache/spark/pull/31980#issuecomment-808905189 **[Test build #136616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136616/testReport)** for PR 31980 at commit [`8f15d15`](https://gi

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31958: [SPARK-34862][SQL] Support nested column in ORC vectorized reader

dongjoon-hyun commented on a change in pull request #31958: URL: https://github.com/apache/spark/pull/31958#discussion_r602918048 ## File path: project/MimaExcludes.scala ## @@ -417,6 +417,21 @@ object MimaExcludes { case _ => true }, +// [SPARK-34862][SQL] Su

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31958: [SPARK-34862][SQL] Support nested column in ORC vectorized reader

dongjoon-hyun commented on a change in pull request #31958: URL: https://github.com/apache/spark/pull/31958#discussion_r602918048 ## File path: project/MimaExcludes.scala ## @@ -417,6 +417,21 @@ object MimaExcludes { case _ => true }, +// [SPARK-34862][SQL] Su

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31958: [SPARK-34862][SQL] Support nested column in ORC vectorized reader

dongjoon-hyun commented on a change in pull request #31958: URL: https://github.com/apache/spark/pull/31958#discussion_r602918244 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -838,6 +838,13 @@ object SQLConf { .intConf .

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31958: [SPARK-34862][SQL] Support nested column in ORC vectorized reader

dongjoon-hyun commented on a change in pull request #31958: URL: https://github.com/apache/spark/pull/31958#discussion_r602918532 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala ## @@ -131,11 +131,27 @@ class OrcFileForm

[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them

SparkQA commented on pull request #31677: URL: https://github.com/apache/spark/pull/31677#issuecomment-808945108 **[Test build #136617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136617/testReport)** for PR 31677 at commit [`80c9a9a`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them

SparkQA removed a comment on pull request #31677: URL: https://github.com/apache/spark/pull/31677#issuecomment-808905230 **[Test build #136617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136617/testReport)** for PR 31677 at commit [`80c9a9a`](https://gi

[GitHub] [spark] dongjoon-hyun commented on pull request #31958: [SPARK-34862][SQL] Support nested column in ORC vectorized reader

dongjoon-hyun commented on pull request #31958: URL: https://github.com/apache/spark/pull/31958#issuecomment-808946403 Also, cc @viirya since this is related to the nested columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them

AmplabJenkins commented on pull request #31677: URL: https://github.com/apache/spark/pull/31677#issuecomment-808947114 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136617/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #31980: [SPARK-34807][SQL] Transpose Window nodes with Project between them

AmplabJenkins commented on pull request #31980: URL: https://github.com/apache/spark/pull/31980#issuecomment-808947115 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136616/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them

AmplabJenkins removed a comment on pull request #31677: URL: https://github.com/apache/spark/pull/31677#issuecomment-808947114 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136617/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31980: [SPARK-34807][SQL] Transpose Window nodes with Project between them

AmplabJenkins removed a comment on pull request #31980: URL: https://github.com/apache/spark/pull/31980#issuecomment-808947115 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136616/ -

[GitHub] [spark] tanelk opened a new pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

tanelk opened a new pull request #31983: URL: https://github.com/apache/spark/pull/31983 ### What changes were proposed in this pull request? Replaced the `agg(if (('gid = 1)) 'cat1 else null)` pattern in `RewriteDistinctAggregates` with `agg('cat1) FILTER (WHERE 'gid = 1)`

[GitHub] [spark] SparkQA commented on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

SparkQA commented on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808954952 **[Test build #136618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136618/testReport)** for PR 31983 at commit [`fb91ac0`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

SparkQA commented on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808960938 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41200/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

AmplabJenkins commented on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808962576 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41200/ -- T

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

AmplabJenkins removed a comment on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808962576 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41200/

[GitHub] [spark] SparkQA commented on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

SparkQA commented on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808970531 **[Test build #136618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136618/testReport)** for PR 31983 at commit [`fb91ac0`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

AmplabJenkins commented on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808970591 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136618/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

SparkQA removed a comment on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808954952 **[Test build #136618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136618/testReport)** for PR 31983 at commit [`fb91ac0`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31983: [SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates

AmplabJenkins removed a comment on pull request #31983: URL: https://github.com/apache/spark/pull/31983#issuecomment-808970591 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136618/ -

[GitHub] [spark] srowen commented on pull request #31942: [SPARK-34834][NETWORK] Fix a potential Netty memory leak in TransportResponseHandler.

srowen commented on pull request #31942: URL: https://github.com/apache/spark/pull/31942#issuecomment-808971475 OK, I think the outstanding question - and I don't know enough to feel strongly - is why not just always release the response in all cases and branches in this method? -- This

[GitHub] [spark] dongjoon-hyun commented on pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

dongjoon-hyun commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-808971916 @MaxGekk . Apache Spark master branch doesn't have Hive 1.2. Is there a reason to trigger `[test-hive1.2]`? > @AngersZh Could you add `[test-hive1.2]` to PR's title

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

dongjoon-hyun edited a comment on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-808971916 @MaxGekk . Apache Spark master branch doesn't have Hive 1.2 (or `hive-1.2` profile). Is there a reason to trigger `[test-hive1.2]`? > @AngersZh Could you add

[GitHub] [spark] venkata91 commented on a change in pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

venkata91 commented on a change in pull request #30480: URL: https://github.com/apache/spark/pull/30480#discussion_r602949966 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -449,21 +605,32 @@ private[spark] class MapOutputTrackerMaster( t

[GitHub] [spark] venkata91 commented on a change in pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

venkata91 commented on a change in pull request #30480: URL: https://github.com/apache/spark/pull/30480#discussion_r602949966 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -449,21 +605,32 @@ private[spark] class MapOutputTrackerMaster( t

[GitHub] [spark] maropu commented on pull request #31965: [SPARK-34843][SQL] Calculate more precise partition stride in JDBCRelation

maropu commented on pull request #31965: URL: https://github.com/apache/spark/pull/31965#issuecomment-808979165 late lgtm. thank you, @hanover-fiste -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] maropu commented on pull request #31955: [SPARK-34829][SQL] Fix higher order function results

maropu commented on pull request #31955: URL: https://github.com/apache/spark/pull/31955#issuecomment-808979325 late lgtm, thank you, @peter-toth and all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [spark] SparkQA commented on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

SparkQA commented on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-808979894 **[Test build #136619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136619/testReport)** for PR 30480 at commit [`db36880`](https://github.com

[GitHub] [spark] ron8hu commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

ron8hu commented on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-808980380 > FYI @ron8hu Have rebase to current master's code. > Ping @srowen Can you also take a look at this pr? @AngersZh Thanks Angers for updating the code. @srowen Th

[GitHub] [spark] hanover-fiste commented on pull request #31965: [SPARK-34843][SQL] Calculate more precise partition stride in JDBCRelation

hanover-fiste commented on pull request #31965: URL: https://github.com/apache/spark/pull/31965#issuecomment-808983344 @maropu, thanks again for the comments! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [spark] github-actions[bot] commented on pull request #30470: [SPARK-33495][BUILD] Remove commons-logging.jar's dependency

github-actions[bot] commented on pull request #30470: URL: https://github.com/apache/spark/pull/30470#issuecomment-808987647 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue ma

[GitHub] [spark] github-actions[bot] closed pull request #30225: [SPARK-33187][SQL] Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method

github-actions[bot] closed pull request #30225: URL: https://github.com/apache/spark/pull/30225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this ser

[GitHub] [spark] github-actions[bot] commented on pull request #30836: [SPARK-33791] Support hive legacy grouping id algorithm

github-actions[bot] commented on pull request #30836: URL: https://github.com/apache/spark/pull/30836#issuecomment-808987641 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue ma

[GitHub] [spark] SparkQA commented on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

SparkQA commented on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-808987744 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41201/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

AmplabJenkins commented on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-808991430 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41201/ -- T

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

AmplabJenkins removed a comment on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-808991430 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41201/

[GitHub] [spark] HeartSaVioR commented on pull request #31937: [SPARK-10816][SS] Support session window natively

HeartSaVioR commented on pull request #31937: URL: https://github.com/apache/spark/pull/31937#issuecomment-808997100 Let me go through filing PRs as there doesn't look to be further voice on the overall direction. -- This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] wangyum opened a new pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

wangyum opened a new pull request #31984: URL: https://github.com/apache/spark/pull/31984 ### What changes were proposed in this pull request? Improve dynamic partition pruning evaluation to make filtering side data size must smaller than `math.max(10MB, spark.sql.autoBroadcastJoinTh

[GitHub] [spark] SparkQA commented on pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

SparkQA commented on pull request #31984: URL: https://github.com/apache/spark/pull/31984#issuecomment-809007618 **[Test build #136620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136620/testReport)** for PR 31984 at commit [`b510d7d`](https://github.com

[GitHub] [spark] Ngone51 commented on a change in pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

Ngone51 commented on a change in pull request #30480: URL: https://github.com/apache/spark/pull/30480#discussion_r602970912 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -833,33 +1106,44 @@ private[spark] class MapOutputTrackerWorker(conf: Spa

[GitHub] [spark] maropu commented on a change in pull request #31966: [SPARK-34638][SQL] Single field nested column prune on generator output

maropu commented on a change in pull request #31966: URL: https://github.com/apache/spark/pull/31966#discussion_r602970835 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala ## @@ -241,12 +262,69 @@ object GeneratorNest

[GitHub] [spark] mridulm commented on a change in pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

mridulm commented on a change in pull request #30480: URL: https://github.com/apache/spark/pull/30480#discussion_r602972892 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -449,21 +605,32 @@ private[spark] class MapOutputTrackerMaster( try

[GitHub] [spark] SparkQA commented on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

SparkQA commented on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-809014476 **[Test build #136619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136619/testReport)** for PR 30480 at commit [`db36880`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

SparkQA removed a comment on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-808979894 **[Test build #136619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136619/testReport)** for PR 30480 at commit [`db36880`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

SparkQA commented on pull request #31984: URL: https://github.com/apache/spark/pull/31984#issuecomment-809019244 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41202/ -- This is an automated message from the Apache

[GitHub] [spark] AngersZhuuuu commented on pull request #31179: [SPARK-34113][SQL] Use metric data update metadata statistic's size and rowCount

AngersZh commented on pull request #31179: URL: https://github.com/apache/spark/pull/31179#issuecomment-809020307 Any more suggestion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[GitHub] [spark] AngersZhuuuu commented on pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

AngersZh commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809020632 > @MaxGekk . Apache Spark master branch doesn't have Hive 1.2 (or `hive-1.2` profile). Is there a reason to trigger `[test-hive1.2]`? > > > @AngersZh Could you ad

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31827: [SPARK-34492][DOCS] Add "CSV Files" page for Data Source documents.

HyukjinKwon commented on a change in pull request #31827: URL: https://github.com/apache/spark/pull/31827#discussion_r602979088 ## File path: docs/sql-data-sources-csv.md ## @@ -0,0 +1,54 @@ +--- +layout: global +title: CSV Files +displayTitle: CSV Files +license: | + Licensed

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31979: [SPARK-34879][SQL][test-hive1.2] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

AngersZh commented on a change in pull request #31979: URL: https://github.com/apache/spark/pull/31979#discussion_r602980428 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -528,4 +530,65 @@ class HiveScri

[GitHub] [spark] AmplabJenkins commented on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

AmplabJenkins commented on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-809023952 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136619/ -- This

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30480: [SPARK-32921][SHUFFLE] MapOutputTracker extensions to support push-based shuffle

AmplabJenkins removed a comment on pull request #30480: URL: https://github.com/apache/spark/pull/30480#issuecomment-809023952 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136619/ -

[GitHub] [spark] maropu commented on pull request #31967: [SPARK-34819][SQL]MapType supports orderable semantics

maropu commented on pull request #31967: URL: https://github.com/apache/spark/pull/31967#issuecomment-809024386 Link: #15970 and #19330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [spark] maropu edited a comment on pull request #31967: [SPARK-34819][SQL]MapType supports orderable semantics

maropu edited a comment on pull request #31967: URL: https://github.com/apache/spark/pull/31967#issuecomment-809024386 Links to the previous PRs: #15970 and #19330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [spark] SparkQA commented on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

SparkQA commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809024472 **[Test build #136621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136621/testReport)** for PR 31979 at commit [`4e88bdf`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

SparkQA commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809025371 **[Test build #136622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136622/testReport)** for PR 31979 at commit [`796f1f4`](https://github.com

[GitHub] [spark] maropu commented on pull request #31899: [SPARK-34525][DOCS] Update Spark Create Table DDL and other documentation to reflect alternative key value notation

maropu commented on pull request #31899: URL: https://github.com/apache/spark/pull/31899#issuecomment-809025601 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For qu

[GitHub] [spark] maropu commented on pull request #31899: [SPARK-34525][SQL][DOCS] Update Spark Create Table DDL and other documentation to reflect alternative key value notation

maropu commented on pull request #31899: URL: https://github.com/apache/spark/pull/31899#issuecomment-809026184 Ah, could you update the PR description, too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[GitHub] [spark] AmplabJenkins commented on pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

AmplabJenkins commented on pull request #31984: URL: https://github.com/apache/spark/pull/31984#issuecomment-809027550 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41202/ -- T

[GitHub] [spark] SparkQA commented on pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

SparkQA commented on pull request #31984: URL: https://github.com/apache/spark/pull/31984#issuecomment-809027540 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41202/ -- This is an automated message from the A

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

AmplabJenkins removed a comment on pull request #31984: URL: https://github.com/apache/spark/pull/31984#issuecomment-809027550 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41202/

[GitHub] [spark] HyukjinKwon commented on pull request #31973: [SPARK-34876][SQL] Fill defaultResult of non-nullable aggregates

HyukjinKwon commented on pull request #31973: URL: https://github.com/apache/spark/pull/31973#issuecomment-809028090 Merged to master, branch-3.1, branch-3.0 and branch-2.4 cc @cloud-fan, @maryannxue, @viirya FYI -- This is an automated message from the Apache Git Service. To respo

[GitHub] [spark] AngersZhuuuu commented on pull request #30212: [SPARK-33308][SQL] Refactor current grouping analytics

AngersZh commented on pull request #30212: URL: https://github.com/apache/spark/pull/30212#issuecomment-809028142 Gentle ping @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon closed pull request #31973: [SPARK-34876][SQL] Fill defaultResult of non-nullable aggregates

HyukjinKwon closed pull request #31973: URL: https://github.com/apache/spark/pull/31973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, pl

[GitHub] [spark] HyukjinKwon commented on pull request #31976: [SPARK-34814][SQL] LikeSimplification should handle NULL

HyukjinKwon commented on pull request #31976: URL: https://github.com/apache/spark/pull/31976#issuecomment-809032964 Merged to master and branch-3.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon closed pull request #31976: [SPARK-34814][SQL] LikeSimplification should handle NULL

HyukjinKwon closed pull request #31976: URL: https://github.com/apache/spark/pull/31976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, pl

[GitHub] [spark] HyukjinKwon commented on pull request #31976: [SPARK-34814][SQL] LikeSimplification should handle NULL

HyukjinKwon commented on pull request #31976: URL: https://github.com/apache/spark/pull/31976#issuecomment-809033091 cc @beliefer too FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[GitHub] [spark] zhengruifeng commented on pull request #31693: [SPARK-34858][SPARK-34448][ML] Binary Logistic Regression with intercept support centering

zhengruifeng commented on pull request #31693: URL: https://github.com/apache/spark/pull/31693#issuecomment-809034717 @srowen Thanks for reviewing and merging! I will send another PR for multinominal LR. -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [spark] Ngone51 commented on pull request #31942: [SPARK-34834][NETWORK] Fix a potential Netty memory leak in TransportResponseHandler.

Ngone51 commented on pull request #31942: URL: https://github.com/apache/spark/pull/31942#issuecomment-809035511 I'm also confused with this part. I don't even see a place where the `resp.body()` (a.k.a `ManagedBuffer`) is referenced before the `TransportResponseHandler` handle the `Respon

[GitHub] [spark] SparkQA commented on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

SparkQA commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809037424 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41203/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

SparkQA commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809037801 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41204/ -- This

[GitHub] [spark] HyukjinKwon commented on pull request #31859: [SPARK-34769][SQL]AnsiTypeCoercion: return closest convertible type among TypeCollection

HyukjinKwon commented on pull request #31859: URL: https://github.com/apache/spark/pull/31859#issuecomment-809038253 I just found out that I mistakenly assigned it to myself .. I removed it back now .. -- This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] SparkQA commented on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

SparkQA commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809039750 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41203/ -- This is an automated message from the A

[GitHub] [spark] HyukjinKwon commented on pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

HyukjinKwon commented on pull request #31984: URL: https://github.com/apache/spark/pull/31984#issuecomment-809040667 cc @maryannxue FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

[GitHub] [spark] AmplabJenkins commented on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

AmplabJenkins commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809041420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For q

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

AmplabJenkins removed a comment on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809041420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [spark] SparkQA commented on pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

SparkQA commented on pull request #31984: URL: https://github.com/apache/spark/pull/31984#issuecomment-809041800 **[Test build #136623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136623/testReport)** for PR 31984 at commit [`68ddc7a`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31979: [SPARK-34879][SQL] HiveInspector support DayTimeIntervalType and YearMonthIntervalType

SparkQA commented on pull request #31979: URL: https://github.com/apache/spark/pull/31979#issuecomment-809041822 **[Test build #136624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136624/testReport)** for PR 31979 at commit [`b368584`](https://github.com

[GitHub] [spark] zhengruifeng opened a new pull request #31985: [SPARK-34860][ML] Multinomial Logistic Regression with intercept support centering

zhengruifeng opened a new pull request #31985: URL: https://github.com/apache/spark/pull/31985 ### What changes were proposed in this pull request? 1, use new `MultinomialLogisticBlockAggregator` which support virtual centering 2, remove no-used `BlockLogisticAggregator` ##

<    1   2   3   4   >