[GitHub] [spark] dongjoon-hyun closed pull request #32284: [SPARK-35096][SQL][3.0] SchemaPruning should adhere spark.sql.caseSensitive config

dongjoon-hyun closed pull request #32284: URL: https://github.com/apache/spark/pull/32284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] SparkQA removed a comment on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished

SparkQA removed a comment on pull request #32283: URL: https://github.com/apache/spark/pull/32283#issuecomment-824505223 **[Test build #137764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137764/testReport)** for PR 32283 at commit

[GitHub] [spark] sigmod commented on a change in pull request #32247: [SPARK-35075][SQL] Add traversal pruning for subquery related rules

sigmod commented on a change in pull request #32247: URL: https://github.com/apache/spark/pull/32247#discussion_r618101375 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3790,9 +3793,9 @@ object UpdateOuterReferences

[GitHub] [spark] SparkQA commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished

SparkQA commented on pull request #32283: URL: https://github.com/apache/spark/pull/32283#issuecomment-824561915 **[Test build #137764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137764/testReport)** for PR 32283 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #32284: [SPARK-35096][SQL][3.0] SchemaPruning should adhere spark.sql.caseSensitive config

dongjoon-hyun commented on pull request #32284: URL: https://github.com/apache/spark/pull/32284#issuecomment-824561792 Merged to branch-3.0. Thank you, @sandeep-katta and all! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] sigmod commented on a change in pull request #32247: [SPARK-35075][SQL] Add traversal pruning for subquery related rules

sigmod commented on a change in pull request #32247: URL: https://github.com/apache/spark/pull/32247#discussion_r618100924 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -40,7 +40,8 @@ import

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32287: [SPARK-27991][CORE] Defer the fetch request on Netty OOM

dongjoon-hyun commented on a change in pull request #32287: URL: https://github.com/apache/spark/pull/32287#discussion_r618100388 ## File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyOutOfMemoryError.java ## @@ -0,0 +1,26 @@ +/* + * Licensed to

[GitHub] [spark] MaxGekk commented on a change in pull request #32281: [SPARK-35177][SQL] Fix arithmetic overflow in parsing the minimal interval by `IntervalUtils.fromYearMonthString`

MaxGekk commented on a change in pull request #32281: URL: https://github.com/apache/spark/pull/32281#discussion_r618100249 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala ## @@ -169,6 +169,19 @@ class IntervalUtilsSuite

[GitHub] [spark] mridulm commented on pull request #32287: [SPARK-27991][CORE] Defer the fetch request on Netty OOM

mridulm commented on pull request #32287: URL: https://github.com/apache/spark/pull/32287#issuecomment-824560878 +CC @otterc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] MaxGekk commented on a change in pull request #32281: [SPARK-35177][SQL] Fix arithmetic overflow in parsing the minimal interval by `IntervalUtils.fromYearMonthString`

MaxGekk commented on a change in pull request #32281: URL: https://github.com/apache/spark/pull/32281#discussion_r618099976 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala ## @@ -169,6 +169,19 @@ class IntervalUtilsSuite

[GitHub] [spark] dongjoon-hyun closed pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished

dongjoon-hyun closed pull request #32283: URL: https://github.com/apache/spark/pull/32283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,

[GitHub] [spark] dongjoon-hyun commented on pull request #32283: [SPARK-34674][CORE][K8S] Close SparkContext after the Main method has finished

dongjoon-hyun commented on pull request #32283: URL: https://github.com/apache/spark/pull/32283#issuecomment-824559969 Jenkins also passed Java/Scala/Python tests and currently running irrelevant R test. Merged to master/3.1. Thank you for your patience and persistence, @kotlovs ! --

[GitHub] [spark] AngersZhuuuu commented on pull request #32281: [SPARK-35177][SQL] Fix arithmetic overflow in parsing the minimal interval by `IntervalUtils.fromYearMonthString`

AngersZh commented on pull request #32281: URL: https://github.com/apache/spark/pull/32281#issuecomment-824560009 > Could you make PR's title more precise, for example: > [[SPARK-35177](https://issues.apache.org/jira/browse/SPARK-35177)][SQL] Fix arithmetic overflow in parsing the

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

dongjoon-hyun edited a comment on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824557656 Ya, no worry. This is a beginning. We have only first successful result on the default configuration. As I wrote before

[GitHub] [spark] dongjoon-hyun commented on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

dongjoon-hyun commented on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824558464 When the last commit aa507a4 passed again. I will rename the PR title and run the other combinations. All checkbox are resolved, we can go. -- This is an automated

[GitHub] [spark] Ngone51 commented on pull request #32287: [SPARK-27991][CORE] Defer the fetch request on Netty OOM

Ngone51 commented on pull request #32287: URL: https://github.com/apache/spark/pull/32287#issuecomment-824558087 cc @mridulm @tgravescs @attilapiros could you take a look? thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

HyukjinKwon commented on a change in pull request #32285: URL: https://github.com/apache/spark/pull/32285#discussion_r618097706 ## File path: project/SparkBuild.scala ## @@ -888,6 +892,22 @@ object PySparkAssembly { } +object SparkR { + import scala.sys.process.Process +

[GitHub] [spark] dongjoon-hyun commented on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

dongjoon-hyun commented on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824557656 Ya, no worry. This is a beginning. We have only first successful result on the default configuration. As I wrote before

[GitHub] [spark] Ngone51 opened a new pull request #32287: [SPARK-27991][CORE] Defer the fetch request on Netty OOM

Ngone51 opened a new pull request #32287: URL: https://github.com/apache/spark/pull/32287 ### What changes were proposed in this pull request? This PR proposes a workaround to address the Netty OOM issue (SPARK-24989, SPARK-27991): Basically,

[GitHub] [spark] hiboyang commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

hiboyang commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618097182 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -162,7 +162,7 @@ private class ShuffleStatus(numPartitions: Int) extends

[GitHub] [spark] sarutak commented on a change in pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

sarutak commented on a change in pull request #32285: URL: https://github.com/apache/spark/pull/32285#discussion_r618096798 ## File path: R/README.md ## @@ -17,10 +17,14 @@ export R_HOME=/home/username/R Build Spark -Build Spark with

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

dongjoon-hyun commented on a change in pull request #32285: URL: https://github.com/apache/spark/pull/32285#discussion_r618096280 ## File path: R/README.md ## @@ -17,10 +17,14 @@ export R_HOME=/home/username/R Build Spark -Build Spark with

[GitHub] [spark] andersonm-ibm commented on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

andersonm-ibm commented on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824555329 Thank you, @dongjoon-hyun , for for merging the PRs to unblock this and for following all the changes. I'm afraid Jenkins

[GitHub] [spark] dongjoon-hyun commented on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

dongjoon-hyun commented on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824552081 Thank you for rebasing, @andersonm-ibm . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] c21 commented on pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

c21 commented on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-824550511 Rebased to latest master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA removed a comment on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

SparkQA removed a comment on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824485035 **[Test build #137763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137763/testReport)** for PR 32146 at commit

[GitHub] [spark] c21 commented on pull request #32210: [SPARK-32634][SQL] Introduce sort-based fallback for shuffled hash join (non-code-gen path)

c21 commented on pull request #32210: URL: https://github.com/apache/spark/pull/32210#issuecomment-824550054 Rebased to latest master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32201: [SPARK-35026][SQL] Support nested CUBE/ROLLUP/GROUPING SETS in GROUPING SETS

AngersZh commented on a change in pull request #32201: URL: https://github.com/apache/spark/pull/32201#discussion_r618090727 ## File path: docs/sql-ref-syntax-qry-select-groupby.md ## @@ -113,7 +117,10 @@ aggregate_name ( [ DISTINCT ] expression [ , ... ] ) [ FILTER (

[GitHub] [spark] SparkQA commented on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

SparkQA commented on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824549705 **[Test build #137763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137763/testReport)** for PR 32146 at commit

[GitHub] [spark] c21 commented on a change in pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

c21 commented on a change in pull request #32242: URL: https://github.com/apache/spark/pull/32242#discussion_r618090155 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala ## @@ -128,6 +128,16 @@ case class HashAggregateExec(

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32201: [SPARK-35026][SQL] Support nested CUBE/ROLLUP/GROUPING SETS in GROUPING SETS

AngersZh commented on a change in pull request #32201: URL: https://github.com/apache/spark/pull/32201#discussion_r618089426 ## File path: docs/sql-ref-syntax-qry-select-groupby.md ## @@ -95,13 +95,17 @@ aggregate_name ( [ DISTINCT ] expression [ , ... ] ) [ FILTER (

[GitHub] [spark] dongjoon-hyun opened a new pull request #32286: [SPARK-35181][CORE] Use zstd for spark.io.compression.codec by default

dongjoon-hyun opened a new pull request #32286: URL: https://github.com/apache/spark/pull/32286 ### What changes were proposed in this pull request? This PR aims to use `zstd` as `spark.io.compression.codec` instead of `lz4` in order to reduce the disk IOs and traffic during shuffle

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32284: [SPARK-35096][SQL][3.0] SchemaPruning should adhere spark.sql.caseSensitive config

AmplabJenkins removed a comment on pull request #32284: URL: https://github.com/apache/spark/pull/32284#issuecomment-824547377 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42298/

[GitHub] [spark] SparkQA commented on pull request #32284: [SPARK-35096][SQL][3.0] SchemaPruning should adhere spark.sql.caseSensitive config

SparkQA commented on pull request #32284: URL: https://github.com/apache/spark/pull/32284#issuecomment-824547358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #32284: [SPARK-35096][SQL][3.0] SchemaPruning should adhere spark.sql.caseSensitive config

AmplabJenkins commented on pull request #32284: URL: https://github.com/apache/spark/pull/32284#issuecomment-824547377 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42298/ --

[GitHub] [spark] SparkQA commented on pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

SparkQA commented on pull request #32242: URL: https://github.com/apache/spark/pull/32242#issuecomment-824546920 **[Test build #137775 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137775/testReport)** for PR 32242 at commit

[GitHub] [spark] c21 commented on a change in pull request #32242: [SPARK-35141][SQL] Support two level of hash maps for final hash aggregation

c21 commented on a change in pull request #32242: URL: https://github.com/apache/spark/pull/32242#discussion_r618087306 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala ## @@ -128,6 +128,16 @@ case class HashAggregateExec(

[GitHub] [spark] dongjoon-hyun commented on pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

dongjoon-hyun commented on pull request #32285: URL: https://github.com/apache/spark/pull/32285#issuecomment-824542767 Got it! > mvn -Psparkr runs R/install-dev.sh automatically but SBT didn't. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32167: [SPARK-35067][TESTS]Add test case for function COALESCE().

dongjoon-hyun commented on a change in pull request #32167: URL: https://github.com/apache/spark/pull/32167#discussion_r618083727 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -404,6 +404,9 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] SparkQA commented on pull request #32146: [SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite

SparkQA commented on pull request #32146: URL: https://github.com/apache/spark/pull/32146#issuecomment-824542440 **[Test build #137774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137774/testReport)** for PR 32146 at commit

[GitHub] [spark] SparkQA commented on pull request #32210: [SPARK-32634][SQL] Introduce sort-based fallback for shuffled hash join (non-code-gen path)

SparkQA commented on pull request #32210: URL: https://github.com/apache/spark/pull/32210#issuecomment-824542377 **[Test build #137773 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137773/testReport)** for PR 32210 at commit

[GitHub] [spark] SparkQA commented on pull request #32284: [SPARK-35096][SQL][3.0] SchemaPruning should adhere spark.sql.caseSensitive config

SparkQA commented on pull request #32284: URL: https://github.com/apache/spark/pull/32284#issuecomment-824542328 **[Test build #137772 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137772/testReport)** for PR 32284 at commit

[GitHub] [spark] SparkQA commented on pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

SparkQA commented on pull request #32285: URL: https://github.com/apache/spark/pull/32285#issuecomment-824542276 **[Test build #137771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137771/testReport)** for PR 32285 at commit

[GitHub] [spark] sarutak commented on pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

sarutak commented on pull request #32285: URL: https://github.com/apache/spark/pull/32285#issuecomment-824541998 > It's great. Could you add the build command into R/README.md along with the existing mvn command, @sarutak ? O.K, I'll do it. > BTW, I can do the AS-IS master

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

AmplabJenkins removed a comment on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-824541701 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42295/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32201: [SPARK-35026][SQL] Support nested CUBE/ROLLUP/GROUPING SETS in GROUPING SETS

AmplabJenkins removed a comment on pull request #32201: URL: https://github.com/apache/spark/pull/32201#issuecomment-824541685 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42297/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32274: [SPARK-35175][BUILD] Add linter for JavaScript source files.

AmplabJenkins removed a comment on pull request #32274: URL: https://github.com/apache/spark/pull/32274#issuecomment-824541702 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137762/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32278: [SPARK-34692][SQL][FOLLOWUP] Add INSET to ReplaceNullWithFalseInPredicate's pattern

AmplabJenkins removed a comment on pull request #32278: URL: https://github.com/apache/spark/pull/32278#issuecomment-824541698 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137755/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32281: [SPARK-35177][SQL] IntervalUtils.fromYearMonthString should handle Int.MinValue months correctly

AmplabJenkins removed a comment on pull request #32281: URL: https://github.com/apache/spark/pull/32281#issuecomment-824541704 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42293/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32280: [WIP][SPARK-35078] Add tree traversal pruning in expression rules

AmplabJenkins removed a comment on pull request #32280: URL: https://github.com/apache/spark/pull/32280#issuecomment-824541697 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137757/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32279: [SPARK-34897][SQL][3.1] Support reconcile schemas based on index after nested column pruning

AmplabJenkins removed a comment on pull request #32279: URL: https://github.com/apache/spark/pull/32279#issuecomment-824541700 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137754/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32282: [SPARK-35178][BUILD][FOLLOWUP][3.1] Update Zinc argument

AmplabJenkins removed a comment on pull request #32282: URL: https://github.com/apache/spark/pull/32282#issuecomment-824541699 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137761/

[GitHub] [spark] AmplabJenkins commented on pull request #32280: [WIP][SPARK-35078] Add tree traversal pruning in expression rules

AmplabJenkins commented on pull request #32280: URL: https://github.com/apache/spark/pull/32280#issuecomment-824541697 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137757/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32282: [SPARK-35178][BUILD][FOLLOWUP][3.1] Update Zinc argument

AmplabJenkins commented on pull request #32282: URL: https://github.com/apache/spark/pull/32282#issuecomment-824541699 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137761/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32278: [SPARK-34692][SQL][FOLLOWUP] Add INSET to ReplaceNullWithFalseInPredicate's pattern

AmplabJenkins commented on pull request #32278: URL: https://github.com/apache/spark/pull/32278#issuecomment-824541698 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137755/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32281: [SPARK-35177][SQL] IntervalUtils.fromYearMonthString should handle Int.MinValue months correctly

AmplabJenkins commented on pull request #32281: URL: https://github.com/apache/spark/pull/32281#issuecomment-824541704 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42293/ --

[GitHub] [spark] AmplabJenkins commented on pull request #32274: [SPARK-35175][BUILD] Add linter for JavaScript source files.

AmplabJenkins commented on pull request #32274: URL: https://github.com/apache/spark/pull/32274#issuecomment-824541702 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137762/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32279: [SPARK-34897][SQL][3.1] Support reconcile schemas based on index after nested column pruning

AmplabJenkins commented on pull request #32279: URL: https://github.com/apache/spark/pull/32279#issuecomment-824541700 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137754/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

AmplabJenkins commented on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-824541701 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42295/ --

[GitHub] [spark] SparkQA commented on pull request #32201: [SPARK-35026][SQL] Support nested CUBE/ROLLUP/GROUPING SETS in GROUPING SETS

SparkQA commented on pull request #32201: URL: https://github.com/apache/spark/pull/32201#issuecomment-824541671 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] AmplabJenkins commented on pull request #32201: [SPARK-35026][SQL] Support nested CUBE/ROLLUP/GROUPING SETS in GROUPING SETS

AmplabJenkins commented on pull request #32201: URL: https://github.com/apache/spark/pull/32201#issuecomment-824541685 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42297/ --

[GitHub] [spark] dongjoon-hyun commented on pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

dongjoon-hyun commented on pull request #32285: URL: https://github.com/apache/spark/pull/32285#issuecomment-824540353 BTW, I can do the AS-IS master branch, what is the missing before this PR? ``` $ build/sbt package -Psparkr $ R/install-dev.sh $ bin/sparkR R version

[GitHub] [spark] LuciferYang commented on a change in pull request #32232: [SPARK-35135][CORE] Extract a class implementation of `WritablePartitionedIterator` to remove duplicate code

LuciferYang commented on a change in pull request #32232: URL: https://github.com/apache/spark/pull/32232#discussion_r618069694 ## File path: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala ## @@ -94,3 +83,20 @@ private[spark]

[GitHub] [spark] SparkQA removed a comment on pull request #32280: [WIP][SPARK-35078] Add tree traversal pruning in expression rules

SparkQA removed a comment on pull request #32280: URL: https://github.com/apache/spark/pull/32280#issuecomment-824463605 **[Test build #137757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137757/testReport)** for PR 32280 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #32274: [SPARK-35175][BUILD] Add linter for JavaScript source files.

SparkQA removed a comment on pull request #32274: URL: https://github.com/apache/spark/pull/32274#issuecomment-824484952 **[Test build #137762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137762/testReport)** for PR 32274 at commit

[GitHub] [spark] SparkQA commented on pull request #32280: [WIP][SPARK-35078] Add tree traversal pruning in expression rules

SparkQA commented on pull request #32280: URL: https://github.com/apache/spark/pull/32280#issuecomment-824538403 **[Test build #137757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137757/testReport)** for PR 32280 at commit

[GitHub] [spark] SparkQA commented on pull request #32274: [SPARK-35175][BUILD] Add linter for JavaScript source files.

SparkQA commented on pull request #32274: URL: https://github.com/apache/spark/pull/32274#issuecomment-824538358 **[Test build #137762 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137762/testReport)** for PR 32274 at commit

[GitHub] [spark] gengliangwang commented on pull request #32247: [SPARK-35075][SQL] Add traversal pruning for subquery related rules

gengliangwang commented on pull request #32247: URL: https://github.com/apache/spark/pull/32247#issuecomment-824537559 @sigmod sorry for the late review. Could you rebase the code branch? -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] gengliangwang commented on a change in pull request #32247: [SPARK-35075][SQL] Add traversal pruning for subquery related rules

gengliangwang commented on a change in pull request #32247: URL: https://github.com/apache/spark/pull/32247#discussion_r618079071 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3790,9 +3793,9 @@ object

[GitHub] [spark] SparkQA removed a comment on pull request #32282: [SPARK-35178][BUILD][FOLLOWUP][3.1] Update Zinc argument

SparkQA removed a comment on pull request #32282: URL: https://github.com/apache/spark/pull/32282#issuecomment-824484971 **[Test build #137761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137761/testReport)** for PR 32282 at commit

[GitHub] [spark] gengliangwang commented on a change in pull request #32247: [SPARK-35075][SQL] Add traversal pruning for subquery related rules

gengliangwang commented on a change in pull request #32247: URL: https://github.com/apache/spark/pull/32247#discussion_r618078613 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -40,7 +40,8 @@ import

[GitHub] [spark] SparkQA commented on pull request #32282: [SPARK-35178][BUILD][FOLLOWUP][3.1] Update Zinc argument

SparkQA commented on pull request #32282: URL: https://github.com/apache/spark/pull/32282#issuecomment-824536787 **[Test build #137761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137761/testReport)** for PR 32282 at commit

[GitHub] [spark] c21 commented on pull request #32210: [SPARK-32634][SQL] Introduce sort-based fallback for shuffled hash join (non-code-gen path)

c21 commented on pull request #32210: URL: https://github.com/apache/spark/pull/32210#issuecomment-824536722 Sorry I haven't to rebase to latest master with fix yet, let me rebase in a moment. -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] dongjoon-hyun commented on pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

dongjoon-hyun commented on pull request #32285: URL: https://github.com/apache/spark/pull/32285#issuecomment-824536743 cc @srowen and @HyukjinKwon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #32210: [SPARK-32634][SQL] Introduce sort-based fallback for shuffled hash join (non-code-gen path)

cloud-fan commented on pull request #32210: URL: https://github.com/apache/spark/pull/32210#issuecomment-824536267 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] SparkQA removed a comment on pull request #32279: [SPARK-34897][SQL][3.1] Support reconcile schemas based on index after nested column pruning

SparkQA removed a comment on pull request #32279: URL: https://github.com/apache/spark/pull/32279#issuecomment-824443676 **[Test build #137754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137754/testReport)** for PR 32279 at commit

[GitHub] [spark] SparkQA commented on pull request #32279: [SPARK-34897][SQL][3.1] Support reconcile schemas based on index after nested column pruning

SparkQA commented on pull request #32279: URL: https://github.com/apache/spark/pull/32279#issuecomment-824535933 **[Test build #137754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137754/testReport)** for PR 32279 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618077262 ## File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ## @@ -439,7 +439,14 @@ private[spark] object JsonProtocol { val

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618077262 ## File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ## @@ -439,7 +439,14 @@ private[spark] object JsonProtocol { val

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618077262 ## File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ## @@ -439,7 +439,14 @@ private[spark] object JsonProtocol { val

[GitHub] [spark] SparkQA commented on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

SparkQA commented on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-824535083 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42295/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618076564 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -297,22 +298,25 @@ final class

[GitHub] [spark] viirya commented on a change in pull request #32272: [SPARK-35172][SS] The implementation of RocksDBCheckpointMetadata

viirya commented on a change in pull request #32272: URL: https://github.com/apache/spark/pull/32272#discussion_r618076328 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala ## @@ -0,0 +1,165 @@ +/* + * Licensed to

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618076389 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -297,22 +298,25 @@ final class

[GitHub] [spark] viirya commented on a change in pull request #32272: [SPARK-35172][SS] The implementation of RocksDBCheckpointMetadata

viirya commented on a change in pull request #32272: URL: https://github.com/apache/spark/pull/32272#discussion_r618075956 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala ## @@ -0,0 +1,165 @@ +/* + * Licensed to

[GitHub] [spark] viirya commented on a change in pull request #32272: [SPARK-35172][SS] The implementation of RocksDBCheckpointMetadata

viirya commented on a change in pull request #32272: URL: https://github.com/apache/spark/pull/32272#discussion_r618075800 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala ## @@ -0,0 +1,165 @@ +/* + * Licensed to

[GitHub] [spark] SparkQA commented on pull request #32177: [SPARK-34999][PYTHON] Consolidate PySpark testing utils

SparkQA commented on pull request #32177: URL: https://github.com/apache/spark/pull/32177#issuecomment-824533317 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42295/ -- This is an automated message from the Apache

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618075192 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -297,22 +298,25 @@ final class

[GitHub] [spark] viirya commented on a change in pull request #32272: [SPARK-35172][SS] The implementation of RocksDBCheckpointMetadata

viirya commented on a change in pull request #32272: URL: https://github.com/apache/spark/pull/32272#discussion_r618074831 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala ## @@ -0,0 +1,165 @@ +/* + * Licensed to

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618074826 ## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ## @@ -235,8 +236,9 @@ final class

[GitHub] [spark] SparkQA removed a comment on pull request #32278: [SPARK-34692][SQL][FOLLOWUP] Add INSET to ReplaceNullWithFalseInPredicate's pattern

SparkQA removed a comment on pull request #32278: URL: https://github.com/apache/spark/pull/32278#issuecomment-824443704 **[Test build #137755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137755/testReport)** for PR 32278 at commit

[GitHub] [spark] SparkQA commented on pull request #32278: [SPARK-34692][SQL][FOLLOWUP] Add INSET to ReplaceNullWithFalseInPredicate's pattern

SparkQA commented on pull request #32278: URL: https://github.com/apache/spark/pull/32278#issuecomment-824532620 **[Test build #137755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137755/testReport)** for PR 32278 at commit

[GitHub] [spark] SparkQA commented on pull request #32281: [SPARK-35177][SQL] IntervalUtils.fromYearMonthString should handle Int.MinValue months correctly

SparkQA commented on pull request #32281: URL: https://github.com/apache/spark/pull/32281#issuecomment-824531666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For

[GitHub] [spark] c21 commented on pull request #32210: [SPARK-32634][SQL] Introduce sort-based fallback for shuffled hash join (non-code-gen path)

c21 commented on pull request #32210: URL: https://github.com/apache/spark/pull/32210#issuecomment-824531538 The unit test failure is org.apache.spark.sql.catalyst.optimizer.ReplaceNullWithFalseInPredicateSuite, which is not related to this PR. #32157 introduced the failure and #32278

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618073443 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ## @@ -866,8 +867,11 @@ private[spark] class TaskSetManager(

[GitHub] [spark] c21 commented on pull request #32198: [SPARK-26164][SQL] Allow concurrent writers for writing dynamic partitions and bucket table

c21 commented on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-824531319 The unit test failure is `org.apache.spark.sql.catalyst.optimizer.ReplaceNullWithFalseInPredicateSuite`, which is not related to this PR. https://github.com/apache/spark/pull/32157

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618066683 ## File path: core/src/main/java/org/apache/spark/shuffle/api/Location.java ## @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31876: [SPARK-34942][API][CORE] Abstract Location in MapStatus to enable support for custom storage

dongjoon-hyun commented on a change in pull request #31876: URL: https://github.com/apache/spark/pull/31876#discussion_r618072319 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -52,8 +55,41 @@ private[spark] sealed trait MapStatus { *

[GitHub] [spark] sarutak opened a new pull request #32285: [SPARK-35180][BUILD] Allow to build SparkR with SBT

sarutak opened a new pull request #32285: URL: https://github.com/apache/spark/pull/32285 ### What changes were proposed in this pull request? This PR proposes a change that allows us to build SparkR with SBT. ### Why are the changes needed? In the current master,

[GitHub] [spark] cloud-fan commented on pull request #32284: [SPARK-35096][SQL][3.0] SchemaPruning should adhere spark.sql.caseSensitive config

cloud-fan commented on pull request #32284: URL: https://github.com/apache/spark/pull/32284#issuecomment-824530338 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

  1   2   3   4   5   6   7   8   >