[spark] branch master updated (ad02ced -> e3a768d)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table add e3a768d [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (1aa8f4f -> b905d65)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20 add b905d65 [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ad02ced -> e3a768d)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table add e3a768d [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (1aa8f4f -> b905d65)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20 add b905d65 [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ad02ced -> e3a768d)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table add e3a768d [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (1aa8f4f -> b905d65)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20 add b905d65 [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ad02ced -> e3a768d)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table add e3a768d [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (1aa8f4f -> b905d65)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20 add b905d65 [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ad02ced -> e3a768d)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table add e3a768d [SPARK-33391][SQL] element_at with CreateArray not respect one based index No new revisions were added by this update. Summary of changes: .../expressions/collectionOperations.scala | 30 + .../expressions/CollectionExpressionsSuite.scala | 38 +- 2 files changed, 60 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (90f6f39 -> ad02ced)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier add ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/DataFrameReader.scala| 12 ++-- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 11 +-- 2 files changed, 15 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (90f6f39 -> ad02ced)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier add ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/DataFrameReader.scala| 12 ++-- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 11 +-- 2 files changed, 15 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (90f6f39 -> ad02ced)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier add ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/DataFrameReader.scala| 12 ++-- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 11 +-- 2 files changed, 15 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (90f6f39 -> ad02ced)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier add ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/DataFrameReader.scala| 12 ++-- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 11 +-- 2 files changed, 15 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (90f6f39 -> ad02ced)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier add ad02ced [SPARK-33244][SQL] Unify the code paths for spark.table and spark.read.table No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/DataFrameReader.scala| 12 ++-- .../src/main/scala/org/apache/spark/sql/SparkSession.scala | 11 +-- 2 files changed, 15 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a1f84d8 -> 90f6f39)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata add 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/parser/AstBuilder.scala | 6 +- .../sql/catalyst/plans/logical/statements.scala| 10 .../sql/catalyst/plans/logical/v2Commands.scala| 65 +- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 10 ++-- .../catalyst/analysis/ResolveSessionCatalog.scala | 28 ++ .../datasources/v2/DataSourceV2Strategy.scala | 3 + .../spark/sql/connector/DataSourceV2SQLSuite.scala | 8 +-- .../apache/spark/sql/execution/SQLViewSuite.scala | 15 +++-- .../org/apache/spark/sql/hive/test/TestHive.scala | 5 +- 9 files changed, 82 insertions(+), 68 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a1f84d8 -> 90f6f39)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata add 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/parser/AstBuilder.scala | 6 +- .../sql/catalyst/plans/logical/statements.scala| 10 .../sql/catalyst/plans/logical/v2Commands.scala| 65 +- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 10 ++-- .../catalyst/analysis/ResolveSessionCatalog.scala | 28 ++ .../datasources/v2/DataSourceV2Strategy.scala | 3 + .../spark/sql/connector/DataSourceV2SQLSuite.scala | 8 +-- .../apache/spark/sql/execution/SQLViewSuite.scala | 15 +++-- .../org/apache/spark/sql/hive/test/TestHive.scala | 5 +- 9 files changed, 82 insertions(+), 68 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a1f84d8 -> 90f6f39)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata add 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/parser/AstBuilder.scala | 6 +- .../sql/catalyst/plans/logical/statements.scala| 10 .../sql/catalyst/plans/logical/v2Commands.scala| 65 +- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 10 ++-- .../catalyst/analysis/ResolveSessionCatalog.scala | 28 ++ .../datasources/v2/DataSourceV2Strategy.scala | 3 + .../spark/sql/connector/DataSourceV2SQLSuite.scala | 8 +-- .../apache/spark/sql/execution/SQLViewSuite.scala | 15 +++-- .../org/apache/spark/sql/hive/test/TestHive.scala | 5 +- 9 files changed, 82 insertions(+), 68 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a1f84d8 -> 90f6f39)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata add 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/parser/AstBuilder.scala | 6 +- .../sql/catalyst/plans/logical/statements.scala| 10 .../sql/catalyst/plans/logical/v2Commands.scala| 65 +- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 10 ++-- .../catalyst/analysis/ResolveSessionCatalog.scala | 28 ++ .../datasources/v2/DataSourceV2Strategy.scala | 3 + .../spark/sql/connector/DataSourceV2SQLSuite.scala | 8 +-- .../apache/spark/sql/execution/SQLViewSuite.scala | 15 +++-- .../org/apache/spark/sql/hive/test/TestHive.scala | 5 +- 9 files changed, 82 insertions(+), 68 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a1f84d8 -> 90f6f39)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata add 90f6f39 [SPARK-33366][SQL] Migrate LOAD DATA command to use UnresolvedTable to resolve the identifier No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/parser/AstBuilder.scala | 6 +- .../sql/catalyst/plans/logical/statements.scala| 10 .../sql/catalyst/plans/logical/v2Commands.scala| 65 +- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 10 ++-- .../catalyst/analysis/ResolveSessionCatalog.scala | 28 ++ .../datasources/v2/DataSourceV2Strategy.scala | 3 + .../spark/sql/connector/DataSourceV2SQLSuite.scala | 8 +-- .../apache/spark/sql/execution/SQLViewSuite.scala | 15 +++-- .../org/apache/spark/sql/hive/test/TestHive.scala | 5 +- 9 files changed, 82 insertions(+), 68 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c2caf25 -> a1f84d8)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 add a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata No new revisions were added by this update. Summary of changes: .../spark/sql/connector/catalog/TableProvider.java | 7 +++-- .../org/apache/spark/sql/DataFrameWriter.scala | 11 --- .../spark/sql/streaming/DataStreamWriter.scala | 10 +- ...org.apache.spark.sql.sources.DataSourceRegister | 1 + .../spark/sql/connector/DataSourceV2Suite.scala| 23 ++ .../sources/StreamingDataSourceV2Suite.scala | 36 ++ 6 files changed, 80 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c2caf25 -> a1f84d8)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 add a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata No new revisions were added by this update. Summary of changes: .../spark/sql/connector/catalog/TableProvider.java | 7 +++-- .../org/apache/spark/sql/DataFrameWriter.scala | 11 --- .../spark/sql/streaming/DataStreamWriter.scala | 10 +- ...org.apache.spark.sql.sources.DataSourceRegister | 1 + .../spark/sql/connector/DataSourceV2Suite.scala| 23 ++ .../sources/StreamingDataSourceV2Suite.scala | 36 ++ 6 files changed, 80 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c2caf25 -> a1f84d8)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 add a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata No new revisions were added by this update. Summary of changes: .../spark/sql/connector/catalog/TableProvider.java | 7 +++-- .../org/apache/spark/sql/DataFrameWriter.scala | 11 --- .../spark/sql/streaming/DataStreamWriter.scala | 10 +- ...org.apache.spark.sql.sources.DataSourceRegister | 1 + .../spark/sql/connector/DataSourceV2Suite.scala| 23 ++ .../sources/StreamingDataSourceV2Suite.scala | 36 ++ 6 files changed, 80 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c2caf25 -> a1f84d8)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 add a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata No new revisions were added by this update. Summary of changes: .../spark/sql/connector/catalog/TableProvider.java | 7 +++-- .../org/apache/spark/sql/DataFrameWriter.scala | 11 --- .../spark/sql/streaming/DataStreamWriter.scala | 10 +- ...org.apache.spark.sql.sources.DataSourceRegister | 1 + .../spark/sql/connector/DataSourceV2Suite.scala| 23 ++ .../sources/StreamingDataSourceV2Suite.scala | 36 ++ 6 files changed, 80 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c2caf25 -> a1f84d8)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 add a1f84d8 [SPARK-33369][SQL] DSV2: Skip schema inference in write if table provider supports external metadata No new revisions were added by this update. Summary of changes: .../spark/sql/connector/catalog/TableProvider.java | 7 +++-- .../org/apache/spark/sql/DataFrameWriter.scala | 11 --- .../spark/sql/streaming/DataStreamWriter.scala | 10 +- ...org.apache.spark.sql.sources.DataSourceRegister | 1 + .../spark/sql/connector/DataSourceV2Suite.scala| 23 ++ .../sources/StreamingDataSourceV2Suite.scala | 36 ++ 6 files changed, 80 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (fa1b476 -> bfeaef1)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from fa1b476 [SPARK-3][BUILD][2.4] Upgrade Jetty to 9.4.28.v20200408 add bfeaef1 [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.6 | 2 +- dev/deps/spark-deps-hadoop-2.7 | 2 +- dev/deps/spark-deps-hadoop-3.1 | 2 +- pom.xml| 6 ++ 4 files changed, 9 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (fa1b476 -> bfeaef1)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from fa1b476 [SPARK-3][BUILD][2.4] Upgrade Jetty to 9.4.28.v20200408 add bfeaef1 [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.6 | 2 +- dev/deps/spark-deps-hadoop-2.7 | 2 +- dev/deps/spark-deps-hadoop-3.1 | 2 +- pom.xml| 6 ++ 4 files changed, 9 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new bfeaef1 [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
bfeaef1 is described below
commit bfeaef1bc3d67a16aed401dbf5c91fff5a835a2a
Author: Dongjoon Hyun
AuthorDate: Mon Nov 9 19:55:23 2020 -0800
[SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30307 from dongjoon-hyun/SPARK-33405.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
---
dev/deps/spark-deps-hadoop-2.6 | 2 +-
dev/deps/spark-deps-hadoop-2.7 | 2 +-
dev/deps/spark-deps-hadoop-3.1 | 2 +-
pom.xml| 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/deps/spark-deps-hadoop-2.6
index b4fad23..87c48fa 100644
--- a/dev/deps/spark-deps-hadoop-2.6
+++ b/dev/deps/spark-deps-hadoop-2.6
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7
index 3dcb4d7..80e3ecc 100644
--- a/dev/deps/spark-deps-hadoop-2.7
+++ b/dev/deps/spark-deps-hadoop-2.7
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.1 b/dev/deps/spark-deps-hadoop-3.1
index 01b6224..c925808 100644
--- a/dev/deps/spark-deps-hadoop-3.1
+++ b/dev/deps/spark-deps-hadoop-3.1
@@ -32,7 +32,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index ab2dd91..630979b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -165,6 +165,7 @@
1.1.2
1.2.0-incubating
1.10
+1.20
2.4
2.6
@@ -462,6 +463,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new bfeaef1 [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
bfeaef1 is described below
commit bfeaef1bc3d67a16aed401dbf5c91fff5a835a2a
Author: Dongjoon Hyun
AuthorDate: Mon Nov 9 19:55:23 2020 -0800
[SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30307 from dongjoon-hyun/SPARK-33405.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
---
dev/deps/spark-deps-hadoop-2.6 | 2 +-
dev/deps/spark-deps-hadoop-2.7 | 2 +-
dev/deps/spark-deps-hadoop-3.1 | 2 +-
pom.xml| 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/deps/spark-deps-hadoop-2.6
index b4fad23..87c48fa 100644
--- a/dev/deps/spark-deps-hadoop-2.6
+++ b/dev/deps/spark-deps-hadoop-2.6
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7
index 3dcb4d7..80e3ecc 100644
--- a/dev/deps/spark-deps-hadoop-2.7
+++ b/dev/deps/spark-deps-hadoop-2.7
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.1 b/dev/deps/spark-deps-hadoop-3.1
index 01b6224..c925808 100644
--- a/dev/deps/spark-deps-hadoop-3.1
+++ b/dev/deps/spark-deps-hadoop-3.1
@@ -32,7 +32,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index ab2dd91..630979b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -165,6 +165,7 @@
1.1.2
1.2.0-incubating
1.10
+1.20
2.4
2.6
@@ -462,6 +463,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4ac8133 -> c2caf25)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information add c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 8 dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 8 pom.xml | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new bfeaef1 [SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
bfeaef1 is described below
commit bfeaef1bc3d67a16aed401dbf5c91fff5a835a2a
Author: Dongjoon Hyun
AuthorDate: Mon Nov 9 19:55:23 2020 -0800
[SPARK-33405][BUILD][2.4] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30307 from dongjoon-hyun/SPARK-33405.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
---
dev/deps/spark-deps-hadoop-2.6 | 2 +-
dev/deps/spark-deps-hadoop-2.7 | 2 +-
dev/deps/spark-deps-hadoop-3.1 | 2 +-
pom.xml| 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/deps/spark-deps-hadoop-2.6
index b4fad23..87c48fa 100644
--- a/dev/deps/spark-deps-hadoop-2.6
+++ b/dev/deps/spark-deps-hadoop-2.6
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7
index 3dcb4d7..80e3ecc 100644
--- a/dev/deps/spark-deps-hadoop-2.7
+++ b/dev/deps/spark-deps-hadoop-2.7
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.1 b/dev/deps/spark-deps-hadoop-3.1
index 01b6224..c925808 100644
--- a/dev/deps/spark-deps-hadoop-3.1
+++ b/dev/deps/spark-deps-hadoop-3.1
@@ -32,7 +32,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index ab2dd91..630979b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -165,6 +165,7 @@
1.1.2
1.2.0-incubating
1.10
+1.20
2.4
2.6
@@ -462,6 +463,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4ac8133 -> c2caf25)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information add c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 8 dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 8 pom.xml | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4ac8133 -> c2caf25)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information add c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 8 dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 8 pom.xml | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4360c6f -> 4ac8133)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts add 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information No new revisions were added by this update. Summary of changes: .../ui/StreamingQueryStatisticsPage.scala | 119 - .../spark/sql/streaming/ui/UISeleniumSuite.scala | 15 ++- 2 files changed, 131 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4ac8133 -> c2caf25)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information add c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 8 dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 8 pom.xml | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4360c6f -> 4ac8133)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts add 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information No new revisions were added by this update. Summary of changes: .../ui/StreamingQueryStatisticsPage.scala | 119 - .../spark/sql/streaming/ui/UISeleniumSuite.scala | 15 ++- 2 files changed, 131 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4ac8133 -> c2caf25)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information add c2caf25 [SPARK-33213][BUILD] Upgrade Apache Arrow to 2.0.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 8 dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 8 pom.xml | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4360c6f -> 4ac8133)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts add 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information No new revisions were added by this update. Summary of changes: .../ui/StreamingQueryStatisticsPage.scala | 119 - .../spark/sql/streaming/ui/UISeleniumSuite.scala | 15 ++- 2 files changed, 131 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
1aa8f4f is described below
commit 1aa8f4f39911b855fe9c86c9be86ec1eb720e277
Author: Dongjoon Hyun
AuthorDate: Tue Nov 10 11:14:38 2020 +0900
[SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30305 from dongjoon-hyun/SPARK-33405-3.0.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
---
dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 2 +-
dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +-
pom.xml | 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
index e32ea64..2068d11 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
@@ -36,7 +36,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
index 168d619..06fcff8 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index d730b4a..9c21981 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -31,7 +31,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index de3b69d..d41d61c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -176,6 +176,7 @@
1.1.7.5
1.1.2
1.10
+1.20
2.4
2.6
@@ -531,6 +532,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (35ac314 -> 4360c6f)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 add 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts No new revisions were added by this update. Summary of changes: R/pkg/inst/profile/shell.R | 4 +++- python/pyspark/shell.py| 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4360c6f -> 4ac8133)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts add 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information No new revisions were added by this update. Summary of changes: .../ui/StreamingQueryStatisticsPage.scala | 119 - .../spark/sql/streaming/ui/UISeleniumSuite.scala | 15 ++- 2 files changed, 131 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
1aa8f4f is described below
commit 1aa8f4f39911b855fe9c86c9be86ec1eb720e277
Author: Dongjoon Hyun
AuthorDate: Tue Nov 10 11:14:38 2020 +0900
[SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30305 from dongjoon-hyun/SPARK-33405-3.0.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
---
dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 2 +-
dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +-
pom.xml | 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
index e32ea64..2068d11 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
@@ -36,7 +36,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
index 168d619..06fcff8 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index d730b4a..9c21981 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -31,7 +31,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index de3b69d..d41d61c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -176,6 +176,7 @@
1.1.7.5
1.1.2
1.10
+1.20
2.4
2.6
@@ -531,6 +532,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (35ac314 -> 4360c6f)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 add 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts No new revisions were added by this update. Summary of changes: R/pkg/inst/profile/shell.R | 4 +++- python/pyspark/shell.py| 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
1aa8f4f is described below
commit 1aa8f4f39911b855fe9c86c9be86ec1eb720e277
Author: Dongjoon Hyun
AuthorDate: Tue Nov 10 11:14:38 2020 +0900
[SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30305 from dongjoon-hyun/SPARK-33405-3.0.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
---
dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 2 +-
dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +-
pom.xml | 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
index e32ea64..2068d11 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
@@ -36,7 +36,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
index 168d619..06fcff8 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index d730b4a..9c21981 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -31,7 +31,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index de3b69d..d41d61c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -176,6 +176,7 @@
1.1.7.5
1.1.2
1.10
+1.20
2.4
2.6
@@ -531,6 +532,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (35ac314 -> 4360c6f)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 add 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts No new revisions were added by this update. Summary of changes: R/pkg/inst/profile/shell.R | 4 +++- python/pyspark/shell.py| 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (036c11b -> 35ac314)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url add 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 6 ++ 3 files changed, 8 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4360c6f -> 4ac8133)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts add 4ac8133 [SPARK-33223][SS][UI] Structured Streaming Web UI state information No new revisions were added by this update. Summary of changes: .../ui/StreamingQueryStatisticsPage.scala | 119 - .../spark/sql/streaming/ui/UISeleniumSuite.scala | 15 ++- 2 files changed, 131 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
1aa8f4f is described below
commit 1aa8f4f39911b855fe9c86c9be86ec1eb720e277
Author: Dongjoon Hyun
AuthorDate: Tue Nov 10 11:14:38 2020 +0900
[SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30305 from dongjoon-hyun/SPARK-33405-3.0.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
---
dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 2 +-
dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +-
pom.xml | 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
index e32ea64..2068d11 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
@@ -36,7 +36,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
index 168d619..06fcff8 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index d730b4a..9c21981 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -31,7 +31,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index de3b69d..d41d61c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -176,6 +176,7 @@
1.1.7.5
1.1.2
1.10
+1.20
2.4
2.6
@@ -531,6 +532,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (35ac314 -> 4360c6f)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 add 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts No new revisions were added by this update. Summary of changes: R/pkg/inst/profile/shell.R | 4 +++- python/pyspark/shell.py| 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (036c11b -> 35ac314)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url add 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 6 ++ 3 files changed, 8 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 1aa8f4f [SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
1aa8f4f is described below
commit 1aa8f4f39911b855fe9c86c9be86ec1eb720e277
Author: Dongjoon Hyun
AuthorDate: Tue Nov 10 11:14:38 2020 +0900
[SPARK-33405][BUILD][3.0] Upgrade commons-compress to 1.20
### What changes were proposed in this pull request?
This PR aims to upgrade `commons-compress` from 1.8 to 1.20.
### Why are the changes needed?
- https://commons.apache.org/proper/commons-compress/security-reports.html
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #30305 from dongjoon-hyun/SPARK-33405-3.0.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
---
dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 2 +-
dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +-
dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +-
pom.xml | 6 ++
4 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
index e32ea64..2068d11 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-1.2
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-1.2
@@ -36,7 +36,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
index 168d619..06fcff8 100644
--- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3
@@ -34,7 +34,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration/1.6//commons-configuration-1.6.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-dbcp/1.4//commons-dbcp-1.4.jar
diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
index d730b4a..9c21981 100644
--- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3
@@ -31,7 +31,7 @@ commons-cli/1.2//commons-cli-1.2.jar
commons-codec/1.10//commons-codec-1.10.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-compiler/3.0.16//commons-compiler-3.0.16.jar
-commons-compress/1.8.1//commons-compress-1.8.1.jar
+commons-compress/1.20//commons-compress-1.20.jar
commons-configuration2/2.1.1//commons-configuration2-2.1.1.jar
commons-crypto/1.0.0//commons-crypto-1.0.0.jar
commons-daemon/1.0.13//commons-daemon-1.0.13.jar
diff --git a/pom.xml b/pom.xml
index de3b69d..d41d61c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -176,6 +176,7 @@
1.1.7.5
1.1.2
1.10
+1.20
2.4
2.6
@@ -531,6 +532,11 @@
org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+org.apache.commons
commons-math3
${commons.math3.version}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (35ac314 -> 4360c6f)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 add 4360c6f [SPARK-33363] Add prompt information related to the current task when pyspark/sparkR starts No new revisions were added by this update. Summary of changes: R/pkg/inst/profile/shell.R | 4 +++- python/pyspark/shell.py| 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (036c11b -> 35ac314)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url add 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 6 ++ 3 files changed, 8 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (036c11b -> 35ac314)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url add 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 6 ++ 3 files changed, 8 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (036c11b -> 35ac314)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url add 35ac314 [SPARK-33405][BUILD] Upgrade commons-compress to 1.20 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 6 ++ 3 files changed, 8 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (c157fa3 -> a418495)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning add a418495 [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (c157fa3 -> a418495)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning add a418495 [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (090962c -> 036c11b)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) add 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (090962c -> 036c11b)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) add 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (c157fa3 -> a418495)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning add a418495 [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (090962c -> 036c11b)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) add 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (c157fa3 -> a418495)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning add a418495 [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (090962c -> 036c11b)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) add 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (c157fa3 -> a418495)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning add a418495 [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (090962c -> 036c11b)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) add 036c11b [SPARK-33397][YARN][DOC] Fix generating md to html for available-patterns-for-shs-custom-executor-log-url No new revisions were added by this update. Summary of changes: docs/running-on-yarn.md | 30 +++--- 1 file changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83a8079 -> 090962c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 add 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) No new revisions were added by this update. Summary of changes: python/pyspark/ml/base.py| 94 ++--- python/pyspark/ml/base.pyi | 8 +- python/pyspark/ml/classification.py | 227 ++--- python/pyspark/ml/clustering.py | 133 +++- python/pyspark/ml/evaluation.py | 91 ++--- python/pyspark/ml/feature.py | 379 +++ python/pyspark/ml/fpm.py | 72 +-- python/pyspark/ml/functions.py | 21 +- python/pyspark/ml/image.py | 47 - python/pyspark/ml/linalg/__init__.py | 72 +-- python/pyspark/ml/param/__init__.py | 63 -- python/pyspark/ml/pipeline.py| 42 +++- python/pyspark/ml/recommendation.py | 92 ++--- python/pyspark/ml/regression.py | 219 +--- python/pyspark/ml/stat.py| 312 +--- python/pyspark/ml/tuning.py | 124 +--- python/pyspark/ml/util.py| 27 ++- python/pyspark/ml/wrapper.py | 59 -- 18 files changed, 1427 insertions(+), 655 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83a8079 -> 090962c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 add 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) No new revisions were added by this update. Summary of changes: python/pyspark/ml/base.py| 94 ++--- python/pyspark/ml/base.pyi | 8 +- python/pyspark/ml/classification.py | 227 ++--- python/pyspark/ml/clustering.py | 133 +++- python/pyspark/ml/evaluation.py | 91 ++--- python/pyspark/ml/feature.py | 379 +++ python/pyspark/ml/fpm.py | 72 +-- python/pyspark/ml/functions.py | 21 +- python/pyspark/ml/image.py | 47 - python/pyspark/ml/linalg/__init__.py | 72 +-- python/pyspark/ml/param/__init__.py | 63 -- python/pyspark/ml/pipeline.py| 42 +++- python/pyspark/ml/recommendation.py | 92 ++--- python/pyspark/ml/regression.py | 219 +--- python/pyspark/ml/stat.py| 312 +--- python/pyspark/ml/tuning.py | 124 +--- python/pyspark/ml/util.py| 27 ++- python/pyspark/ml/wrapper.py | 59 -- 18 files changed, 1427 insertions(+), 655 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83a8079 -> 090962c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 add 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) No new revisions were added by this update. Summary of changes: python/pyspark/ml/base.py| 94 ++--- python/pyspark/ml/base.pyi | 8 +- python/pyspark/ml/classification.py | 227 ++--- python/pyspark/ml/clustering.py | 133 +++- python/pyspark/ml/evaluation.py | 91 ++--- python/pyspark/ml/feature.py | 379 +++ python/pyspark/ml/fpm.py | 72 +-- python/pyspark/ml/functions.py | 21 +- python/pyspark/ml/image.py | 47 - python/pyspark/ml/linalg/__init__.py | 72 +-- python/pyspark/ml/param/__init__.py | 63 -- python/pyspark/ml/pipeline.py| 42 +++- python/pyspark/ml/recommendation.py | 92 ++--- python/pyspark/ml/regression.py | 219 +--- python/pyspark/ml/stat.py| 312 +--- python/pyspark/ml/tuning.py | 124 +--- python/pyspark/ml/util.py| 27 ++- python/pyspark/ml/wrapper.py | 59 -- 18 files changed, 1427 insertions(+), 655 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83a8079 -> 090962c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 add 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) No new revisions were added by this update. Summary of changes: python/pyspark/ml/base.py| 94 ++--- python/pyspark/ml/base.pyi | 8 +- python/pyspark/ml/classification.py | 227 ++--- python/pyspark/ml/clustering.py | 133 +++- python/pyspark/ml/evaluation.py | 91 ++--- python/pyspark/ml/feature.py | 379 +++ python/pyspark/ml/fpm.py | 72 +-- python/pyspark/ml/functions.py | 21 +- python/pyspark/ml/image.py | 47 - python/pyspark/ml/linalg/__init__.py | 72 +-- python/pyspark/ml/param/__init__.py | 63 -- python/pyspark/ml/pipeline.py| 42 +++- python/pyspark/ml/recommendation.py | 92 ++--- python/pyspark/ml/regression.py | 219 +--- python/pyspark/ml/stat.py| 312 +--- python/pyspark/ml/tuning.py | 124 +--- python/pyspark/ml/util.py| 27 ++- python/pyspark/ml/wrapper.py | 59 -- 18 files changed, 1427 insertions(+), 655 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (83a8079 -> 090962c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 add 090962c [SPARK-33251][PYTHON][DOCS] Migration to NumPy documentation style in ML (pyspark.ml.*) No new revisions were added by this update. Summary of changes: python/pyspark/ml/base.py| 94 ++--- python/pyspark/ml/base.pyi | 8 +- python/pyspark/ml/classification.py | 227 ++--- python/pyspark/ml/clustering.py | 133 +++- python/pyspark/ml/evaluation.py | 91 ++--- python/pyspark/ml/feature.py | 379 +++ python/pyspark/ml/fpm.py | 72 +-- python/pyspark/ml/functions.py | 21 +- python/pyspark/ml/image.py | 47 - python/pyspark/ml/linalg/__init__.py | 72 +-- python/pyspark/ml/param/__init__.py | 63 -- python/pyspark/ml/pipeline.py| 42 +++- python/pyspark/ml/recommendation.py | 92 ++--- python/pyspark/ml/regression.py | 219 +--- python/pyspark/ml/stat.py| 312 +--- python/pyspark/ml/tuning.py | 124 +--- python/pyspark/ml/util.py| 27 ++- python/pyspark/ml/wrapper.py | 59 -- 18 files changed, 1427 insertions(+), 655 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8113c88 -> 83a8079)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8113c88 [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode add 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8113c88 -> 83a8079)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8113c88 [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode add 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8113c88 -> 83a8079)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8113c88 [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode add 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8113c88 -> 83a8079)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8113c88 [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode add 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8113c88 -> 83a8079)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8113c88 [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode add 83a8079 [SPARK-32691][BUILD] Update commons-crypto to v1.1.0 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 2 +- pom.xml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode
This is an automated email from the ASF dual-hosted git repository.
mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8113c88 [SPARK-32916][SHUFFLE] Implementation of shuffle service that
leverages push-based shuffle in YARN deployment mode
8113c88 is described below
commit 8113c88542ee282b510c7e046d64df1761a85d14
Author: Chandni Singh
AuthorDate: Mon Nov 9 11:00:52 2020 -0600
[SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages
push-based shuffle in YARN deployment mode
### What changes were proposed in this pull request?
This is one of the patches for SPIP
[SPARK-30602](https://issues.apache.org/jira/browse/SPARK-30602) which is
needed for push-based shuffle.
Summary of changes:
- Adds an implementation of `MergedShuffleFileManager` which was introduced
with [Spark 32915](https://issues.apache.org/jira/browse/SPARK-32915).
- Integrated the push-based shuffle service with `YarnShuffleService`.
### Why are the changes needed?
Refer to the SPIP in
[SPARK-30602](https://issues.apache.org/jira/browse/SPARK-30602).
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests.
The reference PR with the consolidated changes covering the complete
implementation is also provided in
[SPARK-30602](https://issues.apache.org/jira/browse/SPARK-30602).
We have already verified the functionality and the improved performance as
documented in the SPIP doc.
Lead-authored-by: Min Shen mshenlinkedin.com
Co-authored-by: Chandni Singh chsinghlinkedin.com
Co-authored-by: Ye Zhou yezhoulinkedin.com
Closes #30062 from otterc/SPARK-32916.
Lead-authored-by: Chandni Singh
Co-authored-by: Chandni Singh
Co-authored-by: Ye Zhou
Co-authored-by: Min Shen
Signed-off-by: Mridul Muralidharan gmail.com>
---
.../apache/spark/network/protocol/Encoders.java| 26 +-
.../apache/spark/network/util/TransportConf.java | 35 +
.../spark/network/protocol/EncodersSuite.java | 68 ++
common/network-shuffle/pom.xml | 10 +-
.../apache/spark/network/shuffle/ErrorHandler.java | 8 +-
.../network/shuffle/ExternalBlockHandler.java | 25 +-
.../spark/network/shuffle/MergedBlockMeta.java | 2 +
.../network/shuffle/MergedShuffleFileManager.java | 28 +-
.../network/shuffle/OneForOneBlockPusher.java | 11 +-
.../network/shuffle/RemoteBlockPushResolver.java | 934 +
.../shuffle/protocol/FinalizeShuffleMerge.java | 2 +
.../network/shuffle/protocol/MergeStatuses.java| 2 +
.../network/shuffle/protocol/PushBlockStream.java | 37 +-
.../network/shuffle/ExternalBlockHandlerSuite.java | 2 +-
.../network/shuffle/OneForOneBlockPusherSuite.java | 66 +-
.../shuffle/RemoteBlockPushResolverSuite.java | 496 +++
.../spark/network/yarn/YarnShuffleService.java | 23 +-
.../network/yarn/YarnShuffleServiceSuite.java | 61 ++
18 files changed, 1748 insertions(+), 88 deletions(-)
diff --git
a/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
b/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
index 4fa191b..8bab808 100644
---
a/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
+++
b/common/network-common/src/main/java/org/apache/spark/network/protocol/Encoders.java
@@ -18,6 +18,7 @@
package org.apache.spark.network.protocol;
import java.io.IOException;
+import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import io.netty.buffer.ByteBuf;
@@ -46,7 +47,11 @@ public class Encoders {
}
}
- /** Bitmaps are encoded with their serialization length followed by the
serialization bytes. */
+ /**
+ * Bitmaps are encoded with their serialization length followed by the
serialization bytes.
+ *
+ * @since 3.1.0
+ */
public static class Bitmaps {
public static int encodedLength(RoaringBitmap b) {
// Compress the bitmap before serializing it. Note that since
BlockTransferMessage
@@ -57,13 +62,20 @@ public class Encoders {
return b.serializedSizeInBytes();
}
+/**
+ * The input ByteBuf for this encoder should have enough write capacity to
fit the serialized
+ * bitmap. Other encoders which use {@link
io.netty.buffer.AbstractByteBuf#writeBytes(byte[])}
+ * to write can expand the buf as writeBytes calls {@link
ByteBuf#ensureWritable} internally.
+ * However, this encoder doesn't rely on netty's writeBytes and will fail
if the input buf
+ * doesn't have enough write capacity.
+ */
public static void encode(ByteBuf buf, RoaringBitmap b) {
- int encodedLength = b.serializedSizeInBytes();
// RoaringBitmap requires nio
[spark] branch master updated (4e1c894 -> 84dc374)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules add 84dc374 [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls No new revisions were added by this update. Summary of changes: .../sql/execution/python/ExtractPythonUDFs.scala | 20 +++- .../python/BatchEvalPythonExecSuite.scala | 7 ++ .../execution/python/ExtractPythonUDFsSuite.scala | 27 ++ 3 files changed, 48 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1c894 -> 84dc374)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules add 84dc374 [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls No new revisions were added by this update. Summary of changes: .../sql/execution/python/ExtractPythonUDFs.scala | 20 +++- .../python/BatchEvalPythonExecSuite.scala | 7 ++ .../execution/python/ExtractPythonUDFsSuite.scala | 27 ++ 3 files changed, 48 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1c894 -> 84dc374)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules add 84dc374 [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls No new revisions were added by this update. Summary of changes: .../sql/execution/python/ExtractPythonUDFs.scala | 20 +++- .../python/BatchEvalPythonExecSuite.scala | 7 ++ .../execution/python/ExtractPythonUDFsSuite.scala | 27 ++ 3 files changed, 48 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1c894 -> 84dc374)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules add 84dc374 [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls No new revisions were added by this update. Summary of changes: .../sql/execution/python/ExtractPythonUDFs.scala | 20 +++- .../python/BatchEvalPythonExecSuite.scala | 7 ++ .../execution/python/ExtractPythonUDFsSuite.scala | 27 ++ 3 files changed, 48 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1c894 -> 84dc374)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules add 84dc374 [SPARK-33303][SQL] Deduplicate deterministic PythonUDF calls No new revisions were added by this update. Summary of changes: .../sql/execution/python/ExtractPythonUDFs.scala | 20 +++- .../python/BatchEvalPythonExecSuite.scala | 7 ++ .../execution/python/ExtractPythonUDFsSuite.scala | 27 ++ 3 files changed, 48 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7a5647a -> 4e1c894)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN add 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7a5647a -> 4e1c894)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN add 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7a5647a -> 4e1c894)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN add 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7a5647a -> 4e1c894)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN add 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (69799c5 -> 7a5647a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning add 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN No new revisions were added by this update. Summary of changes: .../execution/datasources/FileSourceStrategy.scala | 7 +++ .../spark/sql/sources/BucketedReadSuite.scala | 21 - 2 files changed, 27 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7a5647a -> 4e1c894)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN add 4e1c894 [SPARK-33140][SQL][FOLLOW-UP] Use sparkSession in AQE context when applying rules No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (69799c5 -> 7a5647a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning add 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN No new revisions were added by this update. Summary of changes: .../execution/datasources/FileSourceStrategy.scala | 7 +++ .../spark/sql/sources/BucketedReadSuite.scala | 21 - 2 files changed, 27 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (69799c5 -> 7a5647a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning add 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN No new revisions were added by this update. Summary of changes: .../execution/datasources/FileSourceStrategy.scala | 7 +++ .../spark/sql/sources/BucketedReadSuite.scala | 21 - 2 files changed, 27 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (69799c5 -> 7a5647a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning add 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN No new revisions were added by this update. Summary of changes: .../execution/datasources/FileSourceStrategy.scala | 7 +++ .../spark/sql/sources/BucketedReadSuite.scala | 21 - 2 files changed, 27 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (69799c5 -> 7a5647a)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning add 7a5647a [SPARK-33385][SQL] Support bucket pruning for IsNaN No new revisions were added by this update. Summary of changes: .../execution/datasources/FileSourceStrategy.scala | 7 +++ .../spark/sql/sources/BucketedReadSuite.scala | 21 - 2 files changed, 27 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33372][SQL] Fix InSet bucket pruning
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning
c157fa3 is described below
commit c157fa36e8e1aa18dcf87b6c774970b8122df0dc
Author: Yuming Wang
AuthorDate: Mon Nov 9 08:32:51 2020 +
[SPARK-33372][SQL] Fix InSet bucket pruning
### What changes were proposed in this pull request?
This pr fix `InSet` bucket pruning because of it's values should not be
`Literal`:
https://github.com/apache/spark/blob/cbd3fdea62dab73fc4a96702de8fd1f07722da66/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L255
### Why are the changes needed?
Fix bug.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and manual test:
```scala
spark.sql("select id as a, id as b from range(1)").write.bucketBy(100,
"a").saveAsTable("t")
spark.sql("select * from t where a in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11)").show
```
Before this PR | After this PR
-- | --
|
Closes #30279 from wangyum/SPARK-33372.
Authored-by: Yuming Wang
Signed-off-by: Wenchen Fan
(cherry picked from commit 69799c514ff9874c57bf94d4de21ea4cd0cbbf8d)
Signed-off-by: Wenchen Fan
---
.../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++---
.../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +-
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
index b1e03fe..4b9e0c6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
@@ -89,9 +89,8 @@ object FileSourceStrategy extends Strategy with Logging {
case expressions.In(a: Attribute, list)
if list.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
getBucketSetFromIterable(a, list.map(e => e.eval(EmptyRow)))
- case expressions.InSet(a: Attribute, hset)
-if hset.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
-getBucketSetFromIterable(a, hset.map(e =>
expressions.Literal(e).eval(EmptyRow)))
+ case expressions.InSet(a: Attribute, hset) if a.name == bucketColumnName
=>
+getBucketSetFromIterable(a, hset)
case expressions.IsNull(a: Attribute) if a.name == bucketColumnName =>
getBucketSetFromValue(a, null)
case expressions.And(left, right) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
index 558bfd7..df8ca33 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
@@ -188,7 +188,7 @@ abstract class BucketedReadSuite extends QueryTest with
SQLTestUtils {
// Case 4: InSet
val inSetExpr = expressions.InSet($"j".expr,
-Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue +
3).map(lit(_).expr))
+Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue + 3))
checkPrunedAnswers(
bucketSpec,
bucketValues = Seq(bucketValue, bucketValue + 1, bucketValue + 2,
bucketValue + 3),
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33372][SQL] Fix InSet bucket pruning
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning
c157fa3 is described below
commit c157fa36e8e1aa18dcf87b6c774970b8122df0dc
Author: Yuming Wang
AuthorDate: Mon Nov 9 08:32:51 2020 +
[SPARK-33372][SQL] Fix InSet bucket pruning
### What changes were proposed in this pull request?
This pr fix `InSet` bucket pruning because of it's values should not be
`Literal`:
https://github.com/apache/spark/blob/cbd3fdea62dab73fc4a96702de8fd1f07722da66/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L255
### Why are the changes needed?
Fix bug.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and manual test:
```scala
spark.sql("select id as a, id as b from range(1)").write.bucketBy(100,
"a").saveAsTable("t")
spark.sql("select * from t where a in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11)").show
```
Before this PR | After this PR
-- | --
|
Closes #30279 from wangyum/SPARK-33372.
Authored-by: Yuming Wang
Signed-off-by: Wenchen Fan
(cherry picked from commit 69799c514ff9874c57bf94d4de21ea4cd0cbbf8d)
Signed-off-by: Wenchen Fan
---
.../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++---
.../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +-
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
index b1e03fe..4b9e0c6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
@@ -89,9 +89,8 @@ object FileSourceStrategy extends Strategy with Logging {
case expressions.In(a: Attribute, list)
if list.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
getBucketSetFromIterable(a, list.map(e => e.eval(EmptyRow)))
- case expressions.InSet(a: Attribute, hset)
-if hset.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
-getBucketSetFromIterable(a, hset.map(e =>
expressions.Literal(e).eval(EmptyRow)))
+ case expressions.InSet(a: Attribute, hset) if a.name == bucketColumnName
=>
+getBucketSetFromIterable(a, hset)
case expressions.IsNull(a: Attribute) if a.name == bucketColumnName =>
getBucketSetFromValue(a, null)
case expressions.And(left, right) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
index 558bfd7..df8ca33 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
@@ -188,7 +188,7 @@ abstract class BucketedReadSuite extends QueryTest with
SQLTestUtils {
// Case 4: InSet
val inSetExpr = expressions.InSet($"j".expr,
-Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue +
3).map(lit(_).expr))
+Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue + 3))
checkPrunedAnswers(
bucketSpec,
bucketValues = Seq(bucketValue, bucketValue + 1, bucketValue + 2,
bucketValue + 3),
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (98730b7 -> 69799c5)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 98730b7 [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer add 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33372][SQL] Fix InSet bucket pruning
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning
c157fa3 is described below
commit c157fa36e8e1aa18dcf87b6c774970b8122df0dc
Author: Yuming Wang
AuthorDate: Mon Nov 9 08:32:51 2020 +
[SPARK-33372][SQL] Fix InSet bucket pruning
### What changes were proposed in this pull request?
This pr fix `InSet` bucket pruning because of it's values should not be
`Literal`:
https://github.com/apache/spark/blob/cbd3fdea62dab73fc4a96702de8fd1f07722da66/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L255
### Why are the changes needed?
Fix bug.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and manual test:
```scala
spark.sql("select id as a, id as b from range(1)").write.bucketBy(100,
"a").saveAsTable("t")
spark.sql("select * from t where a in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11)").show
```
Before this PR | After this PR
-- | --
|
Closes #30279 from wangyum/SPARK-33372.
Authored-by: Yuming Wang
Signed-off-by: Wenchen Fan
(cherry picked from commit 69799c514ff9874c57bf94d4de21ea4cd0cbbf8d)
Signed-off-by: Wenchen Fan
---
.../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++---
.../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +-
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
index b1e03fe..4b9e0c6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
@@ -89,9 +89,8 @@ object FileSourceStrategy extends Strategy with Logging {
case expressions.In(a: Attribute, list)
if list.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
getBucketSetFromIterable(a, list.map(e => e.eval(EmptyRow)))
- case expressions.InSet(a: Attribute, hset)
-if hset.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
-getBucketSetFromIterable(a, hset.map(e =>
expressions.Literal(e).eval(EmptyRow)))
+ case expressions.InSet(a: Attribute, hset) if a.name == bucketColumnName
=>
+getBucketSetFromIterable(a, hset)
case expressions.IsNull(a: Attribute) if a.name == bucketColumnName =>
getBucketSetFromValue(a, null)
case expressions.And(left, right) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
index 558bfd7..df8ca33 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
@@ -188,7 +188,7 @@ abstract class BucketedReadSuite extends QueryTest with
SQLTestUtils {
// Case 4: InSet
val inSetExpr = expressions.InSet($"j".expr,
-Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue +
3).map(lit(_).expr))
+Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue + 3))
checkPrunedAnswers(
bucketSpec,
bucketValues = Seq(bucketValue, bucketValue + 1, bucketValue + 2,
bucketValue + 3),
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (98730b7 -> 69799c5)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 98730b7 [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer add 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33372][SQL] Fix InSet bucket pruning
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning
c157fa3 is described below
commit c157fa36e8e1aa18dcf87b6c774970b8122df0dc
Author: Yuming Wang
AuthorDate: Mon Nov 9 08:32:51 2020 +
[SPARK-33372][SQL] Fix InSet bucket pruning
### What changes were proposed in this pull request?
This pr fix `InSet` bucket pruning because of it's values should not be
`Literal`:
https://github.com/apache/spark/blob/cbd3fdea62dab73fc4a96702de8fd1f07722da66/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L255
### Why are the changes needed?
Fix bug.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and manual test:
```scala
spark.sql("select id as a, id as b from range(1)").write.bucketBy(100,
"a").saveAsTable("t")
spark.sql("select * from t where a in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11)").show
```
Before this PR | After this PR
-- | --
|
Closes #30279 from wangyum/SPARK-33372.
Authored-by: Yuming Wang
Signed-off-by: Wenchen Fan
(cherry picked from commit 69799c514ff9874c57bf94d4de21ea4cd0cbbf8d)
Signed-off-by: Wenchen Fan
---
.../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++---
.../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +-
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
index b1e03fe..4b9e0c6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
@@ -89,9 +89,8 @@ object FileSourceStrategy extends Strategy with Logging {
case expressions.In(a: Attribute, list)
if list.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
getBucketSetFromIterable(a, list.map(e => e.eval(EmptyRow)))
- case expressions.InSet(a: Attribute, hset)
-if hset.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
-getBucketSetFromIterable(a, hset.map(e =>
expressions.Literal(e).eval(EmptyRow)))
+ case expressions.InSet(a: Attribute, hset) if a.name == bucketColumnName
=>
+getBucketSetFromIterable(a, hset)
case expressions.IsNull(a: Attribute) if a.name == bucketColumnName =>
getBucketSetFromValue(a, null)
case expressions.And(left, right) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
index 558bfd7..df8ca33 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
@@ -188,7 +188,7 @@ abstract class BucketedReadSuite extends QueryTest with
SQLTestUtils {
// Case 4: InSet
val inSetExpr = expressions.InSet($"j".expr,
-Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue +
3).map(lit(_).expr))
+Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue + 3))
checkPrunedAnswers(
bucketSpec,
bucketValues = Seq(bucketValue, bucketValue + 1, bucketValue + 2,
bucketValue + 3),
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (98730b7 -> 69799c5)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 98730b7 [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer add 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33372][SQL] Fix InSet bucket pruning
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c157fa3 [SPARK-33372][SQL] Fix InSet bucket pruning
c157fa3 is described below
commit c157fa36e8e1aa18dcf87b6c774970b8122df0dc
Author: Yuming Wang
AuthorDate: Mon Nov 9 08:32:51 2020 +
[SPARK-33372][SQL] Fix InSet bucket pruning
### What changes were proposed in this pull request?
This pr fix `InSet` bucket pruning because of it's values should not be
`Literal`:
https://github.com/apache/spark/blob/cbd3fdea62dab73fc4a96702de8fd1f07722da66/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala#L253-L255
### Why are the changes needed?
Fix bug.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and manual test:
```scala
spark.sql("select id as a, id as b from range(1)").write.bucketBy(100,
"a").saveAsTable("t")
spark.sql("select * from t where a in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11)").show
```
Before this PR | After this PR
-- | --
|
Closes #30279 from wangyum/SPARK-33372.
Authored-by: Yuming Wang
Signed-off-by: Wenchen Fan
(cherry picked from commit 69799c514ff9874c57bf94d4de21ea4cd0cbbf8d)
Signed-off-by: Wenchen Fan
---
.../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++---
.../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +-
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
index b1e03fe..4b9e0c6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
@@ -89,9 +89,8 @@ object FileSourceStrategy extends Strategy with Logging {
case expressions.In(a: Attribute, list)
if list.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
getBucketSetFromIterable(a, list.map(e => e.eval(EmptyRow)))
- case expressions.InSet(a: Attribute, hset)
-if hset.forall(_.isInstanceOf[Literal]) && a.name == bucketColumnName
=>
-getBucketSetFromIterable(a, hset.map(e =>
expressions.Literal(e).eval(EmptyRow)))
+ case expressions.InSet(a: Attribute, hset) if a.name == bucketColumnName
=>
+getBucketSetFromIterable(a, hset)
case expressions.IsNull(a: Attribute) if a.name == bucketColumnName =>
getBucketSetFromValue(a, null)
case expressions.And(left, right) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
index 558bfd7..df8ca33 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala
@@ -188,7 +188,7 @@ abstract class BucketedReadSuite extends QueryTest with
SQLTestUtils {
// Case 4: InSet
val inSetExpr = expressions.InSet($"j".expr,
-Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue +
3).map(lit(_).expr))
+Set(bucketValue, bucketValue + 1, bucketValue + 2, bucketValue + 3))
checkPrunedAnswers(
bucketSpec,
bucketValues = Seq(bucketValue, bucketValue + 1, bucketValue + 2,
bucketValue + 3),
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (98730b7 -> 69799c5)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 98730b7 [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer add 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (98730b7 -> 69799c5)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 98730b7 [SPARK-33087][SQL] DataFrameWriterV2 should delegate table resolution to the analyzer add 69799c5 [SPARK-33372][SQL] Fix InSet bucket pruning No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/FileSourceStrategy.scala | 5 ++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
