[spark] branch master updated (21c02ee -> e736c62)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 21c02ee [SPARK-30864][SQL][DOC] add the user guide for Adaptive Query Execution add e736c62 [SPARK-31116][SQL] Fix nested schema case-sensitivity in ParquetRowConverter No new revisions were added by this update. Summary of changes: .../datasources/parquet/ParquetRowConverter.scala | 12 +-- .../spark/sql/FileBasedDataSourceSuite.scala | 40 ++ 2 files changed, 50 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31116][SQL] Fix nested schema case-sensitivity in ParquetRowConverter
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new da1f95b [SPARK-31116][SQL] Fix nested schema case-sensitivity in
ParquetRowConverter
da1f95b is described below
commit da1f95be6b9af59a91a14e01613bdc4e8ac35374
Author: Tae-kyeom, Kim
AuthorDate: Mon Mar 16 10:31:56 2020 -0700
[SPARK-31116][SQL] Fix nested schema case-sensitivity in ParquetRowConverter
### What changes were proposed in this pull request?
This PR (SPARK-31116) add caseSensitive parameter to ParquetRowConverter so
that it handle materialize parquet properly with respect to case sensitivity
### Why are the changes needed?
From spark 3.0.0, below statement throws IllegalArgumentException in
caseInsensitive mode because of explicit field index searching in
ParquetRowConverter. As we already constructed parquet requested schema and
catalyst requested schema during schema clipping in ParquetReadSupport, just
follow these behavior.
```scala
val path = "/some/temp/path"
spark
.range(1L)
.selectExpr("NAMED_STRUCT('lowercase', id, 'camelCase', id + 1) AS
StructColumn")
.write.parquet(path)
val caseInsensitiveSchema = new StructType()
.add(
"StructColumn",
new StructType()
.add("LowerCase", LongType)
.add("camelcase", LongType))
spark.read.schema(caseInsensitiveSchema).parquet(path).show()
```
### Does this PR introduce any user-facing change?
No. The changes are only in unreleased branches (`master` and `branch-3.0`).
### How was this patch tested?
Passed new test cases that check parquet column selection with respect to
schemas and case sensitivities
Closes #27888 from kimtkyeom/parquet_row_converter_case_sensitivity.
Authored-by: Tae-kyeom, Kim
Signed-off-by: Dongjoon Hyun
(cherry picked from commit e736c62764137b2c3af90d2dc8a77e391891200a)
Signed-off-by: Dongjoon Hyun
---
.../datasources/parquet/ParquetRowConverter.scala | 12 +--
.../spark/sql/FileBasedDataSourceSuite.scala | 40 ++
2 files changed, 50 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
index 850adae..22422c0 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala
@@ -33,8 +33,9 @@ import
org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.{BINARY, DOUBLE
import org.apache.spark.internal.Logging
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, DateTimeUtils,
GenericArrayData}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData,
CaseInsensitiveMap, DateTimeUtils, GenericArrayData}
import org.apache.spark.sql.catalyst.util.DateTimeUtils.SQLTimestamp
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.UTF8String
@@ -178,8 +179,15 @@ private[parquet] class ParquetRowConverter(
// Converters for each field.
private[this] val fieldConverters: Array[Converter with
HasParentContainerUpdater] = {
+// (SPARK-31116) Use case insensitive map if spark.sql.caseSensitive is
false
+// to prevent throwing IllegalArgumentException when searching catalyst
type's field index
+val catalystFieldNameToIndex = if (SQLConf.get.caseSensitiveAnalysis) {
+ catalystType.fieldNames.zipWithIndex.toMap
+} else {
+ CaseInsensitiveMap(catalystType.fieldNames.zipWithIndex.toMap)
+}
parquetType.getFields.asScala.map { parquetField =>
- val fieldIndex = catalystType.fieldIndex(parquetField.getName)
+ val fieldIndex = catalystFieldNameToIndex(parquetField.getName)
val catalystField = catalystType(fieldIndex)
// Converted field value should be set to the `fieldIndex`-th cell of
`currentRow`
newConverter(parquetField, catalystField.dataType, new
RowUpdater(currentRow, fieldIndex))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
index c870958..cb410b4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSui
[spark] branch branch-3.0 updated: [SPARK-31146][SQL] Leverage the helper method for aliasing in built-in SQL expressions
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 9cf5d17 [SPARK-31146][SQL] Leverage the helper method for aliasing in
built-in SQL expressions
9cf5d17 is described below
commit 9cf5d170910a65792d894d129a976ec486b4abc6
Author: HyukjinKwon
AuthorDate: Mon Mar 16 11:22:34 2020 -0700
[SPARK-31146][SQL] Leverage the helper method for aliasing in built-in SQL
expressions
### What changes were proposed in this pull request?
This PR is kind of a followup of #26808. It leverages the helper method for
aliasing in built-in SQL expressions to use the alias as its output column name
where it's applicable.
- `Expression`, `UnaryMathExpression` and `BinaryMathExpression` search the
alias in the tags by default.
- When the naming is different in its implementation, it has to be
overwritten for the expression specifically. E.g., `CallMethodViaReflection`,
`Remainder`, `CurrentTimestamp`,
`FormatString` and `XPathDouble`.
This PR fixes the aliases of the functions below:
| class| alias|
|--|--|
|`Rand`|`random` |
|`Ceil`|`ceiling` |
|`Remainder` |`mod` |
|`Pow` |`pow` |
|`Signum` |`sign`|
|`Chr` |`char`|
|`Length` |`char_length` |
|`Length` |`character_length`|
|`FormatString`|`printf` |
|`Substring` |`substr` |
|`Upper` |`ucase` |
|`XPathDouble` |`xpath_number`|
|`DayOfMonth` |`day` |
|`CurrentTimestamp`|`now` |
|`Size`|`cardinality` |
|`Sha1`|`sha` |
|`CallMethodViaReflection` |`java_method` |
Note: `EqualTo`, `=` and `==` aliases were excluded because it's unable to
leverage this helper method. It should fix the parser.
Note: this PR also excludes some instances such as `ToDegrees`,
`ToRadians`, `UnaryMinus` and `UnaryPositive` that needs an explicit name
overwritten to make the scope of this PR smaller.
### Why are the changes needed?
To respect expression name.
### Does this PR introduce any user-facing change?
Yes, it will change the output column name.
### How was this patch tested?
Manually tested, and unittests were added.
Closes #27901 from HyukjinKwon/31146.
Authored-by: HyukjinKwon
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 6704103499d2003b1879ff0b4b8e29141e401b9f)
Signed-off-by: Dongjoon Hyun
---
.../sql/catalyst/analysis/FunctionRegistry.scala | 38 --
.../expressions/CallMethodViaReflection.scala | 4 +--
.../sql/catalyst/expressions/Expression.scala | 5 +--
.../sql/catalyst/expressions/aggregate/First.scala | 2 --
.../sql/catalyst/expressions/aggregate/Last.scala | 2 --
.../sql/catalyst/expressions/arithmetic.scala | 14 +++-
.../catalyst/expressions/datetimeExpressions.scala | 4 ++-
.../sql/catalyst/expressions/mathExpressions.scala | 10 +++---
.../catalyst/expressions/stringExpressions.scala | 7 ++--
.../spark/sql/catalyst/expressions/xml/xpath.scala | 5 +--
.../resources/sql-tests/results/operators.sql.out | 14
.../sql-tests/results/postgreSQL/insert.sql.out| 2 +-
.../sql-tests/results/postgreSQL/numeric.sql.out | 2 +-
.../sql-tests/results/postgreSQL/strings.sql.out | 4 +--
.../sql-tests/results/string-functions.sql.out | 6 ++--
.../typeCoercion/native/implicitTypeCasts.sql.out | 2 +-
.../scala/org/apache/spark/sql/ExplainSuite.scala | 4 +--
17 files changed, 70 insertions(+), 55 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index 6c4aee4..c11186e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -218,7 +218,7 @@ object FunctionRegistry {
expression[PosExplode]("posexplode"),
expressionGeneratorOuter[PosExplode]("posexplode_outer"),
expression[Rand]("rand"),
-expression[Rand]("random"),
+expression[Rand]("random", true),
expression[Randn]("randn"),
expressio
[spark] branch master updated (3ce1dff -> 6704103)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3ce1dff [SPARK-30930][ML] Remove ML/MLLIB DeveloperApi annotations add 6704103 [SPARK-31146][SQL] Leverage the helper method for aliasing in built-in SQL expressions No new revisions were added by this update. Summary of changes: .../sql/catalyst/analysis/FunctionRegistry.scala | 38 -- .../expressions/CallMethodViaReflection.scala | 4 +-- .../sql/catalyst/expressions/Expression.scala | 5 +-- .../sql/catalyst/expressions/aggregate/First.scala | 2 -- .../sql/catalyst/expressions/aggregate/Last.scala | 2 -- .../sql/catalyst/expressions/arithmetic.scala | 14 +++- .../catalyst/expressions/datetimeExpressions.scala | 4 ++- .../sql/catalyst/expressions/mathExpressions.scala | 10 +++--- .../catalyst/expressions/stringExpressions.scala | 7 ++-- .../spark/sql/catalyst/expressions/xml/xpath.scala | 5 +-- .../resources/sql-tests/results/operators.sql.out | 14 .../sql-tests/results/postgreSQL/insert.sql.out| 2 +- .../sql-tests/results/postgreSQL/numeric.sql.out | 2 +- .../sql-tests/results/postgreSQL/strings.sql.out | 4 +-- .../sql-tests/results/string-functions.sql.out | 6 ++-- .../typeCoercion/native/implicitTypeCasts.sql.out | 2 +- .../scala/org/apache/spark/sql/ExplainSuite.scala | 4 +-- 17 files changed, 70 insertions(+), 55 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle non-existed path
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new cb26f63 [SPARK-31163][SQL] TruncateTableCommand with acl/permission
should handle non-existed path
cb26f63 is described below
commit cb26f636b08aea4c5c6bf5035a359cd3cbf335c0
Author: yi.wu
AuthorDate: Mon Mar 16 11:45:25 2020 -0700
[SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle
non-existed path
### What changes were proposed in this pull request?
This fix #26956
Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of
the path doesn't exist.
### Why are the changes needed?
`truncate table` may fail to re-create path in case of interruption or
something else. As a result, next time we `truncate table` on the same table
with acl/permission, it will fail due to `FileNotFoundException`. And it also
brings behavior change compares to previous Spark version, which could still
`truncate table` successfully even if the path doesn't exist.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added UT.
Closes #27923 from Ngone51/fix_truncate.
Authored-by: yi.wu
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/execution/command/tables.scala | 2 +-
.../spark/sql/execution/command/DDLSuite.scala | 21 +
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
index 6243261..d4de822 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
@@ -508,8 +508,8 @@ case class TruncateTableCommand(
var optPermission: Option[FsPermission] = None
var optAcls: Option[java.util.List[AclEntry]] = None
if (!ignorePermissionAcl) {
-val fileStatus = fs.getFileStatus(path)
try {
+ val fileStatus = fs.getFileStatus(path)
optPermission = Some(fileStatus.getPermission())
} catch {
case NonFatal(_) => // do nothing
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
index 5a67dce..10ad8ac 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
@@ -2084,6 +2084,27 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
}
}
+ test("SPARK-31163: acl/permission should handle non-existed path when
truncating table") {
+withSQLConf(SQLConf.TRUNCATE_TABLE_IGNORE_PERMISSION_ACL.key -> "false") {
+ withTable("tab1") {
+sql("CREATE TABLE tab1 (col1 STRING, col2 INT) USING parquet
PARTITIONED BY (col2)")
+sql("INSERT INTO tab1 SELECT 'one', 1")
+checkAnswer(spark.table("tab1"), Row("one", 1))
+val part =
spark.sessionState.catalog.listPartitions(TableIdentifier("tab1")).head
+val path = new File(part.location.getPath)
+sql("TRUNCATE TABLE tab1")
+// simulate incomplete/unsuccessful truncate
+assert(path.exists())
+path.delete()
+assert(!path.exists())
+// execute without java.io.FileNotFoundException
+sql("TRUNCATE TABLE tab1")
+// partition path should be re-created
+assert(path.exists())
+ }
+}
+ }
+
test("create temporary view with mismatched schema") {
withTable("tab1") {
spark.range(10).write.saveAsTable("tab1")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle non-existed path
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new fed4a68 [SPARK-31163][SQL] TruncateTableCommand with acl/permission
should handle non-existed path
fed4a68 is described below
commit fed4a680e3c45462d0d69aeac53780a677f988ae
Author: yi.wu
AuthorDate: Mon Mar 16 11:45:25 2020 -0700
[SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle
non-existed path
### What changes were proposed in this pull request?
This fix #26956
Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of
the path doesn't exist.
### Why are the changes needed?
`truncate table` may fail to re-create path in case of interruption or
something else. As a result, next time we `truncate table` on the same table
with acl/permission, it will fail due to `FileNotFoundException`. And it also
brings behavior change compares to previous Spark version, which could still
`truncate table` successfully even if the path doesn't exist.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added UT.
Closes #27923 from Ngone51/fix_truncate.
Authored-by: yi.wu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit cb26f636b08aea4c5c6bf5035a359cd3cbf335c0)
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/execution/command/tables.scala | 2 +-
.../spark/sql/execution/command/DDLSuite.scala | 21 +
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
index 6243261..d4de822 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
@@ -508,8 +508,8 @@ case class TruncateTableCommand(
var optPermission: Option[FsPermission] = None
var optAcls: Option[java.util.List[AclEntry]] = None
if (!ignorePermissionAcl) {
-val fileStatus = fs.getFileStatus(path)
try {
+ val fileStatus = fs.getFileStatus(path)
optPermission = Some(fileStatus.getPermission())
} catch {
case NonFatal(_) => // do nothing
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
index 5a67dce..10ad8ac 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
@@ -2084,6 +2084,27 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
}
}
+ test("SPARK-31163: acl/permission should handle non-existed path when
truncating table") {
+withSQLConf(SQLConf.TRUNCATE_TABLE_IGNORE_PERMISSION_ACL.key -> "false") {
+ withTable("tab1") {
+sql("CREATE TABLE tab1 (col1 STRING, col2 INT) USING parquet
PARTITIONED BY (col2)")
+sql("INSERT INTO tab1 SELECT 'one', 1")
+checkAnswer(spark.table("tab1"), Row("one", 1))
+val part =
spark.sessionState.catalog.listPartitions(TableIdentifier("tab1")).head
+val path = new File(part.location.getPath)
+sql("TRUNCATE TABLE tab1")
+// simulate incomplete/unsuccessful truncate
+assert(path.exists())
+path.delete()
+assert(!path.exists())
+// execute without java.io.FileNotFoundException
+sql("TRUNCATE TABLE tab1")
+// partition path should be re-created
+assert(path.exists())
+ }
+}
+ }
+
test("create temporary view with mismatched schema") {
withTable("tab1") {
spark.range(10).write.saveAsTable("tab1")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle non-existed path
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 26ad3fe [SPARK-31163][SQL] TruncateTableCommand with acl/permission
should handle non-existed path
26ad3fe is described below
commit 26ad3fe2604506a2cbc5de0ecd1b5771f3965cb4
Author: yi.wu
AuthorDate: Mon Mar 16 11:45:25 2020 -0700
[SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle
non-existed path
### What changes were proposed in this pull request?
This fix #26956
Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of
the path doesn't exist.
### Why are the changes needed?
`truncate table` may fail to re-create path in case of interruption or
something else. As a result, next time we `truncate table` on the same table
with acl/permission, it will fail due to `FileNotFoundException`. And it also
brings behavior change compares to previous Spark version, which could still
`truncate table` successfully even if the path doesn't exist.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added UT.
Closes #27923 from Ngone51/fix_truncate.
Authored-by: yi.wu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit cb26f636b08aea4c5c6bf5035a359cd3cbf335c0)
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/execution/command/tables.scala | 2 +-
.../spark/sql/execution/command/DDLSuite.scala | 21 +
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
index 28dc4a4..8f504d3 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
@@ -470,8 +470,8 @@ case class TruncateTableCommand(
var optPermission: Option[FsPermission] = None
var optAcls: Option[java.util.List[AclEntry]] = None
if (!ignorePermissionAcl) {
-val fileStatus = fs.getFileStatus(path)
try {
+ val fileStatus = fs.getFileStatus(path)
optPermission = Some(fileStatus.getPermission())
} catch {
case NonFatal(_) => // do nothing
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
index 0aabe86..73565f2 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
@@ -2009,6 +2009,27 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
}
}
+ test("SPARK-31163: acl/permission should handle non-existed path when
truncating table") {
+withSQLConf(SQLConf.TRUNCATE_TABLE_IGNORE_PERMISSION_ACL.key -> "false") {
+ withTable("tab1") {
+sql("CREATE TABLE tab1 (col1 STRING, col2 INT) USING parquet
PARTITIONED BY (col2)")
+sql("INSERT INTO tab1 SELECT 'one', 1")
+checkAnswer(spark.table("tab1"), Row("one", 1))
+val part =
spark.sessionState.catalog.listPartitions(TableIdentifier("tab1")).head
+val path = new File(part.location.getPath)
+sql("TRUNCATE TABLE tab1")
+// simulate incomplete/unsuccessful truncate
+assert(path.exists())
+path.delete()
+assert(!path.exists())
+// execute without java.io.FileNotFoundException
+sql("TRUNCATE TABLE tab1")
+// partition path should be re-created
+assert(path.exists())
+ }
+}
+ }
+
test("create temporary view with mismatched schema") {
withTable("tab1") {
spark.range(10).write.saveAsTable("tab1")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle non-existed path
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 26ad3fe [SPARK-31163][SQL] TruncateTableCommand with acl/permission
should handle non-existed path
26ad3fe is described below
commit 26ad3fe2604506a2cbc5de0ecd1b5771f3965cb4
Author: yi.wu
AuthorDate: Mon Mar 16 11:45:25 2020 -0700
[SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle
non-existed path
### What changes were proposed in this pull request?
This fix #26956
Wrap try-catch on `fs.getFileStatus(path)` within acl/permission in case of
the path doesn't exist.
### Why are the changes needed?
`truncate table` may fail to re-create path in case of interruption or
something else. As a result, next time we `truncate table` on the same table
with acl/permission, it will fail due to `FileNotFoundException`. And it also
brings behavior change compares to previous Spark version, which could still
`truncate table` successfully even if the path doesn't exist.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added UT.
Closes #27923 from Ngone51/fix_truncate.
Authored-by: yi.wu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit cb26f636b08aea4c5c6bf5035a359cd3cbf335c0)
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/execution/command/tables.scala | 2 +-
.../spark/sql/execution/command/DDLSuite.scala | 21 +
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
index 28dc4a4..8f504d3 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
@@ -470,8 +470,8 @@ case class TruncateTableCommand(
var optPermission: Option[FsPermission] = None
var optAcls: Option[java.util.List[AclEntry]] = None
if (!ignorePermissionAcl) {
-val fileStatus = fs.getFileStatus(path)
try {
+ val fileStatus = fs.getFileStatus(path)
optPermission = Some(fileStatus.getPermission())
} catch {
case NonFatal(_) => // do nothing
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
index 0aabe86..73565f2 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
@@ -2009,6 +2009,27 @@ abstract class DDLSuite extends QueryTest with
SQLTestUtils {
}
}
+ test("SPARK-31163: acl/permission should handle non-existed path when
truncating table") {
+withSQLConf(SQLConf.TRUNCATE_TABLE_IGNORE_PERMISSION_ACL.key -> "false") {
+ withTable("tab1") {
+sql("CREATE TABLE tab1 (col1 STRING, col2 INT) USING parquet
PARTITIONED BY (col2)")
+sql("INSERT INTO tab1 SELECT 'one', 1")
+checkAnswer(spark.table("tab1"), Row("one", 1))
+val part =
spark.sessionState.catalog.listPartitions(TableIdentifier("tab1")).head
+val path = new File(part.location.getPath)
+sql("TRUNCATE TABLE tab1")
+// simulate incomplete/unsuccessful truncate
+assert(path.exists())
+path.delete()
+assert(!path.exists())
+// execute without java.io.FileNotFoundException
+sql("TRUNCATE TABLE tab1")
+// partition path should be re-created
+assert(path.exists())
+ }
+}
+ }
+
test("create temporary view with mismatched schema") {
withTable("tab1") {
spark.range(10).write.saveAsTable("tab1")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d7b97a1 -> ed06d98)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d7b97a1 [SPARK-31166][SQL] UNION map and other maps should not fail add ed06d98 [SPARK-25355][K8S] Add proxy user to driver if present on spark-submit No new revisions were added by this update. Summary of changes: .../org/apache/spark/deploy/SparkSubmit.scala | 4 ++ .../org/apache/spark/deploy/SparkSubmitSuite.scala | 2 + .../apache/spark/deploy/k8s/KubernetesConf.scala | 14 -- .../k8s/features/DriverCommandFeatureStep.scala| 6 +++ .../k8s/submit/KubernetesClientApplication.scala | 12 +++-- .../spark/deploy/k8s/KubernetesConfSuite.scala | 3 +- .../spark/deploy/k8s/KubernetesTestConf.scala | 5 +- .../features/DriverCommandFeatureStepSuite.scala | 54 +- 8 files changed, 89 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [MINOR][SQL] Update the DataFrameWriter.bucketBy comment
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 124b4ce [MINOR][SQL] Update the DataFrameWriter.bucketBy comment
124b4ce is described below
commit 124b4ce2e6e8f84294f8fc13d3e731a82325dacb
Author: Takeshi Yamamuro
AuthorDate: Tue Mar 17 00:52:45 2020 -0700
[MINOR][SQL] Update the DataFrameWriter.bucketBy comment
### What changes were proposed in this pull request?
This PR intends to update the `DataFrameWriter.bucketBy` comment for
clearly describing that the bucketBy scheme follows a Spark "specific" one.
I saw the questions about the current bucketing compatibility with Hive in
[SPARK-31162](https://issues.apache.org/jira/browse/SPARK-31162?focusedCommentId=17060408&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17060408)
and
[SPARK-17495](https://issues.apache.org/jira/browse/SPARK-17495?focusedCommentId=17059847&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17059847)
from users and IMHO the comment is a bit confusing [...]
### Why are the changes needed?
To make users understood smoothly.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
N/A
Closes #27930 from maropu/UpdateBucketByComment.
Authored-by: Takeshi Yamamuro
Signed-off-by: Dongjoon Hyun
---
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
index 22b26ca..6946c1f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -198,7 +198,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T])
{
/**
* Buckets the output by the given columns. If specified, the output is laid
out on the file
- * system similar to Hive's bucketing scheme.
+ * system similar to Hive's bucketing scheme, but with a different bucket
hash function
+ * and is not compatible with Hive's bucketing.
*
* This is applicable for all file-based data sources (e.g. Parquet, JSON)
starting with Spark
* 2.1.0.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][SQL] Update the DataFrameWriter.bucketBy comment
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 26ea213 [MINOR][SQL] Update the DataFrameWriter.bucketBy comment
26ea213 is described below
commit 26ea213f3c4f2acb07045bf0f6b476ddfb635436
Author: Takeshi Yamamuro
AuthorDate: Tue Mar 17 00:52:45 2020 -0700
[MINOR][SQL] Update the DataFrameWriter.bucketBy comment
### What changes were proposed in this pull request?
This PR intends to update the `DataFrameWriter.bucketBy` comment for
clearly describing that the bucketBy scheme follows a Spark "specific" one.
I saw the questions about the current bucketing compatibility with Hive in
[SPARK-31162](https://issues.apache.org/jira/browse/SPARK-31162?focusedCommentId=17060408&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17060408)
and
[SPARK-17495](https://issues.apache.org/jira/browse/SPARK-17495?focusedCommentId=17059847&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17059847)
from users and IMHO the comment is a bit confusing [...]
### Why are the changes needed?
To make users understood smoothly.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
N/A
Closes #27930 from maropu/UpdateBucketByComment.
Authored-by: Takeshi Yamamuro
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 124b4ce2e6e8f84294f8fc13d3e731a82325dacb)
Signed-off-by: Dongjoon Hyun
---
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
index 22b26ca..6946c1f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -198,7 +198,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T])
{
/**
* Buckets the output by the given columns. If specified, the output is laid
out on the file
- * system similar to Hive's bucketing scheme.
+ * system similar to Hive's bucketing scheme, but with a different bucket
hash function
+ * and is not compatible with Hive's bucketing.
*
* This is applicable for all file-based data sources (e.g. Parquet, JSON)
starting with Spark
* 2.1.0.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][SQL] Update the DataFrameWriter.bucketBy comment
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 26ea213 [MINOR][SQL] Update the DataFrameWriter.bucketBy comment
26ea213 is described below
commit 26ea213f3c4f2acb07045bf0f6b476ddfb635436
Author: Takeshi Yamamuro
AuthorDate: Tue Mar 17 00:52:45 2020 -0700
[MINOR][SQL] Update the DataFrameWriter.bucketBy comment
### What changes were proposed in this pull request?
This PR intends to update the `DataFrameWriter.bucketBy` comment for
clearly describing that the bucketBy scheme follows a Spark "specific" one.
I saw the questions about the current bucketing compatibility with Hive in
[SPARK-31162](https://issues.apache.org/jira/browse/SPARK-31162?focusedCommentId=17060408&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17060408)
and
[SPARK-17495](https://issues.apache.org/jira/browse/SPARK-17495?focusedCommentId=17059847&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17059847)
from users and IMHO the comment is a bit confusing [...]
### Why are the changes needed?
To make users understood smoothly.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
N/A
Closes #27930 from maropu/UpdateBucketByComment.
Authored-by: Takeshi Yamamuro
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 124b4ce2e6e8f84294f8fc13d3e731a82325dacb)
Signed-off-by: Dongjoon Hyun
---
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
index 22b26ca..6946c1f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -198,7 +198,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T])
{
/**
* Buckets the output by the given columns. If specified, the output is laid
out on the file
- * system similar to Hive's bucketing scheme.
+ * system similar to Hive's bucketing scheme, but with a different bucket
hash function
+ * and is not compatible with Hive's bucketing.
*
* This is applicable for all file-based data sources (e.g. Parquet, JSON)
starting with Spark
* 2.1.0.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (26ad3fe -> 6a60c66)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 26ad3fe [SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle non-existed path add 6a60c66 [MINOR][SQL] Update the DataFrameWriter.bucketBy comment No new revisions were added by this update. Summary of changes: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (26ad3fe -> 6a60c66)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 26ad3fe [SPARK-31163][SQL] TruncateTableCommand with acl/permission should handle non-existed path add 6a60c66 [MINOR][SQL] Update the DataFrameWriter.bucketBy comment No new revisions were added by this update. Summary of changes: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event log folder and appStatusFile creation
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9f27a54 [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at
rolling event log folder and appStatusFile creation
9f27a54 is described below
commit 9f27a5495d0114723701c932f20f32d308a571cc
Author: Adam Binford
AuthorDate: Tue Mar 17 11:20:10 2020 -0700
[SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event
log folder and appStatusFile creation
### What changes were proposed in this pull request?
This pull request fixes an issue with rolling event logs. The rolling event
log directory is created ignoring the dfs umask setting. This allows the
history server to prune old rolling logs when run as the group owner of the
event log folder.
### Why are the changes needed?
For non-rolling event logs, log files are created ignoring the umask
setting by calling setPermission after creating the file. The default umask of
022 currently causes rolling log directories to be created without group write
permissions, preventing the history server from pruning logs of applications
not run as the same user as the history server. This adds the same behavior for
rolling event logs so users don't need to worry about the umask setting causing
different behavior.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Manually. The folder is created with the correct 770 permission. The status
file is still affected by the umask setting, but that doesn't stop the folder
from being deleted by the history server. I'm not sure if that causes any other
issues. I'm not sure how to test something involving a Hadoop setting.
Closes #27764 from Kimahriman/bug/rolling-log-permissions.
Authored-by: Adam Binford
Signed-off-by: Dongjoon Hyun
---
.../org/apache/spark/deploy/history/EventLogFileWriters.scala | 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
index 1d58d05..7d44cbd 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
@@ -166,7 +166,8 @@ object EventLogFileWriter {
val IN_PROGRESS = ".inprogress"
val COMPACTED = ".compact"
- val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("770",
8).toShort)
+ val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("660",
8).toShort)
+ val LOG_FOLDER_PERMISSIONS = new FsPermission(Integer.parseInt("770",
8).toShort)
def apply(
appId: String,
@@ -317,7 +318,8 @@ class RollingEventLogFilesWriter(
throw new IOException(s"Target log directory already exists
($logDirForAppPath)")
}
-fileSystem.mkdirs(logDirForAppPath,
EventLogFileWriter.LOG_FILE_PERMISSIONS)
+// SPARK-30860: use the class method to avoid the umask causing permission
issues
+FileSystem.mkdirs(fileSystem, logDirForAppPath,
EventLogFileWriter.LOG_FOLDER_PERMISSIONS)
createAppStatusFile(inProgress = true)
rollEventLogFile()
}
@@ -361,7 +363,9 @@ class RollingEventLogFilesWriter(
private def createAppStatusFile(inProgress: Boolean): Unit = {
val appStatusPath = getAppStatusFilePath(logDirForAppPath, appId,
appAttemptId, inProgress)
-val outputStream = fileSystem.create(appStatusPath)
+// SPARK-30860: use the class method to avoid the umask causing permission
issues
+val outputStream = FileSystem.create(fileSystem, appStatusPath,
+ EventLogFileWriter.LOG_FILE_PERMISSIONS)
// we intentionally create zero-byte file to minimize the cost
outputStream.close()
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (5bc0d76 -> 9f27a54)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 5bc0d76 [SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir add 9f27a54 [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event log folder and appStatusFile creation No new revisions were added by this update. Summary of changes: .../org/apache/spark/deploy/history/EventLogFileWriters.scala | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event log folder and appStatusFile creation
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 314627e [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at
rolling event log folder and appStatusFile creation
314627e is described below
commit 314627ee148264e170b0feaa623e1e1284932166
Author: Adam Binford
AuthorDate: Tue Mar 17 11:20:10 2020 -0700
[SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event
log folder and appStatusFile creation
### What changes were proposed in this pull request?
This pull request fixes an issue with rolling event logs. The rolling event
log directory is created ignoring the dfs umask setting. This allows the
history server to prune old rolling logs when run as the group owner of the
event log folder.
### Why are the changes needed?
For non-rolling event logs, log files are created ignoring the umask
setting by calling setPermission after creating the file. The default umask of
022 currently causes rolling log directories to be created without group write
permissions, preventing the history server from pruning logs of applications
not run as the same user as the history server. This adds the same behavior for
rolling event logs so users don't need to worry about the umask setting causing
different behavior.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Manually. The folder is created with the correct 770 permission. The status
file is still affected by the umask setting, but that doesn't stop the folder
from being deleted by the history server. I'm not sure if that causes any other
issues. I'm not sure how to test something involving a Hadoop setting.
Closes #27764 from Kimahriman/bug/rolling-log-permissions.
Authored-by: Adam Binford
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 9f27a5495d0114723701c932f20f32d308a571cc)
Signed-off-by: Dongjoon Hyun
---
.../org/apache/spark/deploy/history/EventLogFileWriters.scala | 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
index 1d58d05..7d44cbd 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
@@ -166,7 +166,8 @@ object EventLogFileWriter {
val IN_PROGRESS = ".inprogress"
val COMPACTED = ".compact"
- val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("770",
8).toShort)
+ val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("660",
8).toShort)
+ val LOG_FOLDER_PERMISSIONS = new FsPermission(Integer.parseInt("770",
8).toShort)
def apply(
appId: String,
@@ -317,7 +318,8 @@ class RollingEventLogFilesWriter(
throw new IOException(s"Target log directory already exists
($logDirForAppPath)")
}
-fileSystem.mkdirs(logDirForAppPath,
EventLogFileWriter.LOG_FILE_PERMISSIONS)
+// SPARK-30860: use the class method to avoid the umask causing permission
issues
+FileSystem.mkdirs(fileSystem, logDirForAppPath,
EventLogFileWriter.LOG_FOLDER_PERMISSIONS)
createAppStatusFile(inProgress = true)
rollEventLogFile()
}
@@ -361,7 +363,9 @@ class RollingEventLogFilesWriter(
private def createAppStatusFile(inProgress: Boolean): Unit = {
val appStatusPath = getAppStatusFilePath(logDirForAppPath, appId,
appAttemptId, inProgress)
-val outputStream = fileSystem.create(appStatusPath)
+// SPARK-30860: use the class method to avoid the umask causing permission
issues
+val outputStream = FileSystem.create(fileSystem, appStatusPath,
+ EventLogFileWriter.LOG_FILE_PERMISSIONS)
// we intentionally create zero-byte file to minimize the cost
outputStream.close()
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event log folder and appStatusFile creation
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 314627e [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at
rolling event log folder and appStatusFile creation
314627e is described below
commit 314627ee148264e170b0feaa623e1e1284932166
Author: Adam Binford
AuthorDate: Tue Mar 17 11:20:10 2020 -0700
[SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event
log folder and appStatusFile creation
### What changes were proposed in this pull request?
This pull request fixes an issue with rolling event logs. The rolling event
log directory is created ignoring the dfs umask setting. This allows the
history server to prune old rolling logs when run as the group owner of the
event log folder.
### Why are the changes needed?
For non-rolling event logs, log files are created ignoring the umask
setting by calling setPermission after creating the file. The default umask of
022 currently causes rolling log directories to be created without group write
permissions, preventing the history server from pruning logs of applications
not run as the same user as the history server. This adds the same behavior for
rolling event logs so users don't need to worry about the umask setting causing
different behavior.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Manually. The folder is created with the correct 770 permission. The status
file is still affected by the umask setting, but that doesn't stop the folder
from being deleted by the history server. I'm not sure if that causes any other
issues. I'm not sure how to test something involving a Hadoop setting.
Closes #27764 from Kimahriman/bug/rolling-log-permissions.
Authored-by: Adam Binford
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 9f27a5495d0114723701c932f20f32d308a571cc)
Signed-off-by: Dongjoon Hyun
---
.../org/apache/spark/deploy/history/EventLogFileWriters.scala | 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
index 1d58d05..7d44cbd 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
@@ -166,7 +166,8 @@ object EventLogFileWriter {
val IN_PROGRESS = ".inprogress"
val COMPACTED = ".compact"
- val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("770",
8).toShort)
+ val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("660",
8).toShort)
+ val LOG_FOLDER_PERMISSIONS = new FsPermission(Integer.parseInt("770",
8).toShort)
def apply(
appId: String,
@@ -317,7 +318,8 @@ class RollingEventLogFilesWriter(
throw new IOException(s"Target log directory already exists
($logDirForAppPath)")
}
-fileSystem.mkdirs(logDirForAppPath,
EventLogFileWriter.LOG_FILE_PERMISSIONS)
+// SPARK-30860: use the class method to avoid the umask causing permission
issues
+FileSystem.mkdirs(fileSystem, logDirForAppPath,
EventLogFileWriter.LOG_FOLDER_PERMISSIONS)
createAppStatusFile(inProgress = true)
rollEventLogFile()
}
@@ -361,7 +363,9 @@ class RollingEventLogFilesWriter(
private def createAppStatusFile(inProgress: Boolean): Unit = {
val appStatusPath = getAppStatusFilePath(logDirForAppPath, appId,
appAttemptId, inProgress)
-val outputStream = fileSystem.create(appStatusPath)
+// SPARK-30860: use the class method to avoid the umask causing permission
issues
+val outputStream = FileSystem.create(fileSystem, appStatusPath,
+ EventLogFileWriter.LOG_FILE_PERMISSIONS)
// we intentionally create zero-byte file to minimize the cost
outputStream.close()
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9f27a54 -> dc5ebc2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9f27a54 [SPARK-30860][CORE] Use FileSystem.mkdirs to avoid umask at rolling event log folder and appStatusFile creation add dc5ebc2 [SPARK-31171][SQL] size(null) should return null under ansi mode No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/internal/SQLConf.scala | 9 ++--- .../catalyst/expressions/CollectionExpressionsSuite.scala| 6 ++ .../scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala | 12 3 files changed, 24 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31171][SQL] size(null) should return null under ansi mode
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b9371e3 [SPARK-31171][SQL] size(null) should return null under ansi
mode
b9371e3 is described below
commit b9371e38abbc351106453b14172d6919be0eca82
Author: Wenchen Fan
AuthorDate: Tue Mar 17 11:48:54 2020 -0700
[SPARK-31171][SQL] size(null) should return null under ansi mode
Make `size(null)` return null under ANSI mode, regardless of the
`spark.sql.legacy.sizeOfNull` config.
In https://github.com/apache/spark/pull/27834, we change the result of
`size(null)` to be -1 to match the 2.4 behavior and avoid breaking changes.
However, it's true that the "return -1" behavior is error-prone when being
used with aggregate functions. The current ANSI mode controls a bunch of
"better behaviors" like failing on overflow. We don't enable these "better
behaviors" by default because they are too breaking. The "return null" behavior
of `size(null)` is a good fit of the ANSI mode.
No as ANSI mode is off by default.
new tests
Closes #27936 from cloud-fan/null.
Authored-by: Wenchen Fan
Signed-off-by: Dongjoon Hyun
(cherry picked from commit dc5ebc2d5b8122121d89a9175737bea95ae10126)
Signed-off-by: Dongjoon Hyun
---
.../main/scala/org/apache/spark/sql/internal/SQLConf.scala | 9 ++---
.../catalyst/expressions/CollectionExpressionsSuite.scala| 6 ++
.../scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala | 12
3 files changed, 24 insertions(+), 3 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index e49593e..1331350 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1940,8 +1940,8 @@ object SQLConf {
val LEGACY_SIZE_OF_NULL = buildConf("spark.sql.legacy.sizeOfNull")
.internal()
-.doc("If it is set to true, size of null returns -1. This behavior was
inherited from Hive. " +
- "The size function returns null for null input if the flag is disabled.")
+.doc(s"If it is set to false, or ${ANSI_ENABLED.key} is true, then size of
null returns " +
+ "null. Otherwise, it returns -1, which was inherited from Hive.")
.booleanConf
.createWithDefault(true)
@@ -2759,7 +2759,10 @@ class SQLConf extends Serializable with Logging {
def csvColumnPruning: Boolean = getConf(SQLConf.CSV_PARSER_COLUMN_PRUNING)
- def legacySizeOfNull: Boolean = getConf(SQLConf.LEGACY_SIZE_OF_NULL)
+ def legacySizeOfNull: Boolean = {
+// size(null) should return null under ansi mode.
+getConf(SQLConf.LEGACY_SIZE_OF_NULL) && !getConf(ANSI_ENABLED)
+ }
def isReplEagerEvalEnabled: Boolean =
getConf(SQLConf.REPL_EAGER_EVAL_ENABLED)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index 3cfc66f..173f248 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -74,6 +74,12 @@ class CollectionExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper
withSQLConf(SQLConf.LEGACY_SIZE_OF_NULL.key -> "false") {
testSize(sizeOfNull = null)
}
+// size(null) should return null under ansi mode.
+withSQLConf(
+ SQLConf.LEGACY_SIZE_OF_NULL.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ testSize(sizeOfNull = null)
+}
}
test("MapKeys/MapValues") {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index a613c33..c41eb98 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -490,6 +490,12 @@ class DataFrameFunctionsSuite extends QueryTest with
SharedSparkSession {
withSQLConf(SQLConf.LEGACY_SIZE_OF_NULL.key -> "false") {
testSizeOfArray(sizeOfNull = null)
}
+// size(null) should return null under ansi mode.
+withSQLConf(
+ SQLConf.LEGACY_SIZE_OF_NULL.key -> "true",
+ SQLConf.ANSI_ENABLED.key -> "true") {
+ testSize
[spark] branch master updated: [SPARK-31125][K8S] Terminating pods have a deletion timestamp but they are not yet dead
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 57d27e9 [SPARK-31125][K8S] Terminating pods have a deletion timestamp
but they are not yet dead
57d27e9 is described below
commit 57d27e900f79e6c5699b9a23db236aae98e761ad
Author: Holden Karau
AuthorDate: Tue Mar 17 12:04:06 2020 -0700
[SPARK-31125][K8S] Terminating pods have a deletion timestamp but they are
not yet dead
### What changes were proposed in this pull request?
Change what we consider a deleted pod to not include "Terminating"
### Why are the changes needed?
If we get a new snapshot while a pod is in the process of being cleaned up
we shouldn't delete the executor until it is fully terminated.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
This should be covered by the decommissioning tests in that they currently
are flaky because we sometimes delete the executor instead of allowing it to
decommission all the way.
I also ran this in a loop locally ~80 times with the only failures being
the PV suite because of unrelated minikube mount issues.
Closes #27905 from holdenk/SPARK-31125-Processing-state-snapshots-incorrect.
Authored-by: Holden Karau
Signed-off-by: Dongjoon Hyun
---
.../spark/scheduler/cluster/k8s/ExecutorPodStates.scala | 2 ++
.../spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala| 11 ++-
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodStates.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodStates.scala
index 83daddf..34fca29 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodStates.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodStates.scala
@@ -34,4 +34,6 @@ case class PodFailed(pod: Pod) extends FinalPodState
case class PodDeleted(pod: Pod) extends FinalPodState
+case class PodTerminating(pod: Pod) extends FinalPodState
+
case class PodUnknown(pod: Pod) extends ExecutorPodState
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
index 435a5f1..30030ab 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
@@ -64,6 +64,8 @@ object ExecutorPodsSnapshot extends Logging {
PodFailed(pod)
case "succeeded" =>
PodSucceeded(pod)
+case "terminating" =>
+ PodTerminating(pod)
case _ =>
logWarning(s"Received unknown phase $phase for executor pod with
name" +
s" ${pod.getMetadata.getName} in namespace
${pod.getMetadata.getNamespace}")
@@ -72,5 +74,12 @@ object ExecutorPodsSnapshot extends Logging {
}
}
- private def isDeleted(pod: Pod): Boolean =
pod.getMetadata.getDeletionTimestamp != null
+ private def isDeleted(pod: Pod): Boolean = {
+(pod.getMetadata.getDeletionTimestamp != null &&
+ (
+pod.getStatus == null ||
+pod.getStatus.getPhase == null ||
+pod.getStatus.getPhase.toLowerCase(Locale.ROOT) != "terminating"
+ ))
+ }
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31047][SQL] Improve file listing for ViewFileSystem
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4e4e08f [SPARK-31047][SQL] Improve file listing for ViewFileSystem
4e4e08f is described below
commit 4e4e08f372db888797fd23faca88ac02d9466d5a
Author: manuzhang
AuthorDate: Tue Mar 17 14:23:28 2020 -0700
[SPARK-31047][SQL] Improve file listing for ViewFileSystem
### What changes were proposed in this pull request?
Use `listLocatedStatus` when `lnMemoryFileIndex` is listing files from a
`ViewFileSystem` which should delegate to that of `DistributedFileSystem`.
### Why are the changes needed?
When `ViewFileSystem` is used to manage several `DistributedFileSystem`,
the change will improve performance of file listing, especially when there are
many files.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #27801 from manuzhang/spark-31047.
Authored-by: manuzhang
Signed-off-by: Dongjoon Hyun
---
.../execution/datasources/InMemoryFileIndex.scala | 3 ++-
.../sql/execution/datasources/FileIndexSuite.scala | 25 +-
2 files changed, 26 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
index cac2d6e..84160f3 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
@@ -23,6 +23,7 @@ import scala.collection.mutable
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs._
+import org.apache.hadoop.fs.viewfs.ViewFileSystem
import org.apache.hadoop.hdfs.DistributedFileSystem
import org.apache.hadoop.mapred.{FileInputFormat, JobConf}
@@ -313,7 +314,7 @@ object InMemoryFileIndex extends Logging {
// to retrieve the file status with the file block location. The
reason to still fallback
// to listStatus is because the default implementation would
potentially throw a
// FileNotFoundException which is better handled by doing the lookups
manually below.
-case _: DistributedFileSystem if !ignoreLocality =>
+case (_: DistributedFileSystem | _: ViewFileSystem) if !ignoreLocality
=>
val remoteIter = fs.listLocatedStatus(path)
new Iterator[LocatedFileStatus]() {
def next(): LocatedFileStatus = remoteIter.next
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
index 553773e..ea15f18 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
@@ -22,7 +22,11 @@ import java.net.URI
import scala.collection.mutable
-import org.apache.hadoop.fs.{BlockLocation, FileStatus, LocatedFileStatus,
Path, RawLocalFileSystem}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{BlockLocation, FileStatus, LocatedFileStatus,
Path, RawLocalFileSystem, RemoteIterator}
+import org.apache.hadoop.fs.viewfs.ViewFileSystem
+import org.mockito.ArgumentMatchers.any
+import org.mockito.Mockito.{mock, when}
import org.apache.spark.SparkException
import org.apache.spark.metrics.source.HiveCatalogMetrics
@@ -465,6 +469,25 @@ class FileIndexSuite extends SharedSparkSession {
}
}
}
+
+ test("SPARK-31047 - Improve file listing for ViewFileSystem") {
+val path = mock(classOf[Path])
+val dfs = mock(classOf[ViewFileSystem])
+when(path.getFileSystem(any[Configuration])).thenReturn(dfs)
+val statuses =
+ Seq(
+new LocatedFileStatus(
+ new FileStatus(0, false, 0, 100, 0,
+new Path("file")), Array(new BlockLocation()))
+ )
+when(dfs.listLocatedStatus(path)).thenReturn(new
RemoteIterator[LocatedFileStatus] {
+ val iter = statuses.toIterator
+ override def hasNext: Boolean = iter.hasNext
+ override def next(): LocatedFileStatus = iter.next
+})
+val fileIndex = new TestInMemoryFileIndex(spark, path)
+assert(fileIndex.leafFileStatuses.toSeq == statuses)
+ }
}
object DeletionRaceFileSystem {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-30954][ML][R] Make file name the same as class name
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6f0b0f1 [SPARK-30954][ML][R] Make file name the same as class name
6f0b0f1 is described below
commit 6f0b0f1655e5d8fbfb20ed0a58b740e6c9a6ae50
Author: Qianyang Yu
AuthorDate: Tue Mar 17 16:15:02 2020 -0700
[SPARK-30954][ML][R] Make file name the same as class name
This pr solved the same issue as
[pr27919](https://github.com/apache/spark/pull/27919), but this one changes the
file names based on comment from previous pr.
### What changes were proposed in this pull request?
Make some of file names the same as class name in R package.
### Why are the changes needed?
Make the file consistence
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
run `./R/run-tests.sh`
Closes #27940 from kevinyu98/spark-30954-r-v2.
Authored-by: Qianyang Yu
Signed-off-by: Dongjoon Hyun
---
...reeClassificationWrapper.scala => DecisionTreeClassifierWrapper.scala} | 0
...sionTreeRegressionWrapper.scala => DecisionTreeRegressorWrapper.scala} | 0
.../ml/r/{GBTClassificationWrapper.scala => GBTClassifierWrapper.scala} | 0
.../spark/ml/r/{GBTRegressionWrapper.scala => GBTRegressorWrapper.scala} | 0
...estClassificationWrapper.scala => RandomForestClassifierWrapper.scala} | 0
...omForestRegressionWrapper.scala => RandomForestRegressorWrapper.scala} | 0
6 files changed, 0 insertions(+), 0 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassificationWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressionWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressionWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
rename to mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressionWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressionWrapper.scala
rename to mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressionWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressionWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30954][ML][R] Make file name the same as class name
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a7c2525 [SPARK-30954][ML][R] Make file name the same as class name
a7c2525 is described below
commit a7c25259c97085f6518f88fbcc3f0f4bde2995af
Author: Qianyang Yu
AuthorDate: Tue Mar 17 16:15:02 2020 -0700
[SPARK-30954][ML][R] Make file name the same as class name
This pr solved the same issue as
[pr27919](https://github.com/apache/spark/pull/27919), but this one changes the
file names based on comment from previous pr.
### What changes were proposed in this pull request?
Make some of file names the same as class name in R package.
### Why are the changes needed?
Make the file consistence
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
run `./R/run-tests.sh`
Closes #27940 from kevinyu98/spark-30954-r-v2.
Authored-by: Qianyang Yu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 6f0b0f1655e5d8fbfb20ed0a58b740e6c9a6ae50)
Signed-off-by: Dongjoon Hyun
---
...reeClassificationWrapper.scala => DecisionTreeClassifierWrapper.scala} | 0
...sionTreeRegressionWrapper.scala => DecisionTreeRegressorWrapper.scala} | 0
.../ml/r/{GBTClassificationWrapper.scala => GBTClassifierWrapper.scala} | 0
.../spark/ml/r/{GBTRegressionWrapper.scala => GBTRegressorWrapper.scala} | 0
...estClassificationWrapper.scala => RandomForestClassifierWrapper.scala} | 0
...omForestRegressionWrapper.scala => RandomForestRegressorWrapper.scala} | 0
6 files changed, 0 insertions(+), 0 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassificationWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressionWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressionWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/GBTClassificationWrapper.scala
rename to mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressionWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressionWrapper.scala
rename to mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassificationWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressionWrapper.scala
b/mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala
similarity index 100%
rename from
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressionWrapper.scala
rename to
mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac262cb -> c6a6d5e)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac262cb [SPARK-30292][SQL][FOLLOWUP] ansi cast from strings to integral numbers (byte/short/int/long) should fail with fraction add c6a6d5e Revert "[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir" No new revisions were added by this update. Summary of changes: .../apache/spark/sql/internal/SharedState.scala| 80 +++--- .../sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 - .../spark/sql/hive/thriftserver/CliSuite.scala | 12 .../spark/sql/hive/HiveSharedStateSuite.scala | 1 + .../spark/sql/hive/HiveSparkSubmitSuite.scala | 2 +- 5 files changed, 42 insertions(+), 55 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac262cb -> c6a6d5e)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac262cb [SPARK-30292][SQL][FOLLOWUP] ansi cast from strings to integral numbers (byte/short/int/long) should fail with fraction add c6a6d5e Revert "[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir" No new revisions were added by this update. Summary of changes: .../apache/spark/sql/internal/SharedState.scala| 80 +++--- .../sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 - .../spark/sql/hive/thriftserver/CliSuite.scala | 12 .../spark/sql/hive/HiveSharedStateSuite.scala | 1 + .../spark/sql/hive/HiveSparkSubmitSuite.scala | 2 +- 5 files changed, 42 insertions(+), 55 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: Revert "[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir"
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 476aaee Revert "[SPARK-31170][SQL] Spark SQL Cli should respect
hive-site.xml and spark.sql.warehouse.dir"
476aaee is described below
commit 476aaee7089d051dae49d2a44de69df3b79248a0
Author: Dongjoon Hyun
AuthorDate: Thu Mar 19 16:13:50 2020 -0700
Revert "[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and
spark.sql.warehouse.dir"
This reverts commit 321341a4c3104380035350631c82a4b385f117e4.
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/internal/SharedState.scala| 80 +++---
.../sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 -
.../spark/sql/hive/thriftserver/CliSuite.scala | 12
.../spark/sql/hive/HiveSharedStateSuite.scala | 1 +
.../spark/sql/hive/HiveSparkSubmitSuite.scala | 2 +-
5 files changed, 42 insertions(+), 55 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
b/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
index eb74e96..5347264 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
@@ -41,6 +41,7 @@ import
org.apache.spark.sql.streaming.ui.{StreamingQueryStatusListener, Streamin
import org.apache.spark.status.ElementTrackingStore
import org.apache.spark.util.Utils
+
/**
* A class that holds all state shared across sessions in a given
[[SQLContext]].
*
@@ -54,10 +55,45 @@ private[sql] class SharedState(
SharedState.setFsUrlStreamHandlerFactory(sparkContext.conf)
+ // Load hive-site.xml into hadoopConf and determine the warehouse path we
want to use, based on
+ // the config from both hive and Spark SQL. Finally set the warehouse config
value to sparkConf.
+ val warehousePath: String = {
+val configFile =
Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
+if (configFile != null) {
+ logInfo(s"loading hive config file: $configFile")
+ sparkContext.hadoopConfiguration.addResource(configFile)
+}
+
+// hive.metastore.warehouse.dir only stay in hadoopConf
+sparkContext.conf.remove("hive.metastore.warehouse.dir")
+// Set the Hive metastore warehouse path to the one we use
+val hiveWarehouseDir =
sparkContext.hadoopConfiguration.get("hive.metastore.warehouse.dir")
+if (hiveWarehouseDir != null &&
!sparkContext.conf.contains(WAREHOUSE_PATH.key)) {
+ // If hive.metastore.warehouse.dir is set and spark.sql.warehouse.dir is
not set,
+ // we will respect the value of hive.metastore.warehouse.dir.
+ sparkContext.conf.set(WAREHOUSE_PATH.key, hiveWarehouseDir)
+ logInfo(s"${WAREHOUSE_PATH.key} is not set, but
hive.metastore.warehouse.dir " +
+s"is set. Setting ${WAREHOUSE_PATH.key} to the value of " +
+s"hive.metastore.warehouse.dir ('$hiveWarehouseDir').")
+ hiveWarehouseDir
+} else {
+ // If spark.sql.warehouse.dir is set, we will override
hive.metastore.warehouse.dir using
+ // the value of spark.sql.warehouse.dir.
+ // When neither spark.sql.warehouse.dir nor hive.metastore.warehouse.dir
is set,
+ // we will set hive.metastore.warehouse.dir to the default value of
spark.sql.warehouse.dir.
+ val sparkWarehouseDir = sparkContext.conf.get(WAREHOUSE_PATH)
+ logInfo(s"Setting hive.metastore.warehouse.dir ('$hiveWarehouseDir') to
the value of " +
+s"${WAREHOUSE_PATH.key} ('$sparkWarehouseDir').")
+ sparkContext.hadoopConfiguration.set("hive.metastore.warehouse.dir",
sparkWarehouseDir)
+ sparkWarehouseDir
+}
+ }
+ logInfo(s"Warehouse path is '$warehousePath'.")
+
+ // These 2 variables should be initiated after `warehousePath`, because in
the first place we need
+ // to load hive-site.xml into hadoopConf and determine the warehouse path
which will be set into
+ // both spark conf and hadoop conf avoiding be affected by any SparkSession
level options
private val (conf, hadoopConf) = {
-// Load hive-site.xml into hadoopConf and determine the warehouse path
which will be set into
-// both spark conf and hadoop conf avoiding be affected by any
SparkSession level options
-SharedState.loadHiveConfFile(sparkContext.conf,
sparkContext.hadoopConfiguration)
val confClone = sparkContext.conf.clone()
val hadoopConfClone = new Configuration(sparkContext.hadoopConfiguration)
// If `SparkSession` is instantiated using an existing `SparkContext`
instance and no existing
@@ -130,7 +166,7 @@
[spark] branch master updated (c6a6d5e -> ca499e9)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c6a6d5e Revert "[SPARK-31170][SQL] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir" add ca499e9 [SPARK-25121][SQL] Supports multi-part table names for broadcast hint resolution No new revisions were added by this update. Summary of changes: .../sql/catalyst/analysis/HintErrorLogger.scala| 7 +- .../spark/sql/catalyst/analysis/ResolveHints.scala | 71 +++- .../spark/sql/catalyst/expressions/package.scala | 2 +- .../spark/sql/catalyst/plans/logical/hints.scala | 3 +- .../spark/sql/catalyst/analysis/AnalysisTest.scala | 2 + .../sql/catalyst/analysis/ResolveHintsSuite.scala | 48 +++ .../sql/catalyst/analysis/TestRelations.scala | 2 + .../org/apache/spark/sql/DataFrameJoinSuite.scala | 98 +- .../spark/sql/execution/GlobalTempViewSuite.scala | 24 +- 9 files changed, 230 insertions(+), 27 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-25121][SQL] Supports multi-part table names for broadcast hint resolution
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 7d4c1b8 [SPARK-25121][SQL] Supports multi-part table names for
broadcast hint resolution
7d4c1b8 is described below
commit 7d4c1b894ef32170b421f51c42cf30198c35c21b
Author: Takeshi Yamamuro
AuthorDate: Thu Mar 19 20:11:04 2020 -0700
[SPARK-25121][SQL] Supports multi-part table names for broadcast hint
resolution
### What changes were proposed in this pull request?
This pr fixed code to respect a database name for broadcast table hint
resolution.
Currently, spark ignores a database name in multi-part names;
```
scala> sql("CREATE DATABASE testDb")
scala> spark.range(10).write.saveAsTable("testDb.t")
// without this patch
scala> spark.range(10).join(spark.table("testDb.t"),
"id").hint("broadcast", "testDb.t").explain
== Physical Plan ==
*(2) Project [id#24L]
+- *(2) BroadcastHashJoin [id#24L], [id#26L], Inner, BuildLeft
:- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint,
false]))
: +- *(1) Range (0, 10, step=1, splits=4)
+- *(2) Project [id#26L]
+- *(2) Filter isnotnull(id#26L)
+- *(2) FileScan parquet testdb.t[id#26L] Batched: true, Format:
Parquet, Location:
InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-2.3.1-bin-hadoop2.7/spark-warehouse...,
PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema:
struct
// with this patch
scala> spark.range(10).join(spark.table("testDb.t"),
"id").hint("broadcast", "testDb.t").explain
== Physical Plan ==
*(2) Project [id#3L]
+- *(2) BroadcastHashJoin [id#3L], [id#5L], Inner, BuildRight
:- *(2) Range (0, 10, step=1, splits=4)
+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint,
true]))
+- *(1) Project [id#5L]
+- *(1) Filter isnotnull(id#5L)
+- *(1) FileScan parquet testdb.t[id#5L] Batched: true, Format:
Parquet, Location:
InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-master/spark-warehouse/testdb.db/t],
PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema:
struct
```
This PR comes from https://github.com/apache/spark/pull/22198
### Why are the changes needed?
For better usability.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added unit tests.
Closes #27935 from maropu/SPARK-25121-2.
Authored-by: Takeshi Yamamuro
Signed-off-by: Dongjoon Hyun
(cherry picked from commit ca499e94091ae62a6ee76ea779d7b2b4cf2dbc5c)
Signed-off-by: Dongjoon Hyun
---
.../sql/catalyst/analysis/HintErrorLogger.scala| 7 +-
.../spark/sql/catalyst/analysis/ResolveHints.scala | 71 +++-
.../spark/sql/catalyst/expressions/package.scala | 2 +-
.../spark/sql/catalyst/plans/logical/hints.scala | 3 +-
.../spark/sql/catalyst/analysis/AnalysisTest.scala | 2 +
.../sql/catalyst/analysis/ResolveHintsSuite.scala | 48 +++
.../sql/catalyst/analysis/TestRelations.scala | 2 +
.../org/apache/spark/sql/DataFrameJoinSuite.scala | 98 +-
.../spark/sql/execution/GlobalTempViewSuite.scala | 24 +-
9 files changed, 230 insertions(+), 27 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
index c6e0c74..71c6d40 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
@@ -24,15 +24,16 @@ import
org.apache.spark.sql.catalyst.plans.logical.{HintErrorHandler, HintInfo}
* The hint error handler that logs warnings for each hint error.
*/
object HintErrorLogger extends HintErrorHandler with Logging {
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
override def hintNotRecognized(name: String, parameters: Seq[Any]): Unit = {
logWarning(s"Unrecognized hint: ${hintToPrettyString(name, parameters)}")
}
override def hintRelationsNotFound(
- name: String, parameters: Seq[Any], invalidRelations: Set[String]): Unit
= {
-invalidRelations.foreach { n =>
- logWarning(s"Count not find relation '$n' specified in hint " +
+ name: String, parameters: Seq[Any], invalidRelations: Set[Seq[String]]):
Unit = {
+invalidRelations.foreach { ident =>
+ logWarning(s"Count not find relati
[spark] branch branch-3.0 updated: [SPARK-31181][SQL][TESTS] Remove the default value assumption on CREATE TABLE test cases
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c42f9f6 [SPARK-31181][SQL][TESTS] Remove the default value assumption
on CREATE TABLE test cases
c42f9f6 is described below
commit c42f9f61f40d4d796413ab2bfb58fbeff5ceb68b
Author: Dongjoon Hyun
AuthorDate: Fri Mar 20 12:28:57 2020 +0800
[SPARK-31181][SQL][TESTS] Remove the default value assumption on CREATE
TABLE test cases
A few `CREATE TABLE` test cases have some assumption on the default value
of `LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED`. This PR (SPARK-31181) makes
the test cases more explicit from test-case side.
The configuration change was tested via
https://github.com/apache/spark/pull/27894 during discussing SPARK-31136. This
PR has only the test case part from that PR.
This makes our test case more robust in terms of the default value of
`LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED`. Even in the case where we switch
the conf value, that will be one-liner with no test case changes.
No.
Pass the Jenkins with the existing tests.
Closes #27946 from dongjoon-hyun/SPARK-EXPLICIT-TEST.
Authored-by: Dongjoon Hyun
Signed-off-by: Wenchen Fan
(cherry picked from commit f1cc86792f825f76c2689660764d62d7be0d1989)
Signed-off-by: Dongjoon Hyun
---
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 27 --
.../sql-tests/inputs/describe-table-column.sql | 2 +-
.../sql-tests/inputs/postgreSQL/create_view.sql| 2 +-
.../results/describe-table-column.sql.out | 2 +-
.../results/postgreSQL/create_view.sql.out | 2 +-
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 2 ++
.../sql/execution/command/DDLParserSuite.scala | 3 ++-
.../apache/spark/sql/hive/StatisticsSuite.scala| 4 +++-
8 files changed, 26 insertions(+), 18 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
index e3570899..35a54c8 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
@@ -26,6 +26,7 @@ import org.apache.spark.sql.catalyst.expressions.{EqualTo,
Literal}
import org.apache.spark.sql.catalyst.plans.logical._
import
org.apache.spark.sql.connector.catalog.TableChange.ColumnPosition.{after, first}
import org.apache.spark.sql.connector.expressions.{ApplyTransform,
BucketTransform, DaysTransform, FieldReference, HoursTransform,
IdentityTransform, LiteralValue, MonthsTransform, Transform, YearsTransform}
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{IntegerType, LongType, StringType,
StructType, TimestampType}
import org.apache.spark.unsafe.types.UTF8String
@@ -2117,18 +2118,20 @@ class DDLParserSuite extends AnalysisTest {
}
test("create table - without using") {
-val sql = "CREATE TABLE 1m.2g(a INT)"
-val expectedTableSpec = TableSpec(
- Seq("1m", "2g"),
- Some(new StructType().add("a", IntegerType)),
- Seq.empty[Transform],
- None,
- Map.empty[String, String],
- None,
- Map.empty[String, String],
- None,
- None)
+withSQLConf(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key ->
"false") {
+ val sql = "CREATE TABLE 1m.2g(a INT)"
+ val expectedTableSpec = TableSpec(
+Seq("1m", "2g"),
+Some(new StructType().add("a", IntegerType)),
+Seq.empty[Transform],
+None,
+Map.empty[String, String],
+None,
+Map.empty[String, String],
+None,
+None)
-testCreateOrReplaceDdl(sql, expectedTableSpec, expectedIfNotExists = false)
+ testCreateOrReplaceDdl(sql, expectedTableSpec, expectedIfNotExists =
false)
+}
}
}
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/describe-table-column.sql
b/sql/core/src/test/resources/sql-tests/inputs/describe-table-column.sql
index 821cb47..d55e398 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/describe-table-column.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/describe-table-column.sql
@@ -52,7 +52,7 @@ DROP TABLE desc_complex_col_table;
--Test case insensitive
-CREATE TABLE customer(CName STRING);
+CREATE TABLE customer(CName STRING) USING PARQUET;
INSERT INTO customer VALUES('Maria');
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/postgreSQL/create_view.sql
b/sql/core/src/test/resources/sql-tests/inputs/postgreSQL/create_view.sql
index 39e7084..21f
[spark] branch master updated: [SPARK-31184][SQL] Support getTablesByType API of Hive Client
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3a48ea1 [SPARK-31184][SQL] Support getTablesByType API of Hive Client
3a48ea1 is described below
commit 3a48ea1fe0fb85253f12d86caea01ffcb7e678d0
Author: Eric Wu <492960...@qq.com>
AuthorDate: Sat Mar 21 17:41:23 2020 -0700
[SPARK-31184][SQL] Support getTablesByType API of Hive Client
### What changes were proposed in this pull request?
Hive 2.3+ supports `getTablesByType` API, which will provide an efficient
way to get HiveTable with specific type. Now, we have following mappings when
using `HiveExternalCatalog`.
```
CatalogTableType.EXTERNAL => HiveTableType.EXTERNAL_TABLE
CatalogTableType.MANAGED => HiveTableType.MANAGED_TABLE
CatalogTableType.VIEW => HiveTableType.VIRTUAL_VIEW
```
Without this API, we need to achieve the goal by `getTables` +
`getTablesByName` + `filter with type`.
This PR add `getTablesByType` in `HiveShim`. For those hive versions don't
support this API, `UnsupportedOperationException` will be thrown. And the upper
logic should catch the exception and fallback to the filter solution mentioned
above.
Since the JDK11 related fix in `Hive` is not released yet, manual tests
against hive 2.3.7-SNAPSHOT is done by following the instructions of
SPARK-29245.
### Why are the changes needed?
This API will provide better usability and performance if we want to get a
list of hiveTables with specific type. For example `HiveTableType.VIRTUAL_VIEW`
corresponding to `CatalogTableType.VIEW`.
### Does this PR introduce any user-facing change?
No, this is a support function.
### How was this patch tested?
Add tests in VersionsSuite and manually run JDK11 test with following
settings:
- Hive 2.3.6 Metastore on JDK8
- Hive 2.3.7-SNAPSHOT library build from source of Hive 2.3 branch
- Spark build with Hive 2.3.7-SNAPSHOT on jdk-11.0.6
Closes #27952 from Eric5553/GetTableByType.
Authored-by: Eric Wu <492960...@qq.com>
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/hive/client/HiveClient.scala | 9 +
.../spark/sql/hive/client/HiveClientImpl.scala | 42 --
.../apache/spark/sql/hive/client/HiveShim.scala| 35 +-
.../spark/sql/hive/client/VersionsSuite.scala | 22 +---
4 files changed, 92 insertions(+), 16 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
index e31dffa..3ea80ea 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
@@ -61,6 +61,15 @@ private[hive] trait HiveClient {
/** Returns the names of tables in the given database that matches the given
pattern. */
def listTables(dbName: String, pattern: String): Seq[String]
+ /**
+ * Returns the names of tables with specific tableType in the given database
that matches
+ * the given pattern.
+ */
+ def listTablesByType(
+ dbName: String,
+ pattern: String,
+ tableType: CatalogTableType): Seq[String]
+
/** Sets the name of current database. */
def setCurrentDatabase(databaseName: String): Unit
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 4a3e813..6ad5e9d 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -755,6 +755,22 @@ private[hive] class HiveClientImpl(
client.getTablesByPattern(dbName, pattern).asScala
}
+ override def listTablesByType(
+ dbName: String,
+ pattern: String,
+ tableType: CatalogTableType): Seq[String] = withHiveState {
+try {
+ // Try with Hive API getTablesByType first, it's supported from Hive
2.3+.
+ shim.getTablesByType(client, dbName, pattern, toHiveTableType(tableType))
+} catch {
+ case _: UnsupportedOperationException =>
+// Fallback to filter logic if getTablesByType not supported.
+val tableNames = client.getTablesByPattern(dbName, pattern).asScala
+val tables = getTablesByName(dbName, tableNames).filter(_.tableType ==
tableType)
+tables.map(_.identifier.table)
+}
+ }
+
/**
* Runs the specified SQL query using Hive.
*/
@@ -1011,25 +1027,29 @@ private[hive] object HiveClientImpl extends Logging {
private def toOutputFormat(name: String) =
Utils.classForName[org.apache.hadoop.hive.q
[spark] branch master updated (fae981e -> 3a48ea1)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from fae981e [SPARK-30773][ML] Support NativeBlas for level-1 routines add 3a48ea1 [SPARK-31184][SQL] Support getTablesByType API of Hive Client No new revisions were added by this update. Summary of changes: .../apache/spark/sql/hive/client/HiveClient.scala | 9 + .../spark/sql/hive/client/HiveClientImpl.scala | 42 -- .../apache/spark/sql/hive/client/HiveShim.scala| 35 +- .../spark/sql/hive/client/VersionsSuite.scala | 22 +--- 4 files changed, 92 insertions(+), 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-30715][K8S][TESTS][FOLLOWUP] Update k8s client version in IT as well
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3799d2b [SPARK-30715][K8S][TESTS][FOLLOWUP] Update k8s client version
in IT as well
3799d2b is described below
commit 3799d2b9d842f4b9f4e78bf701f5e123f0061bad
Author: Prashant Sharma
AuthorDate: Sat Mar 21 18:26:53 2020 -0700
[SPARK-30715][K8S][TESTS][FOLLOWUP] Update k8s client version in IT as well
### What changes were proposed in this pull request?
This is a follow up for SPARK-30715 . Kubernetes client version in sync in
integration-tests and kubernetes/core
### Why are the changes needed?
More than once, the kubernetes client version has gone out of sync between
integration tests and kubernetes/core. So brought them up in sync again and
added a comment to save us from future need of this additional followup.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Manually.
Closes #27948 from ScrapCodes/follow-up-spark-30715.
Authored-by: Prashant Sharma
Signed-off-by: Dongjoon Hyun
---
resource-managers/kubernetes/core/pom.xml | 1 +
resource-managers/kubernetes/integration-tests/pom.xml| 2 +-
.../apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala | 8
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/resource-managers/kubernetes/core/pom.xml
b/resource-managers/kubernetes/core/pom.xml
index 18793de..b527816 100644
--- a/resource-managers/kubernetes/core/pom.xml
+++ b/resource-managers/kubernetes/core/pom.xml
@@ -29,6 +29,7 @@
Spark Project Kubernetes
kubernetes
+
4.7.1
diff --git a/resource-managers/kubernetes/integration-tests/pom.xml
b/resource-managers/kubernetes/integration-tests/pom.xml
index a522e87..7a889c4 100644
--- a/resource-managers/kubernetes/integration-tests/pom.xml
+++ b/resource-managers/kubernetes/integration-tests/pom.xml
@@ -29,7 +29,7 @@
1.3.0
1.4.0
-4.6.4
+4.7.1
3.2.2
1.0
kubernetes-integration-tests
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
index eaaf67d..4de7e70 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
@@ -61,15 +61,15 @@ class KubernetesSuite extends SparkFunSuite
protected var appLocator: String = _
// Default memory limit is 1024M + 384M (minimum overhead constant)
- private val baseMemory = s"${1024 + 384}Mi"
+ private val baseMemory = s"${1024 + 384}"
protected val memOverheadConstant = 0.8
- private val standardNonJVMMemory = s"${(1024 + 0.4*1024).toInt}Mi"
+ private val standardNonJVMMemory = s"${(1024 + 0.4*1024).toInt}"
protected val additionalMemory = 200
// 209715200 is 200Mi
protected val additionalMemoryInBytes = 209715200
- private val extraDriverTotalMemory = s"${(1024 +
memOverheadConstant*1024).toInt}Mi"
+ private val extraDriverTotalMemory = s"${(1024 +
memOverheadConstant*1024).toInt}"
private val extraExecTotalMemory =
-s"${(1024 + memOverheadConstant*1024 + additionalMemory).toInt}Mi"
+s"${(1024 + memOverheadConstant*1024 + additionalMemory).toInt}"
/**
* Build the image ref for the given image name, taking the repo and tag
from the
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3799d2b -> bf342ba)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3799d2b [SPARK-30715][K8S][TESTS][FOLLOWUP] Update k8s client version in IT as well add bf342ba [SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with testRetry No new revisions were added by this update. Summary of changes: external/kafka-0-10-sql/src/test/resources/log4j.properties | 4 +++- .../org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala | 2 +- .../src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala | 4 project/SparkBuild.scala | 3 ++- 4 files changed, 10 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with testRetry
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 4a5f695 [SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with
testRetry
4a5f695 is described below
commit 4a5f6955ba9667298bf13e5cdaa7703d5389b083
Author: Gabor Somogyi
AuthorDate: Sat Mar 21 18:59:29 2020 -0700
[SPARK-30541][TESTS] Implement KafkaDelegationTokenSuite with testRetry
### What changes were proposed in this pull request?
`KafkaDelegationTokenSuite` has been ignored because showed flaky
behaviour. In this PR I've changed the approach how the test executed and
turning it on again. This PR contains the following:
* The test runs in separate JVM in order to avoid modified security context
* The body of the test runs in `testRetry` which reties if failed
* Additional logs to analyse possible failures
* Enhanced clean-up code
### Why are the changes needed?
`KafkaDelegationTokenSuite ` is ignored.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Executed the test in loop 1k+ times in jenkins (locally much harder to
reproduce).
Closes #27877 from gaborgsomogyi/SPARK-30541.
Authored-by: Gabor Somogyi
Signed-off-by: Dongjoon Hyun
(cherry picked from commit bf342bafa81738a47d511d3aa02812a1ccc0ecab)
Signed-off-by: Dongjoon Hyun
---
external/kafka-0-10-sql/src/test/resources/log4j.properties | 4 +++-
.../org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala | 2 +-
.../src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala | 4
project/SparkBuild.scala | 3 ++-
4 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/external/kafka-0-10-sql/src/test/resources/log4j.properties
b/external/kafka-0-10-sql/src/test/resources/log4j.properties
index 75e3b53..daf0572 100644
--- a/external/kafka-0-10-sql/src/test/resources/log4j.properties
+++ b/external/kafka-0-10-sql/src/test/resources/log4j.properties
@@ -25,4 +25,6 @@ log4j.appender.file.layout.ConversionPattern=%d{yy/MM/dd
HH:mm:ss.SSS} %t %p %c{
# Ignore messages below warning level from Jetty, because it's a bit verbose
log4j.logger.org.spark-project.jetty=WARN
-
+log4j.logger.org.apache.spark.sql.kafka010.KafkaTestUtils=DEBUG
+log4j.logger.org.apache.directory.server.kerberos.kdc.authentication=DEBUG
+log4j.logger.org.apache.directory.server.core.DefaultDirectoryService=DEBUG
diff --git
a/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala
b/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala
index 79239e5..702bd4f 100644
---
a/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala
+++
b/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDelegationTokenSuite.scala
@@ -62,7 +62,7 @@ class KafkaDelegationTokenSuite extends StreamTest with
SharedSparkSession with
}
}
- ignore("Roundtrip") {
+ testRetry("Roundtrip", 3) {
val hadoopConf = new Configuration()
val manager = new HadoopDelegationTokenManager(spark.sparkContext.conf,
hadoopConf, null)
val credentials = new Credentials()
diff --git
a/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
b/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
index 7b972fe..c1ca557 100644
---
a/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
+++
b/external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
@@ -170,6 +170,7 @@ class KafkaTestUtils(
kdc.getKrb5conf.delete()
Files.write(krb5confStr, kdc.getKrb5conf, StandardCharsets.UTF_8)
+logDebug(s"krb5.conf file content: $krb5confStr")
}
private def addedKrb5Config(key: String, value: String): String = {
@@ -309,6 +310,7 @@ class KafkaTestUtils(
}
brokerReady = false
zkReady = false
+kdcReady = false
if (producer != null) {
producer.close()
@@ -317,6 +319,7 @@ class KafkaTestUtils(
if (adminClient != null) {
adminClient.close()
+ adminClient = null
}
if (server != null) {
@@ -351,6 +354,7 @@ class KafkaTestUtils(
Configuration.getConfiguration.refresh()
if (kdc != null) {
kdc.stop()
+ kdc = null
}
UserGroupInformation.reset()
teardownKrbDebug()
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 1a2a7c3..3889013 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -479,7 +4
[spark] branch master updated: [SPARK-31101][BUILD] Upgrade Janino to 3.0.16
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f55f6b5 [SPARK-31101][BUILD] Upgrade Janino to 3.0.16 f55f6b5 is described below commit f55f6b569beea3636549f8a71949cd2bca2a813b Author: Jungtaek Lim (HeartSaVioR) AuthorDate: Sat Mar 21 19:10:23 2020 -0700 [SPARK-31101][BUILD] Upgrade Janino to 3.0.16 ### What changes were proposed in this pull request? This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released recently. * Merged pull request janino-compiler/janino#114 "Grow the code for relocatables, and do fixup, and relocate". Please see the commit log. - https://github.com/janino-compiler/janino/commits/3.0.16 You can see the changelog from the link: http://janino-compiler.github.io/janino/changelog.html / though release note for Janino 3.0.16 is actually incorrect. ### Why are the changes needed? We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: janino-compiler/janino#113 It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details. Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable Janino to succeed to compile user's query properly. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing UTs. Closes #27932 from HeartSaVioR/SPARK-31101-janino-3.0.16. Authored-by: Jungtaek Lim (HeartSaVioR) Signed-off-by: Dongjoon Hyun --- dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 4 ++-- dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 4 ++-- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 4 ++-- pom.xml | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-1.2 b/dev/deps/spark-deps-hadoop-2.7-hive-1.2 index 6ab937f..8f4d48f 100644 --- a/dev/deps/spark-deps-hadoop-2.7-hive-1.2 +++ b/dev/deps/spark-deps-hadoop-2.7-hive-1.2 @@ -35,7 +35,7 @@ commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar commons-cli/1.2//commons-cli-1.2.jar commons-codec/1.10//commons-codec-1.10.jar commons-collections/3.2.2//commons-collections-3.2.2.jar -commons-compiler/3.0.15//commons-compiler-3.0.15.jar +commons-compiler/3.0.16//commons-compiler-3.0.16.jar commons-compress/1.8.1//commons-compress-1.8.1.jar commons-configuration/1.6//commons-configuration-1.6.jar commons-crypto/1.0.0//commons-crypto-1.0.0.jar @@ -105,7 +105,7 @@ jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar -janino/3.0.15//janino-3.0.15.jar +janino/3.0.16//janino-3.0.16.jar javassist/3.25.0-GA//javassist-3.25.0-GA.jar javax.inject/1//javax.inject-1.jar javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar diff --git a/dev/deps/spark-deps-hadoop-2.7-hive-2.3 b/dev/deps/spark-deps-hadoop-2.7-hive-2.3 index 6f56381..3b1d3ad 100644 --- a/dev/deps/spark-deps-hadoop-2.7-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-2.7-hive-2.3 @@ -33,7 +33,7 @@ commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar commons-cli/1.2//commons-cli-1.2.jar commons-codec/1.10//commons-codec-1.10.jar commons-collections/3.2.2//commons-collections-3.2.2.jar -commons-compiler/3.0.15//commons-compiler-3.0.15.jar +commons-compiler/3.0.16//commons-compiler-3.0.16.jar commons-compress/1.8.1//commons-compress-1.8.1.jar commons-configuration/1.6//commons-configuration-1.6.jar commons-crypto/1.0.0//commons-crypto-1.0.0.jar @@ -118,7 +118,7 @@ jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar -janino/3.0.15//janino-3.0.15.jar +janino/3.0.16//janino-3.0.16.jar javassist/3.25.0-GA//javassist-3.25.0-GA.jar javax.inject/1//javax.inject-1.jar javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar diff --git a/dev/deps/spark-deps-hadoop-3.2-hive-2.3 b/dev/deps/spark-deps-hadoop-3.2-hive-2.3 index 90cbd02..3486ed1 100644 --- a/dev/deps/spark-deps-hadoop-3.2-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3.2-hive-2.3 @@ -30,7 +30,7 @@ commons-beanutils/1.9.4//commons-beanutils-1.9.4.jar commons-cli/1.2//commons-cli-1.2.jar commons-codec/1.10//commons-codec-1.10.jar commons-collections/3.2.2//commons-collections-3.2.2.jar -commons-compiler/3.0.15//commons-compiler-3.0.15.jar +commons-compiler/3.0.16//commons-compiler-3.0.16.jar commons-compress/1.8.1//commons-compress-1.8.1.jar commons-configuration2/2.1.1//commons-conf
[spark] branch branch-3.0 updated: Revert "[SPARK-31183][SQL][FOLLOWUP] Move rebase tests to `AvroSuite` and check the rebase flag out of function bodies"
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new f0bfdc5 Revert "[SPARK-31183][SQL][FOLLOWUP] Move rebase tests to
`AvroSuite` and check the rebase flag out of function bodies"
f0bfdc5 is described below
commit f0bfdc513a15884de8f3ffc79cc1845991082642
Author: Dongjoon Hyun
AuthorDate: Sat Mar 21 20:52:04 2020 -0700
Revert "[SPARK-31183][SQL][FOLLOWUP] Move rebase tests to `AvroSuite` and
check the rebase flag out of function bodies"
This reverts commit a6f3e3b096e2d7a39e0b2fdec6452e6d633baf7e.
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/avro/AvroDeserializer.scala | 21 ++--
.../org/apache/spark/sql/avro/AvroSerializer.scala | 18 +--
.../spark/sql/avro/AvroLogicalTypeSuite.scala | 98 +++-
.../org/apache/spark/sql/avro/AvroSuite.scala | 124 +++--
4 files changed, 130 insertions(+), 131 deletions(-)
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
index 3e8a7f9..b98f303 100644
---
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
+++
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
@@ -106,22 +106,21 @@ class AvroDeserializer(rootAvroType: Schema,
rootCatalystType: DataType) {
case (LONG, TimestampType) => avroType.getLogicalType match {
// For backward compatibility, if the Avro type is Long and it is not
logical type
// (the `null` case), the value is processed as timestamp type with
millisecond precision.
-case null | _: TimestampMillis if rebaseDateTime => (updater, ordinal,
value) =>
- val millis = value.asInstanceOf[Long]
- val micros = DateTimeUtils.fromMillis(millis)
- val rebasedMicros =
DateTimeUtils.rebaseJulianToGregorianMicros(micros)
- updater.setLong(ordinal, rebasedMicros)
case null | _: TimestampMillis => (updater, ordinal, value) =>
val millis = value.asInstanceOf[Long]
val micros = DateTimeUtils.fromMillis(millis)
- updater.setLong(ordinal, micros)
-case _: TimestampMicros if rebaseDateTime => (updater, ordinal, value)
=>
- val micros = value.asInstanceOf[Long]
- val rebasedMicros =
DateTimeUtils.rebaseJulianToGregorianMicros(micros)
- updater.setLong(ordinal, rebasedMicros)
+ if (rebaseDateTime) {
+updater.setLong(ordinal,
DateTimeUtils.rebaseJulianToGregorianMicros(micros))
+ } else {
+updater.setLong(ordinal, micros)
+ }
case _: TimestampMicros => (updater, ordinal, value) =>
val micros = value.asInstanceOf[Long]
- updater.setLong(ordinal, micros)
+ if (rebaseDateTime) {
+updater.setLong(ordinal,
DateTimeUtils.rebaseJulianToGregorianMicros(micros))
+ } else {
+updater.setLong(ordinal, micros)
+ }
case other => throw new IncompatibleSchemaException(
s"Cannot convert Avro logical type ${other} to Catalyst Timestamp
type.")
}
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
index 68df7c0..af9e3a5 100644
---
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
+++
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala
@@ -149,15 +149,17 @@ class AvroSerializer(rootCatalystType: DataType,
rootAvroType: Schema, nullable:
case (TimestampType, LONG) => avroType.getLogicalType match {
// For backward compatibility, if the Avro type is Long and it is
not logical type
// (the `null` case), output the timestamp value as with millisecond
precision.
- case null | _: TimestampMillis if rebaseDateTime => (getter,
ordinal) =>
-val micros = getter.getLong(ordinal)
-val rebasedMicros =
DateTimeUtils.rebaseGregorianToJulianMicros(micros)
-DateTimeUtils.fromMillis(rebasedMicros)
case null | _: TimestampMillis => (getter, ordinal) =>
-DateTimeUtils.fromMillis(getter.getLong(ordinal))
- case _: TimestampMicros if rebaseDateTime => (getter, ordinal) =>
-
DateTimeUtils.rebaseGregorianToJulianMicros(getter.getLong(ordinal))
- case _: TimestampMicros => (getter, ordinal) =>
getter.getLong(ordinal)
+val micros = getter.getLong(ordinal)
+val rebasedMicros = if (rebaseDateTime) {
+ DateTimeUtils.rebaseGregori
[spark] branch master updated: [SPARK-30494][SQL] Fix cached data leakage during replacing an existing view
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 929b794 [SPARK-30494][SQL] Fix cached data leakage during replacing
an existing view
929b794 is described below
commit 929b794e25ff5454dadde7da304e6df25526d60e
Author: LantaoJin
AuthorDate: Sun Mar 22 22:22:13 2020 -0700
[SPARK-30494][SQL] Fix cached data leakage during replacing an existing view
### What changes were proposed in this pull request?
The cached RDD for plan "select 1" stays in memory forever until the
session close. This cached data cannot be used since the view temp1 has been
replaced by another plan. It's a memory leak.
We can reproduce by below commands:
```
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.0-SNAPSHOT
/_/
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_201)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.sql("create or replace temporary view temp1 as select 1")
scala> spark.sql("cache table temp1")
scala> spark.sql("create or replace temporary view temp1 as select 1, 2")
scala> spark.sql("cache table temp1")
scala> assert(spark.sharedState.cacheManager.lookupCachedData(sql("select
1, 2")).isDefined)
scala> assert(spark.sharedState.cacheManager.lookupCachedData(sql("select
1")).isDefined)
```
### Why are the changes needed?
Fix the memory leak, specially for long running mode.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Add an unit test.
Closes #27185 from LantaoJin/SPARK-30494.
Authored-by: LantaoJin
Signed-off-by: Dongjoon Hyun
---
.../spark/sql/execution/command/CommandUtils.scala | 8 +
.../spark/sql/execution/command/tables.scala | 13 ++-
.../apache/spark/sql/execution/command/views.scala | 15
.../org/apache/spark/sql/CachedTableSuite.scala| 42 ++
.../sql/hive/execution/InsertIntoHiveTable.scala | 2 +-
5 files changed, 68 insertions(+), 12 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
index 81157ca..c047be7 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
@@ -386,4 +386,12 @@ object CommandUtils extends Logging {
private def isDataPath(path: Path, stagingDir: String): Boolean = {
!path.getName.startsWith(stagingDir) && DataSourceUtils.isDataPath(path)
}
+
+ def uncacheTableOrView(sparkSession: SparkSession, name: String): Unit = {
+try {
+ sparkSession.catalog.uncacheTable(name)
+} catch {
+ case NonFatal(e) => logWarning("Exception when attempting to uncache
$name", e)
+}
+ }
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
index d4de822..61955ba 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
@@ -196,11 +196,7 @@ case class AlterTableRenameCommand(
// this can happen with Hive tables when the underlying catalog is
in-memory.
val wasCached =
Try(sparkSession.catalog.isCached(oldName.unquotedString)).getOrElse(false)
if (wasCached) {
-try {
- sparkSession.catalog.uncacheTable(oldName.unquotedString)
-} catch {
- case NonFatal(e) => log.warn(e.toString, e)
-}
+CommandUtils.uncacheTableOrView(sparkSession, oldName.unquotedString)
}
// Invalidate the table last, otherwise uncaching the table would load
the logical plan
// back into the hive metastore cache
@@ -230,12 +226,7 @@ case class AlterTableAddColumnsCommand(
val catalog = sparkSession.sessionState.catalog
val catalogTable =
verifyAlterTableAddColumn(sparkSession.sessionState.conf, catalog, table)
-try {
- sparkSession.catalog.uncacheTable(table.quotedString)
-} catch {
- case NonFatal(e) =>
-log.warn(s"Exception when attempting to uncache table
${table.quotedString}", e)
-}
+CommandUtils.uncacheTableOrView(sparkSession, table.quotedString)
catalog.refreshTable(table)
SchemaUtils.checkCol
[spark] branch branch-3.0 updated: [SPARK-30494][SQL] Fix cached data leakage during replacing an existing view
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 8c09160 [SPARK-30494][SQL] Fix cached data leakage during replacing
an existing view
8c09160 is described below
commit 8c09160cb8ec750826932988e1c87385a496ba22
Author: LantaoJin
AuthorDate: Sun Mar 22 22:22:13 2020 -0700
[SPARK-30494][SQL] Fix cached data leakage during replacing an existing view
### What changes were proposed in this pull request?
The cached RDD for plan "select 1" stays in memory forever until the
session close. This cached data cannot be used since the view temp1 has been
replaced by another plan. It's a memory leak.
We can reproduce by below commands:
```
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.0-SNAPSHOT
/_/
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java
1.8.0_201)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.sql("create or replace temporary view temp1 as select 1")
scala> spark.sql("cache table temp1")
scala> spark.sql("create or replace temporary view temp1 as select 1, 2")
scala> spark.sql("cache table temp1")
scala> assert(spark.sharedState.cacheManager.lookupCachedData(sql("select
1, 2")).isDefined)
scala> assert(spark.sharedState.cacheManager.lookupCachedData(sql("select
1")).isDefined)
```
### Why are the changes needed?
Fix the memory leak, specially for long running mode.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Add an unit test.
Closes #27185 from LantaoJin/SPARK-30494.
Authored-by: LantaoJin
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 929b794e25ff5454dadde7da304e6df25526d60e)
Signed-off-by: Dongjoon Hyun
---
.../spark/sql/execution/command/CommandUtils.scala | 8 +
.../spark/sql/execution/command/tables.scala | 13 ++-
.../apache/spark/sql/execution/command/views.scala | 15
.../org/apache/spark/sql/CachedTableSuite.scala| 42 ++
.../sql/hive/execution/InsertIntoHiveTable.scala | 2 +-
5 files changed, 68 insertions(+), 12 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
index b229b23..7e456a6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
@@ -385,4 +385,12 @@ object CommandUtils extends Logging {
private def isDataPath(path: Path, stagingDir: String): Boolean = {
!path.getName.startsWith(stagingDir) && DataSourceUtils.isDataPath(path)
}
+
+ def uncacheTableOrView(sparkSession: SparkSession, name: String): Unit = {
+try {
+ sparkSession.catalog.uncacheTable(name)
+} catch {
+ case NonFatal(e) => logWarning("Exception when attempting to uncache
$name", e)
+}
+ }
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
index d4de822..61955ba 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
@@ -196,11 +196,7 @@ case class AlterTableRenameCommand(
// this can happen with Hive tables when the underlying catalog is
in-memory.
val wasCached =
Try(sparkSession.catalog.isCached(oldName.unquotedString)).getOrElse(false)
if (wasCached) {
-try {
- sparkSession.catalog.uncacheTable(oldName.unquotedString)
-} catch {
- case NonFatal(e) => log.warn(e.toString, e)
-}
+CommandUtils.uncacheTableOrView(sparkSession, oldName.unquotedString)
}
// Invalidate the table last, otherwise uncaching the table would load
the logical plan
// back into the hive metastore cache
@@ -230,12 +226,7 @@ case class AlterTableAddColumnsCommand(
val catalog = sparkSession.sessionState.catalog
val catalogTable =
verifyAlterTableAddColumn(sparkSession.sessionState.conf, catalog, table)
-try {
- sparkSession.catalog.uncacheTable(table.quotedString)
-} catch {
- case NonFatal(e) =>
-log.warn(s"Exception when attempting to uncache table
${table.quotedString}", e)
-}
+CommandUt
[spark] branch branch-2.4 updated: Revert "[SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip package test"
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new e37f664 Revert "[SPARK-31231][BUILD] Explicitly setuptools version as
46.0.0 in pip package test"
e37f664 is described below
commit e37f66420bbb73da1a1a4d3ad02b175eedbb5052
Author: Dongjoon Hyun
AuthorDate: Tue Mar 24 11:55:57 2020 -0700
Revert "[SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip
package test"
This reverts commit 223b9fb1eadeba0e05b1a300512c31c4f99f41e8.
Signed-off-by: Dongjoon Hyun
---
dev/run-pip-tests | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev/run-pip-tests b/dev/run-pip-tests
index f9cd94d..60cf4d8 100755
--- a/dev/run-pip-tests
+++ b/dev/run-pip-tests
@@ -81,7 +81,7 @@ for python in "${PYTHON_EXECS[@]}"; do
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
rm -rf "$VIRTUALENV_PATH"
if [ -n "$USE_CONDA" ]; then
- conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip
setuptools=46.0.0
+ conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip
setuptools
source activate "$VIRTUALENV_PATH"
else
mkdir -p "$VIRTUALENV_PATH"
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (e37f664 -> 4381ad5)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from e37f664 Revert "[SPARK-31231][BUILD] Explicitly setuptools version as 46.0.0 in pip package test" add 4381ad5 [SPARK-30494][SQL][2.4] Fix cached data leakage during replacing an existing view No new revisions were added by this update. Summary of changes: .../spark/sql/execution/command/CommandUtils.scala | 8 + .../spark/sql/execution/command/tables.scala | 13 ++- .../apache/spark/sql/execution/command/views.scala | 17 - .../org/apache/spark/sql/CachedTableSuite.scala| 42 ++ .../sql/hive/execution/InsertIntoHiveTable.scala | 2 +- 5 files changed, 69 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (d278960 -> 4b97009)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from d278960 [SPARK-31081][UI][SQL] Make display of stageId/stageAttemptId/taskId of sql metrics toggleable add 4b97009 [SPARK-31101][BUILD][3.0] Upgrade Janino to 3.0.16 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 4 ++-- dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 4 ++-- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 4 ++-- pom.xml | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31184][SQL] Support getTablesByType API of Hive Client
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b3fd5c9 [SPARK-31184][SQL] Support getTablesByType API of Hive Client
b3fd5c9 is described below
commit b3fd5c90ae00807f3a49e2b9f1f0c8d026cc4219
Author: Eric Wu <492960...@qq.com>
AuthorDate: Sat Mar 21 17:41:23 2020 -0700
[SPARK-31184][SQL] Support getTablesByType API of Hive Client
### What changes were proposed in this pull request?
Hive 2.3+ supports `getTablesByType` API, which will provide an efficient
way to get HiveTable with specific type. Now, we have following mappings when
using `HiveExternalCatalog`.
```
CatalogTableType.EXTERNAL => HiveTableType.EXTERNAL_TABLE
CatalogTableType.MANAGED => HiveTableType.MANAGED_TABLE
CatalogTableType.VIEW => HiveTableType.VIRTUAL_VIEW
```
Without this API, we need to achieve the goal by `getTables` +
`getTablesByName` + `filter with type`.
This PR add `getTablesByType` in `HiveShim`. For those hive versions don't
support this API, `UnsupportedOperationException` will be thrown. And the upper
logic should catch the exception and fallback to the filter solution mentioned
above.
Since the JDK11 related fix in `Hive` is not released yet, manual tests
against hive 2.3.7-SNAPSHOT is done by following the instructions of
SPARK-29245.
### Why are the changes needed?
This API will provide better usability and performance if we want to get a
list of hiveTables with specific type. For example `HiveTableType.VIRTUAL_VIEW`
corresponding to `CatalogTableType.VIEW`.
### Does this PR introduce any user-facing change?
No, this is a support function.
### How was this patch tested?
Add tests in VersionsSuite and manually run JDK11 test with following
settings:
- Hive 2.3.6 Metastore on JDK8
- Hive 2.3.7-SNAPSHOT library build from source of Hive 2.3 branch
- Spark build with Hive 2.3.7-SNAPSHOT on jdk-11.0.6
Closes #27952 from Eric5553/GetTableByType.
Authored-by: Eric Wu <492960...@qq.com>
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 3a48ea1fe0fb85253f12d86caea01ffcb7e678d0)
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/sql/hive/client/HiveClient.scala | 9 +
.../spark/sql/hive/client/HiveClientImpl.scala | 42 --
.../apache/spark/sql/hive/client/HiveShim.scala| 35 +-
.../spark/sql/hive/client/VersionsSuite.scala | 22 +---
4 files changed, 92 insertions(+), 16 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
index e31dffa..3ea80ea 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
@@ -61,6 +61,15 @@ private[hive] trait HiveClient {
/** Returns the names of tables in the given database that matches the given
pattern. */
def listTables(dbName: String, pattern: String): Seq[String]
+ /**
+ * Returns the names of tables with specific tableType in the given database
that matches
+ * the given pattern.
+ */
+ def listTablesByType(
+ dbName: String,
+ pattern: String,
+ tableType: CatalogTableType): Seq[String]
+
/** Sets the name of current database. */
def setCurrentDatabase(databaseName: String): Unit
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 4a3e813..6ad5e9d 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -755,6 +755,22 @@ private[hive] class HiveClientImpl(
client.getTablesByPattern(dbName, pattern).asScala
}
+ override def listTablesByType(
+ dbName: String,
+ pattern: String,
+ tableType: CatalogTableType): Seq[String] = withHiveState {
+try {
+ // Try with Hive API getTablesByType first, it's supported from Hive
2.3+.
+ shim.getTablesByType(client, dbName, pattern, toHiveTableType(tableType))
+} catch {
+ case _: UnsupportedOperationException =>
+// Fallback to filter logic if getTablesByType not supported.
+val tableNames = client.getTablesByPattern(dbName, pattern).asScala
+val tables = getTablesByName(dbName, tableNames).filter(_.tableType ==
tableType)
+tables.map(_.identifier.table)
+}
+ }
+
/**
* Runs the specified SQL query using Hive.
*/
@@ -1011,25 +1027,29 @@ private[hive] object HiveClientImp
[spark] branch master updated: [SPARK-25121][SQL][FOLLOWUP] Add more unit tests for multi-part identifiers in join strategy hints
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new da49f50 [SPARK-25121][SQL][FOLLOWUP] Add more unit tests for
multi-part identifiers in join strategy hints
da49f50 is described below
commit da49f50621b201822a64358624c19be3fae0a855
Author: Takeshi Yamamuro
AuthorDate: Wed Mar 25 08:37:28 2020 -0700
[SPARK-25121][SQL][FOLLOWUP] Add more unit tests for multi-part identifiers
in join strategy hints
### What changes were proposed in this pull request?
This pr intends to add unit tests for the other join hints (`MERGEJOIN`,
`SHUFFLE_HASH`, and `SHUFFLE_REPLICATE_NL`). This is a followup PR of #27935.
### Why are the changes needed?
For better test coverage.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Added unit tests.
Closes #28013 from maropu/SPARK-25121-FOLLOWUP.
Authored-by: Takeshi Yamamuro
Signed-off-by: Dongjoon Hyun
---
.../sql/catalyst/analysis/ResolveHintsSuite.scala | 93 --
1 file changed, 49 insertions(+), 44 deletions(-)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveHintsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveHintsSuite.scala
index ca7d284..d3bd5d0 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveHintsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveHintsSuite.scala
@@ -242,51 +242,56 @@ class ResolveHintsSuite extends AnalysisTest {
caseSensitive = false)
}
- test("Supports multi-part table names for broadcast hint resolution") {
-// local temp table (single-part identifier case)
-checkAnalysis(
- UnresolvedHint("MAPJOIN", Seq("table", "table2"),
-table("TaBlE").join(table("TaBlE2"))),
- Join(
-ResolvedHint(testRelation, HintInfo(strategy = Some(BROADCAST))),
-ResolvedHint(testRelation2, HintInfo(strategy = Some(BROADCAST))),
-Inner,
-None,
-JoinHint.NONE),
- caseSensitive = false)
-
-checkAnalysis(
- UnresolvedHint("MAPJOIN", Seq("TaBlE", "table2"),
-table("TaBlE").join(table("TaBlE2"))),
- Join(
-ResolvedHint(testRelation, HintInfo(strategy = Some(BROADCAST))),
-testRelation2,
-Inner,
-None,
-JoinHint.NONE),
- caseSensitive = true)
+ test("Supports multi-part table names for join strategy hint resolution") {
+Seq(("MAPJOIN", BROADCAST),
+("MERGEJOIN", SHUFFLE_MERGE),
+("SHUFFLE_HASH", SHUFFLE_HASH),
+("SHUFFLE_REPLICATE_NL", SHUFFLE_REPLICATE_NL)).foreach { case
(hintName, st) =>
+ // local temp table (single-part identifier case)
+ checkAnalysis(
+UnresolvedHint(hintName, Seq("table", "table2"),
+ table("TaBlE").join(table("TaBlE2"))),
+Join(
+ ResolvedHint(testRelation, HintInfo(strategy = Some(st))),
+ ResolvedHint(testRelation2, HintInfo(strategy = Some(st))),
+ Inner,
+ None,
+ JoinHint.NONE),
+caseSensitive = false)
-// global temp table (multi-part identifier case)
-checkAnalysis(
- UnresolvedHint("MAPJOIN", Seq("GlOBal_TeMP.table4", "table5"),
-table("global_temp", "table4").join(table("global_temp", "table5"))),
- Join(
-ResolvedHint(testRelation4, HintInfo(strategy = Some(BROADCAST))),
-ResolvedHint(testRelation5, HintInfo(strategy = Some(BROADCAST))),
-Inner,
-None,
-JoinHint.NONE),
- caseSensitive = false)
+ checkAnalysis(
+UnresolvedHint(hintName, Seq("TaBlE", "table2"),
+ table("TaBlE").join(table("TaBlE2"))),
+Join(
+ ResolvedHint(testRelation, HintInfo(strategy = Some(st))),
+ testRelation2,
+ Inner,
+ None,
+ JoinHint.NONE),
+caseSensitive = true)
+
+ // global temp table (multi-part identifier case)
+ checkAnalysis(
+UnresolvedHint(hintName, Seq("GlOBal_TeMP.table4", "table5"),
+ table("global_temp", "table4").join(table("global_temp", "table5"))),
+Join(
+ ResolvedHint(testRelation4, HintInfo(strategy = Some(st))),
+ ResolvedHint(testRelation5, HintInfo(strategy = Some(st))),
+ Inner,
+
[spark] branch master updated (27d53de -> da49f50)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 27d53de [SPARK-31232][SQL][DOCS] Specify formats of `spark.sql.session.timeZone` add da49f50 [SPARK-25121][SQL][FOLLOWUP] Add more unit tests for multi-part identifiers in join strategy hints No new revisions were added by this update. Summary of changes: .../sql/catalyst/analysis/ResolveHintsSuite.scala | 93 -- 1 file changed, 49 insertions(+), 44 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (bd99994 -> e6c85b3)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from bd4 [SPARK-31232][SQL][DOCS] Specify formats of `spark.sql.session.timeZone` add e6c85b3 [SPARK-25121][SQL][FOLLOWUP] Add more unit tests for multi-part identifiers in join strategy hints No new revisions were added by this update. Summary of changes: .../sql/catalyst/analysis/ResolveHintsSuite.scala | 93 -- 1 file changed, 49 insertions(+), 44 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (bd99994 -> e6c85b3)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from bd4 [SPARK-31232][SQL][DOCS] Specify formats of `spark.sql.session.timeZone` add e6c85b3 [SPARK-25121][SQL][FOLLOWUP] Add more unit tests for multi-part identifiers in join strategy hints No new revisions were added by this update. Summary of changes: .../sql/catalyst/analysis/ResolveHintsSuite.scala | 93 -- 1 file changed, 49 insertions(+), 44 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (da49f50 -> 4f274a4)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from da49f50 [SPARK-25121][SQL][FOLLOWUP] Add more unit tests for multi-part identifiers in join strategy hints add 4f274a4 [SPARK-31147][SQL] Forbid CHAR type in non-Hive-Serde tables No new revisions were added by this update. Summary of changes: docs/sql-migration-guide.md| 18 +++--- .../sql/catalyst/analysis/ResolveCatalogs.scala| 5 ++ .../spark/sql/catalyst/parser/AstBuilder.scala | 4 ++ .../spark/sql/catalyst/parser/ParseDriver.scala| 5 ++ .../sql/catalyst/parser/ParserInterface.scala | 6 ++ .../sql/connector/catalog/CatalogV2Util.scala | 19 +- .../apache/spark/sql/types/HiveStringType.scala| 7 +++ .../catalyst/analysis/ResolveSessionCatalog.scala | 15 + .../spark/sql/SparkSessionExtensionSuite.scala | 3 + .../execution/command/PlanResolutionSuite.scala| 70 ++ .../spark/sql/hive/HiveMetastoreCatalogSuite.scala | 2 +- 11 files changed, 134 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 53221cd [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite 53221cd is described below commit 53221cda408e9be5d0d2ff5946c200cb43647dd9 Author: Dongjoon Hyun AuthorDate: Wed Mar 25 12:38:15 2020 -0700 [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite ### What changes were proposed in this pull request? This PR (SPARK-31244) replaces `Ceph` with `Minio` in K8S `DepsTestSuite`. ### Why are the changes needed? Currently, `DepsTestsSuite` is using `ceph` for S3 storage. However, the used version and all new releases are broken on new `minikube` releases. We had better use more robust and small one. ``` $ minikube version minikube version: v1.8.2 $ minikube -p minikube docker-env | source $ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.3-stable-4.0-nautilus-centos-7-x86_64 /bin/sh 2020-03-25 04:26:21 /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings $ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.11-stable-4.0-nautilus-centos-7 /bin/sh 2020-03-25 04:20:30 /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings ``` Also, the image size is unnecessarily big (almost `1GB`) and growing while `minio` is `55.8MB` with the same features. ``` $ docker images | grep ceph ceph/daemon v4.0.3-stable-4.0-nautilus-centos-7-x86_64 a6a05ccdf924 6 months ago 852MB ceph/daemon v4.0.11-stable-4.0-nautilus-centos-7 87f695550d8e 12 hours ago 901MB $ docker images | grep minio minio/minio latest 95c226551ea6 5 days ago 55.8MB ``` ### Does this PR introduce any user-facing change? No. (This is a test case change) ### How was this patch tested? Pass the existing Jenkins K8s integration test job and test with the latest minikube. ``` $ minikube version minikube version: v1.8.2 $ kubectl version --short Client Version: v1.17.4 Server Version: v1.17.4 $ NO_MANUAL=1 ./dev/make-distribution.sh --r --pip --tgz -Pkubernetes $ resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --spark-tgz $PWD/spark-*.tgz ... KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage *** FAILED *** // This is irrelevant to this PR. - Launcher client dependencies // This is the fixed test case by this PR. - Test basic decommissioning - Run SparkR on simple dataframe.R example Run completed in 12 minutes, 4 seconds. ... ``` The following is the working snapshot of `DepsTestSuite` test. ``` $ kubectl get all -ncf9438dd8a65436686b1196a6b73000f NAME READY STATUS RESTARTS AGE pod/minio-0 1/1 Running 0 70s pod/spark-test-app-8494bddca3754390b9e59a2ef47584eb 1/1 Running 0 55s NAME TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE service/minio-s3 NodePort 10.109.54.180 9000:30678/TCP 70s service/spark-test-app-fd916b711061c7b8-driver-svc ClusterIP None 7078/TCP,7079/TCP,4040/TCP 55s NAME READY AGE statefulset.apps/minio 1/1 70s ``` Closes #28015 from dongjoon-hyun/SPARK-31244. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit
[spark] branch master updated: [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f206bbd [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite f206bbd is described below commit f206bbde3a8f64650236013d61680faba492d7a4 Author: Dongjoon Hyun AuthorDate: Wed Mar 25 12:38:15 2020 -0700 [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite ### What changes were proposed in this pull request? This PR (SPARK-31244) replaces `Ceph` with `Minio` in K8S `DepsTestSuite`. ### Why are the changes needed? Currently, `DepsTestsSuite` is using `ceph` for S3 storage. However, the used version and all new releases are broken on new `minikube` releases. We had better use more robust and small one. ``` $ minikube version minikube version: v1.8.2 $ minikube -p minikube docker-env | source $ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.3-stable-4.0-nautilus-centos-7-x86_64 /bin/sh 2020-03-25 04:26:21 /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings $ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.11-stable-4.0-nautilus-centos-7 /bin/sh 2020-03-25 04:20:30 /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings ``` Also, the image size is unnecessarily big (almost `1GB`) and growing while `minio` is `55.8MB` with the same features. ``` $ docker images | grep ceph ceph/daemon v4.0.3-stable-4.0-nautilus-centos-7-x86_64 a6a05ccdf924 6 months ago 852MB ceph/daemon v4.0.11-stable-4.0-nautilus-centos-7 87f695550d8e 12 hours ago 901MB $ docker images | grep minio minio/minio latest 95c226551ea6 5 days ago 55.8MB ``` ### Does this PR introduce any user-facing change? No. (This is a test case change) ### How was this patch tested? Pass the existing Jenkins K8s integration test job and test with the latest minikube. ``` $ minikube version minikube version: v1.8.2 $ kubectl version --short Client Version: v1.17.4 Server Version: v1.17.4 $ NO_MANUAL=1 ./dev/make-distribution.sh --r --pip --tgz -Pkubernetes $ resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --spark-tgz $PWD/spark-*.tgz ... KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage *** FAILED *** // This is irrelevant to this PR. - Launcher client dependencies // This is the fixed test case by this PR. - Test basic decommissioning - Run SparkR on simple dataframe.R example Run completed in 12 minutes, 4 seconds. ... ``` The following is the working snapshot of `DepsTestSuite` test. ``` $ kubectl get all -ncf9438dd8a65436686b1196a6b73000f NAME READY STATUS RESTARTS AGE pod/minio-0 1/1 Running 0 70s pod/spark-test-app-8494bddca3754390b9e59a2ef47584eb 1/1 Running 0 55s NAME TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE service/minio-s3 NodePort 10.109.54.180 9000:30678/TCP 70s service/spark-test-app-fd916b711061c7b8-driver-svc ClusterIP None 7078/TCP,7079/TCP,4040/TCP 55s NAME READY AGE statefulset.apps/minio 1/1 70s ``` Closes #28015 from dongjoon-hyun/SPARK-31244. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- .../k8s/integrationtest
[spark] branch branch-3.0 updated: [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 53221cd [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite 53221cd is described below commit 53221cda408e9be5d0d2ff5946c200cb43647dd9 Author: Dongjoon Hyun AuthorDate: Wed Mar 25 12:38:15 2020 -0700 [SPARK-31244][K8S][TEST] Use Minio instead of Ceph in K8S DepsTestsSuite ### What changes were proposed in this pull request? This PR (SPARK-31244) replaces `Ceph` with `Minio` in K8S `DepsTestSuite`. ### Why are the changes needed? Currently, `DepsTestsSuite` is using `ceph` for S3 storage. However, the used version and all new releases are broken on new `minikube` releases. We had better use more robust and small one. ``` $ minikube version minikube version: v1.8.2 $ minikube -p minikube docker-env | source $ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.3-stable-4.0-nautilus-centos-7-x86_64 /bin/sh 2020-03-25 04:26:21 /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings $ docker run -it --rm -e NETWORK_AUTO_DETECT=4 -e RGW_FRONTEND_PORT=8000 -e SREE_PORT=5001 -e CEPH_DEMO_UID=nano -e CEPH_DAEMON=demo ceph/daemon:v4.0.11-stable-4.0-nautilus-centos-7 /bin/sh 2020-03-25 04:20:30 /opt/ceph-container/bin/entrypoint.sh: ERROR- it looks like we have not been able to discover the network settings ``` Also, the image size is unnecessarily big (almost `1GB`) and growing while `minio` is `55.8MB` with the same features. ``` $ docker images | grep ceph ceph/daemon v4.0.3-stable-4.0-nautilus-centos-7-x86_64 a6a05ccdf924 6 months ago 852MB ceph/daemon v4.0.11-stable-4.0-nautilus-centos-7 87f695550d8e 12 hours ago 901MB $ docker images | grep minio minio/minio latest 95c226551ea6 5 days ago 55.8MB ``` ### Does this PR introduce any user-facing change? No. (This is a test case change) ### How was this patch tested? Pass the existing Jenkins K8s integration test job and test with the latest minikube. ``` $ minikube version minikube version: v1.8.2 $ kubectl version --short Client Version: v1.17.4 Server Version: v1.17.4 $ NO_MANUAL=1 ./dev/make-distribution.sh --r --pip --tgz -Pkubernetes $ resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --spark-tgz $PWD/spark-*.tgz ... KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage *** FAILED *** // This is irrelevant to this PR. - Launcher client dependencies // This is the fixed test case by this PR. - Test basic decommissioning - Run SparkR on simple dataframe.R example Run completed in 12 minutes, 4 seconds. ... ``` The following is the working snapshot of `DepsTestSuite` test. ``` $ kubectl get all -ncf9438dd8a65436686b1196a6b73000f NAME READY STATUS RESTARTS AGE pod/minio-0 1/1 Running 0 70s pod/spark-test-app-8494bddca3754390b9e59a2ef47584eb 1/1 Running 0 55s NAME TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE service/minio-s3 NodePort 10.109.54.180 9000:30678/TCP 70s service/spark-test-app-fd916b711061c7b8-driver-svc ClusterIP None 7078/TCP,7079/TCP,4040/TCP 55s NAME READY AGE statefulset.apps/minio 1/1 70s ``` Closes #28015 from dongjoon-hyun/SPARK-31244. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit
[spark] branch branch-3.0 updated: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 8f93dc2 [SPARK-31259][CORE] Fix log message about fetch request size
in ShuffleBlockFetcherIterator
8f93dc2 is described below
commit 8f93dc2f1dd8bd09d52fd3dc07a4c10e70bd237c
Author: yi.wu
AuthorDate: Thu Mar 26 09:11:13 2020 -0700
[SPARK-31259][CORE] Fix log message about fetch request size in
ShuffleBlockFetcherIterator
### What changes were proposed in this pull request?
Fix incorrect log of `cureRequestSize`.
### Why are the changes needed?
In batch mode, `curRequestSize` can be the total size of several block
groups. And each group should have its own request size instead of using the
total size.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
It's only affect log.
Closes #28028 from Ngone51/fix_curRequestSize.
Authored-by: yi.wu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 33f532a9f201fb9c7895d685b3dce82cf042dc61)
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/storage/ShuffleBlockFetcherIterator.scala | 14 ++
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
index f1a7d88..404e055 100644
---
a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
+++
b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
@@ -329,9 +329,8 @@ final class ShuffleBlockFetcherIterator(
private def createFetchRequest(
blocks: Seq[FetchBlockInfo],
- address: BlockManagerId,
- curRequestSize: Long): FetchRequest = {
-logDebug(s"Creating fetch request of $curRequestSize at $address "
+ address: BlockManagerId): FetchRequest = {
+logDebug(s"Creating fetch request of ${blocks.map(_.size).sum} at $address
"
+ s"with ${blocks.size} blocks")
FetchRequest(address, blocks)
}
@@ -339,17 +338,16 @@ final class ShuffleBlockFetcherIterator(
private def createFetchRequests(
curBlocks: Seq[FetchBlockInfo],
address: BlockManagerId,
- curRequestSize: Long,
isLast: Boolean,
collectedRemoteRequests: ArrayBuffer[FetchRequest]): Seq[FetchBlockInfo]
= {
val mergedBlocks = mergeContinuousShuffleBlockIdsIfNeeded(curBlocks)
var retBlocks = Seq.empty[FetchBlockInfo]
if (mergedBlocks.length <= maxBlocksInFlightPerAddress) {
- collectedRemoteRequests += createFetchRequest(mergedBlocks, address,
curRequestSize)
+ collectedRemoteRequests += createFetchRequest(mergedBlocks, address)
} else {
mergedBlocks.grouped(maxBlocksInFlightPerAddress).foreach { blocks =>
if (blocks.length == maxBlocksInFlightPerAddress || isLast) {
- collectedRemoteRequests += createFetchRequest(blocks, address,
curRequestSize)
+ collectedRemoteRequests += createFetchRequest(blocks, address)
} else {
// The last group does not exceed `maxBlocksInFlightPerAddress`. Put
it back
// to `curBlocks`.
@@ -377,14 +375,14 @@ final class ShuffleBlockFetcherIterator(
// For batch fetch, the actual block in flight should count for merged
block.
val mayExceedsMaxBlocks = !doBatchFetch && curBlocks.size >=
maxBlocksInFlightPerAddress
if (curRequestSize >= targetRemoteRequestSize || mayExceedsMaxBlocks) {
-curBlocks = createFetchRequests(curBlocks, address, curRequestSize,
isLast = false,
+curBlocks = createFetchRequests(curBlocks, address, isLast = false,
collectedRemoteRequests).to[ArrayBuffer]
curRequestSize = curBlocks.map(_.size).sum
}
}
// Add in the final request
if (curBlocks.nonEmpty) {
- curBlocks = createFetchRequests(curBlocks, address, curRequestSize,
isLast = true,
+ curBlocks = createFetchRequests(curBlocks, address, isLast = true,
collectedRemoteRequests).to[ArrayBuffer]
curRequestSize = curBlocks.map(_.size).sum
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 33f532a [SPARK-31259][CORE] Fix log message about fetch request size
in ShuffleBlockFetcherIterator
33f532a is described below
commit 33f532a9f201fb9c7895d685b3dce82cf042dc61
Author: yi.wu
AuthorDate: Thu Mar 26 09:11:13 2020 -0700
[SPARK-31259][CORE] Fix log message about fetch request size in
ShuffleBlockFetcherIterator
### What changes were proposed in this pull request?
Fix incorrect log of `cureRequestSize`.
### Why are the changes needed?
In batch mode, `curRequestSize` can be the total size of several block
groups. And each group should have its own request size instead of using the
total size.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
It's only affect log.
Closes #28028 from Ngone51/fix_curRequestSize.
Authored-by: yi.wu
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/storage/ShuffleBlockFetcherIterator.scala | 14 ++
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
index f1a7d88..404e055 100644
---
a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
+++
b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
@@ -329,9 +329,8 @@ final class ShuffleBlockFetcherIterator(
private def createFetchRequest(
blocks: Seq[FetchBlockInfo],
- address: BlockManagerId,
- curRequestSize: Long): FetchRequest = {
-logDebug(s"Creating fetch request of $curRequestSize at $address "
+ address: BlockManagerId): FetchRequest = {
+logDebug(s"Creating fetch request of ${blocks.map(_.size).sum} at $address
"
+ s"with ${blocks.size} blocks")
FetchRequest(address, blocks)
}
@@ -339,17 +338,16 @@ final class ShuffleBlockFetcherIterator(
private def createFetchRequests(
curBlocks: Seq[FetchBlockInfo],
address: BlockManagerId,
- curRequestSize: Long,
isLast: Boolean,
collectedRemoteRequests: ArrayBuffer[FetchRequest]): Seq[FetchBlockInfo]
= {
val mergedBlocks = mergeContinuousShuffleBlockIdsIfNeeded(curBlocks)
var retBlocks = Seq.empty[FetchBlockInfo]
if (mergedBlocks.length <= maxBlocksInFlightPerAddress) {
- collectedRemoteRequests += createFetchRequest(mergedBlocks, address,
curRequestSize)
+ collectedRemoteRequests += createFetchRequest(mergedBlocks, address)
} else {
mergedBlocks.grouped(maxBlocksInFlightPerAddress).foreach { blocks =>
if (blocks.length == maxBlocksInFlightPerAddress || isLast) {
- collectedRemoteRequests += createFetchRequest(blocks, address,
curRequestSize)
+ collectedRemoteRequests += createFetchRequest(blocks, address)
} else {
// The last group does not exceed `maxBlocksInFlightPerAddress`. Put
it back
// to `curBlocks`.
@@ -377,14 +375,14 @@ final class ShuffleBlockFetcherIterator(
// For batch fetch, the actual block in flight should count for merged
block.
val mayExceedsMaxBlocks = !doBatchFetch && curBlocks.size >=
maxBlocksInFlightPerAddress
if (curRequestSize >= targetRemoteRequestSize || mayExceedsMaxBlocks) {
-curBlocks = createFetchRequests(curBlocks, address, curRequestSize,
isLast = false,
+curBlocks = createFetchRequests(curBlocks, address, isLast = false,
collectedRemoteRequests).to[ArrayBuffer]
curRequestSize = curBlocks.map(_.size).sum
}
}
// Add in the final request
if (curBlocks.nonEmpty) {
- curBlocks = createFetchRequests(curBlocks, address, curRequestSize,
isLast = true,
+ curBlocks = createFetchRequests(curBlocks, address, isLast = true,
collectedRemoteRequests).to[ArrayBuffer]
curRequestSize = curBlocks.map(_.size).sum
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 33f532a [SPARK-31259][CORE] Fix log message about fetch request size
in ShuffleBlockFetcherIterator
33f532a is described below
commit 33f532a9f201fb9c7895d685b3dce82cf042dc61
Author: yi.wu
AuthorDate: Thu Mar 26 09:11:13 2020 -0700
[SPARK-31259][CORE] Fix log message about fetch request size in
ShuffleBlockFetcherIterator
### What changes were proposed in this pull request?
Fix incorrect log of `cureRequestSize`.
### Why are the changes needed?
In batch mode, `curRequestSize` can be the total size of several block
groups. And each group should have its own request size instead of using the
total size.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
It's only affect log.
Closes #28028 from Ngone51/fix_curRequestSize.
Authored-by: yi.wu
Signed-off-by: Dongjoon Hyun
---
.../apache/spark/storage/ShuffleBlockFetcherIterator.scala | 14 ++
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
index f1a7d88..404e055 100644
---
a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
+++
b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
@@ -329,9 +329,8 @@ final class ShuffleBlockFetcherIterator(
private def createFetchRequest(
blocks: Seq[FetchBlockInfo],
- address: BlockManagerId,
- curRequestSize: Long): FetchRequest = {
-logDebug(s"Creating fetch request of $curRequestSize at $address "
+ address: BlockManagerId): FetchRequest = {
+logDebug(s"Creating fetch request of ${blocks.map(_.size).sum} at $address
"
+ s"with ${blocks.size} blocks")
FetchRequest(address, blocks)
}
@@ -339,17 +338,16 @@ final class ShuffleBlockFetcherIterator(
private def createFetchRequests(
curBlocks: Seq[FetchBlockInfo],
address: BlockManagerId,
- curRequestSize: Long,
isLast: Boolean,
collectedRemoteRequests: ArrayBuffer[FetchRequest]): Seq[FetchBlockInfo]
= {
val mergedBlocks = mergeContinuousShuffleBlockIdsIfNeeded(curBlocks)
var retBlocks = Seq.empty[FetchBlockInfo]
if (mergedBlocks.length <= maxBlocksInFlightPerAddress) {
- collectedRemoteRequests += createFetchRequest(mergedBlocks, address,
curRequestSize)
+ collectedRemoteRequests += createFetchRequest(mergedBlocks, address)
} else {
mergedBlocks.grouped(maxBlocksInFlightPerAddress).foreach { blocks =>
if (blocks.length == maxBlocksInFlightPerAddress || isLast) {
- collectedRemoteRequests += createFetchRequest(blocks, address,
curRequestSize)
+ collectedRemoteRequests += createFetchRequest(blocks, address)
} else {
// The last group does not exceed `maxBlocksInFlightPerAddress`. Put
it back
// to `curBlocks`.
@@ -377,14 +375,14 @@ final class ShuffleBlockFetcherIterator(
// For batch fetch, the actual block in flight should count for merged
block.
val mayExceedsMaxBlocks = !doBatchFetch && curBlocks.size >=
maxBlocksInFlightPerAddress
if (curRequestSize >= targetRemoteRequestSize || mayExceedsMaxBlocks) {
-curBlocks = createFetchRequests(curBlocks, address, curRequestSize,
isLast = false,
+curBlocks = createFetchRequests(curBlocks, address, isLast = false,
collectedRemoteRequests).to[ArrayBuffer]
curRequestSize = curBlocks.map(_.size).sum
}
}
// Add in the final request
if (curBlocks.nonEmpty) {
- curBlocks = createFetchRequests(curBlocks, address, curRequestSize,
isLast = true,
+ curBlocks = createFetchRequests(curBlocks, address, isLast = true,
collectedRemoteRequests).to[ArrayBuffer]
curRequestSize = curBlocks.map(_.size).sum
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for ORC datasource
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d72ec85 [SPARK-31238][SQL] Rebase dates to/from Julian calendar in
write/read for ORC datasource
d72ec85 is described below
commit d72ec8574113f9a7e87f3d7ec56c8447267b0506
Author: Maxim Gekk
AuthorDate: Thu Mar 26 13:14:28 2020 -0700
[SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for
ORC datasource
### What changes were proposed in this pull request?
This PR (SPARK-31238) aims the followings.
1. Modified ORC Vectorized Reader, in particular, OrcColumnVector v1.2 and
v2.3. After the changes, it uses `DateTimeUtils. rebaseJulianToGregorianDays()`
added by https://github.com/apache/spark/pull/27915 . The method performs
rebasing days from the hybrid calendar (Julian + Gregorian) to Proleptic
Gregorian calendar. It builds a local date in the original calendar, extracts
date fields `year`, `month` and `day` from the local date, and builds another
local date in the target calend [...]
2. Introduced rebasing dates while saving ORC files, in particular, I
modified `OrcShimUtils. getDateWritable` v1.2 and v2.3, and returned
`DaysWritable` instead of Hive's `DateWritable`. The `DaysWritable` class was
added by the PR https://github.com/apache/spark/pull/27890 (and fixed by
https://github.com/apache/spark/pull/27962). I moved `DaysWritable` from
`sql/hive` to `sql/core` to re-use it in ORC datasource.
### Why are the changes needed?
For the backward compatibility with Spark 2.4 and earlier versions. The
changes allow users to read dates/timestamps saved by previous version, and get
the same result.
### Does this PR introduce any user-facing change?
Yes. Before the changes, loading the date `1200-01-01` saved by Spark 2.4.5
returns the following:
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-08|
+--+
```
After the changes
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
### How was this patch tested?
- By running `OrcSourceSuite` and `HiveOrcSourceSuite`.
- Add new test `SPARK-31238: compatibility with Spark 2.4 in reading dates`
to `OrcSuite` which reads an ORC file saved by Spark 2.4.5 via the commands:
```shell
$ export TZ="America/Los_Angeles"
```
```scala
scala> sql("select cast('1200-01-01' as date)
dt").write.mode("overwrite").orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc")
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
- Add round trip test `SPARK-31238: rebasing dates in write`. The test
`SPARK-31238: compatibility with Spark 2.4 in reading dates` confirms rebasing
in read. So, we can check rebasing in write.
Closes #28016 from MaxGekk/rebase-date-orc.
Authored-by: Maxim Gekk
Signed-off-by: Dongjoon Hyun
---
.../sql/execution/datasources}/DaysWritable.scala | 10 ++--
.../test-data/before_1582_date_v2_4.snappy.orc | Bin 0 -> 201 bytes
.../execution/datasources/orc/OrcSourceSuite.scala | 28 -
.../sql/execution/datasources/orc/OrcTest.scala| 5
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc}/DaysWritable.scala | 17 ++---
.../execution/datasources/orc/OrcShimUtils.scala | 4 +--
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc/OrcShimUtils.scala | 5 ++--
.../org/apache/spark/sql/hive/HiveInspectors.scala | 1 +
10 files changed, 88 insertions(+), 12 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
similarity index 92%
copy from sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
copy to
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
index 1eec8d7..00b710f 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.spark.sql.hive
+package org.apache.spark.sql.execution.datasources
import java.io.{DataInput, DataOutput, IOException
[spark] branch branch-3.0 updated: [SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for ORC datasource
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 78cc2ef [SPARK-31238][SQL] Rebase dates to/from Julian calendar in
write/read for ORC datasource
78cc2ef is described below
commit 78cc2ef5b663d6d605e3d4febc6fb99e20b7f165
Author: Maxim Gekk
AuthorDate: Thu Mar 26 13:14:28 2020 -0700
[SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for
ORC datasource
### What changes were proposed in this pull request?
This PR (SPARK-31238) aims the followings.
1. Modified ORC Vectorized Reader, in particular, OrcColumnVector v1.2 and
v2.3. After the changes, it uses `DateTimeUtils. rebaseJulianToGregorianDays()`
added by https://github.com/apache/spark/pull/27915 . The method performs
rebasing days from the hybrid calendar (Julian + Gregorian) to Proleptic
Gregorian calendar. It builds a local date in the original calendar, extracts
date fields `year`, `month` and `day` from the local date, and builds another
local date in the target calend [...]
2. Introduced rebasing dates while saving ORC files, in particular, I
modified `OrcShimUtils. getDateWritable` v1.2 and v2.3, and returned
`DaysWritable` instead of Hive's `DateWritable`. The `DaysWritable` class was
added by the PR https://github.com/apache/spark/pull/27890 (and fixed by
https://github.com/apache/spark/pull/27962). I moved `DaysWritable` from
`sql/hive` to `sql/core` to re-use it in ORC datasource.
### Why are the changes needed?
For the backward compatibility with Spark 2.4 and earlier versions. The
changes allow users to read dates/timestamps saved by previous version, and get
the same result.
### Does this PR introduce any user-facing change?
Yes. Before the changes, loading the date `1200-01-01` saved by Spark 2.4.5
returns the following:
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-08|
+--+
```
After the changes
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
### How was this patch tested?
- By running `OrcSourceSuite` and `HiveOrcSourceSuite`.
- Add new test `SPARK-31238: compatibility with Spark 2.4 in reading dates`
to `OrcSuite` which reads an ORC file saved by Spark 2.4.5 via the commands:
```shell
$ export TZ="America/Los_Angeles"
```
```scala
scala> sql("select cast('1200-01-01' as date)
dt").write.mode("overwrite").orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc")
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
- Add round trip test `SPARK-31238: rebasing dates in write`. The test
`SPARK-31238: compatibility with Spark 2.4 in reading dates` confirms rebasing
in read. So, we can check rebasing in write.
Closes #28016 from MaxGekk/rebase-date-orc.
Authored-by: Maxim Gekk
Signed-off-by: Dongjoon Hyun
(cherry picked from commit d72ec8574113f9a7e87f3d7ec56c8447267b0506)
Signed-off-by: Dongjoon Hyun
---
.../sql/execution/datasources}/DaysWritable.scala | 10 ++--
.../test-data/before_1582_date_v2_4.snappy.orc | Bin 0 -> 201 bytes
.../execution/datasources/orc/OrcSourceSuite.scala | 28 -
.../sql/execution/datasources/orc/OrcTest.scala| 5
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc}/DaysWritable.scala | 17 ++---
.../execution/datasources/orc/OrcShimUtils.scala | 4 +--
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc/OrcShimUtils.scala | 5 ++--
.../org/apache/spark/sql/hive/HiveInspectors.scala | 1 +
10 files changed, 88 insertions(+), 12 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
similarity index 92%
copy from sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
copy to
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
index 1eec8d7..00b710f 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.spark.s
[spark] branch master updated (33f532a -> d72ec85)
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 33f532a [SPARK-31259][CORE] Fix log message about fetch request size
in ShuffleBlockFetcherIterator
add d72ec85 [SPARK-31238][SQL] Rebase dates to/from Julian calendar in
write/read for ORC datasource
No new revisions were added by this update.
Summary of changes:
.../sql/execution/datasources}/DaysWritable.scala | 10 ++--
.../test-data/before_1582_date_v2_4.snappy.orc | Bin 0 -> 201 bytes
.../execution/datasources/orc/OrcSourceSuite.scala | 28 -
.../sql/execution/datasources/orc/OrcTest.scala| 5
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc}/DaysWritable.scala | 17 ++---
.../execution/datasources/orc/OrcShimUtils.scala | 4 +--
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc/OrcShimUtils.scala | 5 ++--
.../org/apache/spark/sql/hive/HiveInspectors.scala | 1 +
10 files changed, 88 insertions(+), 12 deletions(-)
copy sql/{hive/src/main/scala/org/apache/spark/sql/hive =>
core/src/main/scala/org/apache/spark/sql/execution/datasources}/DaysWritable.scala
(92%)
create mode 100644
sql/core/src/test/resources/test-data/before_1582_date_v2_4.snappy.orc
rename sql/{hive/src/main/scala/org/apache/spark/sql/hive =>
core/v1.2/src/main/scala/org/apache/spark/sql/execution/datasources/orc}/DaysWritable.scala
(81%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for ORC datasource
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 78cc2ef [SPARK-31238][SQL] Rebase dates to/from Julian calendar in
write/read for ORC datasource
78cc2ef is described below
commit 78cc2ef5b663d6d605e3d4febc6fb99e20b7f165
Author: Maxim Gekk
AuthorDate: Thu Mar 26 13:14:28 2020 -0700
[SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for
ORC datasource
### What changes were proposed in this pull request?
This PR (SPARK-31238) aims the followings.
1. Modified ORC Vectorized Reader, in particular, OrcColumnVector v1.2 and
v2.3. After the changes, it uses `DateTimeUtils. rebaseJulianToGregorianDays()`
added by https://github.com/apache/spark/pull/27915 . The method performs
rebasing days from the hybrid calendar (Julian + Gregorian) to Proleptic
Gregorian calendar. It builds a local date in the original calendar, extracts
date fields `year`, `month` and `day` from the local date, and builds another
local date in the target calend [...]
2. Introduced rebasing dates while saving ORC files, in particular, I
modified `OrcShimUtils. getDateWritable` v1.2 and v2.3, and returned
`DaysWritable` instead of Hive's `DateWritable`. The `DaysWritable` class was
added by the PR https://github.com/apache/spark/pull/27890 (and fixed by
https://github.com/apache/spark/pull/27962). I moved `DaysWritable` from
`sql/hive` to `sql/core` to re-use it in ORC datasource.
### Why are the changes needed?
For the backward compatibility with Spark 2.4 and earlier versions. The
changes allow users to read dates/timestamps saved by previous version, and get
the same result.
### Does this PR introduce any user-facing change?
Yes. Before the changes, loading the date `1200-01-01` saved by Spark 2.4.5
returns the following:
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-08|
+--+
```
After the changes
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
### How was this patch tested?
- By running `OrcSourceSuite` and `HiveOrcSourceSuite`.
- Add new test `SPARK-31238: compatibility with Spark 2.4 in reading dates`
to `OrcSuite` which reads an ORC file saved by Spark 2.4.5 via the commands:
```shell
$ export TZ="America/Los_Angeles"
```
```scala
scala> sql("select cast('1200-01-01' as date)
dt").write.mode("overwrite").orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc")
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
- Add round trip test `SPARK-31238: rebasing dates in write`. The test
`SPARK-31238: compatibility with Spark 2.4 in reading dates` confirms rebasing
in read. So, we can check rebasing in write.
Closes #28016 from MaxGekk/rebase-date-orc.
Authored-by: Maxim Gekk
Signed-off-by: Dongjoon Hyun
(cherry picked from commit d72ec8574113f9a7e87f3d7ec56c8447267b0506)
Signed-off-by: Dongjoon Hyun
---
.../sql/execution/datasources}/DaysWritable.scala | 10 ++--
.../test-data/before_1582_date_v2_4.snappy.orc | Bin 0 -> 201 bytes
.../execution/datasources/orc/OrcSourceSuite.scala | 28 -
.../sql/execution/datasources/orc/OrcTest.scala| 5
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc}/DaysWritable.scala | 17 ++---
.../execution/datasources/orc/OrcShimUtils.scala | 4 +--
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc/OrcShimUtils.scala | 5 ++--
.../org/apache/spark/sql/hive/HiveInspectors.scala | 1 +
10 files changed, 88 insertions(+), 12 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
similarity index 92%
copy from sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
copy to
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
index 1eec8d7..00b710f 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.spark.s
[spark] branch branch-3.0 updated: [SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for ORC datasource
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 78cc2ef [SPARK-31238][SQL] Rebase dates to/from Julian calendar in
write/read for ORC datasource
78cc2ef is described below
commit 78cc2ef5b663d6d605e3d4febc6fb99e20b7f165
Author: Maxim Gekk
AuthorDate: Thu Mar 26 13:14:28 2020 -0700
[SPARK-31238][SQL] Rebase dates to/from Julian calendar in write/read for
ORC datasource
### What changes were proposed in this pull request?
This PR (SPARK-31238) aims the followings.
1. Modified ORC Vectorized Reader, in particular, OrcColumnVector v1.2 and
v2.3. After the changes, it uses `DateTimeUtils. rebaseJulianToGregorianDays()`
added by https://github.com/apache/spark/pull/27915 . The method performs
rebasing days from the hybrid calendar (Julian + Gregorian) to Proleptic
Gregorian calendar. It builds a local date in the original calendar, extracts
date fields `year`, `month` and `day` from the local date, and builds another
local date in the target calend [...]
2. Introduced rebasing dates while saving ORC files, in particular, I
modified `OrcShimUtils. getDateWritable` v1.2 and v2.3, and returned
`DaysWritable` instead of Hive's `DateWritable`. The `DaysWritable` class was
added by the PR https://github.com/apache/spark/pull/27890 (and fixed by
https://github.com/apache/spark/pull/27962). I moved `DaysWritable` from
`sql/hive` to `sql/core` to re-use it in ORC datasource.
### Why are the changes needed?
For the backward compatibility with Spark 2.4 and earlier versions. The
changes allow users to read dates/timestamps saved by previous version, and get
the same result.
### Does this PR introduce any user-facing change?
Yes. Before the changes, loading the date `1200-01-01` saved by Spark 2.4.5
returns the following:
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-08|
+--+
```
After the changes
```scala
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
### How was this patch tested?
- By running `OrcSourceSuite` and `HiveOrcSourceSuite`.
- Add new test `SPARK-31238: compatibility with Spark 2.4 in reading dates`
to `OrcSuite` which reads an ORC file saved by Spark 2.4.5 via the commands:
```shell
$ export TZ="America/Los_Angeles"
```
```scala
scala> sql("select cast('1200-01-01' as date)
dt").write.mode("overwrite").orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc")
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_date_orc").show(false)
+--+
|dt|
+--+
|1200-01-01|
+--+
```
- Add round trip test `SPARK-31238: rebasing dates in write`. The test
`SPARK-31238: compatibility with Spark 2.4 in reading dates` confirms rebasing
in read. So, we can check rebasing in write.
Closes #28016 from MaxGekk/rebase-date-orc.
Authored-by: Maxim Gekk
Signed-off-by: Dongjoon Hyun
(cherry picked from commit d72ec8574113f9a7e87f3d7ec56c8447267b0506)
Signed-off-by: Dongjoon Hyun
---
.../sql/execution/datasources}/DaysWritable.scala | 10 ++--
.../test-data/before_1582_date_v2_4.snappy.orc | Bin 0 -> 201 bytes
.../execution/datasources/orc/OrcSourceSuite.scala | 28 -
.../sql/execution/datasources/orc/OrcTest.scala| 5
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc}/DaysWritable.scala | 17 ++---
.../execution/datasources/orc/OrcShimUtils.scala | 4 +--
.../execution/datasources/orc/OrcColumnVector.java | 15 ++-
.../execution/datasources/orc/OrcShimUtils.scala | 5 ++--
.../org/apache/spark/sql/hive/HiveInspectors.scala | 1 +
10 files changed, 88 insertions(+), 12 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
similarity index 92%
copy from sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
copy to
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
index 1eec8d7..00b710f 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/DaysWritable.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.spark.s
[spark] branch branch-2.4 updated: Revert "[SPARK-31258][BUILD] Pin the avro version in SBT"
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 4217f75 Revert "[SPARK-31258][BUILD] Pin the avro version in SBT"
4217f75 is described below
commit 4217f75b3f05f323018a3a9986ecb9ae587688a8
Author: Dongjoon Hyun
AuthorDate: Thu Mar 26 13:49:39 2020 -0700
Revert "[SPARK-31258][BUILD] Pin the avro version in SBT"
This reverts commit 916a25a46bca7196416372bacc3fc260a6ef658f.
---
project/SparkBuild.scala | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 4578857..7ee079c 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -554,8 +554,7 @@ object DependencyOverrides {
dependencyOverrides += "com.google.guava" % "guava" % "14.0.1",
dependencyOverrides += "commons-io" % "commons-io" % "2.4",
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind"
% "2.6.7.3",
-dependencyOverrides += "jline" % "jline" % "2.14.6",
-dependencyOverrides += "org.apache.avro" % "avro" % "1.8.2")
+dependencyOverrides += "jline" % "jline" % "2.14.6")
}
/**
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (4868b4d -> f94d13f)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 4868b4d [SPARK-31225][SQL] Override sql method of OuterReference add f94d13f [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][FOLLOWUP][3.0] Fix build error due to conf version No new revisions were added by this update. Summary of changes: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 1 - 1 file changed, 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9f0c010 -> fc2a974)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9f0c010 [SPARK-31277][SQL][TESTS] Migrate `DateTimeTestUtils` from `TimeZone` to `ZoneId` add fc2a974 [SPARK-31284][SQL][TESTS] Check rebasing of timestamps in ORC datasource No new revisions were added by this update. Summary of changes: .../test-data/before_1582_ts_v2_4.snappy.orc | Bin 0 -> 251 bytes .../execution/datasources/orc/OrcSourceSuite.scala | 28 + 2 files changed, 28 insertions(+) create mode 100644 sql/core/src/test/resources/test-data/before_1582_ts_v2_4.snappy.orc - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31284][SQL][TESTS] Check rebasing of timestamps in ORC datasource
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b6e8f64 [SPARK-31284][SQL][TESTS] Check rebasing of timestamps in ORC
datasource
b6e8f64 is described below
commit b6e8f64d49caf1f0a1f1b910d603e8e000270d01
Author: Maxim Gekk
AuthorDate: Fri Mar 27 09:06:59 2020 -0700
[SPARK-31284][SQL][TESTS] Check rebasing of timestamps in ORC datasource
### What changes were proposed in this pull request?
In the PR, I propose 2 tests to check that rebasing of timestamps from/to
the hybrid calendar (Julian + Gregorian) to/from Proleptic Gregorian calendar
works correctly.
1. The test `compatibility with Spark 2.4 in reading timestamps` load ORC
file saved by Spark 2.4.5 via:
```shell
$ export TZ="America/Los_Angeles"
```
```scala
scala> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
scala> val df = Seq("1001-01-01
01:02:03.123456").toDF("tsS").select($"tsS".cast("timestamp").as("ts"))
df: org.apache.spark.sql.DataFrame = [ts: timestamp]
scala> df.write.orc("/Users/maxim/tmp/before_1582/2_4_5_ts_orc")
scala>
spark.read.orc("/Users/maxim/tmp/before_1582/2_4_5_ts_orc").show(false)
+--+
|ts|
+--+
|1001-01-01 01:02:03.123456|
+--+
```
2. The test `rebasing timestamps in write` is round trip test. Since the
previous test confirms correct rebasing of timestamps in read. This test should
pass only if rebasing works correctly in write.
### Why are the changes needed?
To guarantee that rebasing works correctly for timestamps in ORC datasource.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
By running `OrcSourceSuite` for Hive 1.2 and 2.3 via the commands:
```
$ build/sbt -Phive-2.3 "test:testOnly *OrcSourceSuite"
```
and
```
$ build/sbt -Phive-1.2 "test:testOnly *OrcSourceSuite"
```
Closes #28047 from MaxGekk/rebase-ts-orc-test.
Authored-by: Maxim Gekk
Signed-off-by: Dongjoon Hyun
(cherry picked from commit fc2a974e030c82bf500a81c3908f853c3eeb761d)
Signed-off-by: Dongjoon Hyun
---
.../test-data/before_1582_ts_v2_4.snappy.orc | Bin 0 -> 251 bytes
.../execution/datasources/orc/OrcSourceSuite.scala | 28 +
2 files changed, 28 insertions(+)
diff --git
a/sql/core/src/test/resources/test-data/before_1582_ts_v2_4.snappy.orc
b/sql/core/src/test/resources/test-data/before_1582_ts_v2_4.snappy.orc
new file mode 100644
index 000..af9ef04
Binary files /dev/null and
b/sql/core/src/test/resources/test-data/before_1582_ts_v2_4.snappy.orc differ
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
index b5e002f..0b7500c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
@@ -508,6 +508,34 @@ abstract class OrcSuite extends OrcTest with
BeforeAndAfterAll {
}
}
}
+
+ test("SPARK-31284: compatibility with Spark 2.4 in reading timestamps") {
+Seq(false, true).foreach { vectorized =>
+ withSQLConf(SQLConf.ORC_VECTORIZED_READER_ENABLED.key ->
vectorized.toString) {
+checkAnswer(
+ readResourceOrcFile("test-data/before_1582_ts_v2_4.snappy.orc"),
+ Row(java.sql.Timestamp.valueOf("1001-01-01 01:02:03.123456")))
+ }
+}
+ }
+
+ test("SPARK-31284: rebasing timestamps in write") {
+withTempPath { dir =>
+ val path = dir.getAbsolutePath
+ Seq("1001-01-01 01:02:03.123456").toDF("tsS")
+.select($"tsS".cast("timestamp").as("ts"))
+.write
+.orc(path)
+
+ Seq(false, true).foreach { vectorized =>
+withSQLConf(SQLConf.ORC_VECTORIZED_READER_ENABLED.key ->
vectorized.toString) {
+ checkAnswer(
+spark.read.orc(path),
+Row(java.sql.Timestamp.valueOf("1001-01-01 01:02:03.123456")))
+}
+ }
+}
+ }
}
class OrcSourceSuite extends OrcSuite with SharedSparkSession {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (fc2a974 -> f879573)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from fc2a974 [SPARK-31284][SQL][TESTS] Check rebasing of timestamps in ORC datasource add f879573 [SPARK-31200][K8S] Enforce to use `https` in /etc/apt/sources.list No new revisions were added by this update. Summary of changes: .../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31200][K8S] Enforce to use `https` in /etc/apt/sources.list
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c1a03b2 [SPARK-31200][K8S] Enforce to use `https` in
/etc/apt/sources.list
c1a03b2 is described below
commit c1a03b2233aee77f957dbc94180d2334c61ac088
Author: Prashant Sharma
AuthorDate: Fri Mar 27 09:13:55 2020 -0700
[SPARK-31200][K8S] Enforce to use `https` in /etc/apt/sources.list
…n progress errors.
### What changes were proposed in this pull request?
Switching to `https` instead of `http` in the debian mirror urls.
### Why are the changes needed?
My ISP was trying to intercept (or trying to serve from cache) the `http`
traffic and this was causing a very confusing errors while building the spark
image. I thought by posting this, I can help someone save his time and energy,
if he encounters the same issue.
```
bash-3.2$ bin/docker-image-tool.sh -r scrapcodes -t v3.1.0-f1cc86 build
Sending build context to Docker daemon 203.4MB
Step 1/18 : ARG java_image_tag=8-jre-slim
Step 2/18 : FROM openjdk:${java_image_tag}
---> 381b20190cf7
Step 3/18 : ARG spark_uid=185
---> Using cache
---> 65c06f86753c
Step 4/18 : RUN set -ex && apt-get update && ln -s /lib /lib64 &&
apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps &&
mkdir -p /opt/spark && mkdir -p /opt/spark/examples && mkdir -p
/opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln
-sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >>
/etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&
rm -rf /var/cache/apt/*
---> Running in 96bcbe927d35
+ apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Err:3 http://deb.debian.org/debian buster/main amd64 Packages
File has unexpected size (13217 != 7906744). Mirror sync in progress?
[IP: 151.101.10.133 80]
Hashes of expected file:
- Filesize:7906744 [weak]
- SHA256:80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc
- MD5Sum:80b6d9c1b6630b2234161e42f4040ab3 [weak]
Release file created at: Sat, 08 Feb 2020 10:57:10 +
Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages [7380
B]
Err:5 http://deb.debian.org/debian buster-updates/main amd64 Packages
File has unexpected size (13233 != 7380). Mirror sync in progress? [IP:
151.101.10.133 80]
Hashes of expected file:
- Filesize:7380 [weak]
- SHA256:6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d
Release file created at: Fri, 20 Mar 2020 02:28:11 +
Get:4 http://security-cdn.debian.org/debian-security buster/updates
InRelease [65.4 kB]
Get:6 http://security-cdn.debian.org/debian-security buster/updates/main
amd64 Packages [183 kB]
Fetched 419 kB in 1s (327 kB/s)
Reading package lists...
E: Failed to fetch
http://deb.debian.org/debian/dists/buster/main/binary-amd64/by-hash/SHA256/80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc
File has unexpected size (13217 != 7906744). Mirror sync in progress? [IP:
151.101.10.133 80]
Hashes of expected file:
- Filesize:7906744 [weak]
-
SHA256:80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc
- MD5Sum:80b6d9c1b6630b2234161e42f4040ab3 [weak]
Release file created at: Sat, 08 Feb 2020 10:57:10 +
E: Failed to fetch
http://deb.debian.org/debian/dists/buster-updates/main/binary-amd64/by-hash/SHA256/6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d
File has unexpected size (13233 != 7380). Mirror sync in progress? [IP:
151.101.10.133 80]
Hashes of expected file:
- Filesize:7380 [weak]
-
SHA256:6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d
Release file created at: Fri, 20 Mar 2020 02:28:11 +
E: Some index files failed to download. They have been ignored, or old ones
used instead.
The command '/bin/sh -c set -ex && apt-get update && ln -s /lib
/lib64 && apt install -y bash tini libc6 libpam-modules krb5-user libnss3
procps && mkdir -p /opt/spark && mkdir -p /opt/spark/examples &&
mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh
&& ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so
use_uid&quo
[spark] branch master updated (f879573 -> 8a5d496)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f879573 [SPARK-31200][K8S] Enforce to use `https` in /etc/apt/sources.list add 8a5d496 [MINOR][DOC] Refine comments of QueryPlan regarding subquery No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/plans/QueryPlan.scala | 32 ++ 1 file changed, 21 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOC] Refine comments of QueryPlan regarding subquery
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 7435f45 [MINOR][DOC] Refine comments of QueryPlan regarding subquery
7435f45 is described below
commit 7435f4543ea6f2b927da6055c1cfb75f4a62f19d
Author: Wenchen Fan
AuthorDate: Fri Mar 27 09:35:35 2020 -0700
[MINOR][DOC] Refine comments of QueryPlan regarding subquery
### What changes were proposed in this pull request?
The query plan of Spark SQL is a mutually recursive structure: QueryPlan ->
Expression (PlanExpression) -> QueryPlan, but the transformations do not take
this into account.
This PR refines the comments of `QueryPlan` to highlight this fact.
### Why are the changes needed?
better document.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
N/A
Closes #28050 from cloud-fan/comment.
Authored-by: Wenchen Fan
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 8a5d49610d875c473114781e92300c79e24a53cc)
Signed-off-by: Dongjoon Hyun
---
.../spark/sql/catalyst/plans/QueryPlan.scala | 32 ++
1 file changed, 21 insertions(+), 11 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
index 1248266..9f86fb2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
@@ -23,6 +23,16 @@ import org.apache.spark.sql.catalyst.trees.{CurrentOrigin,
TreeNode, TreeNodeTag
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{DataType, StructType}
+/**
+ * An abstraction of the Spark SQL query plan tree, which can be logical or
physical. This class
+ * defines some basic properties of a query plan node, as well as some new
transform APIs to
+ * transform the expressions of the plan node.
+ *
+ * Note that, the query plan is a mutually recursive structure:
+ * QueryPlan -> Expression (subquery) -> QueryPlan
+ * The tree traverse APIs like `transform`, `foreach`, `collect`, etc. that are
+ * inherited from `TreeNode`, do not traverse into query plans inside
subqueries.
+ */
abstract class QueryPlan[PlanType <: QueryPlan[PlanType]] extends
TreeNode[PlanType] {
self: PlanType =>
@@ -133,7 +143,7 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
extends TreeNode[PlanT
/**
* Returns the result of running [[transformExpressions]] on this node
- * and all its children.
+ * and all its children. Note that this method skips expressions inside
subqueries.
*/
def transformAllExpressions(rule: PartialFunction[Expression, Expression]):
this.type = {
transform {
@@ -204,7 +214,7 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
extends TreeNode[PlanT
}
/**
- * All the subqueries of current plan.
+ * All the top-level subqueries of the current plan node. Nested subqueries
are not included.
*/
def subqueries: Seq[PlanType] = {
expressions.flatMap(_.collect {
@@ -213,21 +223,21 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
extends TreeNode[PlanT
}
/**
- * Returns a sequence containing the result of applying a partial function
to all elements in this
- * plan, also considering all the plans in its (nested) subqueries
- */
- def collectInPlanAndSubqueries[B](f: PartialFunction[PlanType, B]): Seq[B] =
-(this +: subqueriesAll).flatMap(_.collect(f))
-
- /**
- * Returns a sequence containing the subqueries in this plan, also including
the (nested)
- * subquries in its children
+ * All the subqueries of the current plan node and all its children. Nested
subqueries are also
+ * included.
*/
def subqueriesAll: Seq[PlanType] = {
val subqueries = this.flatMap(_.subqueries)
subqueries ++ subqueries.flatMap(_.subqueriesAll)
}
+ /**
+ * Returns a sequence containing the result of applying a partial function
to all elements in this
+ * plan, also considering all the plans in its (nested) subqueries
+ */
+ def collectInPlanAndSubqueries[B](f: PartialFunction[PlanType, B]): Seq[B] =
+(this +: subqueriesAll).flatMap(_.collect(f))
+
override def innerChildren: Seq[QueryPlan[_]] = subqueries
/**
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8a5d496 -> aa8776b)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8a5d496 [MINOR][DOC] Refine comments of QueryPlan regarding subquery add aa8776b [SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project No new revisions were added by this update. Summary of changes: .../catalyst/optimizer/NestedColumnAliasing.scala | 53 ++ .../spark/sql/catalyst/optimizer/Optimizer.scala | 20 +- .../optimizer/NestedColumnAliasingSuite.scala | 80 ++ .../execution/datasources/SchemaPruningSuite.scala | 37 ++ 4 files changed, 172 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8a5d496 -> aa8776b)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8a5d496 [MINOR][DOC] Refine comments of QueryPlan regarding subquery add aa8776b [SPARK-29721][SQL] Prune unnecessary nested fields from Generate without Project No new revisions were added by this update. Summary of changes: .../catalyst/optimizer/NestedColumnAliasing.scala | 53 ++ .../spark/sql/catalyst/optimizer/Optimizer.scala | 20 +- .../optimizer/NestedColumnAliasingSuite.scala | 80 ++ .../execution/datasources/SchemaPruningSuite.scala | 37 ++ 4 files changed, 172 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31271][UI] fix web ui for driver side SQL metrics
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c4e98c0 [SPARK-31271][UI] fix web ui for driver side SQL metrics
c4e98c0 is described below
commit c4e98c065c99d2cf840e6006ee5414fbaaba9937
Author: Wenchen Fan
AuthorDate: Fri Mar 27 15:45:35 2020 -0700
[SPARK-31271][UI] fix web ui for driver side SQL metrics
### What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/23551, we changed the metrics type
of driver-side SQL metrics to size/time etc. which comes with max/min/median
info.
This doesn't make sense for driver side SQL metrics as they have only one
value. It makes the web UI hard to read:
This PR updates the SQL metrics UI to only display max/min/median if there
are more than one metrics values:
### Why are the changes needed?
Makes the UI easier to read
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
manual test
Closes #28037 from cloud-fan/ui.
Authored-by: Wenchen Fan
Signed-off-by: Dongjoon Hyun
---
.../spark/sql/execution/metric/SQLMetrics.scala| 60 +++---
.../spark/sql/execution/ui/SparkPlanGraph.scala| 7 ++-
.../sql/execution/metric/SQLMetricsSuite.scala | 33 +++-
.../sql/execution/metric/SQLMetricsTestUtils.scala | 12 ++---
.../execution/ui/SQLAppStatusListenerSuite.scala | 9 ++--
5 files changed, 68 insertions(+), 53 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
index 1394e0f..92d2179 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
@@ -116,26 +116,23 @@ object SQLMetrics {
// data size total (min, med, max):
// 100GB (100MB, 1GB, 10GB)
val acc = new SQLMetric(SIZE_METRIC, -1)
-acc.register(sc, name = Some(s"$name total (min, med, max (stageId:
taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
def createTimingMetric(sc: SparkContext, name: String): SQLMetric = {
// The final result of this metric in physical operator UI may looks like:
-// duration(min, med, max):
+// duration total (min, med, max):
// 5s (800ms, 1s, 2s)
val acc = new SQLMetric(TIMING_METRIC, -1)
-acc.register(sc, name = Some(s"$name total (min, med, max (stageId:
taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
def createNanoTimingMetric(sc: SparkContext, name: String): SQLMetric = {
// Same with createTimingMetric, just normalize the unit of time to
millisecond.
val acc = new SQLMetric(NS_TIMING_METRIC, -1)
-acc.register(sc, name = Some(s"$name total (min, med, max (stageId:
taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
@@ -150,8 +147,7 @@ object SQLMetrics {
// probe avg (min, med, max):
// (1.2, 2.2, 6.3)
val acc = new SQLMetric(AVERAGE_METRIC)
-acc.register(sc, name = Some(s"$name (min, med, max (stageId: taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
@@ -164,13 +160,15 @@ object SQLMetrics {
metricsType != SUM_METRIC
}
+ private val METRICS_NAME_SUFFIX = "(min, med, max (stageId: taskId))"
+
/**
* A function that defines how we aggregate the final accumulator results
among all tasks,
* and represent it in string for a SQL physical operator.
*/
def stringValue(metricsType: String, values: Array[Long], maxMetrics:
Array[Long]): String = {
-// stringMetric = "(driver)" OR (stage ${stageId}.${attemptId}: task
$taskId)
-val stringMetric = if (maxMetrics.isEmpty) {
+// taskInfo = "(driver)" OR (stage ${stageId}.${attemptId}: task $taskId)
+val taskInfo = if (maxMetrics.isEmpty) {
"(driver)"
} else {
s"(stage ${maxMetrics(1)}.${maxMetrics(2)}: task ${maxMetrics(3)})"
@@ -180,18 +178,20 @@ object SQLMetrics {
numberFormat.format(values.sum)
} else if (metricsType == AVERAGE_METRIC) {
val validValues = value
[spark] branch branch-3.0 updated: [SPARK-31271][UI] fix web ui for driver side SQL metrics
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 7c90ec0 [SPARK-31271][UI] fix web ui for driver side SQL metrics
7c90ec0 is described below
commit 7c90ec065f81c3933eef1f0dd172f1a518b1232b
Author: Wenchen Fan
AuthorDate: Fri Mar 27 15:45:35 2020 -0700
[SPARK-31271][UI] fix web ui for driver side SQL metrics
### What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/23551, we changed the metrics type
of driver-side SQL metrics to size/time etc. which comes with max/min/median
info.
This doesn't make sense for driver side SQL metrics as they have only one
value. It makes the web UI hard to read:
This PR updates the SQL metrics UI to only display max/min/median if there
are more than one metrics values:
### Why are the changes needed?
Makes the UI easier to read
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
manual test
Closes #28037 from cloud-fan/ui.
Authored-by: Wenchen Fan
Signed-off-by: Dongjoon Hyun
(cherry picked from commit c4e98c065c99d2cf840e6006ee5414fbaaba9937)
Signed-off-by: Dongjoon Hyun
---
.../spark/sql/execution/metric/SQLMetrics.scala| 60 +++---
.../spark/sql/execution/ui/SparkPlanGraph.scala| 7 ++-
.../sql/execution/metric/SQLMetricsSuite.scala | 33 +++-
.../sql/execution/metric/SQLMetricsTestUtils.scala | 12 ++---
.../execution/ui/SQLAppStatusListenerSuite.scala | 9 ++--
5 files changed, 68 insertions(+), 53 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
index 1394e0f..92d2179 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
@@ -116,26 +116,23 @@ object SQLMetrics {
// data size total (min, med, max):
// 100GB (100MB, 1GB, 10GB)
val acc = new SQLMetric(SIZE_METRIC, -1)
-acc.register(sc, name = Some(s"$name total (min, med, max (stageId:
taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
def createTimingMetric(sc: SparkContext, name: String): SQLMetric = {
// The final result of this metric in physical operator UI may looks like:
-// duration(min, med, max):
+// duration total (min, med, max):
// 5s (800ms, 1s, 2s)
val acc = new SQLMetric(TIMING_METRIC, -1)
-acc.register(sc, name = Some(s"$name total (min, med, max (stageId:
taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
def createNanoTimingMetric(sc: SparkContext, name: String): SQLMetric = {
// Same with createTimingMetric, just normalize the unit of time to
millisecond.
val acc = new SQLMetric(NS_TIMING_METRIC, -1)
-acc.register(sc, name = Some(s"$name total (min, med, max (stageId:
taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
@@ -150,8 +147,7 @@ object SQLMetrics {
// probe avg (min, med, max):
// (1.2, 2.2, 6.3)
val acc = new SQLMetric(AVERAGE_METRIC)
-acc.register(sc, name = Some(s"$name (min, med, max (stageId: taskId))"),
- countFailedValues = false)
+acc.register(sc, name = Some(name), countFailedValues = false)
acc
}
@@ -164,13 +160,15 @@ object SQLMetrics {
metricsType != SUM_METRIC
}
+ private val METRICS_NAME_SUFFIX = "(min, med, max (stageId: taskId))"
+
/**
* A function that defines how we aggregate the final accumulator results
among all tasks,
* and represent it in string for a SQL physical operator.
*/
def stringValue(metricsType: String, values: Array[Long], maxMetrics:
Array[Long]): String = {
-// stringMetric = "(driver)" OR (stage ${stageId}.${attemptId}: task
$taskId)
-val stringMetric = if (maxMetrics.isEmpty) {
+// taskInfo = "(driver)" OR (stage ${stageId}.${attemptId}: task $taskId)
+val taskInfo = if (maxMetrics.isEmpty) {
"(driver)"
} else {
s"(stage ${maxMetrics(1)}.${maxMetrics(2)}: task ${maxMetrics(3)})"
@@ -180,18 +178,20 @@ object SQLMetrics {
[spark] branch branch-3.0 updated: [SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix readResourceOrcFile to create a local file from resource
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 4e13ba9 [SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix
readResourceOrcFile to create a local file from resource
4e13ba9 is described below
commit 4e13ba90446745fc5a9f46ed1f80c6eefb738795
Author: Dongjoon Hyun
AuthorDate: Fri Mar 27 18:44:53 2020 -0700
[SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix readResourceOrcFile to
create a local file from resource
### What changes were proposed in this pull request?
This PR aims to copy a test resource file to a local file in `OrcTest`
suite before reading it.
### Why are the changes needed?
SPARK-31238 and SPARK-31284 added test cases to access the resouce file in
`sql/core` module from `sql/hive` module. In **Maven** test environment, this
causes a failure.
```
- SPARK-31238: compatibility with Spark 2.4 in reading dates *** FAILED ***
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
path in absolute URI:
jar:file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/sql/core/target/spark-sql_2.12-3.1.0-SNAPSHOT-tests.jar!/test-data/before_1582_date_v2_4.snappy.orc
```
```
- SPARK-31284: compatibility with Spark 2.4 in reading timestamps ***
FAILED ***
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
path in absolute URI:
jar:file:/home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3/sql/core/target/spark-sql_2.12-3.1.0-SNAPSHOT-tests.jar!/test-data/before_1582_ts_v2_4.snappy.orc
```
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Pass the Jenkins with Maven.
Closes #28059 from dongjoon-hyun/SPARK-31238.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e)
Signed-off-by: Dongjoon Hyun
---
.../org/apache/spark/sql/execution/datasources/orc/OrcTest.scala | 7 ++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
index 16772fe..e929f90 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
@@ -22,6 +22,7 @@ import java.io.File
import scala.reflect.ClassTag
import scala.reflect.runtime.universe.TypeTag
+import org.apache.commons.io.FileUtils
import org.scalatest.BeforeAndAfterAll
import org.apache.spark.sql._
@@ -136,6 +137,10 @@ abstract class OrcTest extends QueryTest with
FileBasedDataSourceTest with Befor
protected def readResourceOrcFile(name: String): DataFrame = {
val url = Thread.currentThread().getContextClassLoader.getResource(name)
-spark.read.orc(url.toString)
+// Copy to avoid URISyntaxException when `sql/hive` accesses the resources
in `sql/core`
+val file = File.createTempFile("orc-test", ".orc")
+file.deleteOnExit();
+FileUtils.copyURLToFile(url, file)
+spark.read.orc(file.getAbsolutePath)
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c4e98c0 -> d025ddba)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c4e98c0 [SPARK-31271][UI] fix web ui for driver side SQL metrics add d025ddba [SPARK-31238][SPARK-31284][TEST][FOLLOWUP] Fix readResourceOrcFile to create a local file from resource No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/execution/datasources/orc/OrcTest.scala | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0b237bd -> 34c7476)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0b237bd [SPARK-31292][CORE][SQL] Replace toSet.toSeq with distinct for readability add 34c7476 [SPARK-30722][DOCS][FOLLOW-UP] Add Pandas Function API into the menu No new revisions were added by this update. Summary of changes: docs/_data/menu-sql.yaml | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30722][DOCS][FOLLOW-UP] Add Pandas Function API into the menu
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 1c4fe31 [SPARK-30722][DOCS][FOLLOW-UP] Add Pandas Function API into the menu 1c4fe31 is described below commit 1c4fe31a5a697c80ca59ed6286ad1423d8541e6a Author: HyukjinKwon AuthorDate: Sat Mar 28 18:36:34 2020 -0700 [SPARK-30722][DOCS][FOLLOW-UP] Add Pandas Function API into the menu ### What changes were proposed in this pull request? This PR adds "Pandas Function API" into the menu. ### Why are the changes needed? To be consistent and to make easier to navigate. ### Does this PR introduce any user-facing change? No, master only.  ### How was this patch tested? Manually verified by `SKIP_API=1 jekyll build`. Closes #28054 from HyukjinKwon/followup-spark-30722. Authored-by: HyukjinKwon Signed-off-by: Dongjoon Hyun (cherry picked from commit 34c7476cb5da98d5f3be354669dcd762df2b75e1) Signed-off-by: Dongjoon Hyun --- docs/_data/menu-sql.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/_data/menu-sql.yaml b/docs/_data/menu-sql.yaml index c17bfd3..8a1d937 100644 --- a/docs/_data/menu-sql.yaml +++ b/docs/_data/menu-sql.yaml @@ -67,6 +67,8 @@ url: sql-pyspark-pandas-with-arrow.html#enabling-for-conversion-tofrom-pandas - text: "Pandas UDFs (a.k.a. Vectorized UDFs)" url: sql-pyspark-pandas-with-arrow.html#pandas-udfs-aka-vectorized-udfs +- text: "Pandas Function APIs" + url: sql-pyspark-pandas-with-arrow.html#pandas-function-apis - text: Usage Notes url: sql-pyspark-pandas-with-arrow.html#usage-notes - text: Migration Guide - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31280][SQL] Perform propagating empty relation after RewritePredicateSubquery
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f376d24 [SPARK-31280][SQL] Perform propagating empty relation after
RewritePredicateSubquery
f376d24 is described below
commit f376d24ea1f40740864d38ceb424713372e7e6ce
Author: Kent Yao
AuthorDate: Sun Mar 29 11:32:22 2020 -0700
[SPARK-31280][SQL] Perform propagating empty relation after
RewritePredicateSubquery
### What changes were proposed in this pull request?
```sql
scala> spark.sql(" select * from values(1), (2) t(key) where key in (select
1 as key where 1=0)").queryExecution
res15: org.apache.spark.sql.execution.QueryExecution =
== Parsed Logical Plan ==
'Project [*]
+- 'Filter 'key IN (list#39 [])
: +- Project [1 AS key#38]
: +- Filter (1 = 0)
:+- OneRowRelation
+- 'SubqueryAlias t
+- 'UnresolvedInlineTable [key], [List(1), List(2)]
== Analyzed Logical Plan ==
key: int
Project [key#40]
+- Filter key#40 IN (list#39 [])
: +- Project [1 AS key#38]
: +- Filter (1 = 0)
:+- OneRowRelation
+- SubqueryAlias t
+- LocalRelation [key#40]
== Optimized Logical Plan ==
Join LeftSemi, (key#40 = key#38)
:- LocalRelation [key#40]
+- LocalRelation , [key#38]
== Physical Plan ==
*(1) BroadcastHashJoin [key#40], [key#38], LeftSemi, BuildRight
:- *(1) LocalTableScan [key#40]
+- Br...
```
`LocalRelation ` should be able to propagate after subqueries are
lift up to joins
### Why are the changes needed?
optimize query
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
add new tests
Closes #28043 from yaooqinn/SPARK-31280.
Authored-by: Kent Yao
Signed-off-by: Dongjoon Hyun
---
.../spark/sql/catalyst/optimizer/Optimizer.scala | 2 ++
.../catalyst/optimizer/RewriteSubquerySuite.scala | 17 +++---
.../apache/spark/sql/execution/PlannerSuite.scala | 36 ++
3 files changed, 51 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index da147dd..827f528 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -198,6 +198,8 @@ abstract class Optimizer(catalogManager: CatalogManager)
CheckCartesianProducts) :+
Batch("RewriteSubquery", Once,
RewritePredicateSubquery,
+ ConvertToLocalRelation,
+ PropagateEmptyRelation,
ColumnPruning,
CollapseProject,
RemoveNoopOperators) :+
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala
index f00d22e..2238afd 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala
@@ -22,17 +22,17 @@ import org.apache.spark.sql.catalyst.dsl.plans._
import org.apache.spark.sql.catalyst.expressions.ListQuery
import org.apache.spark.sql.catalyst.plans.{LeftSemi, PlanTest}
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
-import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.catalyst.rules.{Rule, RuleExecutor}
class RewriteSubquerySuite extends PlanTest {
- object Optimize extends RuleExecutor[LogicalPlan] {
+ case class Optimize(addOn: Rule[LogicalPlan]) extends
RuleExecutor[LogicalPlan] {
val batches =
Batch("Column Pruning", FixedPoint(100), ColumnPruning) ::
Batch("Rewrite Subquery", FixedPoint(1),
RewritePredicateSubquery,
-ColumnPruning,
+addOn,
CollapseProject,
RemoveNoopOperators) :: Nil
}
@@ -43,7 +43,7 @@ class RewriteSubquerySuite extends PlanTest {
val query =
relation.where('a.in(ListQuery(relInSubquery.select('x.select('a)
-val optimized = Optimize.execute(query.analyze)
+val optimized = Optimize(ColumnPruning).execute(query.analyze)
val correctAnswer = relation
.select('a)
.join(relInSubquery.select('x), LeftSemi, Some('a === 'x))
@@ -52,4 +52,13 @@ class RewriteSubquerySuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+ test("SPARK-31280: Per
[spark] branch master updated (e656e99 -> f376d24)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e656e99 [SPARK-30363][SQL][DOCS][FOLLOWUP] Fix a broken link in SQL Reference add f376d24 [SPARK-31280][SQL] Perform propagating empty relation after RewritePredicateSubquery No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/Optimizer.scala | 2 ++ .../catalyst/optimizer/RewriteSubquerySuite.scala | 17 +++--- .../apache/spark/sql/execution/PlannerSuite.scala | 36 ++ 3 files changed, 51 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and help messages for Kinesis integration
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 83f943c [SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and
help messages for Kinesis integration
83f943c is described below
commit 83f943c4b922413159cdcc09dfee7ef67d586215
Author: Kengo Seki
AuthorDate: Sun Mar 29 14:27:19 2020 -0700
[SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and help messages
for Kinesis integration
### What changes were proposed in this pull request?
This PR (SPARK-31293) fixes wrong command examples, parameter descriptions
and help message format for Amazon Kinesis integration with Spark Streaming.
### Why are the changes needed?
To improve usability of those commands.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
I ran the fixed commands manually and confirmed they worked as expected.
Closes #28063 from sekikn/SPARK-31293.
Authored-by: Kengo Seki
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 60dd1a690fed62b1d6442cdc8cf3f89ef4304d5a)
Signed-off-by: Dongjoon Hyun
---
docs/streaming-kinesis-integration.md | 3 +--
.../main/python/examples/streaming/kinesis_wordcount_asl.py| 7 ---
.../apache/spark/examples/streaming/KinesisWordCountASL.scala | 10 +-
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/docs/streaming-kinesis-integration.md
b/docs/streaming-kinesis-integration.md
index e68d513..db813c4 100644
--- a/docs/streaming-kinesis-integration.md
+++ b/docs/streaming-kinesis-integration.md
@@ -246,8 +246,7 @@ To run the example,
-./bin/spark-submit --jars external/kinesis-asl/target/scala-*/\
-spark-streaming-kinesis-asl-assembly_*.jar \
+./bin/spark-submit --jars
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
[Kinesis app name] [Kinesis stream name] [endpoint URL] [region
name]
diff --git
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
index 49794fa..777a332 100644
---
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
+++
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
@@ -26,6 +26,7 @@
name of the Kinesis stream (ie. mySparkStream)
endpoint of the Kinesis service
(e.g. https://kinesis.us-east-1.amazonaws.com)
+ region name of the Kinesis endpoint (e.g. us-east-1)
Example:
@@ -34,10 +35,10 @@
$ export AWS_SECRET_KEY=
# run the example
- $ bin/spark-submit -jars external/kinesis-asl/target/scala-*/\
-spark-streaming-kinesis-asl-assembly_*.jar \
+ $ bin/spark-submit --jars \
+
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
-myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
+myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
us-east-1
There is a companion helper class called KinesisWordProducerASL which puts
dummy data
onto the Kinesis stream.
diff --git
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
index a5d5ac7..32f4a67 100644
---
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
+++
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
@@ -78,7 +78,7 @@ object KinesisWordCountASL extends Logging {
if (args.length != 3) {
System.err.println(
"""
- |Usage: KinesisWordCountASL
+ |Usage: KinesisWordCountASL
|
| is the name of the consumer app, used to track the
read data in DynamoDB
| is the name of the Kinesis stream
@@ -171,11 +171,11 @@ object KinesisWordCountASL extends Logging {
*is the endpoint of the Kinesis service
* (ie. https://kinesis.us-east-1.amazonaws.com)
*is the rate of records per second to put onto the stream
- *is the rate of records per second to put onto the
stream
+ *is the number of words per record
*
* Example:
*$ SPARK_HOME/bin/run-example streaming.KinesisWordProducerASL
mySparkStream \
- * https://kinesis.us-
[spark] branch master updated: [SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and help messages for Kinesis integration
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 60dd1a6 [SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and
help messages for Kinesis integration
60dd1a6 is described below
commit 60dd1a690fed62b1d6442cdc8cf3f89ef4304d5a
Author: Kengo Seki
AuthorDate: Sun Mar 29 14:27:19 2020 -0700
[SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and help messages
for Kinesis integration
### What changes were proposed in this pull request?
This PR (SPARK-31293) fixes wrong command examples, parameter descriptions
and help message format for Amazon Kinesis integration with Spark Streaming.
### Why are the changes needed?
To improve usability of those commands.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
I ran the fixed commands manually and confirmed they worked as expected.
Closes #28063 from sekikn/SPARK-31293.
Authored-by: Kengo Seki
Signed-off-by: Dongjoon Hyun
---
docs/streaming-kinesis-integration.md | 3 +--
.../main/python/examples/streaming/kinesis_wordcount_asl.py| 7 ---
.../apache/spark/examples/streaming/KinesisWordCountASL.scala | 10 +-
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/docs/streaming-kinesis-integration.md
b/docs/streaming-kinesis-integration.md
index e68d513..db813c4 100644
--- a/docs/streaming-kinesis-integration.md
+++ b/docs/streaming-kinesis-integration.md
@@ -246,8 +246,7 @@ To run the example,
-./bin/spark-submit --jars external/kinesis-asl/target/scala-*/\
-spark-streaming-kinesis-asl-assembly_*.jar \
+./bin/spark-submit --jars
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
[Kinesis app name] [Kinesis stream name] [endpoint URL] [region
name]
diff --git
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
index 49794fa..777a332 100644
---
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
+++
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
@@ -26,6 +26,7 @@
name of the Kinesis stream (ie. mySparkStream)
endpoint of the Kinesis service
(e.g. https://kinesis.us-east-1.amazonaws.com)
+ region name of the Kinesis endpoint (e.g. us-east-1)
Example:
@@ -34,10 +35,10 @@
$ export AWS_SECRET_KEY=
# run the example
- $ bin/spark-submit -jars external/kinesis-asl/target/scala-*/\
-spark-streaming-kinesis-asl-assembly_*.jar \
+ $ bin/spark-submit --jars \
+
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
-myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
+myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
us-east-1
There is a companion helper class called KinesisWordProducerASL which puts
dummy data
onto the Kinesis stream.
diff --git
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
index a5d5ac7..32f4a67 100644
---
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
+++
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
@@ -78,7 +78,7 @@ object KinesisWordCountASL extends Logging {
if (args.length != 3) {
System.err.println(
"""
- |Usage: KinesisWordCountASL
+ |Usage: KinesisWordCountASL
|
| is the name of the consumer app, used to track the
read data in DynamoDB
| is the name of the Kinesis stream
@@ -171,11 +171,11 @@ object KinesisWordCountASL extends Logging {
*is the endpoint of the Kinesis service
* (ie. https://kinesis.us-east-1.amazonaws.com)
*is the rate of records per second to put onto the stream
- *is the rate of records per second to put onto the
stream
+ *is the number of words per record
*
* Example:
*$ SPARK_HOME/bin/run-example streaming.KinesisWordProducerASL
mySparkStream \
- * https://kinesis.us-east-1.amazonaws.com us-east-1 10 5
+ * https://kinesis.us-east-1.amazonaws.com 10 5
*/
object Kinesi
[spark] branch branch-2.4 updated: [SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and help messages for Kinesis integration
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new f05ac28a [SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and
help messages for Kinesis integration
f05ac28a is described below
commit f05ac28a949b9ad5b154ac959b9da6a24a4e09e6
Author: Kengo Seki
AuthorDate: Sun Mar 29 14:27:19 2020 -0700
[SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and help messages
for Kinesis integration
This PR (SPARK-31293) fixes wrong command examples, parameter descriptions
and help message format for Amazon Kinesis integration with Spark Streaming.
To improve usability of those commands.
No
I ran the fixed commands manually and confirmed they worked as expected.
Closes #28063 from sekikn/SPARK-31293.
Authored-by: Kengo Seki
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 60dd1a690fed62b1d6442cdc8cf3f89ef4304d5a)
Signed-off-by: Dongjoon Hyun
---
docs/streaming-kinesis-integration.md | 3 +--
.../main/python/examples/streaming/kinesis_wordcount_asl.py| 7 ---
.../apache/spark/examples/streaming/KinesisWordCountASL.scala | 10 +-
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/docs/streaming-kinesis-integration.md
b/docs/streaming-kinesis-integration.md
index 6a52e8a..0685cd8 100644
--- a/docs/streaming-kinesis-integration.md
+++ b/docs/streaming-kinesis-integration.md
@@ -217,8 +217,7 @@ To run the example,
-bin/spark-submit --jars external/kinesis-asl/target/scala-*/\
-spark-streaming-kinesis-asl-assembly_*.jar \
+./bin/spark-submit --jars
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
[Kinesis app name] [Kinesis stream name] [endpoint URL] [region
name]
diff --git
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
index 49794fa..777a332 100644
---
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
+++
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
@@ -26,6 +26,7 @@
name of the Kinesis stream (ie. mySparkStream)
endpoint of the Kinesis service
(e.g. https://kinesis.us-east-1.amazonaws.com)
+ region name of the Kinesis endpoint (e.g. us-east-1)
Example:
@@ -34,10 +35,10 @@
$ export AWS_SECRET_KEY=
# run the example
- $ bin/spark-submit -jars external/kinesis-asl/target/scala-*/\
-spark-streaming-kinesis-asl-assembly_*.jar \
+ $ bin/spark-submit --jars \
+
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
-myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
+myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
us-east-1
There is a companion helper class called KinesisWordProducerASL which puts
dummy data
onto the Kinesis stream.
diff --git
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
index fcb790e..d97ab74 100644
---
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
+++
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
@@ -78,7 +78,7 @@ object KinesisWordCountASL extends Logging {
if (args.length != 3) {
System.err.println(
"""
- |Usage: KinesisWordCountASL
+ |Usage: KinesisWordCountASL
|
| is the name of the consumer app, used to track the
read data in DynamoDB
| is the name of the Kinesis stream
@@ -171,11 +171,11 @@ object KinesisWordCountASL extends Logging {
*is the endpoint of the Kinesis service
* (ie. https://kinesis.us-east-1.amazonaws.com)
*is the rate of records per second to put onto the stream
- *is the rate of records per second to put onto the
stream
+ *is the number of words per record
*
* Example:
*$ SPARK_HOME/bin/run-example streaming.KinesisWordProducerASL
mySparkStream \
- * https://kinesis.us-east-1.amazonaws.com us-east-1 10 5
+ * https://kinesis.us-east-1.amazonaws.com 10 5
*/
object KinesisWordProducerASL {
def main(args: Array[String]) {
@@ -183,13 +183,
[spark] branch branch-2.4 updated: [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 4add8ad [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16
4add8ad is described below
commit 4add8ad70c19518722d643948de3b78c1fcfefe5
Author: Jungtaek Lim (HeartSaVioR)
AuthorDate: Sun Mar 29 19:25:57 2020 -0700
[SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16
### What changes were proposed in this pull request?
This PR(SPARK-31101) proposes to upgrade Janino to 3.0.16 which is released
recently.
* Merged pull request janino-compiler/janino#114 "Grow the code for
relocatables, and do fixup, and relocate".
Please see the commit log.
- https://github.com/janino-compiler/janino/commits/3.0.16
You can see the changelog from the link:
http://janino-compiler.github.io/janino/changelog.html / though release note
for Janino 3.0.16 is actually incorrect.
### Why are the changes needed?
We got some report on failure on user's query which Janino throws error on
compiling generated code. The issue is here: janino-compiler/janino#113 It
contains the information of generated code, symptom (error), and analysis of
the bug, so please refer the link for more details.
Janino 3.0.16 contains the PR janino-compiler/janino#114 which would enable
Janino to succeed to compile user's query properly.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Existing UTs.
Below test code fails on branch-2.4 and passes with this patch.
(Note that there seems to be the case where another UT affects this UT to
not fail - adding this to SQLQuerySuite won't fail this UT, but adding this to
DateFunctionsSuite will fail this UT, and if you run this UT solely in
SQLQuerySuite via `build/sbt "sql/testOnly *.SQLQuerySuite -- -z SPARK-31115"`
then it fails.)
```
/**
* NOTE: The test code tries to control the size of for/switch statement
in expand_doConsume,
* as well as the overall size of expand_doConsume, so that the query
triggers known Janino
* bug - https://github.com/janino-compiler/janino/issues/113.
*
* The expected exception message from Janino when we use switch
statement for "ExpandExec":
* - "Operand stack inconsistent at offset xxx: Previous size 1, now 0"
* which will not happen when we use if-else-if statement for
"ExpandExec".
*
* "The number of fields" and "The number of distinct aggregation
functions" are the major
* factors to increase the size of generated code: while these values
should be large enough
* to trigger the Janino bug, these values should not also too big;
otherwise one of below
* exceptions might be thrown:
* - "expand_doConsume would be beyond 64KB"
* - "java.lang.ClassFormatError: Too many arguments in method signature
in class file"
*/
test("SPARK-31115 Lots of columns and distinct aggregations shouldn't
break code generation") {
withSQLConf(
(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key, "true"),
(SQLConf.WHOLESTAGE_MAX_NUM_FIELDS.key, "1"),
(SQLConf.CODEGEN_FALLBACK.key, "false"),
(SQLConf.CODEGEN_LOGGING_MAX_LINES.key, "-1")
) {
var df = Seq(("1", "2", 1), ("1", "2", 2), ("2", "3", 3), ("2", "3",
4)).toDF("a", "b", "c")
// The value is tested under commit
"244405fe57d7737d81c34ba9e8917df6285889eb":
// the query fails with switch statement, whereas it passes with
if-else statement.
// Note that the value depends on the Spark logic as well - different
Spark versions may
// require different value to ensure the test failing with switch
statement.
val numNewFields = 100
df = df.withColumns(
(1 to numNewFields).map { idx => s"a$idx" },
(1 to numNewFields).map { idx =>
when(col("c").mod(lit(2)).===(lit(0)),
lit(idx)).otherwise(col("c"))
}
)
val aggExprs: Array[Column] = Range(1, numNewFields).map { idx =>
if (idx % 2 == 0) {
coalesce(countDistinct(s"a$idx"), lit(0))
} else {
coalesce(count(s"a$idx"), lit(0))
}
}.toArray
val aggDf = df
.groupBy("a", "b")
.agg(aggExprs.he
[spark] branch master updated (0d997e5 -> 1d0fc9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0d997e5 [SPARK-31219][YARN] Enable closeIdleConnections in YarnShuffleService add 1d0fc9a [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker entrypoint.sh No new revisions were added by this update. Summary of changes: .../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0d997e5 -> 1d0fc9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0d997e5 [SPARK-31219][YARN] Enable closeIdleConnections in YarnShuffleService add 1d0fc9a [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker entrypoint.sh No new revisions were added by this update. Summary of changes: .../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0d997e5 -> 1d0fc9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0d997e5 [SPARK-31219][YARN] Enable closeIdleConnections in YarnShuffleService add 1d0fc9a [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker entrypoint.sh No new revisions were added by this update. Summary of changes: .../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0d997e5 -> 1d0fc9a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0d997e5 [SPARK-31219][YARN] Enable closeIdleConnections in YarnShuffleService add 1d0fc9a [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker entrypoint.sh No new revisions were added by this update. Summary of changes: .../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1d0fc9a -> aa98ac5)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1d0fc9a [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker entrypoint.sh add aa98ac5 [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation No new revisions were added by this update. Summary of changes: docs/monitoring.md | 58 +++--- 1 file changed, 51 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (aa98ac5 -> cda2e30)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from aa98ac5 [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation add cda2e30 Revert "[SPARK-31280][SQL] Perform propagating empty relation after RewritePredicateSubquery" No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/Optimizer.scala | 2 -- .../catalyst/optimizer/RewriteSubquerySuite.scala | 17 +++--- .../apache/spark/sql/execution/PlannerSuite.scala | 36 -- 3 files changed, 4 insertions(+), 51 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (aa98ac5 -> cda2e30)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from aa98ac5 [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation add cda2e30 Revert "[SPARK-31280][SQL] Perform propagating empty relation after RewritePredicateSubquery" No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/Optimizer.scala | 2 -- .../catalyst/optimizer/RewriteSubquerySuite.scala | 17 +++--- .../apache/spark/sql/execution/PlannerSuite.scala | 36 -- 3 files changed, 4 insertions(+), 51 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker entrypoint.sh
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 5a96ee7 [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in
Docker entrypoint.sh
5a96ee7 is described below
commit 5a96ee7619ea07edefd030c66641e6e473a890e0
Author: Đặng Minh Dũng
AuthorDate: Mon Mar 30 15:41:57 2020 -0700
[SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker
entrypoint.sh
A small change to fix an error in Docker `entrypoint.sh`
When spark running on Kubernetes, I got the following logs:
```log
+ '[' -n ']'
+ '[' -z ']'
++ /bin/hadoop classpath
/opt/entrypoint.sh: line 62: /bin/hadoop: No such file or directory
+ export SPARK_DIST_CLASSPATH=
+ SPARK_DIST_CLASSPATH=
```
This is because you are missing some quotes on bash comparisons.
No
CI
Closes #28075 from dungdm93/patch-1.
Authored-by: Đặng Minh Dũng
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 1d0fc9aa85b3ad3326b878de49b748413dee1dd9)
Signed-off-by: Dongjoon Hyun
---
.../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
index 6ee3523..8218c29 100755
---
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
+++
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
@@ -58,8 +58,8 @@ fi
# If HADOOP_HOME is set and SPARK_DIST_CLASSPATH is not set, set it here so
Hadoop jars are available to the executor.
# It does not set SPARK_DIST_CLASSPATH if already set, to avoid overriding
customizations of this value from elsewhere e.g. Docker/K8s.
-if [ -n ${HADOOP_HOME} ] && [ -z ${SPARK_DIST_CLASSPATH} ]; then
- export SPARK_DIST_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath)
+if [ -n "${HADOOP_HOME}" ] && [ -z "${SPARK_DIST_CLASSPATH}" ]; then
+ export SPARK_DIST_CLASSPATH="$($HADOOP_HOME/bin/hadoop classpath)"
fi
if ! [ -z ${HADOOP_CONF_DIR+x} ]; then
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker entrypoint.sh
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 5a96ee7 [SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in
Docker entrypoint.sh
5a96ee7 is described below
commit 5a96ee7619ea07edefd030c66641e6e473a890e0
Author: Đặng Minh Dũng
AuthorDate: Mon Mar 30 15:41:57 2020 -0700
[SPARK-29574][K8S][FOLLOWUP] Fix bash comparison error in Docker
entrypoint.sh
A small change to fix an error in Docker `entrypoint.sh`
When spark running on Kubernetes, I got the following logs:
```log
+ '[' -n ']'
+ '[' -z ']'
++ /bin/hadoop classpath
/opt/entrypoint.sh: line 62: /bin/hadoop: No such file or directory
+ export SPARK_DIST_CLASSPATH=
+ SPARK_DIST_CLASSPATH=
```
This is because you are missing some quotes on bash comparisons.
No
CI
Closes #28075 from dungdm93/patch-1.
Authored-by: Đặng Minh Dũng
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 1d0fc9aa85b3ad3326b878de49b748413dee1dd9)
Signed-off-by: Dongjoon Hyun
---
.../kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
index 6ee3523..8218c29 100755
---
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
+++
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
@@ -58,8 +58,8 @@ fi
# If HADOOP_HOME is set and SPARK_DIST_CLASSPATH is not set, set it here so
Hadoop jars are available to the executor.
# It does not set SPARK_DIST_CLASSPATH if already set, to avoid overriding
customizations of this value from elsewhere e.g. Docker/K8s.
-if [ -n ${HADOOP_HOME} ] && [ -z ${SPARK_DIST_CLASSPATH} ]; then
- export SPARK_DIST_CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath)
+if [ -n "${HADOOP_HOME}" ] && [ -z "${SPARK_DIST_CLASSPATH}" ]; then
+ export SPARK_DIST_CLASSPATH="$($HADOOP_HOME/bin/hadoop classpath)"
fi
if ! [ -z ${HADOOP_CONF_DIR+x} ]; then
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new ca3887a [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation ca3887a is described below commit ca3887a0de31fa78097ca7ee92ead914a3ce050c Author: Luca Canali AuthorDate: Mon Mar 30 18:00:54 2020 -0700 [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation ### What changes were proposed in this pull request? This PR (SPARK-30775) aims to improve the description of the executor metrics in the monitoring documentation. ### Why are the changes needed? Improve and clarify monitoring documentation by: - adding reference to the Prometheus end point, as implemented in [SPARK-29064] - extending the list and descripion of executor metrics, following up from [SPARK-27157] ### Does this PR introduce any user-facing change? Documentation update. ### How was this patch tested? n.a. Closes #27526 from LucaCanali/docPrometheusMetricsFollowupSpark29064. Authored-by: Luca Canali Signed-off-by: Dongjoon Hyun (cherry picked from commit aa98ac52dbbe3fc2d3b152af9324a71f48439a38) Signed-off-by: Dongjoon Hyun --- docs/monitoring.md | 58 +++--- 1 file changed, 51 insertions(+), 7 deletions(-) diff --git a/docs/monitoring.md b/docs/monitoring.md index ba3f1dc..131cd2a 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -689,9 +689,12 @@ A list of the available metrics, with a short description: ### Executor Metrics Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe the performance metrics of Executor itself like JVM heap memory, GC information. -Executor metric values and their measured peak values per executor are exposed via the REST API at the end point `/applications/[app-id]/executors`. -In addition, aggregated per-stage peak values of the executor metrics are written to the event log if `spark.eventLog.logStageExecutorMetrics` is true. -Executor metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. +Executor metric values and their measured memory peak values per executor are exposed via the REST API in JSON format and in Prometheus format. +The JSON end point is exposed at: `/applications/[app-id]/executors`, and the Prometheus endpoint at: `/metrics/executors/prometheus`. +The Prometheus endpoint is conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`). +In addition, aggregated per-stage peak values of the executor memory metrics are written to the event log if +`spark.eventLog.logStageExecutorMetrics` is true. +Executor memory metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. A list of the available metrics, with a short description: @@ -699,21 +702,62 @@ A list of the available metrics, with a short description: Short description +rddBlocks +RDD blocks in the block manager of this executor. + + +memoryUsed +Storage memory used by this executor. + + +diskUsed +Disk space used for RDD storage by this executor. + + +totalCores +Number of cores available in this executor. + + +maxTasks +Maximum number of tasks that can run concurrently in this executor. + + +activeTasks +Number of tasks currently executing. + + +failedTasks +Number of tasks that have failed in this executor. + + +completedTasks +Number of tasks that have completed in this executor. + + +totalTasks +Total number of tasks (running, failed and completed) in this executor. + + +totalDuration +Elapsed time the JVM spent executing tasks in this executor. +The value is expressed in milliseconds. + + totalGCTime -Elapsed time the JVM spent in garbage collection summed in this Executor. +Elapsed time the JVM spent in garbage collection summed in this executor. The value is expressed in milliseconds. totalInputBytes -Total input bytes summed in this Executor. +Total input bytes summed in this executor. totalShuffleRead -Total shuffer read bytes summed in this Executor. +Total shuffle read bytes summed in this executor. totalShuffleWrite -Total shuffer write bytes summed in this Executor. +Total shuffle write bytes summed in this executor. maxMemory - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h
[spark] branch branch-3.0 updated: [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new ca3887a [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation ca3887a is described below commit ca3887a0de31fa78097ca7ee92ead914a3ce050c Author: Luca Canali AuthorDate: Mon Mar 30 18:00:54 2020 -0700 [SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation ### What changes were proposed in this pull request? This PR (SPARK-30775) aims to improve the description of the executor metrics in the monitoring documentation. ### Why are the changes needed? Improve and clarify monitoring documentation by: - adding reference to the Prometheus end point, as implemented in [SPARK-29064] - extending the list and descripion of executor metrics, following up from [SPARK-27157] ### Does this PR introduce any user-facing change? Documentation update. ### How was this patch tested? n.a. Closes #27526 from LucaCanali/docPrometheusMetricsFollowupSpark29064. Authored-by: Luca Canali Signed-off-by: Dongjoon Hyun (cherry picked from commit aa98ac52dbbe3fc2d3b152af9324a71f48439a38) Signed-off-by: Dongjoon Hyun --- docs/monitoring.md | 58 +++--- 1 file changed, 51 insertions(+), 7 deletions(-) diff --git a/docs/monitoring.md b/docs/monitoring.md index ba3f1dc..131cd2a 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -689,9 +689,12 @@ A list of the available metrics, with a short description: ### Executor Metrics Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe the performance metrics of Executor itself like JVM heap memory, GC information. -Executor metric values and their measured peak values per executor are exposed via the REST API at the end point `/applications/[app-id]/executors`. -In addition, aggregated per-stage peak values of the executor metrics are written to the event log if `spark.eventLog.logStageExecutorMetrics` is true. -Executor metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. +Executor metric values and their measured memory peak values per executor are exposed via the REST API in JSON format and in Prometheus format. +The JSON end point is exposed at: `/applications/[app-id]/executors`, and the Prometheus endpoint at: `/metrics/executors/prometheus`. +The Prometheus endpoint is conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`). +In addition, aggregated per-stage peak values of the executor memory metrics are written to the event log if +`spark.eventLog.logStageExecutorMetrics` is true. +Executor memory metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. A list of the available metrics, with a short description: @@ -699,21 +702,62 @@ A list of the available metrics, with a short description: Short description +rddBlocks +RDD blocks in the block manager of this executor. + + +memoryUsed +Storage memory used by this executor. + + +diskUsed +Disk space used for RDD storage by this executor. + + +totalCores +Number of cores available in this executor. + + +maxTasks +Maximum number of tasks that can run concurrently in this executor. + + +activeTasks +Number of tasks currently executing. + + +failedTasks +Number of tasks that have failed in this executor. + + +completedTasks +Number of tasks that have completed in this executor. + + +totalTasks +Total number of tasks (running, failed and completed) in this executor. + + +totalDuration +Elapsed time the JVM spent executing tasks in this executor. +The value is expressed in milliseconds. + + totalGCTime -Elapsed time the JVM spent in garbage collection summed in this Executor. +Elapsed time the JVM spent in garbage collection summed in this executor. The value is expressed in milliseconds. totalInputBytes -Total input bytes summed in this Executor. +Total input bytes summed in this executor. totalShuffleRead -Total shuffer read bytes summed in this Executor. +Total shuffle read bytes summed in this executor. totalShuffleWrite -Total shuffer write bytes summed in this Executor. +Total shuffle write bytes summed in this executor. maxMemory - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h
[spark] branch master updated (1a7f964 -> 5ec1814)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1a7f964 [SPARK-31305][SQL][DOCS] Add a page to list all commands in SQL Reference add 5ec1814 [SPARK-31248][CORE][TEST] Fix flaky ExecutorAllocationManagerSuite.interleaving add and remove No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala | 2 +- .../test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala | 5 - 2 files changed, 5 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31308][PYSPARK] Merging pyFiles to files argument for Non-PySpark applications
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 20fc6fa [SPARK-31308][PYSPARK] Merging pyFiles to files argument for
Non-PySpark applications
20fc6fa is described below
commit 20fc6fa8398b9dc47b9ae7df52133a306f89b25f
Author: Liang-Chi Hsieh
AuthorDate: Tue Mar 31 18:08:55 2020 -0700
[SPARK-31308][PYSPARK] Merging pyFiles to files argument for Non-PySpark
applications
### What changes were proposed in this pull request?
This PR (SPARK-31308) proposed to add python dependencies even it is not
Python applications.
### Why are the changes needed?
For now, we add `pyFiles` argument to `files` argument only for Python
applications, in SparkSubmit. Like the reason in #21420, "for some Spark
applications, though they're a java program, they require not only jar
dependencies, but also python dependencies.", we need to add `pyFiles` to
`files` even it is not Python applications.
### Does this PR introduce any user-facing change?
Yes. After this change, for non-PySpark applications, the Python files
specified by `pyFiles` are also added to `files` like PySpark applications.
### How was this patch tested?
Manually test on jupyter notebook or do `spark-submit` with `--verbose`.
```
Spark config:
...
(spark.files,file:/Users/dongjoon/PRS/SPARK-PR-28077/a.py)
(spark.submit.deployMode,client)
(spark.master,local[*])
```
Closes #28077 from viirya/pyfile.
Lead-authored-by: Liang-Chi Hsieh
Co-authored-by: Liang-Chi Hsieh
Signed-off-by: Dongjoon Hyun
---
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index 4d67dfa..1271a3d 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -474,10 +474,12 @@ private[spark] class SparkSubmit extends Logging {
args.mainClass = "org.apache.spark.deploy.PythonRunner"
args.childArgs = ArrayBuffer(localPrimaryResource, localPyFiles) ++
args.childArgs
}
- if (clusterManager != YARN) {
-// The YARN backend handles python files differently, so don't merge
the lists.
-args.files = mergeFileLists(args.files, args.pyFiles)
- }
+}
+
+// Non-PySpark applications can need Python dependencies.
+if (deployMode == CLIENT && clusterManager != YARN) {
+ // The YARN backend handles python files differently, so don't merge the
lists.
+ args.files = mergeFileLists(args.files, args.pyFiles)
}
if (localPyFiles != null) {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 08a88e0 [SPARK-31313][K8S][TEST] Add `m01` node name to support
Minikube 1.8.x
08a88e0 is described below
commit 08a88e04f4417c5a0168592d1f01c300b4ae9de5
Author: Dongjoon Hyun
AuthorDate: Wed Apr 1 03:42:26 2020 +
[SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x
### What changes were proposed in this pull request?
This PR aims to add `m01` as a node name additionally to `PVTestsSuite`.
### Why are the changes needed?
minikube 1.8.0 ~ 1.8.2 generate a cluster with a nodename `m01` while all
the other versions have `minikube`. This causes `PVTestSuite` failure.
```
$ minikube --vm-driver=hyperkit start --memory 6000 --cpus 8
* minikube v1.8.2 on Darwin 10.15.3
- MINIKUBE_ACTIVE_DOCKERD=minikube
* Using the hyperkit driver based on user configuration
* Creating hyperkit VM (CPUs=8, Memory=6000MB, Disk=2MB) ...
* Preparing Kubernetes v1.18.0 on Docker 19.03.6 ...
* Launching Kubernetes ...
* Enabling addons: default-storageclass, storage-provisioner
* Waiting for cluster to come online ...
* Done! kubectl is now configured to use "minikube"
$ kubectl get nodes
NAME STATUS ROLESAGE VERSION
m01Readymaster 22s v1.17.3
```
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
This only adds a new node name. So, K8S Jenkins job should passed.
In addition, `K8s` integration test suite should be tested on `minikube
1.8.2` manually.
```
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Test basic decommissioning
- Run SparkR on simple dataframe.R example
Run completed in 10 minutes, 23 seconds.
Total number of tests run: 20
Suites: completed 2, aborted 0
Tests: succeeded 20, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
For the above test, Minikube 1.8.2 and K8s v1.18.0 is used.
```
$ minikube version
minikube version: v1.8.2
commit: eb13446e786c9ef70cb0a9f85a633194e62396a1
$ kubectl version --short
Client Version: v1.18.0
Server Version: v1.18.0
```
Closes #28080 from dongjoon-hyun/SPARK-31313.
Authored-by: Dongjoon Hyun
Signed-off-by: DB Tsai
(cherry picked from commit dba525c997b0033ac1b6fd24236cd72938f94bbf)
Signed-off-by: Dongjoon Hyun
---
.../org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
index f021821..a7cb84e 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
@@ -56,7 +56,8 @@ private[spark] trait PVTestsSuite { k8sSuite: KubernetesSuite
=>
.withMatchExpressions(new NodeSelectorRequirementBuilder()
.withKey("kubernetes.io/hostname")
.withOperator("In")
- .withValues("minikube", "docker-for-desktop",
"docker-desktop").build()).build())
+ .withValues("minikube", "m01", "docker-for-desktop",
"docker-desktop")
+ .build()).build())
.endRequired()
.endNodeAffinity()
.endSpec()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 08a88e0 [SPARK-31313][K8S][TEST] Add `m01` node name to support
Minikube 1.8.x
08a88e0 is described below
commit 08a88e04f4417c5a0168592d1f01c300b4ae9de5
Author: Dongjoon Hyun
AuthorDate: Wed Apr 1 03:42:26 2020 +
[SPARK-31313][K8S][TEST] Add `m01` node name to support Minikube 1.8.x
### What changes were proposed in this pull request?
This PR aims to add `m01` as a node name additionally to `PVTestsSuite`.
### Why are the changes needed?
minikube 1.8.0 ~ 1.8.2 generate a cluster with a nodename `m01` while all
the other versions have `minikube`. This causes `PVTestSuite` failure.
```
$ minikube --vm-driver=hyperkit start --memory 6000 --cpus 8
* minikube v1.8.2 on Darwin 10.15.3
- MINIKUBE_ACTIVE_DOCKERD=minikube
* Using the hyperkit driver based on user configuration
* Creating hyperkit VM (CPUs=8, Memory=6000MB, Disk=2MB) ...
* Preparing Kubernetes v1.18.0 on Docker 19.03.6 ...
* Launching Kubernetes ...
* Enabling addons: default-storageclass, storage-provisioner
* Waiting for cluster to come online ...
* Done! kubectl is now configured to use "minikube"
$ kubectl get nodes
NAME STATUS ROLESAGE VERSION
m01Readymaster 22s v1.17.3
```
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
This only adds a new node name. So, K8S Jenkins job should passed.
In addition, `K8s` integration test suite should be tested on `minikube
1.8.2` manually.
```
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Test basic decommissioning
- Run SparkR on simple dataframe.R example
Run completed in 10 minutes, 23 seconds.
Total number of tests run: 20
Suites: completed 2, aborted 0
Tests: succeeded 20, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
For the above test, Minikube 1.8.2 and K8s v1.18.0 is used.
```
$ minikube version
minikube version: v1.8.2
commit: eb13446e786c9ef70cb0a9f85a633194e62396a1
$ kubectl version --short
Client Version: v1.18.0
Server Version: v1.18.0
```
Closes #28080 from dongjoon-hyun/SPARK-31313.
Authored-by: Dongjoon Hyun
Signed-off-by: DB Tsai
(cherry picked from commit dba525c997b0033ac1b6fd24236cd72938f94bbf)
Signed-off-by: Dongjoon Hyun
---
.../org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
index f021821..a7cb84e 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PVTestsSuite.scala
@@ -56,7 +56,8 @@ private[spark] trait PVTestsSuite { k8sSuite: KubernetesSuite
=>
.withMatchExpressions(new NodeSelectorRequirementBuilder()
.withKey("kubernetes.io/hostname")
.withOperator("In")
- .withValues("minikube", "docker-for-desktop",
"docker-desktop").build()).build())
+ .withValues("minikube", "m01", "docker-for-desktop",
"docker-desktop")
+ .build()).build())
.endRequired()
.endNodeAffinity()
.endSpec()
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31285][CORE] uppercase schedule mode string at config
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2c0e15e [SPARK-31285][CORE] uppercase schedule mode string at config
2c0e15e is described below
commit 2c0e15e1d0c774d57a801038de4fc826702a7c5c
Author: ulysses
AuthorDate: Wed Apr 1 11:46:41 2020 -0700
[SPARK-31285][CORE] uppercase schedule mode string at config
### What changes were proposed in this pull request?
In `TaskSchedulerImpl`, Spark will upper schedule mode
`SchedulingMode.withName(schedulingModeConf.toUpperCase(Locale.ROOT))`.
But at other place, Spark does not. Such as
[AllJobsPage](https://github.com/apache/spark/blob/5945d46c11a86fd85f9e65f24c2e88f368eee01f/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala#L304).
We should have the same behavior and uppercase schema mode string at config.
### Why are the changes needed?
Before this pr, it's ok to set `spark.scheduler.mode=fair` logically.
But Spark will throw warn log
```
java.util.NoSuchElementException: No value found for 'fair'
at scala.Enumeration.withName(Enumeration.scala:124)
at
org.apache.spark.ui.jobs.AllJobsPage$$anonfun$22.apply(AllJobsPage.scala:314)
at
org.apache.spark.ui.jobs.AllJobsPage$$anonfun$22.apply(AllJobsPage.scala:314)
at scala.Option.map(Option.scala:146)
at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:314)
at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:90)
at org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:90)
at org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90)
```
### Does this PR introduce any user-facing change?
Almost no.
### How was this patch tested?
Exists Test.
Closes #28049 from ulysses-you/SPARK-31285.
Authored-by: ulysses
Signed-off-by: Dongjoon Hyun
---
core/src/main/scala/org/apache/spark/internal/config/package.scala | 2 ++
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala | 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index f70ee2e..8f8b6ad 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -17,6 +17,7 @@
package org.apache.spark.internal
+import java.util.Locale
import java.util.concurrent.TimeUnit
import org.apache.spark.launcher.SparkLauncher
@@ -1756,6 +1757,7 @@ package object config {
ConfigBuilder("spark.scheduler.mode")
.version("0.8.0")
.stringConf
+ .transform(_.toUpperCase(Locale.ROOT))
.createWithDefault(SchedulingMode.FIFO.toString)
private[spark] val SCHEDULER_REVIVE_INTERVAL =
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
index f0f84fe..718c571 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
@@ -141,7 +141,7 @@ private[spark] class TaskSchedulerImpl(
private val schedulingModeConf = conf.get(SCHEDULER_MODE)
val schedulingMode: SchedulingMode =
try {
- SchedulingMode.withName(schedulingModeConf.toUpperCase(Locale.ROOT))
+ SchedulingMode.withName(schedulingModeConf)
} catch {
case e: java.util.NoSuchElementException =>
throw new SparkException(s"Unrecognized $SCHEDULER_MODE_PROPERTY:
$schedulingModeConf")
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31073][DOC][FOLLOWUP] Add description for Shuffle Write Time metric in StagePage to web-ui.md
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b9b1b54 [SPARK-31073][DOC][FOLLOWUP] Add description for Shuffle Write Time metric in StagePage to web-ui.md b9b1b54 is described below commit b9b1b549af4dd57767fc0c23f5fe7695ca9fa9d5 Author: Kousuke Saruta AuthorDate: Wed Apr 1 12:03:41 2020 -0700 [SPARK-31073][DOC][FOLLOWUP] Add description for Shuffle Write Time metric in StagePage to web-ui.md ### What changes were proposed in this pull request? This PR adds description for `Shuffle Write Time` to `web-ui.md`. ### Why are the changes needed? #27837 added `Shuffle Write Time` metric to task metrics summary but it's not documented yet. ### Does this PR introduce any user-facing change? Yes. We can see the description for `Shuffle Write Time` in the new `web-ui.html`. https://user-images.githubusercontent.com/4736016/78175342-a9722280-7495-11ea-9cc6-62c6f3619aa3.png";> ### How was this patch tested? Built docs by `SKIP_API=1 jekyll build` in `doc` directory and then confirmed `web-ui.html`. Closes #28093 from sarutak/SPARK-31073-doc. Authored-by: Kousuke Saruta Signed-off-by: Dongjoon Hyun --- docs/web-ui.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/web-ui.md b/docs/web-ui.md index e28a689..c53af80 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -143,6 +143,7 @@ Summary metrics for all task are represented in a table and in a timeline. * **Shuffle Read Size / Records**. Total shuffle bytes read, includes both data read locally and data read from remote executors. * **Shuffle Read Blocked Time** is the time that tasks spent blocked waiting for shuffle data to be read from remote machines. * **Shuffle Remote Reads** is the total shuffle bytes read from remote executors. +* **Shuffle Write Time** is the time that tasks spent writing shuffle data. * **Shuffle spill (memory)** is the size of the deserialized form of the shuffled data in memory. * **Shuffle spill (disk)** is the size of the serialized form of the data on disk. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31073][DOC][FOLLOWUP] Add description for Shuffle Write Time metric in StagePage to web-ui.md
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 5a7395d [SPARK-31073][DOC][FOLLOWUP] Add description for Shuffle Write Time metric in StagePage to web-ui.md 5a7395d is described below commit 5a7395d296b57c8e06bbf100b4ebdaa1100c3a91 Author: Kousuke Saruta AuthorDate: Wed Apr 1 12:03:41 2020 -0700 [SPARK-31073][DOC][FOLLOWUP] Add description for Shuffle Write Time metric in StagePage to web-ui.md ### What changes were proposed in this pull request? This PR adds description for `Shuffle Write Time` to `web-ui.md`. ### Why are the changes needed? #27837 added `Shuffle Write Time` metric to task metrics summary but it's not documented yet. ### Does this PR introduce any user-facing change? Yes. We can see the description for `Shuffle Write Time` in the new `web-ui.html`. https://user-images.githubusercontent.com/4736016/78175342-a9722280-7495-11ea-9cc6-62c6f3619aa3.png";> ### How was this patch tested? Built docs by `SKIP_API=1 jekyll build` in `doc` directory and then confirmed `web-ui.html`. Closes #28093 from sarutak/SPARK-31073-doc. Authored-by: Kousuke Saruta Signed-off-by: Dongjoon Hyun (cherry picked from commit b9b1b549af4dd57767fc0c23f5fe7695ca9fa9d5) Signed-off-by: Dongjoon Hyun --- docs/web-ui.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/web-ui.md b/docs/web-ui.md index e28a689..c53af80 100644 --- a/docs/web-ui.md +++ b/docs/web-ui.md @@ -143,6 +143,7 @@ Summary metrics for all task are represented in a table and in a timeline. * **Shuffle Read Size / Records**. Total shuffle bytes read, includes both data read locally and data read from remote executors. * **Shuffle Read Blocked Time** is the time that tasks spent blocked waiting for shuffle data to be read from remote machines. * **Shuffle Remote Reads** is the total shuffle bytes read from remote executors. +* **Shuffle Write Time** is the time that tasks spent writing shuffle data. * **Shuffle spill (memory)** is the size of the deserialized form of the shuffled data in memory. * **Shuffle spill (disk)** is the size of the serialized form of the data on disk. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-31322][SQL] rename QueryPlan.collectInPlanAndSubqueries to collectWithSubqueries
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 09f036a [SPARK-31322][SQL] rename
QueryPlan.collectInPlanAndSubqueries to collectWithSubqueries
09f036a is described below
commit 09f036a14cee4825edc73b463e1eebe85ff1c915
Author: Wenchen Fan
AuthorDate: Wed Apr 1 12:04:40 2020 -0700
[SPARK-31322][SQL] rename QueryPlan.collectInPlanAndSubqueries to
collectWithSubqueries
### What changes were proposed in this pull request?
rename `QueryPlan.collectInPlanAndSubqueries` to `collectWithSubqueries`
### Why are the changes needed?
The old name is too verbose. `QueryPlan` is internal but it's the core of
catalyst and we'd better make the API name clearer before we release it.
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
N/A
Closes #28092 from cloud-fan/rename.
Authored-by: Wenchen Fan
Signed-off-by: Dongjoon Hyun
---
.../main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala| 4 ++--
.../scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala| 2 +-
.../scala/org/apache/spark/sql/execution/CollectMetricsExec.scala | 2 +-
.../scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala | 2 +-
4 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
index 9f86fb2..13e5b12 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
@@ -232,10 +232,10 @@ abstract class QueryPlan[PlanType <: QueryPlan[PlanType]]
extends TreeNode[PlanT
}
/**
- * Returns a sequence containing the result of applying a partial function
to all elements in this
+ * A variant of `collect`. This method not only apply the given function to
all elements in this
* plan, also considering all the plans in its (nested) subqueries
*/
- def collectInPlanAndSubqueries[B](f: PartialFunction[PlanType, B]): Seq[B] =
+ def collectWithSubqueries[B](f: PartialFunction[PlanType, B]): Seq[B] =
(this +: subqueriesAll).flatMap(_.collect(f))
override def innerChildren: Seq[QueryPlan[_]] = subqueries
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala
index d96f808..91ce187 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala
@@ -78,7 +78,7 @@ class QueryPlanSuite extends SparkFunSuite {
val countRelationsInPlan = plan.collect({ case _: UnresolvedRelation => 1
}).sum
val countRelationsInPlanAndSubqueries =
- plan.collectInPlanAndSubqueries({ case _: UnresolvedRelation => 1 }).sum
+ plan.collectWithSubqueries({ case _: UnresolvedRelation => 1 }).sum
assert(countRelationsInPlan == 2)
assert(countRelationsInPlanAndSubqueries == 5)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/CollectMetricsExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/CollectMetricsExec.scala
index e482bc9..e1b9c8f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/CollectMetricsExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/CollectMetricsExec.scala
@@ -87,7 +87,7 @@ object CollectMetricsExec {
* Recursively collect all collected metrics from a query tree.
*/
def collect(plan: SparkPlan): Map[String, Row] = {
-val metrics = plan.collectInPlanAndSubqueries {
+val metrics = plan.collectWithSubqueries {
case collector: CollectMetricsExec => collector.name ->
collector.collectedMetrics
}
metrics.toMap
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
index baa9f5e..cdf9ea4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
@@ -1234,7 +1234,7 @@ abstract class DynamicPartitionPruningSuiteBase
val plan = df.queryExecution.executedPlan
val countSubqueryBroadcasts =
-plan.collectInPlanAndSubqueries({ case _: SubqueryBroadcastExec => 1
}).sum
+plan.collectWithSubqueries({ case _: SubqueryBroadcastExec => 1 }).sum
a
