[spark] branch master updated (1c781a4 -> 7d6e3fb)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection add 7d6e3fb [SPARK-33074][SQL] Classify dialect exceptions in JDBC v2 Table Catalog No new revisions were added by this update. Summary of changes: .../catalyst/analysis/AlreadyExistException.scala | 3 +- .../catalyst/analysis/NoSuchItemException.scala| 6 ++- .../datasources/v2/jdbc/JDBCTableCatalog.scala | 24 +-- .../org/apache/spark/sql/jdbc/H2Dialect.scala | 48 ++ .../org/apache/spark/sql/jdbc/JdbcDialects.scala | 12 ++ .../v2/jdbc/JDBCTableCatalogSuite.scala| 47 - .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 11 +++-- .../org/apache/spark/sql/jdbc/JDBCWriteSuite.scala | 37 + 8 files changed, 142 insertions(+), 46 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1c781a4 -> 7d6e3fb)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection add 7d6e3fb [SPARK-33074][SQL] Classify dialect exceptions in JDBC v2 Table Catalog No new revisions were added by this update. Summary of changes: .../catalyst/analysis/AlreadyExistException.scala | 3 +- .../catalyst/analysis/NoSuchItemException.scala| 6 ++- .../datasources/v2/jdbc/JDBCTableCatalog.scala | 24 +-- .../org/apache/spark/sql/jdbc/H2Dialect.scala | 48 ++ .../org/apache/spark/sql/jdbc/JdbcDialects.scala | 12 ++ .../v2/jdbc/JDBCTableCatalogSuite.scala| 47 - .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 11 +++-- .../org/apache/spark/sql/jdbc/JDBCWriteSuite.scala | 37 + 8 files changed, 142 insertions(+), 46 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1c781a4 -> 7d6e3fb)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection add 7d6e3fb [SPARK-33074][SQL] Classify dialect exceptions in JDBC v2 Table Catalog No new revisions were added by this update. Summary of changes: .../catalyst/analysis/AlreadyExistException.scala | 3 +- .../catalyst/analysis/NoSuchItemException.scala| 6 ++- .../datasources/v2/jdbc/JDBCTableCatalog.scala | 24 +-- .../org/apache/spark/sql/jdbc/H2Dialect.scala | 48 ++ .../org/apache/spark/sql/jdbc/JdbcDialects.scala | 12 ++ .../v2/jdbc/JDBCTableCatalogSuite.scala| 47 - .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 11 +++-- .../org/apache/spark/sql/jdbc/JDBCWriteSuite.scala | 37 + 8 files changed, 142 insertions(+), 46 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1c781a4 -> 7d6e3fb)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection add 7d6e3fb [SPARK-33074][SQL] Classify dialect exceptions in JDBC v2 Table Catalog No new revisions were added by this update. Summary of changes: .../catalyst/analysis/AlreadyExistException.scala | 3 +- .../catalyst/analysis/NoSuchItemException.scala| 6 ++- .../datasources/v2/jdbc/JDBCTableCatalog.scala | 24 +-- .../org/apache/spark/sql/jdbc/H2Dialect.scala | 48 ++ .../org/apache/spark/sql/jdbc/JdbcDialects.scala | 12 ++ .../v2/jdbc/JDBCTableCatalogSuite.scala| 47 - .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 11 +++-- .../org/apache/spark/sql/jdbc/JDBCWriteSuite.scala | 37 + 8 files changed, 142 insertions(+), 46 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1c781a4 -> 7d6e3fb)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection add 7d6e3fb [SPARK-33074][SQL] Classify dialect exceptions in JDBC v2 Table Catalog No new revisions were added by this update. Summary of changes: .../catalyst/analysis/AlreadyExistException.scala | 3 +- .../catalyst/analysis/NoSuchItemException.scala| 6 ++- .../datasources/v2/jdbc/JDBCTableCatalog.scala | 24 +-- .../org/apache/spark/sql/jdbc/H2Dialect.scala | 48 ++ .../org/apache/spark/sql/jdbc/JdbcDialects.scala | 12 ++ .../v2/jdbc/JDBCTableCatalogSuite.scala| 47 - .../org/apache/spark/sql/jdbc/JDBCSuite.scala | 11 +++-- .../org/apache/spark/sql/jdbc/JDBCWriteSuite.scala | 37 + 8 files changed, 142 insertions(+), 46 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bbc887b -> 1c781a4)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system add 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection No new revisions were added by this update. Summary of changes: .../execution/exchange/EnsureRequirements.scala| 58 -- .../exchange/EnsureRequirementsSuite.scala | 122 + 2 files changed, 168 insertions(+), 12 deletions(-) create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bbc887b -> 1c781a4)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system add 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection No new revisions were added by this update. Summary of changes: .../execution/exchange/EnsureRequirements.scala| 58 -- .../exchange/EnsureRequirementsSuite.scala | 122 + 2 files changed, 168 insertions(+), 12 deletions(-) create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bbc887b -> 1c781a4)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system add 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection No new revisions were added by this update. Summary of changes: .../execution/exchange/EnsureRequirements.scala| 58 -- .../exchange/EnsureRequirementsSuite.scala | 122 + 2 files changed, 168 insertions(+), 12 deletions(-) create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bbc887b -> 1c781a4)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system add 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection No new revisions were added by this update. Summary of changes: .../execution/exchange/EnsureRequirements.scala| 58 -- .../exchange/EnsureRequirementsSuite.scala | 122 + 2 files changed, 168 insertions(+), 12 deletions(-) create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bbc887b -> 1c781a4)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system add 1c781a4 [SPARK-32282][SQL] Improve EnsureRquirement.reorderJoinKeys to handle more scenarios such as PartitioningCollection No new revisions were added by this update. Summary of changes: .../execution/exchange/EnsureRequirements.scala| 58 -- .../exchange/EnsureRequirementsSuite.scala | 122 + 2 files changed, 168 insertions(+), 12 deletions(-) create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a7e4318 [SPARK-33089][SQL] make avro format propagate Hadoop config
from DS options to underlying HDFS file system
a7e4318 is described below
commit a7e43185715549f14decc018f7a58e2119c99aae
Author: Yuning Zhang
AuthorDate: Thu Oct 8 12:18:06 2020 +0900
[SPARK-33089][SQL] make avro format propagate Hadoop config from DS options
to underlying HDFS file system
### What changes were proposed in this pull request?
In `AvroUtils`'s `inferSchema()`, propagate Hadoop config from DS options
to underlying HDFS file system.
### Why are the changes needed?
There is a bug that when running:
```scala
spark.read.format("avro").options(conf).load(path)
```
The underlying file system will not receive the `conf` options.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test added
Closes #29971 from yuningzh-db/avro_options.
Authored-by: Yuning Zhang
Signed-off-by: HyukjinKwon
(cherry picked from commit bbc887bf73233b8c65ace05929290c0de4f63de8)
Signed-off-by: HyukjinKwon
---
.../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +-
.../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
index 9ff89f6..a9f34bb 100644
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
@@ -42,7 +42,7 @@ private[sql] object AvroUtils extends Logging {
spark: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
-val conf = spark.sessionState.newHadoopConf()
+val conf = spark.sessionState.newHadoopConfWithOptions(options)
val parsedOptions = new AvroOptions(options, conf)
if (parsedOptions.parameters.contains(ignoreExtensionKey)) {
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index e2ae489..d2f49ae 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -1799,6 +1799,16 @@ abstract class AvroSuite extends QueryTest with
SharedSparkSession {
assert(version === SPARK_VERSION_SHORT)
}
}
+
+ test("SPARK-33089: should propagate Hadoop config from DS options to
underlying file system") {
+withSQLConf(
+ "fs.file.impl" -> classOf[FakeFileSystemRequiringDSOption].getName,
+ "fs.file.impl.disable.cache" -> "true") {
+ val conf = Map("ds_option" -> "value")
+ val path = "file:" + testAvro.stripPrefix("file:")
+ spark.read.format("avro").options(conf).load(path)
+}
+ }
}
class AvroV1Suite extends AvroSuite {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a7e4318 [SPARK-33089][SQL] make avro format propagate Hadoop config
from DS options to underlying HDFS file system
a7e4318 is described below
commit a7e43185715549f14decc018f7a58e2119c99aae
Author: Yuning Zhang
AuthorDate: Thu Oct 8 12:18:06 2020 +0900
[SPARK-33089][SQL] make avro format propagate Hadoop config from DS options
to underlying HDFS file system
### What changes were proposed in this pull request?
In `AvroUtils`'s `inferSchema()`, propagate Hadoop config from DS options
to underlying HDFS file system.
### Why are the changes needed?
There is a bug that when running:
```scala
spark.read.format("avro").options(conf).load(path)
```
The underlying file system will not receive the `conf` options.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test added
Closes #29971 from yuningzh-db/avro_options.
Authored-by: Yuning Zhang
Signed-off-by: HyukjinKwon
(cherry picked from commit bbc887bf73233b8c65ace05929290c0de4f63de8)
Signed-off-by: HyukjinKwon
---
.../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +-
.../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
index 9ff89f6..a9f34bb 100644
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
@@ -42,7 +42,7 @@ private[sql] object AvroUtils extends Logging {
spark: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
-val conf = spark.sessionState.newHadoopConf()
+val conf = spark.sessionState.newHadoopConfWithOptions(options)
val parsedOptions = new AvroOptions(options, conf)
if (parsedOptions.parameters.contains(ignoreExtensionKey)) {
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index e2ae489..d2f49ae 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -1799,6 +1799,16 @@ abstract class AvroSuite extends QueryTest with
SharedSparkSession {
assert(version === SPARK_VERSION_SHORT)
}
}
+
+ test("SPARK-33089: should propagate Hadoop config from DS options to
underlying file system") {
+withSQLConf(
+ "fs.file.impl" -> classOf[FakeFileSystemRequiringDSOption].getName,
+ "fs.file.impl.disable.cache" -> "true") {
+ val conf = Map("ds_option" -> "value")
+ val path = "file:" + testAvro.stripPrefix("file:")
+ spark.read.format("avro").options(conf).load(path)
+}
+ }
}
class AvroV1Suite extends AvroSuite {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (39510b0 -> bbc887b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true add bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +- .../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++ 2 files changed, 11 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a7e4318 [SPARK-33089][SQL] make avro format propagate Hadoop config
from DS options to underlying HDFS file system
a7e4318 is described below
commit a7e43185715549f14decc018f7a58e2119c99aae
Author: Yuning Zhang
AuthorDate: Thu Oct 8 12:18:06 2020 +0900
[SPARK-33089][SQL] make avro format propagate Hadoop config from DS options
to underlying HDFS file system
### What changes were proposed in this pull request?
In `AvroUtils`'s `inferSchema()`, propagate Hadoop config from DS options
to underlying HDFS file system.
### Why are the changes needed?
There is a bug that when running:
```scala
spark.read.format("avro").options(conf).load(path)
```
The underlying file system will not receive the `conf` options.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test added
Closes #29971 from yuningzh-db/avro_options.
Authored-by: Yuning Zhang
Signed-off-by: HyukjinKwon
(cherry picked from commit bbc887bf73233b8c65ace05929290c0de4f63de8)
Signed-off-by: HyukjinKwon
---
.../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +-
.../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
index 9ff89f6..a9f34bb 100644
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
@@ -42,7 +42,7 @@ private[sql] object AvroUtils extends Logging {
spark: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
-val conf = spark.sessionState.newHadoopConf()
+val conf = spark.sessionState.newHadoopConfWithOptions(options)
val parsedOptions = new AvroOptions(options, conf)
if (parsedOptions.parameters.contains(ignoreExtensionKey)) {
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index e2ae489..d2f49ae 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -1799,6 +1799,16 @@ abstract class AvroSuite extends QueryTest with
SharedSparkSession {
assert(version === SPARK_VERSION_SHORT)
}
}
+
+ test("SPARK-33089: should propagate Hadoop config from DS options to
underlying file system") {
+withSQLConf(
+ "fs.file.impl" -> classOf[FakeFileSystemRequiringDSOption].getName,
+ "fs.file.impl.disable.cache" -> "true") {
+ val conf = Map("ds_option" -> "value")
+ val path = "file:" + testAvro.stripPrefix("file:")
+ spark.read.format("avro").options(conf).load(path)
+}
+ }
}
class AvroV1Suite extends AvroSuite {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (39510b0 -> bbc887b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true add bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +- .../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++ 2 files changed, 11 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a7e4318 [SPARK-33089][SQL] make avro format propagate Hadoop config
from DS options to underlying HDFS file system
a7e4318 is described below
commit a7e43185715549f14decc018f7a58e2119c99aae
Author: Yuning Zhang
AuthorDate: Thu Oct 8 12:18:06 2020 +0900
[SPARK-33089][SQL] make avro format propagate Hadoop config from DS options
to underlying HDFS file system
### What changes were proposed in this pull request?
In `AvroUtils`'s `inferSchema()`, propagate Hadoop config from DS options
to underlying HDFS file system.
### Why are the changes needed?
There is a bug that when running:
```scala
spark.read.format("avro").options(conf).load(path)
```
The underlying file system will not receive the `conf` options.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test added
Closes #29971 from yuningzh-db/avro_options.
Authored-by: Yuning Zhang
Signed-off-by: HyukjinKwon
(cherry picked from commit bbc887bf73233b8c65ace05929290c0de4f63de8)
Signed-off-by: HyukjinKwon
---
.../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +-
.../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
index 9ff89f6..a9f34bb 100644
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
@@ -42,7 +42,7 @@ private[sql] object AvroUtils extends Logging {
spark: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
-val conf = spark.sessionState.newHadoopConf()
+val conf = spark.sessionState.newHadoopConfWithOptions(options)
val parsedOptions = new AvroOptions(options, conf)
if (parsedOptions.parameters.contains(ignoreExtensionKey)) {
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index e2ae489..d2f49ae 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -1799,6 +1799,16 @@ abstract class AvroSuite extends QueryTest with
SharedSparkSession {
assert(version === SPARK_VERSION_SHORT)
}
}
+
+ test("SPARK-33089: should propagate Hadoop config from DS options to
underlying file system") {
+withSQLConf(
+ "fs.file.impl" -> classOf[FakeFileSystemRequiringDSOption].getName,
+ "fs.file.impl.disable.cache" -> "true") {
+ val conf = Map("ds_option" -> "value")
+ val path = "file:" + testAvro.stripPrefix("file:")
+ spark.read.format("avro").options(conf).load(path)
+}
+ }
}
class AvroV1Suite extends AvroSuite {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (39510b0 -> bbc887b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true add bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +- .../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++ 2 files changed, 11 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a7e4318 [SPARK-33089][SQL] make avro format propagate Hadoop config
from DS options to underlying HDFS file system
a7e4318 is described below
commit a7e43185715549f14decc018f7a58e2119c99aae
Author: Yuning Zhang
AuthorDate: Thu Oct 8 12:18:06 2020 +0900
[SPARK-33089][SQL] make avro format propagate Hadoop config from DS options
to underlying HDFS file system
### What changes were proposed in this pull request?
In `AvroUtils`'s `inferSchema()`, propagate Hadoop config from DS options
to underlying HDFS file system.
### Why are the changes needed?
There is a bug that when running:
```scala
spark.read.format("avro").options(conf).load(path)
```
The underlying file system will not receive the `conf` options.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
unit test added
Closes #29971 from yuningzh-db/avro_options.
Authored-by: Yuning Zhang
Signed-off-by: HyukjinKwon
(cherry picked from commit bbc887bf73233b8c65ace05929290c0de4f63de8)
Signed-off-by: HyukjinKwon
---
.../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +-
.../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
index 9ff89f6..a9f34bb 100644
--- a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
+++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
@@ -42,7 +42,7 @@ private[sql] object AvroUtils extends Logging {
spark: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
-val conf = spark.sessionState.newHadoopConf()
+val conf = spark.sessionState.newHadoopConfWithOptions(options)
val parsedOptions = new AvroOptions(options, conf)
if (parsedOptions.parameters.contains(ignoreExtensionKey)) {
diff --git
a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index e2ae489..d2f49ae 100644
--- a/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -1799,6 +1799,16 @@ abstract class AvroSuite extends QueryTest with
SharedSparkSession {
assert(version === SPARK_VERSION_SHORT)
}
}
+
+ test("SPARK-33089: should propagate Hadoop config from DS options to
underlying file system") {
+withSQLConf(
+ "fs.file.impl" -> classOf[FakeFileSystemRequiringDSOption].getName,
+ "fs.file.impl.disable.cache" -> "true") {
+ val conf = Map("ds_option" -> "value")
+ val path = "file:" + testAvro.stripPrefix("file:")
+ spark.read.format("avro").options(conf).load(path)
+}
+ }
}
class AvroV1Suite extends AvroSuite {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (39510b0 -> bbc887b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true add bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +- .../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++ 2 files changed, 11 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (39510b0 -> bbc887b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true add bbc887b [SPARK-33089][SQL] make avro format propagate Hadoop config from DS options to underlying HDFS file system No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala | 2 +- .../src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala | 10 ++ 2 files changed, 11 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (473b3ba -> 39510b0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark add 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R| 49 + R/pkg/R/generics.R | 8 +++ R/pkg/tests/fulltests/test_sparkSQL.R | 18 + python/docs/source/reference/pyspark.sql.rst | 2 + python/pyspark/sql/functions.py| 55 +- python/pyspark/sql/functions.pyi | 2 + python/pyspark/sql/tests/test_functions.py | 50 + .../sql/catalyst/analysis/FunctionRegistry.scala | 1 + .../spark/sql/catalyst/expressions/misc.scala | 84 +++--- .../catalyst/expressions/CodeGenerationSuite.scala | 2 +- .../expressions/ExpressionEvalHelper.scala | 6 +- .../expressions/MiscExpressionsSuite.scala | 30 .../scala/org/apache/spark/sql/functions.scala | 30 .../sql-functions/sql-expression-schema.md | 5 +- .../resources/sql-tests/inputs/misc-functions.sql | 12 .../sql-tests/results/misc-functions.sql.out | 81 - .../apache/spark/sql/ColumnExpressionSuite.scala | 51 + .../sql/expressions/ExpressionInfoSuite.scala | 17 +++-- 18 files changed, 450 insertions(+), 53 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (473b3ba -> 39510b0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark add 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R| 49 + R/pkg/R/generics.R | 8 +++ R/pkg/tests/fulltests/test_sparkSQL.R | 18 + python/docs/source/reference/pyspark.sql.rst | 2 + python/pyspark/sql/functions.py| 55 +- python/pyspark/sql/functions.pyi | 2 + python/pyspark/sql/tests/test_functions.py | 50 + .../sql/catalyst/analysis/FunctionRegistry.scala | 1 + .../spark/sql/catalyst/expressions/misc.scala | 84 +++--- .../catalyst/expressions/CodeGenerationSuite.scala | 2 +- .../expressions/ExpressionEvalHelper.scala | 6 +- .../expressions/MiscExpressionsSuite.scala | 30 .../scala/org/apache/spark/sql/functions.scala | 30 .../sql-functions/sql-expression-schema.md | 5 +- .../resources/sql-tests/inputs/misc-functions.sql | 12 .../sql-tests/results/misc-functions.sql.out | 81 - .../apache/spark/sql/ColumnExpressionSuite.scala | 51 + .../sql/expressions/ExpressionInfoSuite.scala | 17 +++-- 18 files changed, 450 insertions(+), 53 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (473b3ba -> 39510b0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark add 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R| 49 + R/pkg/R/generics.R | 8 +++ R/pkg/tests/fulltests/test_sparkSQL.R | 18 + python/docs/source/reference/pyspark.sql.rst | 2 + python/pyspark/sql/functions.py| 55 +- python/pyspark/sql/functions.pyi | 2 + python/pyspark/sql/tests/test_functions.py | 50 + .../sql/catalyst/analysis/FunctionRegistry.scala | 1 + .../spark/sql/catalyst/expressions/misc.scala | 84 +++--- .../catalyst/expressions/CodeGenerationSuite.scala | 2 +- .../expressions/ExpressionEvalHelper.scala | 6 +- .../expressions/MiscExpressionsSuite.scala | 30 .../scala/org/apache/spark/sql/functions.scala | 30 .../sql-functions/sql-expression-schema.md | 5 +- .../resources/sql-tests/inputs/misc-functions.sql | 12 .../sql-tests/results/misc-functions.sql.out | 81 - .../apache/spark/sql/ColumnExpressionSuite.scala | 51 + .../sql/expressions/ExpressionInfoSuite.scala | 17 +++-- 18 files changed, 450 insertions(+), 53 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (473b3ba -> 39510b0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark add 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R| 49 + R/pkg/R/generics.R | 8 +++ R/pkg/tests/fulltests/test_sparkSQL.R | 18 + python/docs/source/reference/pyspark.sql.rst | 2 + python/pyspark/sql/functions.py| 55 +- python/pyspark/sql/functions.pyi | 2 + python/pyspark/sql/tests/test_functions.py | 50 + .../sql/catalyst/analysis/FunctionRegistry.scala | 1 + .../spark/sql/catalyst/expressions/misc.scala | 84 +++--- .../catalyst/expressions/CodeGenerationSuite.scala | 2 +- .../expressions/ExpressionEvalHelper.scala | 6 +- .../expressions/MiscExpressionsSuite.scala | 30 .../scala/org/apache/spark/sql/functions.scala | 30 .../sql-functions/sql-expression-schema.md | 5 +- .../resources/sql-tests/inputs/misc-functions.sql | 12 .../sql-tests/results/misc-functions.sql.out | 81 - .../apache/spark/sql/ColumnExpressionSuite.scala | 51 + .../sql/expressions/ExpressionInfoSuite.scala | 17 +++-- 18 files changed, 450 insertions(+), 53 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (473b3ba -> 39510b0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark add 39510b0 [SPARK-32793][SQL] Add raise_error function, adds error message parameter to assert_true No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R| 49 + R/pkg/R/generics.R | 8 +++ R/pkg/tests/fulltests/test_sparkSQL.R | 18 + python/docs/source/reference/pyspark.sql.rst | 2 + python/pyspark/sql/functions.py| 55 +- python/pyspark/sql/functions.pyi | 2 + python/pyspark/sql/tests/test_functions.py | 50 + .../sql/catalyst/analysis/FunctionRegistry.scala | 1 + .../spark/sql/catalyst/expressions/misc.scala | 84 +++--- .../catalyst/expressions/CodeGenerationSuite.scala | 2 +- .../expressions/ExpressionEvalHelper.scala | 6 +- .../expressions/MiscExpressionsSuite.scala | 30 .../scala/org/apache/spark/sql/functions.scala | 30 .../sql-functions/sql-expression-schema.md | 5 +- .../resources/sql-tests/inputs/misc-functions.sql | 12 .../sql-tests/results/misc-functions.sql.out | 81 - .../apache/spark/sql/ColumnExpressionSuite.scala | 51 + .../sql/expressions/ExpressionInfoSuite.scala | 17 +++-- 18 files changed, 450 insertions(+), 53 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37e1b0c -> 473b3ba)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource add 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark No new revisions were added by this update. Summary of changes: R/pkg/NAMESPACE | 1 + R/pkg/R/column.R| 69 + R/pkg/R/generics.R | 3 ++ R/pkg/tests/fulltests/test_sparkSQL.R | 19 - python/pyspark/sql/column.py| 51 python/pyspark/sql/column.pyi | 1 + python/pyspark/sql/tests/test_column.py | 22 +++ 7 files changed, 165 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37e1b0c -> 473b3ba)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource add 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark No new revisions were added by this update. Summary of changes: R/pkg/NAMESPACE | 1 + R/pkg/R/column.R| 69 + R/pkg/R/generics.R | 3 ++ R/pkg/tests/fulltests/test_sparkSQL.R | 19 - python/pyspark/sql/column.py| 51 python/pyspark/sql/column.pyi | 1 + python/pyspark/sql/tests/test_column.py | 22 +++ 7 files changed, 165 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37e1b0c -> 473b3ba)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource add 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark No new revisions were added by this update. Summary of changes: R/pkg/NAMESPACE | 1 + R/pkg/R/column.R| 69 + R/pkg/R/generics.R | 3 ++ R/pkg/tests/fulltests/test_sparkSQL.R | 19 - python/pyspark/sql/column.py| 51 python/pyspark/sql/column.pyi | 1 + python/pyspark/sql/tests/test_column.py | 22 +++ 7 files changed, 165 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6daa2ae -> 37e1b0c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x add 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource No new revisions were added by this update. Summary of changes: python/pyspark/resource/information.pyi | 8 ++--- python/pyspark/resource/profile.pyi | 31 +++-- python/pyspark/resource/requests.pyi| 60 - 3 files changed, 60 insertions(+), 39 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37e1b0c -> 473b3ba)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource add 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark No new revisions were added by this update. Summary of changes: R/pkg/NAMESPACE | 1 + R/pkg/R/column.R| 69 + R/pkg/R/generics.R | 3 ++ R/pkg/tests/fulltests/test_sparkSQL.R | 19 - python/pyspark/sql/column.py| 51 python/pyspark/sql/column.pyi | 1 + python/pyspark/sql/tests/test_column.py | 22 +++ 7 files changed, 165 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6daa2ae -> 37e1b0c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x add 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource No new revisions were added by this update. Summary of changes: python/pyspark/resource/information.pyi | 8 ++--- python/pyspark/resource/profile.pyi | 31 +++-- python/pyspark/resource/requests.pyi| 60 - 3 files changed, 60 insertions(+), 39 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37e1b0c -> 473b3ba)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource add 473b3ba [SPARK-32511][FOLLOW-UP][SQL][R][PYTHON] Add dropFields to SparkR and PySpark No new revisions were added by this update. Summary of changes: R/pkg/NAMESPACE | 1 + R/pkg/R/column.R| 69 + R/pkg/R/generics.R | 3 ++ R/pkg/tests/fulltests/test_sparkSQL.R | 19 - python/pyspark/sql/column.py| 51 python/pyspark/sql/column.pyi | 1 + python/pyspark/sql/tests/test_column.py | 22 +++ 7 files changed, 165 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6daa2ae -> 37e1b0c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x add 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource No new revisions were added by this update. Summary of changes: python/pyspark/resource/information.pyi | 8 ++--- python/pyspark/resource/profile.pyi | 31 +++-- python/pyspark/resource/requests.pyi| 60 - 3 files changed, 60 insertions(+), 39 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6daa2ae -> 37e1b0c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x add 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource No new revisions were added by this update. Summary of changes: python/pyspark/resource/information.pyi | 8 ++--- python/pyspark/resource/profile.pyi | 31 +++-- python/pyspark/resource/requests.pyi| 60 - 3 files changed, 60 insertions(+), 39 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6daa2ae -> 37e1b0c)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x add 37e1b0c [SPARK-33086][PYTHON] Add static annotations for pyspark.resource No new revisions were added by this update. Summary of changes: python/pyspark/resource/information.pyi | 8 ++--- python/pyspark/resource/profile.pyi | 31 +++-- python/pyspark/resource/requests.pyi| 60 - 3 files changed, 60 insertions(+), 39 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (23afc93 -> 6daa2ae)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level add 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x No new revisions were added by this update. Summary of changes: sql/mkdocs.yml => .sbtopts | 9 +- build/sbt-launch-lib.bash | 2 +- project/MimaBuild.scala| 17 ++-- project/MimaExcludes.scala | 30 +++ project/SparkBuild.scala | 96 ++ project/build.properties | 2 +- project/plugins.sbt| 30 ++- .../apache/spark/tools/GenerateMIMAIgnore.scala| 3 +- 8 files changed, 113 insertions(+), 76 deletions(-) copy sql/mkdocs.yml => .sbtopts (84%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (23afc93 -> 6daa2ae)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level add 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x No new revisions were added by this update. Summary of changes: sql/mkdocs.yml => .sbtopts | 9 +- build/sbt-launch-lib.bash | 2 +- project/MimaBuild.scala| 17 ++-- project/MimaExcludes.scala | 30 +++ project/SparkBuild.scala | 96 ++ project/build.properties | 2 +- project/plugins.sbt| 30 ++- .../apache/spark/tools/GenerateMIMAIgnore.scala| 3 +- 8 files changed, 113 insertions(+), 76 deletions(-) copy sql/mkdocs.yml => .sbtopts (84%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (23afc93 -> 6daa2ae)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level add 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x No new revisions were added by this update. Summary of changes: sql/mkdocs.yml => .sbtopts | 9 +- build/sbt-launch-lib.bash | 2 +- project/MimaBuild.scala| 17 ++-- project/MimaExcludes.scala | 30 +++ project/SparkBuild.scala | 96 ++ project/build.properties | 2 +- project/plugins.sbt| 30 ++- .../apache/spark/tools/GenerateMIMAIgnore.scala| 3 +- 8 files changed, 113 insertions(+), 76 deletions(-) copy sql/mkdocs.yml => .sbtopts (84%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (23afc93 -> 6daa2ae)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level add 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x No new revisions were added by this update. Summary of changes: sql/mkdocs.yml => .sbtopts | 9 +- build/sbt-launch-lib.bash | 2 +- project/MimaBuild.scala| 17 ++-- project/MimaExcludes.scala | 30 +++ project/SparkBuild.scala | 96 ++ project/build.properties | 2 +- project/plugins.sbt| 30 ++- .../apache/spark/tools/GenerateMIMAIgnore.scala| 3 +- 8 files changed, 113 insertions(+), 76 deletions(-) copy sql/mkdocs.yml => .sbtopts (84%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (23afc93 -> 6daa2ae)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level add 6daa2ae [SPARK-21708][BUILD] Migrate build to sbt 1.x No new revisions were added by this update. Summary of changes: sql/mkdocs.yml => .sbtopts | 9 +- build/sbt-launch-lib.bash | 2 +- project/MimaBuild.scala| 17 ++-- project/MimaExcludes.scala | 30 +++ project/SparkBuild.scala | 96 ++ project/build.properties | 2 +- project/plugins.sbt| 30 ++- .../apache/spark/tools/GenerateMIMAIgnore.scala| 3 +- 8 files changed, 113 insertions(+), 76 deletions(-) copy sql/mkdocs.yml => .sbtopts (84%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a127387 -> 23afc93)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code add 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level No new revisions were added by this update. Summary of changes: .../execution/datasources/jdbc/connection/ConnectionProvider.scala | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a127387 -> 23afc93)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code add 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level No new revisions were added by this update. Summary of changes: .../execution/datasources/jdbc/connection/ConnectionProvider.scala | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a127387 -> 23afc93)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code add 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level No new revisions were added by this update. Summary of changes: .../execution/datasources/jdbc/connection/ConnectionProvider.scala | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a127387 -> 23afc93)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code add 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level No new revisions were added by this update. Summary of changes: .../execution/datasources/jdbc/connection/ConnectionProvider.scala | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (a127387 -> 23afc93)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code add 23afc93 [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level No new revisions were added by this update. Summary of changes: .../execution/datasources/jdbc/connection/ConnectionProvider.scala | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3099fd9 -> a127387)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template add a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code No new revisions were added by this update. Summary of changes: .../execution/BaseScriptTransformationSuite.scala | 12 +- .../execution/datasources/orc/OrcSourceSuite.scala | 7 ++-- .../thriftserver/SparkGetTablesOperation.scala | 6 +-- .../sql/hive/thriftserver/SparkSQLCLIService.scala | 11 + .../thriftserver/HiveThriftServer2Suites.scala | 18 ++-- .../SparkThriftServerProtocolVersionsSuite.scala | 8 +--- .../hive/execution/HiveCompatibilitySuite.scala| 5 +-- .../scala/org/apache/spark/sql/hive/HiveShim.scala | 48 +++--- .../org/apache/spark/sql/hive/HiveUtils.scala | 3 +- .../spark/sql/hive/client/HiveClientImpl.scala | 11 ++--- .../scala/org/apache/spark/sql/hive/hiveUDFs.scala | 18 +++- .../org/apache/spark/sql/hive/orc/OrcFilters.scala | 16 +--- .../sql/hive/ClasspathDependenciesSuite.scala | 25 +++ .../spark/sql/hive/HiveMetastoreCatalogSuite.scala | 18 ++-- .../org/apache/spark/sql/hive/HiveShimSuite.scala | 12 +- .../apache/spark/sql/hive/StatisticsSuite.scala| 34 --- .../execution/HiveScriptTransformationSuite.scala | 2 - .../spark/sql/hive/execution/HiveUDFSuite.scala| 1 - .../spark/sql/hive/execution/SQLQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcFilterSuite.scala| 19 + .../spark/sql/hive/orc/HiveOrcQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcSourceSuite.scala| 10 + 22 files changed, 60 insertions(+), 226 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3099fd9 -> a127387)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template add a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code No new revisions were added by this update. Summary of changes: .../execution/BaseScriptTransformationSuite.scala | 12 +- .../execution/datasources/orc/OrcSourceSuite.scala | 7 ++-- .../thriftserver/SparkGetTablesOperation.scala | 6 +-- .../sql/hive/thriftserver/SparkSQLCLIService.scala | 11 + .../thriftserver/HiveThriftServer2Suites.scala | 18 ++-- .../SparkThriftServerProtocolVersionsSuite.scala | 8 +--- .../hive/execution/HiveCompatibilitySuite.scala| 5 +-- .../scala/org/apache/spark/sql/hive/HiveShim.scala | 48 +++--- .../org/apache/spark/sql/hive/HiveUtils.scala | 3 +- .../spark/sql/hive/client/HiveClientImpl.scala | 11 ++--- .../scala/org/apache/spark/sql/hive/hiveUDFs.scala | 18 +++- .../org/apache/spark/sql/hive/orc/OrcFilters.scala | 16 +--- .../sql/hive/ClasspathDependenciesSuite.scala | 25 +++ .../spark/sql/hive/HiveMetastoreCatalogSuite.scala | 18 ++-- .../org/apache/spark/sql/hive/HiveShimSuite.scala | 12 +- .../apache/spark/sql/hive/StatisticsSuite.scala| 34 --- .../execution/HiveScriptTransformationSuite.scala | 2 - .../spark/sql/hive/execution/HiveUDFSuite.scala| 1 - .../spark/sql/hive/execution/SQLQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcFilterSuite.scala| 19 + .../spark/sql/hive/orc/HiveOrcQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcSourceSuite.scala| 10 + 22 files changed, 60 insertions(+), 226 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3099fd9 -> a127387)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template add a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code No new revisions were added by this update. Summary of changes: .../execution/BaseScriptTransformationSuite.scala | 12 +- .../execution/datasources/orc/OrcSourceSuite.scala | 7 ++-- .../thriftserver/SparkGetTablesOperation.scala | 6 +-- .../sql/hive/thriftserver/SparkSQLCLIService.scala | 11 + .../thriftserver/HiveThriftServer2Suites.scala | 18 ++-- .../SparkThriftServerProtocolVersionsSuite.scala | 8 +--- .../hive/execution/HiveCompatibilitySuite.scala| 5 +-- .../scala/org/apache/spark/sql/hive/HiveShim.scala | 48 +++--- .../org/apache/spark/sql/hive/HiveUtils.scala | 3 +- .../spark/sql/hive/client/HiveClientImpl.scala | 11 ++--- .../scala/org/apache/spark/sql/hive/hiveUDFs.scala | 18 +++- .../org/apache/spark/sql/hive/orc/OrcFilters.scala | 16 +--- .../sql/hive/ClasspathDependenciesSuite.scala | 25 +++ .../spark/sql/hive/HiveMetastoreCatalogSuite.scala | 18 ++-- .../org/apache/spark/sql/hive/HiveShimSuite.scala | 12 +- .../apache/spark/sql/hive/StatisticsSuite.scala| 34 --- .../execution/HiveScriptTransformationSuite.scala | 2 - .../spark/sql/hive/execution/HiveUDFSuite.scala| 1 - .../spark/sql/hive/execution/SQLQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcFilterSuite.scala| 19 + .../spark/sql/hive/orc/HiveOrcQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcSourceSuite.scala| 10 + 22 files changed, 60 insertions(+), 226 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3099fd9 -> a127387)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template add a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code No new revisions were added by this update. Summary of changes: .../execution/BaseScriptTransformationSuite.scala | 12 +- .../execution/datasources/orc/OrcSourceSuite.scala | 7 ++-- .../thriftserver/SparkGetTablesOperation.scala | 6 +-- .../sql/hive/thriftserver/SparkSQLCLIService.scala | 11 + .../thriftserver/HiveThriftServer2Suites.scala | 18 ++-- .../SparkThriftServerProtocolVersionsSuite.scala | 8 +--- .../hive/execution/HiveCompatibilitySuite.scala| 5 +-- .../scala/org/apache/spark/sql/hive/HiveShim.scala | 48 +++--- .../org/apache/spark/sql/hive/HiveUtils.scala | 3 +- .../spark/sql/hive/client/HiveClientImpl.scala | 11 ++--- .../scala/org/apache/spark/sql/hive/hiveUDFs.scala | 18 +++- .../org/apache/spark/sql/hive/orc/OrcFilters.scala | 16 +--- .../sql/hive/ClasspathDependenciesSuite.scala | 25 +++ .../spark/sql/hive/HiveMetastoreCatalogSuite.scala | 18 ++-- .../org/apache/spark/sql/hive/HiveShimSuite.scala | 12 +- .../apache/spark/sql/hive/StatisticsSuite.scala| 34 --- .../execution/HiveScriptTransformationSuite.scala | 2 - .../spark/sql/hive/execution/HiveUDFSuite.scala| 1 - .../spark/sql/hive/execution/SQLQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcFilterSuite.scala| 19 + .../spark/sql/hive/orc/HiveOrcQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcSourceSuite.scala| 10 + 22 files changed, 60 insertions(+), 226 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3099fd9 -> a127387)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template add a127387 [SPARK-33082][SQL] Remove hive-1.2 workaround code No new revisions were added by this update. Summary of changes: .../execution/BaseScriptTransformationSuite.scala | 12 +- .../execution/datasources/orc/OrcSourceSuite.scala | 7 ++-- .../thriftserver/SparkGetTablesOperation.scala | 6 +-- .../sql/hive/thriftserver/SparkSQLCLIService.scala | 11 + .../thriftserver/HiveThriftServer2Suites.scala | 18 ++-- .../SparkThriftServerProtocolVersionsSuite.scala | 8 +--- .../hive/execution/HiveCompatibilitySuite.scala| 5 +-- .../scala/org/apache/spark/sql/hive/HiveShim.scala | 48 +++--- .../org/apache/spark/sql/hive/HiveUtils.scala | 3 +- .../spark/sql/hive/client/HiveClientImpl.scala | 11 ++--- .../scala/org/apache/spark/sql/hive/hiveUDFs.scala | 18 +++- .../org/apache/spark/sql/hive/orc/OrcFilters.scala | 16 +--- .../sql/hive/ClasspathDependenciesSuite.scala | 25 +++ .../spark/sql/hive/HiveMetastoreCatalogSuite.scala | 18 ++-- .../org/apache/spark/sql/hive/HiveShimSuite.scala | 12 +- .../apache/spark/sql/hive/StatisticsSuite.scala| 34 --- .../execution/HiveScriptTransformationSuite.scala | 2 - .../spark/sql/hive/execution/HiveUDFSuite.scala| 1 - .../spark/sql/hive/execution/SQLQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcFilterSuite.scala| 19 + .../spark/sql/hive/orc/HiveOrcQuerySuite.scala | 1 - .../spark/sql/hive/orc/HiveOrcSourceSuite.scala| 10 + 22 files changed, 60 insertions(+), 226 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (7981f67 -> 45475af)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures add 45475af [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (7981f67 -> 45475af)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures add 45475af [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (94d648d -> 3099fd9)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner add 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (7981f67 -> 45475af)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures add 45475af [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (94d648d -> 3099fd9)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner add 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (7981f67 -> 45475af)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures add 45475af [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (94d648d -> 3099fd9)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner add 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (7981f67 -> 45475af)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures add 45475af [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (94d648d -> 3099fd9)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner add 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (94d648d -> 3099fd9)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner add 3099fd9 [SPARK-32067][K8S] Use unique ConfigMap name for executor pod template No new revisions were added by this update. Summary of changes: .../src/main/scala/org/apache/spark/deploy/k8s/Constants.scala| 2 +- .../spark/deploy/k8s/features/PodTemplateConfigMapStep.scala | 6 -- .../spark/deploy/k8s/features/PodTemplateConfigMapStepSuite.scala | 8 +--- 3 files changed, 10 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (72da6f8 -> 94d648d)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations add 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/subquery.scala| 80 ++ 1 file changed, 51 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (72da6f8 -> 94d648d)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations add 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/subquery.scala| 80 ++ 1 file changed, 51 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (72da6f8 -> 94d648d)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations add 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/subquery.scala| 80 ++ 1 file changed, 51 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (72da6f8 -> 94d648d)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations add 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/subquery.scala| 80 ++ 1 file changed, 51 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (72da6f8 -> 94d648d)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations add 94d648d [SPARK-33036][SQL] Refactor RewriteCorrelatedScalarSubquery code to replace exprIds in a bottom-up manner No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/optimizer/subquery.scala| 80 ++ 1 file changed, 51 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1ded6 -> 72da6f8)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs add 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations No new revisions were added by this update. Summary of changes: python/mypy.ini | 5 +- python/pyspark/_globals.pyi | 27 --- python/pyspark/accumulators.pyi | 4 +- python/pyspark/broadcast.pyi | 4 +- python/pyspark/daemon.pyi | 29 --- python/pyspark/find_spark_home.pyi| 17 - python/pyspark/java_gateway.pyi | 24 -- python/pyspark/join.pyi | 50 python/pyspark/ml/__init__.pyi| 45 --- python/pyspark/mllib/__init__.pyi | 32 python/pyspark/rddsampler.pyi | 54 - python/pyspark/resource/__init__.pyi | 31 python/pyspark/serializers.py | 2 +- python/pyspark/serializers.pyi| 122 -- python/pyspark/shell.py | 4 +- python/pyspark/shell.pyi | 31 python/pyspark/shuffle.pyi| 109 -- python/pyspark/sql/avro/__init__.pyi | 22 -- python/pyspark/sql/pandas/__init__.pyi| 17 - python/pyspark/sql/pandas/serializers.pyi | 65 python/pyspark/sql/pandas/typehints.pyi | 33 python/pyspark/sql/pandas/types.pyi | 41 -- python/pyspark/sql/pandas/utils.pyi | 20 - python/pyspark/sql/utils.pyi | 55 -- python/pyspark/streaming/__init__.pyi | 23 -- python/pyspark/streaming/util.pyi | 48 python/pyspark/traceback_utils.pyi| 29 --- python/pyspark/util.py| 2 +- python/pyspark/util.pyi | 35 - python/pyspark/worker.pyi | 73 -- 30 files changed, 14 insertions(+), 1039 deletions(-) delete mode 100644 python/pyspark/_globals.pyi delete mode 100644 python/pyspark/daemon.pyi delete mode 100644 python/pyspark/find_spark_home.pyi delete mode 100644 python/pyspark/java_gateway.pyi delete mode 100644 python/pyspark/join.pyi delete mode 100644 python/pyspark/ml/__init__.pyi delete mode 100644 python/pyspark/mllib/__init__.pyi delete mode 100644 python/pyspark/rddsampler.pyi delete mode 100644 python/pyspark/resource/__init__.pyi delete mode 100644 python/pyspark/serializers.pyi delete mode 100644 python/pyspark/shell.pyi delete mode 100644 python/pyspark/shuffle.pyi delete mode 100644 python/pyspark/sql/avro/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/serializers.pyi delete mode 100644 python/pyspark/sql/pandas/typehints.pyi delete mode 100644 python/pyspark/sql/pandas/types.pyi delete mode 100644 python/pyspark/sql/pandas/utils.pyi delete mode 100644 python/pyspark/sql/utils.pyi delete mode 100644 python/pyspark/streaming/__init__.pyi delete mode 100644 python/pyspark/streaming/util.pyi delete mode 100644 python/pyspark/traceback_utils.pyi delete mode 100644 python/pyspark/util.pyi delete mode 100644 python/pyspark/worker.pyi - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1ded6 -> 72da6f8)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs add 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations No new revisions were added by this update. Summary of changes: python/mypy.ini | 5 +- python/pyspark/_globals.pyi | 27 --- python/pyspark/accumulators.pyi | 4 +- python/pyspark/broadcast.pyi | 4 +- python/pyspark/daemon.pyi | 29 --- python/pyspark/find_spark_home.pyi| 17 - python/pyspark/java_gateway.pyi | 24 -- python/pyspark/join.pyi | 50 python/pyspark/ml/__init__.pyi| 45 --- python/pyspark/mllib/__init__.pyi | 32 python/pyspark/rddsampler.pyi | 54 - python/pyspark/resource/__init__.pyi | 31 python/pyspark/serializers.py | 2 +- python/pyspark/serializers.pyi| 122 -- python/pyspark/shell.py | 4 +- python/pyspark/shell.pyi | 31 python/pyspark/shuffle.pyi| 109 -- python/pyspark/sql/avro/__init__.pyi | 22 -- python/pyspark/sql/pandas/__init__.pyi| 17 - python/pyspark/sql/pandas/serializers.pyi | 65 python/pyspark/sql/pandas/typehints.pyi | 33 python/pyspark/sql/pandas/types.pyi | 41 -- python/pyspark/sql/pandas/utils.pyi | 20 - python/pyspark/sql/utils.pyi | 55 -- python/pyspark/streaming/__init__.pyi | 23 -- python/pyspark/streaming/util.pyi | 48 python/pyspark/traceback_utils.pyi| 29 --- python/pyspark/util.py| 2 +- python/pyspark/util.pyi | 35 - python/pyspark/worker.pyi | 73 -- 30 files changed, 14 insertions(+), 1039 deletions(-) delete mode 100644 python/pyspark/_globals.pyi delete mode 100644 python/pyspark/daemon.pyi delete mode 100644 python/pyspark/find_spark_home.pyi delete mode 100644 python/pyspark/java_gateway.pyi delete mode 100644 python/pyspark/join.pyi delete mode 100644 python/pyspark/ml/__init__.pyi delete mode 100644 python/pyspark/mllib/__init__.pyi delete mode 100644 python/pyspark/rddsampler.pyi delete mode 100644 python/pyspark/resource/__init__.pyi delete mode 100644 python/pyspark/serializers.pyi delete mode 100644 python/pyspark/shell.pyi delete mode 100644 python/pyspark/shuffle.pyi delete mode 100644 python/pyspark/sql/avro/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/serializers.pyi delete mode 100644 python/pyspark/sql/pandas/typehints.pyi delete mode 100644 python/pyspark/sql/pandas/types.pyi delete mode 100644 python/pyspark/sql/pandas/utils.pyi delete mode 100644 python/pyspark/sql/utils.pyi delete mode 100644 python/pyspark/streaming/__init__.pyi delete mode 100644 python/pyspark/streaming/util.pyi delete mode 100644 python/pyspark/traceback_utils.pyi delete mode 100644 python/pyspark/util.pyi delete mode 100644 python/pyspark/worker.pyi - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1ded6 -> 72da6f8)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs add 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations No new revisions were added by this update. Summary of changes: python/mypy.ini | 5 +- python/pyspark/_globals.pyi | 27 --- python/pyspark/accumulators.pyi | 4 +- python/pyspark/broadcast.pyi | 4 +- python/pyspark/daemon.pyi | 29 --- python/pyspark/find_spark_home.pyi| 17 - python/pyspark/java_gateway.pyi | 24 -- python/pyspark/join.pyi | 50 python/pyspark/ml/__init__.pyi| 45 --- python/pyspark/mllib/__init__.pyi | 32 python/pyspark/rddsampler.pyi | 54 - python/pyspark/resource/__init__.pyi | 31 python/pyspark/serializers.py | 2 +- python/pyspark/serializers.pyi| 122 -- python/pyspark/shell.py | 4 +- python/pyspark/shell.pyi | 31 python/pyspark/shuffle.pyi| 109 -- python/pyspark/sql/avro/__init__.pyi | 22 -- python/pyspark/sql/pandas/__init__.pyi| 17 - python/pyspark/sql/pandas/serializers.pyi | 65 python/pyspark/sql/pandas/typehints.pyi | 33 python/pyspark/sql/pandas/types.pyi | 41 -- python/pyspark/sql/pandas/utils.pyi | 20 - python/pyspark/sql/utils.pyi | 55 -- python/pyspark/streaming/__init__.pyi | 23 -- python/pyspark/streaming/util.pyi | 48 python/pyspark/traceback_utils.pyi| 29 --- python/pyspark/util.py| 2 +- python/pyspark/util.pyi | 35 - python/pyspark/worker.pyi | 73 -- 30 files changed, 14 insertions(+), 1039 deletions(-) delete mode 100644 python/pyspark/_globals.pyi delete mode 100644 python/pyspark/daemon.pyi delete mode 100644 python/pyspark/find_spark_home.pyi delete mode 100644 python/pyspark/java_gateway.pyi delete mode 100644 python/pyspark/join.pyi delete mode 100644 python/pyspark/ml/__init__.pyi delete mode 100644 python/pyspark/mllib/__init__.pyi delete mode 100644 python/pyspark/rddsampler.pyi delete mode 100644 python/pyspark/resource/__init__.pyi delete mode 100644 python/pyspark/serializers.pyi delete mode 100644 python/pyspark/shell.pyi delete mode 100644 python/pyspark/shuffle.pyi delete mode 100644 python/pyspark/sql/avro/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/serializers.pyi delete mode 100644 python/pyspark/sql/pandas/typehints.pyi delete mode 100644 python/pyspark/sql/pandas/types.pyi delete mode 100644 python/pyspark/sql/pandas/utils.pyi delete mode 100644 python/pyspark/sql/utils.pyi delete mode 100644 python/pyspark/streaming/__init__.pyi delete mode 100644 python/pyspark/streaming/util.pyi delete mode 100644 python/pyspark/traceback_utils.pyi delete mode 100644 python/pyspark/util.pyi delete mode 100644 python/pyspark/worker.pyi - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1ded6 -> 72da6f8)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs add 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations No new revisions were added by this update. Summary of changes: python/mypy.ini | 5 +- python/pyspark/_globals.pyi | 27 --- python/pyspark/accumulators.pyi | 4 +- python/pyspark/broadcast.pyi | 4 +- python/pyspark/daemon.pyi | 29 --- python/pyspark/find_spark_home.pyi| 17 - python/pyspark/java_gateway.pyi | 24 -- python/pyspark/join.pyi | 50 python/pyspark/ml/__init__.pyi| 45 --- python/pyspark/mllib/__init__.pyi | 32 python/pyspark/rddsampler.pyi | 54 - python/pyspark/resource/__init__.pyi | 31 python/pyspark/serializers.py | 2 +- python/pyspark/serializers.pyi| 122 -- python/pyspark/shell.py | 4 +- python/pyspark/shell.pyi | 31 python/pyspark/shuffle.pyi| 109 -- python/pyspark/sql/avro/__init__.pyi | 22 -- python/pyspark/sql/pandas/__init__.pyi| 17 - python/pyspark/sql/pandas/serializers.pyi | 65 python/pyspark/sql/pandas/typehints.pyi | 33 python/pyspark/sql/pandas/types.pyi | 41 -- python/pyspark/sql/pandas/utils.pyi | 20 - python/pyspark/sql/utils.pyi | 55 -- python/pyspark/streaming/__init__.pyi | 23 -- python/pyspark/streaming/util.pyi | 48 python/pyspark/traceback_utils.pyi| 29 --- python/pyspark/util.py| 2 +- python/pyspark/util.pyi | 35 - python/pyspark/worker.pyi | 73 -- 30 files changed, 14 insertions(+), 1039 deletions(-) delete mode 100644 python/pyspark/_globals.pyi delete mode 100644 python/pyspark/daemon.pyi delete mode 100644 python/pyspark/find_spark_home.pyi delete mode 100644 python/pyspark/java_gateway.pyi delete mode 100644 python/pyspark/join.pyi delete mode 100644 python/pyspark/ml/__init__.pyi delete mode 100644 python/pyspark/mllib/__init__.pyi delete mode 100644 python/pyspark/rddsampler.pyi delete mode 100644 python/pyspark/resource/__init__.pyi delete mode 100644 python/pyspark/serializers.pyi delete mode 100644 python/pyspark/shell.pyi delete mode 100644 python/pyspark/shuffle.pyi delete mode 100644 python/pyspark/sql/avro/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/serializers.pyi delete mode 100644 python/pyspark/sql/pandas/typehints.pyi delete mode 100644 python/pyspark/sql/pandas/types.pyi delete mode 100644 python/pyspark/sql/pandas/utils.pyi delete mode 100644 python/pyspark/sql/utils.pyi delete mode 100644 python/pyspark/streaming/__init__.pyi delete mode 100644 python/pyspark/streaming/util.pyi delete mode 100644 python/pyspark/traceback_utils.pyi delete mode 100644 python/pyspark/util.pyi delete mode 100644 python/pyspark/worker.pyi - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (4e1ded6 -> 72da6f8)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs add 72da6f8 [SPARK-33002][PYTHON] Remove non-API annotations No new revisions were added by this update. Summary of changes: python/mypy.ini | 5 +- python/pyspark/_globals.pyi | 27 --- python/pyspark/accumulators.pyi | 4 +- python/pyspark/broadcast.pyi | 4 +- python/pyspark/daemon.pyi | 29 --- python/pyspark/find_spark_home.pyi| 17 - python/pyspark/java_gateway.pyi | 24 -- python/pyspark/join.pyi | 50 python/pyspark/ml/__init__.pyi| 45 --- python/pyspark/mllib/__init__.pyi | 32 python/pyspark/rddsampler.pyi | 54 - python/pyspark/resource/__init__.pyi | 31 python/pyspark/serializers.py | 2 +- python/pyspark/serializers.pyi| 122 -- python/pyspark/shell.py | 4 +- python/pyspark/shell.pyi | 31 python/pyspark/shuffle.pyi| 109 -- python/pyspark/sql/avro/__init__.pyi | 22 -- python/pyspark/sql/pandas/__init__.pyi| 17 - python/pyspark/sql/pandas/serializers.pyi | 65 python/pyspark/sql/pandas/typehints.pyi | 33 python/pyspark/sql/pandas/types.pyi | 41 -- python/pyspark/sql/pandas/utils.pyi | 20 - python/pyspark/sql/utils.pyi | 55 -- python/pyspark/streaming/__init__.pyi | 23 -- python/pyspark/streaming/util.pyi | 48 python/pyspark/traceback_utils.pyi| 29 --- python/pyspark/util.py| 2 +- python/pyspark/util.pyi | 35 - python/pyspark/worker.pyi | 73 -- 30 files changed, 14 insertions(+), 1039 deletions(-) delete mode 100644 python/pyspark/_globals.pyi delete mode 100644 python/pyspark/daemon.pyi delete mode 100644 python/pyspark/find_spark_home.pyi delete mode 100644 python/pyspark/java_gateway.pyi delete mode 100644 python/pyspark/join.pyi delete mode 100644 python/pyspark/ml/__init__.pyi delete mode 100644 python/pyspark/mllib/__init__.pyi delete mode 100644 python/pyspark/rddsampler.pyi delete mode 100644 python/pyspark/resource/__init__.pyi delete mode 100644 python/pyspark/serializers.pyi delete mode 100644 python/pyspark/shell.pyi delete mode 100644 python/pyspark/shuffle.pyi delete mode 100644 python/pyspark/sql/avro/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/__init__.pyi delete mode 100644 python/pyspark/sql/pandas/serializers.pyi delete mode 100644 python/pyspark/sql/pandas/typehints.pyi delete mode 100644 python/pyspark/sql/pandas/types.pyi delete mode 100644 python/pyspark/sql/pandas/utils.pyi delete mode 100644 python/pyspark/sql/utils.pyi delete mode 100644 python/pyspark/streaming/__init__.pyi delete mode 100644 python/pyspark/streaming/util.pyi delete mode 100644 python/pyspark/traceback_utils.pyi delete mode 100644 python/pyspark/util.pyi delete mode 100644 python/pyspark/worker.pyi - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (23207fc -> 7981f67)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 23207fc [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput add 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures No new revisions were added by this update. Summary of changes: python/pyspark/sql/pandas/serializers.py| 17 ++--- python/pyspark/sql/tests/test_arrow.py | 9 + python/pyspark/sql/tests/test_pandas_grouped_map.py | 15 --- 3 files changed, 23 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (23207fc -> 7981f67)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 23207fc [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput add 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures No new revisions were added by this update. Summary of changes: python/pyspark/sql/pandas/serializers.py| 17 ++--- python/pyspark/sql/tests/test_arrow.py | 9 + python/pyspark/sql/tests/test_pandas_grouped_map.py | 15 --- 3 files changed, 23 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (23207fc -> 7981f67)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 23207fc [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput add 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures No new revisions were added by this update. Summary of changes: python/pyspark/sql/pandas/serializers.py| 17 ++--- python/pyspark/sql/tests/test_arrow.py | 9 + python/pyspark/sql/tests/test_pandas_grouped_map.py | 15 --- 3 files changed, 23 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (23207fc -> 7981f67)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 23207fc [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput add 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures No new revisions were added by this update. Summary of changes: python/pyspark/sql/pandas/serializers.py| 17 ++--- python/pyspark/sql/tests/test_arrow.py | 9 + python/pyspark/sql/tests/test_pandas_grouped_map.py | 15 --- 3 files changed, 23 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (23207fc -> 7981f67)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from 23207fc [SPARK-33035][SQL][3.0] Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput add 7981f67 [SPARK-33073][PYTHON][3.0] Improve error handling on Pandas to Arrow conversion failures No new revisions were added by this update. Summary of changes: python/pyspark/sql/pandas/serializers.py| 17 ++--- python/pyspark/sql/tests/test_arrow.py | 9 + python/pyspark/sql/tests/test_pandas_grouped_map.py | 15 --- 3 files changed, 23 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7e99fcd -> 4e1ded6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7e99fcd [SPARK-33004][SQL] Migrate DESCRIBE column to use UnresolvedTableOrView to resolve the identifier add 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs No new revisions were added by this update. Summary of changes: python/docs/source/development/debugging.rst | 2 +- python/docs/source/development/setting_ide.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7e99fcd -> 4e1ded6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7e99fcd [SPARK-33004][SQL] Migrate DESCRIBE column to use UnresolvedTableOrView to resolve the identifier add 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs No new revisions were added by this update. Summary of changes: python/docs/source/development/debugging.rst | 2 +- python/docs/source/development/setting_ide.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7e99fcd -> 4e1ded6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7e99fcd [SPARK-33004][SQL] Migrate DESCRIBE column to use UnresolvedTableOrView to resolve the identifier add 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs No new revisions were added by this update. Summary of changes: python/docs/source/development/debugging.rst | 2 +- python/docs/source/development/setting_ide.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7e99fcd -> 4e1ded6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7e99fcd [SPARK-33004][SQL] Migrate DESCRIBE column to use UnresolvedTableOrView to resolve the identifier add 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs No new revisions were added by this update. Summary of changes: python/docs/source/development/debugging.rst | 2 +- python/docs/source/development/setting_ide.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7e99fcd -> 4e1ded6)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7e99fcd [SPARK-33004][SQL] Migrate DESCRIBE column to use UnresolvedTableOrView to resolve the identifier add 4e1ded6 [SPARK-32189][DOCS][PYTHON][FOLLOW-UP] Fixed broken link and typo in PySpark docs No new revisions were added by this update. Summary of changes: python/docs/source/development/debugging.rst | 2 +- python/docs/source/development/setting_ide.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
