[spark] branch master updated (3f6322f -> 48252ba)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 3f6322f  [SPARK-35077][SQL] Migrate to transformWithPruning for 
leftover optimizer rules
 add 48252ba  [SPARK-35583][DOCS] Move JDBC data source options from Python 
and Scala into a single page

No new revisions were added by this update.

Summary of changes:
 docs/sql-data-sources-jdbc.md  | 18 +++
 python/pyspark/sql/readwriter.py   | 37 --
 .../org/apache/spark/sql/DataFrameReader.scala | 29 +++--
 .../org/apache/spark/sql/DataFrameWriter.scala | 14 ++--
 4 files changed, 55 insertions(+), 43 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (c2de0a6 -> 3f6322f)

[gengliang]

This is an automated email from the ASF dual-hosted git repository.

gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from c2de0a6  [SPARK-35100][ML] Refactor AFT - support virtual centering
 add 3f6322f  [SPARK-35077][SQL] Migrate to transformWithPruning for 
leftover optimizer rules

No new revisions were added by this update.

Summary of changes:
 .../apache/spark/sql/catalyst/optimizer/ComplexTypes.scala   |  6 --
 .../sql/catalyst/optimizer/NormalizeFloatingNumbers.scala|  3 ++-
 .../org/apache/spark/sql/catalyst/optimizer/Optimizer.scala  |  9 +++--
 .../org/apache/spark/sql/catalyst/optimizer/joins.scala  |  5 +++--
 .../apache/spark/sql/catalyst/rules/RuleIdCollection.scala   |  1 +
 .../dynamicpruning/CleanupDynamicPruningFilters.scala|  8 ++--
 .../spark/sql/execution/python/ExtractPythonUDFs.scala   | 12 +---
 7 files changed, 32 insertions(+), 12 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (dbf0b50 -> c2de0a6)

[ruifengz]

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from dbf0b50  [SPARK-35560][SQL] Remove redundant subexpression evaluation 
in nested subexpressions
 add c2de0a6  [SPARK-35100][ML] Refactor AFT - support virtual centering

No new revisions were added by this update.

Summary of changes:
 .../spark/ml/optim/aggregator/AFTAggregator.scala  | 256 -
 ...ckAggregator.scala => AFTBlockAggregator.scala} | 117 --
 .../ml/regression/AFTSurvivalRegression.scala  |  52 +++--
 3 files changed, 87 insertions(+), 338 deletions(-)
 delete mode 100644 
mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTAggregator.scala
 copy 
mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/{HingeBlockAggregator.scala
 => AFTBlockAggregator.scala} (52%)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (9d0d4ed -> dbf0b50)

[viirya]

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 9d0d4ed  [SPARK-35595][TESTS] Support multiple loggers in testing 
method withLogAppender
 add dbf0b50  [SPARK-35560][SQL] Remove redundant subexpression evaluation 
in nested subexpressions

No new revisions were added by this update.

Summary of changes:
 .../expressions/codegen/CodeGenerator.scala| 49 +-
 .../org/apache/spark/sql/DataFrameSuite.scala  | 25 +++
 2 files changed, 55 insertions(+), 19 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (0ad5ae5 -> 9d0d4ed)

[gengliang]

This is an automated email from the ASF dual-hosted git repository.

gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 0ad5ae5  [SPARK-35539][PYTHON] Restore to_koalas to keep the backward 
compatibility
 add 9d0d4ed  [SPARK-35595][TESTS] Support multiple loggers in testing 
method withLogAppender

No new revisions were added by this update.

Summary of changes:
 .../scala/org/apache/spark/SparkFunSuite.scala | 24 ++
 .../catalyst/expressions/CodeGenerationSuite.scala |  2 +-
 .../adaptive/AdaptiveQueryExecSuite.scala  |  6 --
 3 files changed, 21 insertions(+), 11 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (6a277bb -> 0ad5ae5)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 6a277bb  [SPARK-35600][TESTS] Move Set command related test cases to 
SetCommandSuite
 add 0ad5ae5  [SPARK-35539][PYTHON] Restore to_koalas to keep the backward 
compatibility

No new revisions were added by this update.

Summary of changes:
 python/pyspark/pandas/__init__.py | 3 +++
 python/pyspark/pandas/frame.py| 8 
 2 files changed, 11 insertions(+)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (35cfabc -> 6a277bb)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 35cfabc  [SPARK-35589][CORE] BlockManagerMasterEndpoint should not 
ignore index-only shuffle file during updating
 add 6a277bb  [SPARK-35600][TESTS] Move Set command related test cases to 
SetCommandSuite

No new revisions were added by this update.

Summary of changes:
 .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 124 +
 .../org/apache/spark/sql/SetCommandSuite.scala | 146 +
 2 files changed, 147 insertions(+), 123 deletions(-)
 create mode 100644 
sql/core/src/test/scala/org/apache/spark/sql/SetCommandSuite.scala

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[GitHub] [spark-website] dongjoon-hyun merged pull request #347: Add 3.1.2 announcement news, release note and download link

[GitBox]

dongjoon-hyun merged pull request #347:
URL: https://github.com/apache/spark-website/pull/347


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[GitHub] [spark-website] dongjoon-hyun commented on pull request #347: Add 3.1.2 announcement news, release note and download link

[GitBox]

dongjoon-hyun commented on pull request #347:
URL: https://github.com/apache/spark-website/pull/347#issuecomment-852607791


   Thank you, @HyukjinKwon !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[GitHub] [spark-website] dongjoon-hyun commented on pull request #347: Add 3.1.2 announcement news, release note and download link

[GitBox]

dongjoon-hyun commented on pull request #347:
URL: https://github.com/apache/spark-website/pull/347#issuecomment-852605291


   Thank you so much, @viirya !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[GitHub] [spark-website] dongjoon-hyun opened a new pull request #347: Add 3.1.2 announcement news, release note and download link

[GitBox]

dongjoon-hyun opened a new pull request #347:
URL: https://github.com/apache/spark-website/pull/347


   This PR aims to update our website.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (0ac5c16 -> 35cfabc)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 0ac5c16  [SPARK-35314][PYTHON] Support arithmetic operations against 
bool IndexOpsMixin
 add 35cfabc  [SPARK-35589][CORE] BlockManagerMasterEndpoint should not 
ignore index-only shuffle file during updating

No new revisions were added by this update.

Summary of changes:
 .../org/apache/spark/shuffle/IndexShuffleBlockResolver.scala  |  2 +-
 .../org/apache/spark/storage/BlockManagerMasterEndpoint.scala |  8 
 .../scala/org/apache/spark/storage/BlockManagerSuite.scala| 11 ++-
 3 files changed, 15 insertions(+), 6 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (a127d91 -> 0ac5c16)

[ueshin]

This is an automated email from the ASF dual-hosted git repository.

ueshin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from a127d91  [SPARK-35402][WEBUI] Increase the max thread pool size of 
jetty server in HistoryServer UI
 add 0ac5c16  [SPARK-35314][PYTHON] Support arithmetic operations against 
bool IndexOpsMixin

No new revisions were added by this update.

Summary of changes:
 python/pyspark/pandas/data_type_ops/base.py|  36 -
 python/pyspark/pandas/data_type_ops/boolean_ops.py | 156 +++-
 .../pyspark/pandas/data_type_ops/datetime_ops.py   |   8 +-
 python/pyspark/pandas/data_type_ops/num_ops.py | 134 ++---
 .../pandas/tests/data_type_ops/test_boolean_ops.py | 163 -
 .../pandas/tests/data_type_ops/test_num_ops.py |  81 +-
 python/pyspark/pandas/tests/test_dataframe.py  |   2 -
 python/pyspark/pandas/tests/test_series.py |   2 -
 8 files changed, 445 insertions(+), 137 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (08e6f63 -> a127d91)

[yao]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 08e6f63  [SPARK-35577][TESTS] Allow to log container output for docker 
integration tests
 add a127d91  [SPARK-35402][WEBUI] Increase the max thread pool size of 
jetty server in HistoryServer UI

No new revisions were added by this update.

Summary of changes:
 .../apache/spark/deploy/history/HistoryServer.scala|  7 ++-
 .../main/scala/org/apache/spark/ui/JettyUtils.scala|  5 +++--
 core/src/main/scala/org/apache/spark/ui/WebUI.scala|  5 +++--
 core/src/test/scala/org/apache/spark/ui/UISuite.scala  | 18 ++
 4 files changed, 30 insertions(+), 5 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (a59063d -> 08e6f63)

[yamamuro]

This is an automated email from the ASF dual-hosted git repository.

yamamuro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from a59063d  [SPARK-35581][SQL] Support special datetime values in typed 
literals only
 add 08e6f63  [SPARK-35577][TESTS] Allow to log container output for docker 
integration tests

No new revisions were added by this update.

Summary of changes:
 .../apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala | 14 +-
 1 file changed, 13 insertions(+), 1 deletion(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-35581][SQL] Support special datetime values in typed literals only

[maxgekk]

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new a59063d  [SPARK-35581][SQL] Support special datetime values in typed 
literals only
a59063d is described below

commit a59063d5446a78d713c19a0d43f86c9f72ffd77d
Author: Max Gekk 
AuthorDate: Tue Jun 1 15:29:05 2021 +0300

[SPARK-35581][SQL] Support special datetime values in typed literals only

### What changes were proposed in this pull request?
In the PR, I propose to support special datetime values introduced by 
#25708 and by #25716 only in typed literals, and don't recognize them in 
parsing strings to dates/timestamps. The following string values are supported 
only in typed timestamp literals:
- `epoch [zoneId]` - `1970-01-01 00:00:00+00 (Unix system time zero)`
- `today [zoneId]` - midnight today.
- `yesterday [zoneId]` - midnight yesterday
- `tomorrow [zoneId]` - midnight tomorrow
- `now` - current query start time.

For example:
```sql
spark-sql> SELECT timestamp 'tomorrow';
2019-09-07 00:00:00
```

Similarly, the following special date values are supported only in typed 
date literals:
- `epoch [zoneId]` - `1970-01-01`
- `today [zoneId]` - the current date in the time zone specified by 
`spark.sql.session.timeZone`.
- `yesterday [zoneId]` - the current date -1
- `tomorrow [zoneId]` - the current date + 1
- `now` - the date of running the current query. It has the same notion as 
`today`.

For example:
```sql
spark-sql> SELECT date 'tomorrow' - date 'yesterday';
2
```

### Why are the changes needed?
In the current implementation, Spark supports the special date/timestamp 
value in any input strings casted to dates/timestamps that leads to the 
following problems:
- If executors have different system time, the result is inconsistent, and 
random. Column values depend on where the conversions were performed.
- The special values play the role of distributed non-deterministic 
functions though users might think of the values as constants.

### Does this PR introduce _any_ user-facing change?
Yes but the probability should be small.

### How was this patch tested?
By running existing test suites:
```
$ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z 
interval.sql"
$ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z 
date.sql"
$ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z 
timestamp.sql"
$ build/sbt "test:testOnly *DateTimeUtilsSuite"
```

Closes #32714 from MaxGekk/remove-datetime-special-values.

Lead-authored-by: Max Gekk 
Co-authored-by: Maxim Gekk 
Signed-off-by: Max Gekk 
---
 docs/sql-migration-guide.md|  2 ++
 .../src/main/scala/org/apache/spark/sql/Row.scala  |  2 +-
 .../spark/sql/catalyst/catalog/interface.scala |  4 +--
 .../sql/catalyst/csv/UnivocityGenerator.scala  |  1 -
 .../spark/sql/catalyst/csv/UnivocityParser.scala   |  3 +-
 .../spark/sql/catalyst/expressions/Cast.scala  | 20 +
 .../spark/sql/catalyst/expressions/literals.scala  |  2 +-
 .../spark/sql/catalyst/json/JacksonGenerator.scala |  1 -
 .../spark/sql/catalyst/json/JacksonParser.scala|  3 +-
 .../spark/sql/catalyst/parser/AstBuilder.scala |  9 --
 .../spark/sql/catalyst/util/DateFormatter.scala| 34 -
 .../spark/sql/catalyst/util/DateTimeUtils.scala| 35 +-
 .../sql/catalyst/util/TimestampFormatter.scala | 23 +++---
 .../expressions/HashExpressionsSuite.scala |  2 +-
 .../sql/catalyst/util/DateFormatterSuite.scala | 34 ++---
 .../sql/catalyst/util/DateTimeUtilsSuite.scala | 32 ++--
 .../sql/catalyst/util/DatetimeFormatterSuite.scala |  2 +-
 .../catalyst/util/TimestampFormatterSuite.scala| 24 +--
 .../spark/sql/catalyst/util/UnsafeArraySuite.scala |  6 ++--
 .../execution/BaseScriptTransformationExec.scala   | 11 +++
 .../apache/spark/sql/execution/HiveResult.scala| 12 ++--
 .../execution/datasources/PartitioningUtils.scala  |  2 +-
 .../execution/datasources/jdbc/JDBCRelation.scala  |  5 ++--
 .../org/apache/spark/sql/jdbc/JdbcDialects.scala   |  4 +--
 .../sql-tests/inputs/postgreSQL/timestamp.sql  | 16 +-
 .../sql-tests/results/postgreSQL/timestamp.sql.out | 16 +-
 .../org/apache/spark/sql/CsvFunctionsSuite.scala   | 19 
 .../org/apache/spark/sql/JsonFunctionsSuite.scala  | 19 
 .../parquet/ParquetPartitionDiscoverySuite.scala   |  2 +-
 .../spark/sql/hive/HiveExternalCatalog.scala   |  8 ++---
 .../apache/spark/sql/hive/client/HiveClient.scala  |  3 +-
 

[spark] branch master updated (d773373 -> b7dd4b3)

[sarutak]

This is an automated email from the ASF dual-hosted git repository.

sarutak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from d773373  [SPARK-35584][CORE][TESTS] Increase the timeout in 
FallbackStorageSuite
 add b7dd4b3  [SPARK-35516][WEBUI] Storage UI tab Storage Level tool tip 
correction

No new revisions were added by this update.

Summary of changes:
 core/src/main/scala/org/apache/spark/ui/storage/ToolTips.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated: [SPARK-35584][CORE][TESTS] Increase the timeout in FallbackStorageSuite

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
 new 77802d9  [SPARK-35584][CORE][TESTS] Increase the timeout in 
FallbackStorageSuite
77802d9 is described below

commit 77802d9643bf9b800e1c4ac4e6c6b0c2fabdd2cd
Author: Yikun Jiang 
AuthorDate: Tue Jun 1 00:45:58 2021 -0700

[SPARK-35584][CORE][TESTS] Increase the timeout in FallbackStorageSuite

### What changes were proposed in this pull request?
```
- Upload multi stages *** FAILED ***
{{ The code passed to eventually never returned normally. Attempted 20 
times over 10.011176743 seconds. Last failure message: 
fallbackStorage.exists(0, file) was false. (FallbackStorageSuite.scala:243)}}
```
The error like above was raised in aarch64 randomly and also in github 
action test[1][2].

[1] https://github.com/apache/spark/actions/runs/489319612
[2]https://github.com/apache/spark/actions/runs/479317320

### Why are the changes needed?
timeout is too short, need to increase to let test case complete.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
build/mvn test -Dtest=none 
-DwildcardSuites=org.apache.spark.storage.FallbackStorageSuite -pl 
:spark-core_2.12

Closes #32719 from Yikun/SPARK-35584.

Authored-by: Yikun Jiang 
Signed-off-by: Dongjoon Hyun 
(cherry picked from commit d77337307489978b0523fcca27e60093bcc2df38)
Signed-off-by: Dongjoon Hyun 
---
 .../test/scala/org/apache/spark/storage/FallbackStorageSuite.scala| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/core/src/test/scala/org/apache/spark/storage/FallbackStorageSuite.scala 
b/core/src/test/scala/org/apache/spark/storage/FallbackStorageSuite.scala
index c07edb6..f155788 100644
--- a/core/src/test/scala/org/apache/spark/storage/FallbackStorageSuite.scala
+++ b/core/src/test/scala/org/apache/spark/storage/FallbackStorageSuite.scala
@@ -221,7 +221,7 @@ class FallbackStorageSuite extends SparkFunSuite with 
LocalSparkContext {
 sched.decommissionExecutor(_, ExecutorDecommissionInfo(""), false)
   }
 
-  eventually(timeout(10.seconds), interval(1.seconds)) {
+  eventually(timeout(20.seconds), interval(1.seconds)) {
 shuffle0_files.foreach { file => assert(fallbackStorage.exists(0, 
file)) }
 shuffle1_files.foreach { file => assert(fallbackStorage.exists(1, 
file)) }
   }
@@ -246,7 +246,7 @@ class FallbackStorageSuite extends SparkFunSuite with 
LocalSparkContext {
 
 // Make it sure that fallback storage are ready
 val fallbackStorage = new FallbackStorage(sc.getConf)
-eventually(timeout(10.seconds), interval(1.seconds)) {
+eventually(timeout(20.seconds), interval(1.seconds)) {
   Seq(
 "shuffle_0_0_0.index", "shuffle_0_0_0.data",
 "shuffle_0_1_0.index", "shuffle_0_1_0.data").foreach { file =>

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (e048838 -> d773373)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from e048838  [SPARK-35586][K8S][TESTS] Set a default value for 
spark.kubernetes.test.sparkTgz in pom.xml for Kubernetes integration tests
 add d773373  [SPARK-35584][CORE][TESTS] Increase the timeout in 
FallbackStorageSuite

No new revisions were added by this update.

Summary of changes:
 .../test/scala/org/apache/spark/storage/FallbackStorageSuite.scala| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (fe09def -> e048838)

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from fe09def  [SPARK-35582][PYTHON][DOCS] Remove # noqa in Python API 
documents
 add e048838  [SPARK-35586][K8S][TESTS] Set a default value for 
spark.kubernetes.test.sparkTgz in pom.xml for Kubernetes integration tests

No new revisions were added by this update.

Summary of changes:
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-35582][PYTHON][DOCS] Remove # noqa in Python API documents

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new fe09def  [SPARK-35582][PYTHON][DOCS] Remove # noqa in Python API 
documents
fe09def is described below

commit fe09def3231600e52cb58aaba5f72af33ab4dc33
Author: itholic 
AuthorDate: Tue Jun 1 15:24:04 2021 +0900

[SPARK-35582][PYTHON][DOCS] Remove # noqa in Python API documents

### What changes were proposed in this pull request?

This PR aims to move `# noqa` in the Python docstring to the proper place 
so that hide them from the official documents.

### Why are the changes needed?

If we don't move `# noqa` to the proper place, it is exposed in the middle 
of the docstring, and it looks a bit wired as below:
https://user-images.githubusercontent.com/44108233/120275617-91da3800-c2ec-11eb-9778-16c5fe789418.png;>

### Does this PR introduce _any_ user-facing change?

Yes, the `# noqa` is no more shown in the documents as below:
https://user-images.githubusercontent.com/44108233/120275927-fbf2dd00-c2ec-11eb-950d-346af2745711.png;>

### How was this patch tested?

Manually build docs and check.

Closes #32728 from itholic/SPARK-35582.

Authored-by: itholic 
Signed-off-by: Hyukjin Kwon 
---
 python/pyspark/sql/functions.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index cd7cadb..49401cc 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -3712,9 +3712,11 @@ def schema_of_json(json, options=None):
 a JSON string or a foldable string column containing a JSON string.
 options : dict, optional
 options to control parsing. accepts the same options as the JSON 
datasource.
-See `Data Source Option 
`_
  # noqa
+See `Data Source Option 
`_
 in the version you use.
 
+.. # noqa
+
 .. versionchanged:: 3.0
It accepts `options` parameter to control schema inferring.
 

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.1 updated (264ce7b -> 92fb23e)

[gengliang]

This is an automated email from the ASF dual-hosted git repository.

gengliang pushed a change to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 264ce7b  [SPARK-35573][R][TESTSt] Make SparkR tests pass with R 4.1+
 add 92fb23e  [SPARK-35576][SQL][3.1] Redact the sensitive info in the 
result of Set command

No new revisions were added by this update.

Summary of changes:
 .../main/scala/org/apache/spark/sql/internal/SQLConf.scala  |  9 -
 .../org/apache/spark/sql/execution/command/SetCommand.scala |  6 --
 .../src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 13 +
 3 files changed, 25 insertions(+), 3 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org