[spark] branch master updated (8c4b535 -> b9aeeb4)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 8c4b535  [SPARK-35678][ML] add a common softmax function
 add b9aeeb4  [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 
'other' to driver side

No new revisions were added by this update.

Summary of changes:
 python/pyspark/pandas/indexes/base.py| 28 ++---
 python/pyspark/pandas/tests/indexes/test_base.py | 31 
 2 files changed, 45 insertions(+), 14 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch branch-3.0 updated (673654b -> f1711af)

[wuyi]

This is an automated email from the ASF dual-hosted git repository.

wuyi pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 673654b  [SPARK-35714][CORE] Bug fix for deadlock during the executor 
shutdown
 add 65ac1e7  Preparing Spark release v3.0.3-rc1
 new f1711af  Preparing development version 3.0.4-SNAPSHOT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 R/pkg/DESCRIPTION  | 2 +-
 assembly/pom.xml   | 2 +-
 common/kvstore/pom.xml | 2 +-
 common/network-common/pom.xml  | 2 +-
 common/network-shuffle/pom.xml | 2 +-
 common/network-yarn/pom.xml| 2 +-
 common/sketch/pom.xml  | 2 +-
 common/tags/pom.xml| 2 +-
 common/unsafe/pom.xml  | 2 +-
 core/pom.xml   | 2 +-
 docs/_config.yml   | 4 ++--
 examples/pom.xml   | 2 +-
 external/avro/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml  | 2 +-
 external/kafka-0-10-assembly/pom.xml   | 2 +-
 external/kafka-0-10-sql/pom.xml| 2 +-
 external/kafka-0-10-token-provider/pom.xml | 2 +-
 external/kafka-0-10/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml  | 2 +-
 external/kinesis-asl/pom.xml   | 2 +-
 external/spark-ganglia-lgpl/pom.xml| 2 +-
 graphx/pom.xml | 2 +-
 hadoop-cloud/pom.xml   | 2 +-
 launcher/pom.xml   | 2 +-
 mllib-local/pom.xml| 2 +-
 mllib/pom.xml  | 2 +-
 pom.xml| 2 +-
 python/pyspark/version.py  | 2 +-
 repl/pom.xml   | 2 +-
 resource-managers/kubernetes/core/pom.xml  | 2 +-
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 resource-managers/mesos/pom.xml| 2 +-
 resource-managers/yarn/pom.xml | 2 +-
 sql/catalyst/pom.xml   | 2 +-
 sql/core/pom.xml   | 2 +-
 sql/hive-thriftserver/pom.xml  | 2 +-
 sql/hive/pom.xml   | 2 +-
 streaming/pom.xml  | 2 +-
 tools/pom.xml  | 2 +-
 39 files changed, 40 insertions(+), 40 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] 01/01: Preparing development version 3.0.4-SNAPSHOT

[wuyi]

This is an automated email from the ASF dual-hosted git repository.

wuyi pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git

commit f1711afda14ccf4e9a39d688f69ebf98a8fda14b
Author: Wu Yi 
AuthorDate: Tue Jun 15 04:04:57 2021 +

Preparing development version 3.0.4-SNAPSHOT
---
 R/pkg/DESCRIPTION  | 2 +-
 assembly/pom.xml   | 2 +-
 common/kvstore/pom.xml | 2 +-
 common/network-common/pom.xml  | 2 +-
 common/network-shuffle/pom.xml | 2 +-
 common/network-yarn/pom.xml| 2 +-
 common/sketch/pom.xml  | 2 +-
 common/tags/pom.xml| 2 +-
 common/unsafe/pom.xml  | 2 +-
 core/pom.xml   | 2 +-
 docs/_config.yml   | 4 ++--
 examples/pom.xml   | 2 +-
 external/avro/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml  | 2 +-
 external/kafka-0-10-assembly/pom.xml   | 2 +-
 external/kafka-0-10-sql/pom.xml| 2 +-
 external/kafka-0-10-token-provider/pom.xml | 2 +-
 external/kafka-0-10/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml  | 2 +-
 external/kinesis-asl/pom.xml   | 2 +-
 external/spark-ganglia-lgpl/pom.xml| 2 +-
 graphx/pom.xml | 2 +-
 hadoop-cloud/pom.xml   | 2 +-
 launcher/pom.xml   | 2 +-
 mllib-local/pom.xml| 2 +-
 mllib/pom.xml  | 2 +-
 pom.xml| 2 +-
 python/pyspark/version.py  | 2 +-
 repl/pom.xml   | 2 +-
 resource-managers/kubernetes/core/pom.xml  | 2 +-
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 resource-managers/mesos/pom.xml| 2 +-
 resource-managers/yarn/pom.xml | 2 +-
 sql/catalyst/pom.xml   | 2 +-
 sql/core/pom.xml   | 2 +-
 sql/hive-thriftserver/pom.xml  | 2 +-
 sql/hive/pom.xml   | 2 +-
 streaming/pom.xml  | 2 +-
 tools/pom.xml  | 2 +-
 39 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index e9354ae..0a5fc41 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: SparkR
 Type: Package
-Version: 3.0.3
+Version: 3.0.4
 Title: R Front End for 'Apache Spark'
 Description: Provides an R Front end for 'Apache Spark' 
.
 Authors@R: c(person("Shivaram", "Venkataraman", role = "aut",
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 00a3981..2a40047 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3
+3.0.4-SNAPSHOT
 ../pom.xml
   
 
diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml
index 7b585da..a6d236a 100644
--- a/common/kvstore/pom.xml
+++ b/common/kvstore/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3
+3.0.4-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index f2d1f11..c9823bd 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3
+3.0.4-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index b15826d..1eb51d8 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3
+3.0.4-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 02dff3a..b449149 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3
+3.0.4-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 5bcad1e..5564e28 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3
+3.0.4-SNAPSHOT
 ../../pom.xml
   
 
diff --git a/common/tags/pom.xml b/common/tags/pom.xml
index 

[spark] tag v3.0.3-rc1 created (now 65ac1e7)

[wuyi]

This is an automated email from the ASF dual-hosted git repository.

wuyi pushed a change to tag v3.0.3-rc1
in repository https://gitbox.apache.org/repos/asf/spark.git.


  at 65ac1e7  (commit)
This tag includes the following new commits:

 new 65ac1e7  Preparing Spark release v3.0.3-rc1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] 01/01: Preparing Spark release v3.0.3-rc1

[wuyi]

This is an automated email from the ASF dual-hosted git repository.

wuyi pushed a commit to tag v3.0.3-rc1
in repository https://gitbox.apache.org/repos/asf/spark.git

commit 65ac1e75dc468f53fc778cd2ce1ba3f21067aab8
Author: Wu Yi 
AuthorDate: Tue Jun 15 04:04:50 2021 +

Preparing Spark release v3.0.3-rc1
---
 assembly/pom.xml   | 2 +-
 common/kvstore/pom.xml | 2 +-
 common/network-common/pom.xml  | 2 +-
 common/network-shuffle/pom.xml | 2 +-
 common/network-yarn/pom.xml| 2 +-
 common/sketch/pom.xml  | 2 +-
 common/tags/pom.xml| 2 +-
 common/unsafe/pom.xml  | 2 +-
 core/pom.xml   | 2 +-
 docs/_config.yml   | 2 +-
 examples/pom.xml   | 2 +-
 external/avro/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml  | 2 +-
 external/kafka-0-10-assembly/pom.xml   | 2 +-
 external/kafka-0-10-sql/pom.xml| 2 +-
 external/kafka-0-10-token-provider/pom.xml | 2 +-
 external/kafka-0-10/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml  | 2 +-
 external/kinesis-asl/pom.xml   | 2 +-
 external/spark-ganglia-lgpl/pom.xml| 2 +-
 graphx/pom.xml | 2 +-
 hadoop-cloud/pom.xml   | 2 +-
 launcher/pom.xml   | 2 +-
 mllib-local/pom.xml| 2 +-
 mllib/pom.xml  | 2 +-
 pom.xml| 2 +-
 python/pyspark/version.py  | 2 +-
 repl/pom.xml   | 2 +-
 resource-managers/kubernetes/core/pom.xml  | 2 +-
 resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
 resource-managers/mesos/pom.xml| 2 +-
 resource-managers/yarn/pom.xml | 2 +-
 sql/catalyst/pom.xml   | 2 +-
 sql/core/pom.xml   | 2 +-
 sql/hive-thriftserver/pom.xml  | 2 +-
 sql/hive/pom.xml   | 2 +-
 streaming/pom.xml  | 2 +-
 tools/pom.xml  | 2 +-
 38 files changed, 38 insertions(+), 38 deletions(-)

diff --git a/assembly/pom.xml b/assembly/pom.xml
index 23c6655..00a3981 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../pom.xml
   
 
diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml
index 346e7db..7b585da 100644
--- a/common/kvstore/pom.xml
+++ b/common/kvstore/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../../pom.xml
   
 
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index f399f82..f2d1f11 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../../pom.xml
   
 
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index 45ca6e8..b15826d 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../../pom.xml
   
 
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 0d32906..02dff3a 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../../pom.xml
   
 
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 68c65a1..5bcad1e 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../../pom.xml
   
 
diff --git a/common/tags/pom.xml b/common/tags/pom.xml
index 6e9c231..9a5c02d 100644
--- a/common/tags/pom.xml
+++ b/common/tags/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../../pom.xml
   
 
diff --git a/common/unsafe/pom.xml b/common/unsafe/pom.xml
index c2a89e7..b0d7e1d 100644
--- a/common/unsafe/pom.xml
+++ b/common/unsafe/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.12
-3.0.3-SNAPSHOT
+3.0.3
 ../../pom.xml
   
 
diff --git 

[spark] branch master updated (2a56cc3 -> 8c4b535)

[ruifengz]

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 2a56cc3  [SPARK-35761][PYTHON] Use type-annotation based pandas_udf or 
avoid specifying udf types to suppress warnings
 add 8c4b535  [SPARK-35678][ML] add a common softmax function

No new revisions were added by this update.

Summary of changes:
 .../scala/org/apache/spark/ml/impl/Utils.scala | 31 +
 .../ml/classification/LogisticRegression.scala | 29 ++--
 .../MultinomialLogisticBlockAggregator.scala   | 32 --
 3 files changed, 40 insertions(+), 52 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (95f36e76 -> 2a56cc3)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 95f36e76 [SPARK-35750][PYTHON][DOCS] Rename "pandas APIs on Spark" to 
"pandas API on Spark"
 add 2a56cc3  [SPARK-35761][PYTHON] Use type-annotation based pandas_udf or 
avoid specifying udf types to suppress warnings

No new revisions were added by this update.

Summary of changes:
 python/pyspark/pandas/accessors.py| 51 ++---
 python/pyspark/pandas/frame.py| 20 
 python/pyspark/pandas/groupby.py  | 32 +++--
 python/pyspark/pandas/internal.py |  4 +-
 python/pyspark/pandas/namespace.py|  6 +--
 python/pyspark/pandas/numpy_compat.py | 86 ---
 python/pyspark/pandas/strings.py  | 32 ++---
 7 files changed, 103 insertions(+), 128 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (2d47fb7 -> 95f36e76)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 2d47fb7  [SPARK-35755][PYTHON][INFRA] Use higher PyArrow versions in 
GitHub Actions build
 add 95f36e76 [SPARK-35750][PYTHON][DOCS] Rename "pandas APIs on Spark" to 
"pandas API on Spark"

No new revisions were added by this update.

Summary of changes:
 python/docs/source/development/index.rst   |  2 +-
 python/docs/source/development/ps_contributing.rst | 10 ++---
 python/docs/source/development/ps_design.rst   | 30 +++---
 python/docs/source/getting_started/index.rst   |  2 +-
 python/docs/source/getting_started/install.rst |  6 +--
 python/docs/source/getting_started/ps_10mins.ipynb | 16 
 python/docs/source/getting_started/ps_install.rst  | 16 
 .../docs/source/getting_started/quickstart.ipynb   |  4 +-
 python/docs/source/index.rst   |  5 +--
 .../docs/source/reference/pyspark.pandas/frame.rst |  2 +-
 .../docs/source/reference/pyspark.pandas/index.rst |  8 ++--
 .../source/reference/pyspark.pandas/series.rst |  4 +-
 .../user_guide/pandas_on_spark/best_practices.rst  | 46 +++---
 .../user_guide/pandas_on_spark/from_to_dbms.rst|  8 ++--
 .../source/user_guide/pandas_on_spark/index.rst|  6 +--
 .../source/user_guide/pandas_on_spark/options.rst  |  8 ++--
 .../user_guide/pandas_on_spark/pandas_pyspark.rst  |  8 ++--
 .../user_guide/pandas_on_spark/transform_apply.rst |  4 +-
 .../user_guide/pandas_on_spark/typehints.rst   | 29 +++---
 .../source/user_guide/pandas_on_spark/types.rst| 28 ++---
 python/pyspark/pandas/groupby.py   |  4 +-
 21 files changed, 122 insertions(+), 124 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (ef7545b -> 2d47fb7)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from ef7545b  [SPARK-35759][PYTHON] Remove the upperbound for numpy for 
pandas-on-Spark
 add 2d47fb7  [SPARK-35755][PYTHON][INFRA] Use higher PyArrow versions in 
GitHub Actions build

No new revisions were added by this update.

Summary of changes:
 .github/workflows/build_and_test.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (0375661 -> ef7545b)

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 0375661  [SPARK-35616][PYTHON] Make `astype` method data-type-based
 add ef7545b  [SPARK-35759][PYTHON] Remove the upperbound for numpy for 
pandas-on-Spark

No new revisions were added by this update.

Summary of changes:
 python/docs/source/getting_started/install.rst | 2 +-
 python/setup.py| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (aab0c2b -> 0375661)

[ueshin]

This is an automated email from the ASF dual-hosted git repository.

ueshin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from aab0c2b  [SPARK-35736][SPARK-35737][SQL][FOLLOWUP] Move a common logic 
to DayTimeIntervalType
 add 0375661  [SPARK-35616][PYTHON] Make `astype` method data-type-based

No new revisions were added by this update.

Summary of changes:
 python/pyspark/pandas/base.py  | 104 +---
 python/pyspark/pandas/data_type_ops/base.py| 106 -
 python/pyspark/pandas/data_type_ops/binary_ops.py  |  29 +-
 python/pyspark/pandas/data_type_ops/boolean_ops.py |  35 ++-
 .../pandas/data_type_ops/categorical_ops.py|  32 +++
 python/pyspark/pandas/data_type_ops/complex_ops.py |  27 +-
 python/pyspark/pandas/data_type_ops/date_ops.py|  28 +-
 .../pyspark/pandas/data_type_ops/datetime_ops.py   |  41 +++-
 python/pyspark/pandas/data_type_ops/null_ops.py|  32 ++-
 python/pyspark/pandas/data_type_ops/num_ops.py |  65 +++--
 python/pyspark/pandas/data_type_ops/string_ops.py  |  34 ++-
 .../pandas/tests/data_type_ops/test_binary_ops.py  |   9 ++
 .../pandas/tests/data_type_ops/test_boolean_ops.py |  24 +
 .../tests/data_type_ops/test_categorical_ops.py|  23 +
 .../pandas/tests/data_type_ops/test_complex_ops.py |   9 +-
 .../pandas/tests/data_type_ops/test_date_ops.py|   9 ++
 .../tests/data_type_ops/test_datetime_ops.py   |   9 ++
 .../pandas/tests/data_type_ops/test_null_ops.py|  10 ++
 .../pandas/tests/data_type_ops/test_num_ops.py |  15 +++
 .../pandas/tests/data_type_ops/test_string_ops.py  |  18 +++-
 .../pandas/tests/data_type_ops/test_udt_ops.py |   3 +
 21 files changed, 532 insertions(+), 130 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (82af318 -> aab0c2b)

[maxgekk]

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 82af318  [SPARK-35748][SS][SQL] Fix StreamingJoinHelper to be able to 
handle day-time interval
 add aab0c2b  [SPARK-35736][SPARK-35737][SQL][FOLLOWUP] Move a common logic 
to DayTimeIntervalType

No new revisions were added by this update.

Summary of changes:
 .../org/apache/spark/sql/catalyst/parser/AstBuilder.scala| 12 
 .../org/apache/spark/sql/types/DayTimeIntervalType.scala |  2 ++
 2 files changed, 6 insertions(+), 8 deletions(-)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated (439e94c -> 82af318)

[maxgekk]

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


from 439e94c  [SPARK-35737][SQL] Parse day-time interval literals to 
tightest types
 add 82af318  [SPARK-35748][SS][SQL] Fix StreamingJoinHelper to be able to 
handle day-time interval

No new revisions were added by this update.

Summary of changes:
 .../spark/sql/catalyst/analysis/StreamingJoinHelper.scala|  3 +++
 .../sql/catalyst/analysis/StreamingJoinHelperSuite.scala | 12 
 2 files changed, 15 insertions(+)

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] branch master updated: [SPARK-35737][SQL] Parse day-time interval literals to tightest types

[maxgekk]

This is an automated email from the ASF dual-hosted git repository.

maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
 new 439e94c  [SPARK-35737][SQL] Parse day-time interval literals to 
tightest types
439e94c is described below

commit 439e94c1712366ff267183d3946f2507ebf3a98e
Author: Kousuke Saruta 
AuthorDate: Mon Jun 14 10:06:19 2021 +0300

[SPARK-35737][SQL] Parse day-time interval literals to tightest types

### What changes were proposed in this pull request?

This PR add a feature which parse day-time interval literals to tightest 
type.

### Why are the changes needed?

To comply with the ANSI behavior.
For example, `INTERVAL '10 20:30' DAY TO MINUTE` should be parsed as 
`DayTimeIntervalType(DAY, MINUTE)` but not as `DayTimeIntervalType(DAY, 
SECOND)`.

### Does this PR introduce _any_ user-facing change?

No because `DayTimeIntervalType` will be introduced in `3.2.0`.

### How was this patch tested?

New tests.

Closes #32892 from sarutak/tight-daytime-interval.

Authored-by: Kousuke Saruta 
Signed-off-by: Max Gekk 
---
 .../org/apache/spark/sql/catalyst/parser/AstBuilder.scala |  9 +++--
 .../resources/sql-tests/results/ansi/interval.sql.out | 12 ++--
 .../src/test/resources/sql-tests/results/interval.sql.out | 12 ++--
 .../test/scala/org/apache/spark/sql/SQLQuerySuite.scala   | 15 +++
 4 files changed, 34 insertions(+), 14 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 9f1d665..4bbd9bd 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -2357,9 +2357,14 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with 
SQLConfHelper with Logg
 Literal(calendarInterval.months, YearMonthIntervalType)
   } else {
 assert(calendarInterval.months == 0)
+val strToFieldIndex = DayTimeIntervalType.dayTimeFields.map(i =>
+  DayTimeIntervalType.fieldToString(i) -> i).toMap
+val fromUnit =
+  
ctx.errorCapturingUnitToUnitInterval.body.from.getText.toLowerCase(Locale.ROOT)
 val micros = IntervalUtils.getDuration(calendarInterval, 
TimeUnit.MICROSECONDS)
-// TODO(SPARK-35737): Parse day-time interval literals to tightest 
types
-Literal(micros, DayTimeIntervalType())
+val start = strToFieldIndex(fromUnit)
+val end = strToFieldIndex(toUnit)
+Literal(micros, DayTimeIntervalType(start, end))
   }
 } else {
   Literal(calendarInterval, CalendarIntervalType)
diff --git 
a/sql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out 
b/sql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out
index 3205259..f2f5d5c 100644
--- a/sql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/ansi/interval.sql.out
@@ -363,7 +363,7 @@ struct
 -- !query
 select interval '20 15' day to hour
 -- !query schema
-struct
+struct
 -- !query output
 20 15:00:00.0
 
@@ -371,7 +371,7 @@ struct
 -- !query
 select interval '20 15:40' day to minute
 -- !query schema
-struct
+struct
 -- !query output
 20 15:40:00.0
 
@@ -387,7 +387,7 @@ struct
 -- !query
 select interval '15:40' hour to minute
 -- !query schema
-struct
+struct
 -- !query output
 0 15:40:00.0
 
@@ -395,7 +395,7 @@ struct
 -- !query
 select interval '15:40:32.9989' hour to second
 -- !query schema
-struct
+struct
 -- !query output
 0 15:40:32.998999000
 
@@ -403,7 +403,7 @@ struct
 -- !query
 select interval '40:32.9989' minute to second
 -- !query schema
-struct
+struct
 -- !query output
 0 00:40:32.998999000
 
@@ -411,7 +411,7 @@ struct
 -- !query
 select interval '40:32' minute to second
 -- !query schema
-struct
+struct
 -- !query output
 0 00:40:32.0
 
diff --git a/sql/core/src/test/resources/sql-tests/results/interval.sql.out 
b/sql/core/src/test/resources/sql-tests/results/interval.sql.out
index ef9ef8f..9b44960 100644
--- a/sql/core/src/test/resources/sql-tests/results/interval.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/interval.sql.out
@@ -357,7 +357,7 @@ struct
 -- !query
 select interval '20 15' day to hour
 -- !query schema
-struct
+struct
 -- !query output
 20 15:00:00.0
 
@@ -365,7 +365,7 @@ struct
 -- !query
 select interval '20 15:40' day to minute
 -- !query schema
-struct
+struct
 -- !query output
 20 15:40:00.0
 
@@ -381,7 +381,7 @@ struct
 -- !query
 select interval '15:40' hour to minute
 -- !query schema
-struct
+struct
 -- !query output
 0 15:40:00.0