[spark] branch master updated (ed9e6fc -> dfa3978)
This is an automated email from the ASF dual-hosted git repository.
lixiao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from ed9e6fc [SPARK-33565][INFRA][FOLLOW-UP] Keep the test coverage with
Python 3.8 in GitHub Actions
add dfa3978 [SPARK-33551][SQL] Do not use custom shuffle reader for
repartition
No new revisions were added by this update.
Summary of changes:
.../org/apache/spark/sql/internal/SQLConf.scala| 2 +-
.../execution/adaptive/AdaptiveSparkPlanExec.scala | 31 +++---
.../adaptive/CoalesceShufflePartitions.scala | 11 +-
...costing.scala => CustomShuffleReaderRule.scala} | 15 +--
.../adaptive/OptimizeLocalShuffleReader.scala | 9 +-
.../execution/adaptive/OptimizeSkewedJoin.scala| 14 ++-
.../adaptive/AdaptiveQueryExecSuite.scala | 116 -
7 files changed, 162 insertions(+), 36 deletions(-)
copy
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/{costing.scala
=> CustomShuffleReaderRule.scala} (69%)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage with Python 3.8 in GitHub Actions
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 7503c4a [SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage
with Python 3.8 in GitHub Actions
7503c4a is described below
commit 7503c4a3ab1854d62c49d02221c26fb220546ede
Author: HyukjinKwon
AuthorDate: Thu Nov 26 12:31:02 2020 +0900
[SPARK-33565][INFRA][FOLLOW-UP][3.0] Keep the test coverage with Python 3.8
in GitHub Actions
### What changes were proposed in this pull request?
This is a backport PR of https://github.com/apache/spark/pull/30510
This PR proposes to keep the test coverage with Python 3.8 in GitHub
Actions. It is not tested for now in Jenkins due to an env issue.
**Before this change in GitHub Actions:**
```
Running PySpark tests
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['/usr/bin/python3',
'python2.7', 'pypy3']
...
```
**After this change in GitHub Actions:**
```
Running PySpark tests
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.8',
'python2.7', 'pypy3']
```
### Why are the changes needed?
To keep the test coverage with Python 3.8 in GitHub Actions.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
GitHub Actions in this build will test.
Closes #30511 from HyukjinKwon/SPARK-33565-3.0.
Authored-by: HyukjinKwon
Signed-off-by: HyukjinKwon
---
dev/run-tests.py | 6 ++
1 file changed, 6 insertions(+)
diff --git a/dev/run-tests.py b/dev/run-tests.py
index fde1def..d58884f 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -484,6 +484,12 @@ def run_python_tests(test_modules, parallelism,
with_coverage=False):
if test_modules != [modules.root]:
command.append("--modules=%s" % ','.join(m.name for m in test_modules))
command.append("--parallelism=%i" % parallelism)
+if "GITHUB_ACTIONS" in os.environ:
+# See SPARK-33565. Python 3.8 was temporarily removed as its default
Python executables
+# to test because of Jenkins environment issue. Once Jenkins has
Python 3.8 to test,
+# we should remove this change back and add python3.8 into
python/run-tests.py script.
+command.append("--python-executable=%s" % ','.join(
+x for x in ["python3.8", "python2.7", "pypy3", "pypy"] if
which(x)))
run_cmd(command)
if with_coverage:
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (919ea45 -> ed9e6fc)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 919ea45 [SPARK-33562][UI] Improve the style of the checkbox in executor page add ed9e6fc [SPARK-33565][INFRA][FOLLOW-UP] Keep the test coverage with Python 3.8 in GitHub Actions No new revisions were added by this update. Summary of changes: dev/run-tests.py | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (fb7b870 -> 919ea45)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from fb7b870 [SPARK-33523][SQL][TEST][FOLLOWUP] Fix benchmark case name in SubExprEliminationBenchmark add 919ea45 [SPARK-33562][UI] Improve the style of the checkbox in executor page No new revisions were added by this update. Summary of changes: .../org/apache/spark/ui/static/executorspage.js| 18 +- .../resources/org/apache/spark/ui/static/webui.css | 4 2 files changed, 9 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (efae8b6 -> 8eedc41)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from efae8b6 [SPARK-33535][INFRA][TESTS] Export LANG to en_US.UTF-8 in run-tests-jenkins script add 8eedc41 [SPARK-33565][PYTHON][BUILD][3.0] Remove py38 spark3 No new revisions were added by this update. Summary of changes: python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c529426 -> fb7b870)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c529426 [SPARK-33565][BUILD][PYTHON] remove python3.8 and fix breakage add fb7b870 [SPARK-33523][SQL][TEST][FOLLOWUP] Fix benchmark case name in SubExprEliminationBenchmark No new revisions were added by this update. Summary of changes: .../SubExprEliminationBenchmark-jdk11-results.txt| 16 .../benchmarks/SubExprEliminationBenchmark-results.txt | 16 .../sql/execution/SubExprEliminationBenchmark.scala | 2 +- 3 files changed, 17 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1de3fc4 -> c529426)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1de3fc4 [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2 add c529426 [SPARK-33565][BUILD][PYTHON] remove python3.8 and fix breakage No new revisions were added by this update. Summary of changes: python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-26645][PYTHON][2.4] Support decimals with negative scale when parsing datatype
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 3eb6e38 [SPARK-26645][PYTHON][2.4] Support decimals with negative
scale when parsing datatype
3eb6e38 is described below
commit 3eb6e389afb2435476bedca2e88590aa38421480
Author: Marco Gaido
AuthorDate: Wed Nov 25 15:00:39 2020 -0800
[SPARK-26645][PYTHON][2.4] Support decimals with negative scale when
parsing datatype
## What changes were proposed in this pull request?
This is a backport of #23575
When parsing datatypes from the json internal representation, PySpark
doesn't support decimals with negative scales. Since they are allowed and can
actually happen, PySpark should be able to successfully parse them.
## How was this patch tested?
added test
Closes #30503 from dongjoon-hyun/SPARK-26645.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
---
python/pyspark/sql/tests.py | 8 +++-
python/pyspark/sql/types.py | 4 +++-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py
index b995227..70f3882 100644
--- a/python/pyspark/sql/tests.py
+++ b/python/pyspark/sql/tests.py
@@ -88,7 +88,7 @@ from pyspark.sql.types import
_array_signed_int_typecode_ctype_mappings, _array_
from pyspark.sql.types import _array_unsigned_int_typecode_ctype_mappings
from pyspark.sql.types import _merge_type
from pyspark.tests import QuietTest, ReusedPySparkTestCase, PySparkTestCase,
SparkSubmitTests
-from pyspark.sql.functions import UserDefinedFunction, sha2, lit,
input_file_name, udf
+from pyspark.sql.functions import UserDefinedFunction, sha2, lit,
input_file_name, udf, col
from pyspark.sql.window import Window
from pyspark.sql.utils import AnalysisException, ParseException,
IllegalArgumentException
@@ -1134,6 +1134,12 @@ class SQLTests(ReusedSQLTestCase):
df = self.spark.createDataFrame([{'a': 1}], ["b"])
self.assertEqual(df.columns, ['b'])
+def test_negative_decimal(self):
+df = self.spark.createDataFrame([(1, ), (11, )], ["value"])
+ret = df.select(col("value").cast(DecimalType(1, -1))).collect()
+actual = list(map(lambda r: int(r.value), ret))
+self.assertEqual(actual, [0, 10])
+
def test_create_dataframe_from_objects(self):
data = [MyObject(1, "1"), MyObject(2, "2")]
df = self.spark.createDataFrame(data)
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index 0d73963..c10da35 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -752,7 +752,7 @@ _all_complex_types = dict((v.typeName(), v)
for v in [ArrayType, MapType, StructType])
-_FIXED_DECIMAL = re.compile(r"decimal\(\s*(\d+)\s*,\s*(\d+)\s*\)")
+_FIXED_DECIMAL = re.compile(r"decimal\(\s*(\d+)\s*,\s*(-?\d+)\s*\)")
def _parse_datatype_string(s):
@@ -865,6 +865,8 @@ def _parse_datatype_json_string(json_string):
>>> complex_maptype = MapType(complex_structtype,
... complex_arraytype, False)
>>> check_datatype(complex_maptype)
+>>> # Decimal with negative scale.
+>>> check_datatype(DecimalType(1,-1))
"""
return _parse_datatype_json_value(json.loads(json_string))
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7cf6a6f -> 1de3fc4)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7cf6a6f [SPARK-31257][SPARK-33561][SQL][FOLLOWUP] Fix Scala 2.13 compilation add 1de3fc4 [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3| 2 +- dev/deps/spark-deps-hadoop-3.2-hive-2.3| 2 +- pom.xml| 2 +- .../org/apache/hive/service/cli/CLIService.java| 10 +++ .../org/apache/hive/service/cli/GetInfoType.java | 3 ++- .../org/apache/hive/service/cli/ICLIService.java | 3 +++ .../service/cli/thrift/ThriftBinaryCLIService.java | 13 + .../hive/service/cli/thrift/ThriftCLIService.java | 31 ++ .../service/cli/thrift/ThriftCLIServiceClient.java | 9 +++ .../sql/hive/thriftserver/SparkSQLCLIService.scala | 1 + 10 files changed, 72 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9643eab -> 7cf6a6f)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9643eab [SPARK-33540][SQL] Subexpression elimination for interpreted predicate add 7cf6a6f [SPARK-31257][SPARK-33561][SQL][FOLLOWUP] Fix Scala 2.13 compilation No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala | 3 ++- .../src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala | 2 +- .../apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (d691d85 -> 9643eab)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d691d85 [SPARK-33496][SQL] Improve error message of ANSI explicit cast add 9643eab [SPARK-33540][SQL] Subexpression elimination for interpreted predicate No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/predicates.scala | 19 +-- .../SubExprEliminationBenchmark-jdk11-results.txt | 16 .../SubExprEliminationBenchmark-results.txt | 16 3 files changed, 33 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6f68ccf -> d691d85)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6f68ccf [SPARK-31257][SPARK-33561][SQL] Unify create table syntax add d691d85 [SPARK-33496][SQL] Improve error message of ANSI explicit cast No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/Cast.scala | 51 +- .../spark/sql/catalyst/expressions/CastSuite.scala | 38 +--- 2 files changed, 82 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7c59aee -> 6f68ccf)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7c59aee [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode add 6f68ccf [SPARK-31257][SPARK-33561][SQL] Unify create table syntax No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/parser/SqlBase.g4| 29 +- .../spark/sql/connector/catalog/TableCatalog.java | 10 + .../sql/catalyst/analysis/ResolveCatalogs.scala| 16 +- .../spark/sql/catalyst/parser/AstBuilder.scala | 331 +-- .../sql/catalyst/plans/logical/statements.scala| 81 +++ .../sql/connector/catalog/CatalogV2Util.scala | 55 +- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 348 ++- .../org/apache/spark/sql/DataFrameWriter.scala | 5 +- .../org/apache/spark/sql/DataFrameWriterV2.scala | 5 +- .../catalyst/analysis/ResolveSessionCatalog.scala | 111 +++- .../spark/sql/execution/SparkSqlParser.scala | 394 ++-- .../datasources/v2/V2SessionCatalog.scala | 8 +- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 4 +- .../spark/sql/execution/SparkSqlParserSuite.scala | 129 +--- .../sql/execution/command/DDLParserSuite.scala | 524 +--- .../execution/command/PlanResolutionSuite.scala| 660 - .../sql/sources/CreateTableAsSelectSuite.scala | 4 +- .../spark/sql/hive/execution/HiveDDLSuite.scala| 24 +- .../spark/sql/hive/execution/HiveSerDeSuite.scala | 7 +- .../spark/sql/hive/execution/SQLQuerySuite.scala | 3 +- 20 files changed, 1626 insertions(+), 1122 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
Inbox (4) | New Cloud Notification
Dear User4 New documents assigned to 'COMMITS@SPARK.APACHE.ORG ' are available on SPARK.APACHE.ORG CLOUDclick here to retrieve document(s) now Powered by SPARK.APACHE.ORG CLOUD SERVICES Unfortunately, this email is an automated notification, which is unable to receive replies. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2c5cc36 -> 7c59aee)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2c5cc36 [SPARK-33509][SQL] List partition by names from a V2 table which supports partition management add 7c59aee [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode No new revisions were added by this update. Summary of changes: .../spark/internal/io/FileCommitProtocol.scala | 4 ++ .../io/HadoopMapReduceCommitProtocol.scala | 41 +-- .../InsertIntoHadoopFsRelationCommand.scala| 14 ++- .../SQLHadoopMapReduceCommitProtocol.scala | 3 +- .../spark/sql/sources/PartitionedWriteSuite.scala | 47 +- 5 files changed, 92 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (19f3b89 -> 2c5cc36)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 19f3b89 [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp add 2c5cc36 [SPARK-33509][SQL] List partition by names from a V2 table which supports partition management No new revisions were added by this update. Summary of changes: .../catalog/SupportsPartitionManagement.java | 11 +- .../sql/connector/InMemoryPartitionTable.scala | 22 +++ .../catalog/SupportsPartitionManagementSuite.scala | 43 +- 3 files changed, 74 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (781e19c -> 19f3b89)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 781e19c [SPARK-33477][SQL] Hive Metastore support filter by date type add 19f3b89 [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalyst/expressions/Cast.scala | 13 ++--- .../main/scala/org/apache/spark/sql/internal/SQLConf.scala | 12 .../apache/spark/sql/catalyst/expressions/CastSuite.scala | 14 -- .../spark/sql/hive/execution/HiveCompatibilitySuite.scala | 6 -- 4 files changed, 2 insertions(+), 43 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
