[spark-docker] branch master updated: [SPARK-43372] Use ; instead of && when enable set -ex
This is an automated email from the ASF dual-hosted git repository.
yikun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-docker.git
The following commit(s) were added to refs/heads/master by this push:
new 7f9b414 [SPARK-43372] Use ; instead of && when enable set -ex
7f9b414 is described below
commit 7f9b414de48639d69c64acfd81e6792517b86f61
Author: Yikun Jiang
AuthorDate: Mon May 8 11:19:36 2023 +0800
[SPARK-43372] Use ; instead of && when enable set -ex
### What changes were proposed in this pull request?
- Use ; instead of && when enable set -ex
- ./add-dockerfiles.sh 3.4.0 to apply changes
### Why are the changes needed?
Address DOI comments: `9. using set -ex means you can use ; instead of &&
(really only matters for complex expressions, like the || in the later RUN that
does use ;)`
https://github.com/docker-library/official-images/pull/13089#issuecomment-1533540388
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
CI passed
Closes #38 from Yikun/SPARK-43372.
Authored-by: Yikun Jiang
Signed-off-by: Yikun Jiang
---
3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile | 10 +++
3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile | 8 +++---
3.4.0/scala2.12-java11-r-ubuntu/Dockerfile | 8 +++---
3.4.0/scala2.12-java11-ubuntu/Dockerfile | 32 +++---
Dockerfile.template| 32 +++---
r-python.template | 10 +++
6 files changed, 50 insertions(+), 50 deletions(-)
diff --git a/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile
b/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile
index 86337c5..12c7a4f 100644
--- a/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile
+++ b/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile
@@ -17,11 +17,11 @@
ARG BASE_IMAGE=spark:3.4.0-scala2.12-java11-ubuntu
FROM $BASE_IMAGE
-RUN set -ex && \
-apt-get update && \
-apt install -y python3 python3-pip && \
-apt install -y r-base r-base-dev && \
-rm -rf /var/cache/apt/* && \
+RUN set -ex; \
+apt-get update; \
+apt install -y python3 python3-pip; \
+apt install -y r-base r-base-dev; \
+rm -rf /var/cache/apt/*; \
rm -rf /var/lib/apt/lists/*
ENV R_HOME /usr/lib/R
diff --git a/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile
b/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile
index 540805f..1f0dd1f 100644
--- a/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile
+++ b/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile
@@ -17,8 +17,8 @@
ARG BASE_IMAGE=spark:3.4.0-scala2.12-java11-ubuntu
FROM $BASE_IMAGE
-RUN set -ex && \
-apt-get update && \
-apt install -y python3 python3-pip && \
-rm -rf /var/cache/apt/* && \
+RUN set -ex; \
+apt-get update; \
+apt install -y python3 python3-pip; \
+rm -rf /var/cache/apt/*; \
rm -rf /var/lib/apt/lists/*
diff --git a/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile
b/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile
index c65c2ce..53647b2 100644
--- a/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile
+++ b/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile
@@ -17,10 +17,10 @@
ARG BASE_IMAGE=spark:3.4.0-scala2.12-java11-ubuntu
FROM $BASE_IMAGE
-RUN set -ex && \
-apt-get update && \
-apt install -y r-base r-base-dev && \
-rm -rf /var/cache/apt/* && \
+RUN set -ex; \
+apt-get update; \
+apt install -y r-base r-base-dev; \
+rm -rf /var/cache/apt/*; \
rm -rf /var/lib/apt/lists/*
ENV R_HOME /usr/lib/R
diff --git a/3.4.0/scala2.12-java11-ubuntu/Dockerfile
b/3.4.0/scala2.12-java11-ubuntu/Dockerfile
index 21d95d4..11f997f 100644
--- a/3.4.0/scala2.12-java11-ubuntu/Dockerfile
+++ b/3.4.0/scala2.12-java11-ubuntu/Dockerfile
@@ -21,22 +21,22 @@ ARG spark_uid=185
RUN groupadd --system --gid=${spark_uid} spark && \
useradd --system --uid=${spark_uid} --gid=spark spark
-RUN set -ex && \
-apt-get update && \
-ln -s /lib /lib64 && \
-apt install -y gnupg2 wget bash tini libc6 libpam-modules krb5-user
libnss3 procps net-tools gosu && \
-mkdir -p /opt/spark && \
-mkdir /opt/spark/python && \
-mkdir -p /opt/spark/examples && \
-mkdir -p /opt/spark/work-dir && \
-chmod g+w /opt/spark/work-dir && \
-touch /opt/spark/RELEASE && \
-chown -R spark:spark /opt/spark && \
-rm /bin/sh && \
-ln -sv /bin/bash /bin/sh && \
-echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
-chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
-rm -rf /var/cache/apt/* && \
+RUN set -ex; \
+apt-get update; \
+ln -s /lib /lib64; \
+apt install -y gnupg2 wget bash tini libc6 libpam-modules krb5-user
libnss3 procps net-tools gosu; \
+mkdir -p /opt/spark; \
+mkdir /opt/spark/python; \
+mkdir -p /opt/spark/examples; \
+mkdir -p /opt/spark/work-dir; \
+chmod
[spark] branch master updated (8d99e646e98 -> d157b2d9a71)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 8d99e646e98 [MINOR][PYTHON] Remove deprecated use of typing.io add d157b2d9a71 [MINOR][PYTHON] Fix MyPy linter failure No new revisions were added by this update. Summary of changes: python/pyspark/broadcast.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [MINOR][PYTHON] Remove deprecated use of typing.io
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8d99e646e98 [MINOR][PYTHON] Remove deprecated use of typing.io 8d99e646e98 is described below commit 8d99e646e983ac1ccb383acf35dcbec745219146 Author: Shantanu <12621235+hauntsani...@users.noreply.github.com> AuthorDate: Mon May 8 10:47:00 2023 +0900 [MINOR][PYTHON] Remove deprecated use of typing.io ### What changes were proposed in this pull request? Use `typing.BinaryIO` instead of `typing.io.BinaryIO`. The latter is deprecated and had questionable type checker support, see https://github.com/python/cpython/issues/92871 ### Why are the changes needed? So Spark is unaffected when `typing.io` is removed in Python 3.13 ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing unit tests / every import of this module Closes #41084 from hauntsaninja/patch-1. Authored-by: Shantanu <12621235+hauntsani...@users.noreply.github.com> Signed-off-by: Hyukjin Kwon --- python/pyspark/broadcast.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/pyspark/broadcast.py b/python/pyspark/broadcast.py index a72bf1e059b..605663a2204 100644 --- a/python/pyspark/broadcast.py +++ b/python/pyspark/broadcast.py @@ -24,6 +24,7 @@ import pickle from typing import ( overload, Any, +BinaryIO, Callable, Dict, Generic, @@ -35,7 +36,6 @@ from typing import ( TYPE_CHECKING, Union, ) -from typing.io import BinaryIO # type: ignore[import] from pyspark.java_gateway import local_connect_and_auth from pyspark.serializers import ChunkedStream, pickle_protocol - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (63cb0939679 -> 45a1935fe38)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 63cb0939679 [SPARK-43362][SQL] Special handling of JSON type for MySQL connector add 45a1935fe38 [SPARK-43364][SS][DOCS] Add docs for RocksDB state store memory management No new revisions were added by this update. Summary of changes: docs/structured-streaming-programming-guide.md | 27 ++ 1 file changed, 27 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43362][SQL] Special handling of JSON type for MySQL connector
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 63cb0939679 [SPARK-43362][SQL] Special handling of JSON type for MySQL
connector
63cb0939679 is described below
commit 63cb09396798da2a16db4bae2b42e0f95bef831b
Author: tianhanhu
AuthorDate: Mon May 8 10:25:37 2023 +0900
[SPARK-43362][SQL] Special handling of JSON type for MySQL connector
### What changes were proposed in this pull request?
MySQL JSON type is converted into JDBC VARCHAR type with precision of -1 on
some MariaDB drivers.
When receiving VARCHAR with negative precision, Spark will throw an error.
This PR special cases this scenario by directly converting JSON type into
StringType in MySQLDialect.
### Why are the changes needed?
Enable reading MySQL tables that has a JSON column.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Update existing integration test
Closes #41040 from tianhanhu/SPARK-43362.
Authored-by: tianhanhu
Signed-off-by: Hyukjin Kwon
---
.../scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala | 8 +---
.../src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala | 4
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala
index c5ca5a72a83..dc3acb66ff1 100644
---
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala
+++
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala
@@ -68,10 +68,10 @@ class MySQLIntegrationSuite extends
DockerJDBCIntegrationSuite {
// TODO: Test locale conversion for strings.
conn.prepareStatement("CREATE TABLE strings (a CHAR(10), b VARCHAR(10), c
TINYTEXT, "
- + "d TEXT, e MEDIUMTEXT, f LONGTEXT, g BINARY(4), h VARBINARY(10), i
BLOB)"
+ + "d TEXT, e MEDIUMTEXT, f LONGTEXT, g BINARY(4), h VARBINARY(10), i
BLOB, j JSON)"
).executeUpdate()
conn.prepareStatement("INSERT INTO strings VALUES ('the', 'quick',
'brown', 'fox', " +
- "'jumps', 'over', 'the', 'lazy', 'dog')").executeUpdate()
+ "'jumps', 'over', 'the', 'lazy', 'dog', '{\"status\":
\"merrily\"}')").executeUpdate()
}
test("Basic test") {
@@ -137,7 +137,7 @@ class MySQLIntegrationSuite extends
DockerJDBCIntegrationSuite {
val rows = df.collect()
assert(rows.length == 1)
val types = rows(0).toSeq.map(x => x.getClass.toString)
-assert(types.length == 9)
+assert(types.length == 10)
assert(types(0).equals("class java.lang.String"))
assert(types(1).equals("class java.lang.String"))
assert(types(2).equals("class java.lang.String"))
@@ -147,6 +147,7 @@ class MySQLIntegrationSuite extends
DockerJDBCIntegrationSuite {
assert(types(6).equals("class [B"))
assert(types(7).equals("class [B"))
assert(types(8).equals("class [B"))
+assert(types(9).equals("class java.lang.String"))
assert(rows(0).getString(0).equals("the".padTo(10, ' ')))
assert(rows(0).getString(1).equals("quick"))
assert(rows(0).getString(2).equals("brown"))
@@ -156,6 +157,7 @@ class MySQLIntegrationSuite extends
DockerJDBCIntegrationSuite {
assert(java.util.Arrays.equals(rows(0).getAs[Array[Byte]](6),
Array[Byte](116, 104, 101, 0)))
assert(java.util.Arrays.equals(rows(0).getAs[Array[Byte]](7),
Array[Byte](108, 97, 122, 121)))
assert(java.util.Arrays.equals(rows(0).getAs[Array[Byte]](8),
Array[Byte](100, 111, 103)))
+assert(rows(0).getString(9).equals("{\"status\": \"merrily\"}"))
}
test("Basic write test") {
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
index 5e85ff3ebf6..d6edb67e57e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala
@@ -98,6 +98,10 @@ private case object MySQLDialect extends JdbcDialect with
SQLConfHelper {
} else if ("TINYTEXT".equalsIgnoreCase(typeName)) {
// TINYTEXT is Types.VARCHAR(63) from mysql jdbc, but keep it AS-IS for
historical reason
Some(StringType)
+} else if (sqlType == Types.VARCHAR && typeName.equals("JSON")) {
+ // Some MySQL JDBC drivers converts JSON type into Types.VARCHAR with a
precision of -1.
+ // Explicitly converts it into StringType here.
+ Some(StringType)
} else None
}
[spark] branch master updated: [SPARK-43032][SS][CONNECT] Python SQM bug fix
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7c5d20b [SPARK-43032][SS][CONNECT] Python SQM bug fix
7c5d20b is described below
commit 7c5d20bb012110d5855e5908cc01658355ed
Author: Wei Liu
AuthorDate: Mon May 8 10:08:27 2023 +0900
[SPARK-43032][SS][CONNECT] Python SQM bug fix
### What changes were proposed in this pull request?
Some bug fix for streaming ***connect*** python SQM
Note that I also changed ***non-connect***'s StreamingQueryManager `get()`
API to return an `Optional[StreamingQuery]`.
Before it looks like this when you get a non-exist query:
```
>>> a = spark.streams.get("--0001--0001")
>>> a
>>> a.id
Traceback (most recent call last):
File "", line 1, in
File "/home/wei.liu/oss-spark/python/pyspark/sql/streaming/query.py",
line 78, in id
return self._jsq.id().toString()
AttributeError: 'NoneType' object has no attribute 'id'
```
But now it looks like:
```
>>> a = spark.streams.get("--0001--0001")
>>> a.id
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'NoneType' object has no attribute 'id'
```
The only difference is the return type, which is not typically honored in
Python... But not very sure if that's a breaking change
### Why are the changes needed?
Bug fix
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually tested. Also verified that it won't throw even without this fix so
it's not that urgent
Closes #41037 from WweiL/SPARK-43032-python-sqm-fix.
Authored-by: Wei Liu
Signed-off-by: Hyukjin Kwon
---
.../src/main/protobuf/spark/connect/commands.proto | 2 +-
.../sql/connect/planner/SparkConnectPlanner.scala | 22 +++--
python/pyspark/sql/connect/proto/commands_pb2.py | 36 +++---
python/pyspark/sql/connect/proto/commands_pb2.pyi | 16 +-
python/pyspark/sql/connect/streaming/query.py | 14 ++---
python/pyspark/sql/streaming/query.py | 8 +++--
6 files changed, 54 insertions(+), 44 deletions(-)
diff --git
a/connector/connect/common/src/main/protobuf/spark/connect/commands.proto
b/connector/connect/common/src/main/protobuf/spark/connect/commands.proto
index b929ffa2564..72bc8b5b6ef 100644
--- a/connector/connect/common/src/main/protobuf/spark/connect/commands.proto
+++ b/connector/connect/common/src/main/protobuf/spark/connect/commands.proto
@@ -330,7 +330,7 @@ message StreamingQueryManagerCommand {
// active() API, returns a list of active queries.
bool active = 1;
// get() API, returns the StreamingQuery identified by id.
-string get = 2;
+string get_query = 2;
// awaitAnyTermination() API, wait until any query terminates or timeout.
AwaitAnyTerminationCommand await_any_termination = 3;
// resetTerminated() API.
diff --git
a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
index 8c43f982ec1..01f1e890630 100644
---
a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
+++
b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
@@ -2466,16 +2466,18 @@ class SparkConnectPlanner(val session: SparkSession) {
.toIterable
.asJava)
- case StreamingQueryManagerCommand.CommandCase.GET =>
-val query = session.streams.get(command.getGet)
-respBuilder.getQueryBuilder
- .setId(
-StreamingQueryInstanceId
- .newBuilder()
- .setId(query.id.toString)
- .setRunId(query.runId.toString)
- .build())
- .setName(SparkConnectService.convertNullString(query.name))
+ case StreamingQueryManagerCommand.CommandCase.GET_QUERY =>
+val query = session.streams.get(command.getGetQuery)
+if (query != null) {
+ respBuilder.getQueryBuilder
+.setId(
+ StreamingQueryInstanceId
+.newBuilder()
+.setId(query.id.toString)
+.setRunId(query.runId.toString)
+.build())
+.setName(SparkConnectService.convertNullString(query.name))
+}
case StreamingQueryManagerCommand.CommandCase.AWAIT_ANY_TERMINATION =>
if (command.getAwaitAnyTermination.hasTimeoutMs) {
diff --git a/python/pyspark/sql/connect/proto/commands_pb2.py
[spark] branch branch-3.3 updated (85ff71f9459 -> 9110c05d54c)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/spark.git from 85ff71f9459 [SPARK-43395][BUILD] Exclude macOS tar extended metadata in make-distribution.sh add 9110c05d54c [SPARK-37829][SQL][3.3] Dataframe.joinWith outer-join should return a null value for unmatched row No new revisions were added by this update. Summary of changes: .../sql/catalyst/encoders/ExpressionEncoder.scala | 19 ++--- .../scala/org/apache/spark/sql/DatasetSuite.scala | 45 ++ 2 files changed, 58 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (92ccc60cde5 -> e05e7e0311a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 92ccc60cde5 [SPARK-43181][SQL] Show UI WebURL in `spark-sql` shell add e05e7e0311a [MINOR][INFRA] Correct the GitHub PR label for DSTREAM No new revisions were added by this update. Summary of changes: .github/labeler.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43181][SQL] Show UI WebURL in `spark-sql` shell
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 92ccc60cde5 [SPARK-43181][SQL] Show UI WebURL in `spark-sql` shell
92ccc60cde5 is described below
commit 92ccc60cde5c6da2c08b44096f9a34f19c72aa1a
Author: panbingkun
AuthorDate: Sun May 7 13:59:23 2023 -0700
[SPARK-43181][SQL] Show UI WebURL in `spark-sql` shell
### What changes were proposed in this pull request?
The pr aims to display `the Spark WEB UI address` when spark-sql startup.
### Why are the changes needed?
Promoting user experience.
Like `spark-shell`, it would be great if `spark-sql` show the UI
information.
```
$ bin/spark-sql
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
23/05/07 13:58:26 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
23/05/07 13:58:27 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout
does not exist
23/05/07 13:58:27 WARN HiveConf: HiveConf of name hive.stats.retries.wait
does not exist
23/05/07 13:58:28 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 2.3.0
23/05/07 13:58:28 WARN ObjectStore: setMetaStoreSchemaVersion called but
recording version is disabled: version = 2.3.0, comment = Set by MetaStore
dongjoon127.0.0.1
Spark Web UI available at http://localhost:4040
Spark master: local[*], Application Id: local-1683493106875
spark-sql (default)>
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA & Manually test.
Closes #40844 from panbingkun/SPARK-43181.
Authored-by: panbingkun
Signed-off-by: Dongjoon Hyun
---
.../org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
index c7c905312b7..2302482bb72 100644
---
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
+++
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
@@ -454,6 +454,9 @@ private[hive] class SparkSQLCLIDriver extends CliDriver
with Logging {
def printMasterAndAppId(): Unit = {
val master = SparkSQLEnv.sparkContext.master
val appId = SparkSQLEnv.sparkContext.applicationId
+SparkSQLEnv.sparkContext.uiWebUrl.foreach {
+ webUrl => console.printInfo(s"Spark Web UI available at $webUrl")
+}
console.printInfo(s"Spark master: $master, Application Id: $appId")
}
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-42780][BUILD] Upgrade `Tink` to 1.9.0
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 96814da9e1a [SPARK-42780][BUILD] Upgrade `Tink` to 1.9.0 96814da9e1a is described below commit 96814da9e1a2400acc7e31cfc6b3956f7d514031 Author: bjornjorgensen AuthorDate: Sun May 7 13:29:37 2023 -0700 [SPARK-42780][BUILD] Upgrade `Tink` to 1.9.0 ### What changes were proposed in this pull request? Upgrade google Tink from 1.7.0 to 1.9.0 [Release note for 1.8.0](https://github.com/tink-crypto/tink-java/releases/tag/v1.8.0) [Release note for 1.9.0](https://github.com/tink-crypto/tink-java/releases/tag/v1.9.0) ### Why are the changes needed? [SNYK-JAVA-COMGOOGLEPROTOBUF-3040284](https://security.snyk.io/vuln/SNYK-JAVA-COMGOOGLEPROTOBUF-3040284) [SNYK-JAVA-COMGOOGLEPROTOBUF-3167772](https://security.snyk.io/vuln/SNYK-JAVA-COMGOOGLEPROTOBUF-3167772) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This have be benchmarks tested With "com.google.crypto.tink" % "tink" % "1.6.1" (min, avg, max) = (75024163.500, 76331532.832, 77324718.069), stdev = 652319.870 With "com.google.crypto.tink" % "tink" % "1.9.0" (min, avg, max) = (76279051.841, 77512667.749, 78590966.453), stdev = 632832.384 Almost the same.. Think 1.9.0 is perhaps a bit slower. Pass GA Closes #40878 from bjornjorgensen/Tink1.9.0. Authored-by: bjornjorgensen Signed-off-by: Dongjoon Hyun --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 12 +++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index 27df72ecaf8..5174b6b0335 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -245,7 +245,7 @@ stax-api/1.0.1//stax-api-1.0.1.jar stream/2.9.6//stream-2.9.6.jar super-csv/2.2.0//super-csv-2.2.0.jar threeten-extra/1.7.1//threeten-extra-1.7.1.jar -tink/1.7.0//tink-1.7.0.jar +tink/1.9.0//tink-1.9.0.jar transaction-api/1.1//transaction-api-1.1.jar univocity-parsers/2.9.1//univocity-parsers-2.9.1.jar wildfly-openssl/1.1.3.Final//wildfly-openssl-1.1.3.Final.jar diff --git a/pom.xml b/pom.xml index 96ee3fb5ed9..04260b51c60 100644 --- a/pom.xml +++ b/pom.xml @@ -215,7 +215,7 @@ 1.1.0 1.5.0 1.60 -1.7.0 +1.9.0 4.1.89.Final
[spark] branch master updated (04ef3d5d0f2 -> 1e090a57f0c)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 04ef3d5d0f2 [SPARK-43394][BUILD] Upgrade maven to 3.8.8 add 1e090a57f0c [SPARK-43347][PYTHON][FOLLOWUP] Change black min target-version to py38 No new revisions were added by this update. Summary of changes: dev/pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43394][BUILD] Upgrade maven to 3.8.8
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 04ef3d5d0f2 [SPARK-43394][BUILD] Upgrade maven to 3.8.8
04ef3d5d0f2 is described below
commit 04ef3d5d0f2bfebce8dd3b48b9861a2aa5ba1c3a
Author: Cheng Pan
AuthorDate: Sun May 7 08:24:12 2023 -0500
[SPARK-43394][BUILD] Upgrade maven to 3.8.8
### What changes were proposed in this pull request?
Upgrade Maven from 3.8.7 to 3.8.8.
### Why are the changes needed?
Maven 3.8.8 is the latest patched version of 3.8.x
https://maven.apache.org/docs/3.8.8/release-notes.html
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GA.
Closes #41073 from pan3793/SPARK-43394.
Authored-by: Cheng Pan
Signed-off-by: Sean Owen
---
dev/appveyor-install-dependencies.ps1 | 2 +-
docs/building-spark.md| 2 +-
pom.xml | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/dev/appveyor-install-dependencies.ps1
b/dev/appveyor-install-dependencies.ps1
index 7f4f027c820..88090149f5c 100644
--- a/dev/appveyor-install-dependencies.ps1
+++ b/dev/appveyor-install-dependencies.ps1
@@ -81,7 +81,7 @@ if (!(Test-Path $tools)) {
# == Maven
# Push-Location $tools
#
-# $mavenVer = "3.8.7"
+# $mavenVer = "3.8.8"
# Start-FileDownload
"https://archive.apache.org/dist/maven/maven-3/$mavenVer/binaries/apache-maven-$mavenVer-bin.zip;
"maven.zip"
#
# # extract
diff --git a/docs/building-spark.md b/docs/building-spark.md
index ba8dddbf6b1..4b8e70655d5 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -27,7 +27,7 @@ license: |
## Apache Maven
The Maven-based build is the build of reference for Apache Spark.
-Building Spark using Maven requires Maven 3.8.7 and Java 8.
+Building Spark using Maven requires Maven 3.8.8 and Java 8/11/17.
Spark requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
3.0.0.
### Setting up Maven's Memory Usage
diff --git a/pom.xml b/pom.xml
index c760eaf0cbb..96ee3fb5ed9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -114,7 +114,7 @@
1.8
${java.version}
${java.version}
-3.8.7
+3.8.8
1.6.0
spark
2.0.7
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
