(spark) branch master updated: [SPARK-46196][PYTHON][DOCS] Add missing `toDegrees/toRadians/atan2/approxCountDistinct` function descriptions
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f0ec157fe833 [SPARK-46196][PYTHON][DOCS] Add missing `toDegrees/toRadians/atan2/approxCountDistinct` function descriptions f0ec157fe833 is described below commit f0ec157fe8339284c1ca301e8af24c144af51e65 Author: Ruifeng Zheng AuthorDate: Fri Dec 1 00:19:32 2023 -0800 [SPARK-46196][PYTHON][DOCS] Add missing `toDegrees/toRadians/atan2/approxCountDistinct` function descriptions ### What changes were proposed in this pull request? Add missing function descriptions ### Why are the changes needed? they are missing in https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/functions.html  ### Does this PR introduce _any_ user-facing change? yes, doc changes ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #44104 from zhengruifeng/py_doc_fix_atan2. Authored-by: Ruifeng Zheng Signed-off-by: Dongjoon Hyun --- python/pyspark/sql/functions/builtin.py | 12 1 file changed, 12 insertions(+) diff --git a/python/pyspark/sql/functions/builtin.py b/python/pyspark/sql/functions/builtin.py index d985b9e6138f..5e5c70322ec9 100644 --- a/python/pyspark/sql/functions/builtin.py +++ b/python/pyspark/sql/functions/builtin.py @@ -2546,6 +2546,9 @@ def tanh(col: "ColumnOrName") -> Column: @_try_remote_functions def toDegrees(col: "ColumnOrName") -> Column: """ +Converts an angle measured in radians to an approximately equivalent angle +measured in degrees. + .. versionadded:: 1.4.0 .. versionchanged:: 3.4.0 @@ -2561,6 +2564,9 @@ def toDegrees(col: "ColumnOrName") -> Column: @_try_remote_functions def toRadians(col: "ColumnOrName") -> Column: """ +Converts an angle measured in degrees to an approximately equivalent angle +measured in radians. + .. versionadded:: 1.4.0 .. versionchanged:: 3.4.0 @@ -4025,6 +4031,9 @@ def radians(col: "ColumnOrName") -> Column: @_try_remote_functions def atan2(col1: Union["ColumnOrName", float], col2: Union["ColumnOrName", float]) -> Column: """ +Compute the angle in radians between the positive x-axis of a plane +and the point given by the coordinates + .. versionadded:: 1.4.0 .. versionchanged:: 3.4.0 @@ -4412,6 +4421,9 @@ def percent_rank() -> Column: @_try_remote_functions def approxCountDistinct(col: "ColumnOrName", rsd: Optional[float] = None) -> Column: """ +This aggregate function returns a new :class:`~pyspark.sql.Column`, which estimates +the approximate distinct count of elements in a specified column or a group of columns. + .. versionadded:: 1.3.0 .. versionchanged:: 3.4.0 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated: [SPARK-46197][INFRA] Upgrade `memory-profiler>=0.61.0` for Python 3.12
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2ff9b7241e46 [SPARK-46197][INFRA] Upgrade `memory-profiler>=0.61.0`
for Python 3.12
2ff9b7241e46 is described below
commit 2ff9b7241e4659c1918b32ceef35aa30206ae077
Author: Ruifeng Zheng
AuthorDate: Fri Dec 1 01:23:11 2023 -0800
[SPARK-46197][INFRA] Upgrade `memory-profiler>=0.61.0` for Python 3.12
### What changes were proposed in this pull request?
Upgrade memory-profiler>=0.61.0 for Python 3.12
### Why are the changes needed?
`memory-profiler==0.60.0` does not support Python 3.12
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #44105 from zhengruifeng/infra_upgrade_memory-profiler.
Authored-by: Ruifeng Zheng
Signed-off-by: Dongjoon Hyun
---
dev/infra/Dockerfile | 8
dev/requirements.txt | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index 7348c6af1e05..f0ca4a47d698 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -93,7 +93,7 @@ RUN Rscript -e "devtools::install_version('preferably',
version='0.4', repos='ht
ENV R_LIBS_SITE
"/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library"
RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas<=2.1.3' scipy coverage
matplotlib
-RUN python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn>=1.3.2'
+RUN python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
# Add Python deps for Spark Connect.
RUN python3.9 -m pip install 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
@@ -110,7 +110,7 @@ RUN apt-get update && apt-get install -y \
python3.10 python3.10-distutils \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
-RUN python3.10 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn>=1.3.2'
+RUN python3.10 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
RUN python3.10 -m pip install 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
RUN python3.10 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
RUN python3.10 -m pip install torcheval
@@ -122,7 +122,7 @@ RUN apt-get update && apt-get install -y \
python3.11 python3.11-distutils \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11
-RUN python3.11 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn>=1.3.2'
+RUN python3.11 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
RUN python3.11 -m pip install 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
RUN python3.11 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
RUN python3.11 -m pip install torcheval
@@ -134,7 +134,7 @@ RUN apt-get update && apt-get install -y \
python3.12 python3.12-distutils \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.12
-RUN python3.12 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'scikit-learn>=1.3.2'
+RUN python3.12 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
RUN python3.12 -m pip install 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
# TODO(SPARK-46078) Use official one instead of nightly build when it's ready
RUN python3.12 -m pip install --pre torch --index-url
https://
(spark) branch master updated: [SPARK-46187][SQL] Align codegen and non-codegen implementation of `StringDecode`
This is an automated email from the ASF dual-hosted git repository.
maxgekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e93bff6fc0bc [SPARK-46187][SQL] Align codegen and non-codegen
implementation of `StringDecode`
e93bff6fc0bc is described below
commit e93bff6fc0bc3a549de02958ffc17b1bca3d50b8
Author: Max Gekk
AuthorDate: Fri Dec 1 10:42:31 2023 +0100
[SPARK-46187][SQL] Align codegen and non-codegen implementation of
`StringDecode`
### What changes were proposed in this pull request?
In the PR, I propose to change the implementation of interpretation mode of
`StringDecode` and apparently of the `decode` function. And make it consistent
to codegen. Both implementation raise the same error with of the error class
`INVALID_PARAMETER_VALUE.CHARSET`.
### Why are the changes needed?
To make codegen and non-codegen of the `StringDecode` expression
consistent. So, users will observe the same behaviour in both modes.
### Does this PR introduce _any_ user-facing change?
Yes, if user code depends on error from `decode()`.
### How was this patch tested?
By running the following test suites:
```
$ PYSPARK_PYTHON=python3 build/sbt "sql/testOnly
org.apache.spark.sql.SQLQueryTestSuite -- -z string-functions.sql"
$ build/sbt "core/testOnly *SparkThrowableSuite"
$ build/sbt "test:testOnly *.StringFunctionsSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44094 from MaxGekk/align-codegen-stringdecode.
Authored-by: Max Gekk
Signed-off-by: Max Gekk
---
.../catalyst/expressions/stringExpressions.scala | 18
.../analyzer-results/ansi/string-functions.sql.out | 15 ++
.../analyzer-results/string-functions.sql.out | 15 ++
.../sql-tests/inputs/string-functions.sql | 2 ++
.../results/ansi/string-functions.sql.out | 34 ++
.../sql-tests/results/string-functions.sql.out | 34 ++
6 files changed, 113 insertions(+), 5 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index 412422f4da4e..84a5eebd70ec 100755
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -2648,18 +2648,26 @@ case class StringDecode(bin: Expression, charset:
Expression)
protected override def nullSafeEval(input1: Any, input2: Any): Any = {
val fromCharset = input2.asInstanceOf[UTF8String].toString
-UTF8String.fromString(new String(input1.asInstanceOf[Array[Byte]],
fromCharset))
+try {
+ UTF8String.fromString(new String(input1.asInstanceOf[Array[Byte]],
fromCharset))
+} catch {
+ case _: UnsupportedEncodingException =>
+throw QueryExecutionErrors.invalidCharsetError(prettyName, fromCharset)
+}
}
override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
-nullSafeCodeGen(ctx, ev, (bytes, charset) =>
+nullSafeCodeGen(ctx, ev, (bytes, charset) => {
+ val fromCharset = ctx.freshName("fromCharset")
s"""
+String $fromCharset = $charset.toString();
try {
- ${ev.value} = UTF8String.fromString(new String($bytes,
$charset.toString()));
+ ${ev.value} = UTF8String.fromString(new String($bytes,
$fromCharset));
} catch (java.io.UnsupportedEncodingException e) {
- org.apache.spark.unsafe.Platform.throwException(e);
+ throw QueryExecutionErrors.invalidCharsetError("$prettyName",
$fromCharset);
}
- """)
+ """
+})
}
override protected def withNewChildrenInternal(
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/ansi/string-functions.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/ansi/string-functions.sql.out
index 9d8705e3e862..7ace31d5 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/ansi/string-functions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/ansi/string-functions.sql.out
@@ -799,6 +799,21 @@ Project [decode(null, 6, Spark, null, SQL, 4, rocks, null,
.) AS decode(NULL, 6,
+- OneRowRelation
+-- !query
+select decode(X'68656c6c6f', 'Windows-xxx')
+-- !query analysis
+Project [decode(0x68656C6C6F, Windows-xxx) AS decode(X'68656C6C6F',
Windows-xxx)#x]
++- OneRowRelation
+
+
+-- !query
+select decode(scol, ecol) from values(X'68656c6c6f', 'Windows-xxx') as t(scol,
ecol)
+-- !query analysis
+Project [decode(scol#x, ecol#x) AS decode(scol, ecol)#x]
++- SubqueryAlias t
+ +- LocalRelation [scol#x, ec
(spark) branch master updated: [SPARK-46193][CORE][TESTS] Add `PersistenceEngineBenchmark`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0e689611f099 [SPARK-46193][CORE][TESTS] Add
`PersistenceEngineBenchmark`
0e689611f099 is described below
commit 0e689611f09968c3a46689294184de29d097302b
Author: Dongjoon Hyun
AuthorDate: Fri Dec 1 02:52:29 2023 -0800
[SPARK-46193][CORE][TESTS] Add `PersistenceEngineBenchmark`
### What changes were proposed in this pull request?
This PR aims to provide a new benchmark, `PersistenceEngineBenchmark`.
### Why are the changes needed?
This is beneficial for both the developers and the users by providing a
consistent measurement.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual review.
```
$ build/sbt "core/Test/runMain
org.apache.spark.deploy.master.PersistenceEngineBenchmark"
...
[info] OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Mac OS X 14.2
[info] Apple M1 Max
[info] 1000 Workers: Best Time(ms) Avg
Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative
[info]
[info] ZooKeeperPersistenceEngine11179
11198 20 0.011179348.5 1.0X
[info] FileSystemPersistenceEngine 416
422 6 0.0 415745.2 26.9X
[info] BlackHolePersistenceEngine0
0 0 22.7 44.1 253597.7X
```
```
$ bin/spark-submit --driver-memory 6g --class
org.apache.spark.deploy.master.PersistenceEngineBenchmark --jars `find
~/Library/Caches/Coursier/v1 -name 'curator-test-*.jar'`
core/target/scala-2.13/spark-core_2.13-4.0.0-SNAPSHOT-tests.jar
...
OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Mac OS X 14.2
Apple M1 Max
1000 Workers: Best Time(ms) Avg Time(ms)
Stdev(ms)Rate(M/s) Per Row(ns) Relative
ZooKeeperPersistenceEngine11565 11857
373 0.011564757.8 1.0X
FileSystemPersistenceEngine 426426
1 0.0 425605.0 27.2X
BlackHolePersistenceEngine0 0
0 27.4 36.5 316478.5X
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44102 from dongjoon-hyun/SPARK-46193.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
---
.github/workflows/benchmark.yml| 2 +-
.../PersistenceEngineBenchmark-jdk21-results.txt | 13 +++
.../PersistenceEngineBenchmark-results.txt | 13 +++
.../deploy/master/PersistenceEngineBenchmark.scala | 114 +
4 files changed, 141 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml
index 8e7551fa7738..3cb63404bcac 100644
--- a/.github/workflows/benchmark.yml
+++ b/.github/workflows/benchmark.yml
@@ -177,7 +177,7 @@ jobs:
# In benchmark, we use local as master so set driver memory only. Note
that GitHub Actions has 7 GB memory limit.
bin/spark-submit \
--driver-memory 6g --class org.apache.spark.benchmark.Benchmarks \
- --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name
'*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \
+ --jars "`find . -name '*-SNAPSHOT-tests.jar' -o -name
'*avro*-SNAPSHOT.jar' | paste -sd ',' -`,`find ~/.cache/coursier -name
'curator-test-*.jar'`" \
"`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
"${{ github.event.inputs.class }}"
# To keep the directory structure and file permissions, tar them
diff --git a/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt
b/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt
new file mode 100644
index ..3312d6feff88
--- /dev/null
+++ b/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt
@@ -0,0 +1,13 @@
+
+PersistenceEngineBenchmark
+
+
+OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1051-azure
+AMD EPYC 7763 64-Core Processor
+1000 Workers: Best Time(ms) Avg Time(ms)
Stdev(ms)Rate(M/s) Per Row(ns)
(spark) branch master updated: [SPARK-46204][K8S] Fix `dev-run-integration-tests.sh` to use 17 for `JAVA_VERSION`
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d78126978799 [SPARK-46204][K8S] Fix `dev-run-integration-tests.sh` to use 17 for `JAVA_VERSION` d78126978799 is described below commit d78126978799e2d125c20b7db16855fdd628e874 Author: hannahkamundson AuthorDate: Fri Dec 1 12:21:25 2023 -0800 [SPARK-46204][K8S] Fix `dev-run-integration-tests.sh` to use 17 for `JAVA_VERSION` ### What changes were proposed in this pull request? Tie the Kubernetes integration dev test script to Java 17. ### Why are the changes needed? The Kubernetes dev integration test shell script was tied to Java 8. When it was run as is (and not forcing Java version >= 14), the build would fail because it couldn't build different features that required Java >= 14. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? I ran `./dev/dev-run-integration-tests.sh` and it worked. ### Was this patch authored or co-authored using generative AI tooling? No Closes #44112 from hannahkamundson/SPARK-46204. Authored-by: hannahkamundson Signed-off-by: Dongjoon Hyun --- .../kubernetes/integration-tests/dev/dev-run-integration-tests.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh index f5f93adeddf6..a0834e1ff23f 100755 --- a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh +++ b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh @@ -38,7 +38,7 @@ CONTEXT= INCLUDE_TAGS="k8s" EXCLUDE_TAGS= DEFAULT_EXCLUDE_TAGS="N/A" -JAVA_VERSION="8" +JAVA_VERSION="17" BUILD_DEPENDENCIES_MVN_FLAG="-am" HADOOP_PROFILE="hadoop-3" MVN="$TEST_ROOT_DIR/build/mvn" - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated: [SPARK-46200][INFRA] re-org the testing dockerfile
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c85d41c558b6 [SPARK-46200][INFRA] re-org the testing dockerfile
c85d41c558b6 is described below
commit c85d41c558b671f815fca980c614a22ca267c28b
Author: Ruifeng Zheng
AuthorDate: Fri Dec 1 12:26:32 2023 -0800
[SPARK-46200][INFRA] re-org the testing dockerfile
### What changes were proposed in this pull request?
re-org the testing dockerfile:
1, move R package installation before the python part;
2, combine pip install commands to make sure no conflict (except the
torch-related pkgs, since we can not specify the `--index-url` to a subset of
pkgs in single pip command):
```
RUN python3.9 -m pip install pkg-a==x
RUN python3.9 -m pip install pkg-b==y
```
pkg-b installation can potentially break `pkg-a==x` by installing another
version
### Why are the changes needed?
to make sure no conflict
### Does this PR introduce _any_ user-facing change?
no, test-only
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #44107 from zhengruifeng/infra_docker_refactor.
Authored-by: Ruifeng Zheng
Signed-off-by: Dongjoon Hyun
---
dev/infra/Dockerfile | 50 +++---
1 file changed, 23 insertions(+), 27 deletions(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index f0ca4a47d698..3e449bcb6c82 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -63,16 +63,6 @@ RUN apt-get update && apt-get install -y \
zlib1g-dev \
&& rm -rf /var/lib/apt/lists/*
-RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
-
-RUN add-apt-repository ppa:pypy/ppa
-
-RUN mkdir -p /usr/local/pypy/pypy3.8 && \
-curl -sqL
https://downloads.python.org/pypy/pypy3.8-v7.3.11-linux64.tar.bz2 | tar xjf -
-C /usr/local/pypy/pypy3.8 --strip-components=1 && \
-ln -sf /usr/local/pypy/pypy3.8/bin/pypy /usr/local/bin/pypy3.8 && \
-ln -sf /usr/local/pypy/pypy3.8/bin/pypy /usr/local/bin/pypy3
-
-RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/' >>
/etc/apt/sources.list
RUN gpg --keyserver hkps://keyserver.ubuntu.com --recv-key
E298A3A825C0D65DFD57CBB651716619E084DAB9
@@ -92,17 +82,28 @@ RUN Rscript -e "devtools::install_version('preferably',
version='0.4', repos='ht
# See more in SPARK-39735
ENV R_LIBS_SITE
"/usr/local/lib/R/site-library:${R_LIBS_SITE}:/usr/lib/R/library"
+
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.9
+
+
+RUN add-apt-repository ppa:pypy/ppa
+RUN mkdir -p /usr/local/pypy/pypy3.8 && \
+curl -sqL
https://downloads.python.org/pypy/pypy3.8-v7.3.11-linux64.tar.bz2 | tar xjf -
-C /usr/local/pypy/pypy3.8 --strip-components=1 && \
+ln -sf /usr/local/pypy/pypy3.8/bin/pypy /usr/local/bin/pypy3.8 && \
+ln -sf /usr/local/pypy/pypy3.8/bin/pypy /usr/local/bin/pypy3
+RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
RUN pypy3 -m pip install numpy 'six==1.16.0' 'pandas<=2.1.3' scipy coverage
matplotlib
-RUN python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'six==1.16.0'
'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.8.1'
coverage matplotlib openpyxl 'memory-profiler>=0.61.0' 'scikit-learn>=1.3.2'
-# Add Python deps for Spark Connect.
-RUN python3.9 -m pip install 'grpcio==1.59.3' 'grpcio-status==1.59.3'
'protobuf==4.25.1' 'googleapis-common-protos==1.56.4'
-# Add torch as a testing dependency for TorchDistributor
+ARG BASIC_PIP_PKGS="numpy pyarrow>=14.0.0 six==1.16.0 pandas<=2.1.3 scipy
unittest-xml-reporting plotly>=4.8 mlflow>=2.8.1 coverage matplotlib openpyxl
memory-profiler>=0.61.0 scikit-learn>=1.3.2"
+# Python deps for Spark Connect
+ARG CONNECT_PIP_PKGS="grpcio==1.59.3 grpcio-status==1.59.3 protobuf==4.25.1
googleapis-common-protos==1.56.4"
+
+
+RUN python3.9 -m pip install $BASIC_PIP_PKGS $CONNECT_PIP_PKGS
+# Add torch as a testing dependency for TorchDistributor and
DeepspeedTorchDistributor
RUN python3.9 -m pip install 'torch<=2.0.1' torchvision --index-url
https://download.pytorch.org/whl/cpu
-RUN python3.9 -m pip install torcheval
-# Add Deepspeed as a testing dependency for DeepspeedTorchDistributor
-RUN python3.9 -m pip install deepspeed
+RUN python3.9 -m pip install deepspeed torcheval
# Install Python 3.10 at the last stage to avoid breaking Python 3.9
RUN add-apt-repository ppa:deadsnakes/ppa
@@ -110,11 +111,9 @@ RUN apt-get update && apt-get install -y \
python3.10 python3.10-distutils \
&& rm -rf /var/lib/apt/lists/*
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10
-RUN python3.10 -m pip install numpy 'pyarrow>=14.0.0' 'six=
(spark) branch master updated: [SPARK-46199][PYTHON][DOCS] Add PyPi link icon to PySpark doc header
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8060e7e73170 [SPARK-46199][PYTHON][DOCS] Add PyPi link icon to PySpark
doc header
8060e7e73170 is described below
commit 8060e7e73170c0122acb2a005f3c54487e226208
Author: panbingkun
AuthorDate: Fri Dec 1 12:29:29 2023 -0800
[SPARK-46199][PYTHON][DOCS] Add PyPi link icon to PySpark doc header
### What changes were proposed in this pull request?
This PR proposes to introduce a `PyPi link icon` in the header of the
PySpark documentation, similar to what is currently implemented in the
`pydata-sphinx-theme` docs.
https://pydata-sphinx-theme.readthedocs.io/en/v0.13.3/user_guide/styling.html#
https://github.com/apache/spark/assets/15246973/4bb66f51-96e7-45d5-890b-03a4700f1ec7";>
### Why are the changes needed?
This change aligns with the practices of other open-source projects like
`pydata-sphinx-theme`, facilitating community engagement and collaboration.
### Does this PR introduce _any_ user-facing change?
No API changes, but a `PyPi link icon` will be added to the top right
corner of the PySpark docs header. This icon will be linked to the `PySpark
PyPi` as below:
https://github.com/apache/spark/assets/15246973/4c5a70a0-cb30-4615-827d-97610a71d5e4";>
### How was this patch tested?
Manually test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44106 from panbingkun/SPARK-46199.
Authored-by: panbingkun
Signed-off-by: Dongjoon Hyun
---
python/docs/source/conf.py | 13 -
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/python/docs/source/conf.py b/python/docs/source/conf.py
index 0ac06f802c95..640155cee5a3 100644
--- a/python/docs/source/conf.py
+++ b/python/docs/source/conf.py
@@ -204,7 +204,18 @@ html_theme_options = {
"image_light": "_static/spark-logo-light.png",
"image_dark": "_static/spark-logo-dark.png",
},
-"github_url": "https://github.com/apache/spark";,
+"icon_links": [
+{
+"name": "GitHub",
+"url": "https://github.com/apache/spark";,
+"icon": "fa-brands fa-github",
+},
+{
+"name": "PyPI",
+"url": "https://pypi.org/project/pyspark";,
+"icon": "fa-solid fa-box",
+},
+]
}
# Add any paths that contain custom themes here, relative to this directory.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
(spark) branch master updated: [SPARK-46201][PYTHON][DOCS] Correct the typing of `schema_of_{csv, json, xml}`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a4678143f6bb [SPARK-46201][PYTHON][DOCS] Correct the typing of
`schema_of_{csv, json, xml}`
a4678143f6bb is described below
commit a4678143f6bb9fd4356bf12a1db993cdfa22c7bd
Author: Ruifeng Zheng
AuthorDate: Fri Dec 1 12:28:10 2023 -0800
[SPARK-46201][PYTHON][DOCS] Correct the typing of `schema_of_{csv, json,
xml}`
### What changes were proposed in this pull request?
Correct the typing of `schema_of_{csv, json, xml}`
### Why are the changes needed?
although `ColumnOrName` is defined as `ColumnOrName = Union[Column, str]`,
we should not use it when the string here is not a column name.
e.g.
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.schema_of_csv.html
in this case, we should follow parameter `schema` in `from_csv`:
### Does this PR introduce _any_ user-facing change?
yes, doc-change
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #44108 from zhengruifeng/py_doc_schema_of.
Authored-by: Ruifeng Zheng
Signed-off-by: Dongjoon Hyun
---
python/pyspark/sql/functions/builtin.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/functions/builtin.py
b/python/pyspark/sql/functions/builtin.py
index 5e5c70322ec9..ac237f10c2e7 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -13753,7 +13753,7 @@ def to_json(col: "ColumnOrName", options:
Optional[Dict[str, str]] = None) -> Co
@_try_remote_functions
-def schema_of_json(json: "ColumnOrName", options: Optional[Dict[str, str]] =
None) -> Column:
+def schema_of_json(json: Union[Column, str], options: Optional[Dict[str, str]]
= None) -> Column:
"""
Parses a JSON string and infers its schema in DDL format.
@@ -13941,7 +13941,7 @@ def from_xml(
@_try_remote_functions
-def schema_of_xml(xml: "ColumnOrName", options: Optional[Dict[str, str]] =
None) -> Column:
+def schema_of_xml(xml: Union[Column, str], options: Optional[Dict[str, str]] =
None) -> Column:
"""
Parses a XML string and infers its schema in DDL format.
@@ -14055,7 +14055,7 @@ def to_xml(col: "ColumnOrName", options:
Optional[Dict[str, str]] = None) -> Col
@_try_remote_functions
-def schema_of_csv(csv: "ColumnOrName", options: Optional[Dict[str, str]] =
None) -> Column:
+def schema_of_csv(csv: Union[Column, str], options: Optional[Dict[str, str]] =
None) -> Column:
"""
Parses a CSV string and infers its schema in DDL format.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
(spark-docker) branch master updated: [SPARK-46185] Add official image Dockerfile for Apache Spark 3.4.2
This is an automated email from the ASF dual-hosted git repository.
yikun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-docker.git
The following commit(s) were added to refs/heads/master by this push:
new ec69b9c [SPARK-46185] Add official image Dockerfile for Apache Spark
3.4.2
ec69b9c is described below
commit ec69b9c77bc733ed5937f5068d23f7407eb51ea9
Author: Yikun Jiang
AuthorDate: Sat Dec 2 10:00:48 2023 +0800
[SPARK-46185] Add official image Dockerfile for Apache Spark 3.4.2
### What changes were proposed in this pull request?
Add Apache Spark 3.4.2 Dockerfiles.
- Add 3.4.2 GPG key
- Add .github/workflows/build_3.4.2.yaml
- `./add-dockerfiles.sh 3.4.2` to generate dockerfiles (and remove master
changes:
https://github.com/apache/spark-docker/pull/55/commits/24cbf40abdc252fdcf48303efa33ba7f84adefaf)
- Add version and tag info
### Why are the changes needed?
Apache Spark 3.4.2 released
### Does this PR introduce _any_ user-facing change?
Docker image will be published.
### How was this patch tested?
Add workflow and CI passed
Closes #57 from Yikun/3.4.2.
Authored-by: Yikun Jiang
Signed-off-by: Yikun Jiang
---
.github/workflows/build_3.4.2.yaml | 41 +++
.github/workflows/publish.yml | 1 +
.github/workflows/test.yml | 1 +
3.4.2/scala2.12-java11-python3-r-ubuntu/Dockerfile | 29 +
3.4.2/scala2.12-java11-python3-ubuntu/Dockerfile | 26 +
3.4.2/scala2.12-java11-r-ubuntu/Dockerfile | 28 +
3.4.2/scala2.12-java11-ubuntu/Dockerfile | 79 +
3.4.2/scala2.12-java11-ubuntu/entrypoint.sh| 126 +
tools/template.py | 2 +
versions.json | 28 +
10 files changed, 361 insertions(+)
diff --git a/.github/workflows/build_3.4.2.yaml
b/.github/workflows/build_3.4.2.yaml
new file mode 100644
index 000..8ae17d1
--- /dev/null
+++ b/.github/workflows/build_3.4.2.yaml
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build and Test (3.4.2)"
+
+on:
+ pull_request:
+branches:
+ - 'master'
+paths:
+ - '3.4.2/**'
+
+jobs:
+ run-build:
+strategy:
+ matrix:
+image-type: ["all", "python", "scala", "r"]
+name: Run
+secrets: inherit
+uses: ./.github/workflows/main.yml
+with:
+ spark: 3.4.2
+ scala: 2.12
+ java: 11
+ image-type: ${{ matrix.image-type }}
diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
index 879a9c2..ec0d66c 100644
--- a/.github/workflows/publish.yml
+++ b/.github/workflows/publish.yml
@@ -29,6 +29,7 @@ on:
type: choice
options:
- 3.5.0
+- 3.4.2
- 3.4.1
- 3.4.0
- 3.3.3
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 689981a..df79364 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -29,6 +29,7 @@ on:
type: choice
options:
- 3.5.0
+- 3.4.2
- 3.4.1
- 3.4.0
- 3.3.3
diff --git a/3.4.2/scala2.12-java11-python3-r-ubuntu/Dockerfile
b/3.4.2/scala2.12-java11-python3-r-ubuntu/Dockerfile
new file mode 100644
index 000..7c7e96a
--- /dev/null
+++ b/3.4.2/scala2.12-java11-python3-r-ubuntu/Dockerfile
@@ -0,0 +1,29 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the Licen
(spark) branch master updated: [SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer`
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0d40b1aea758 [SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer` 0d40b1aea758 is described below commit 0d40b1aea758b95a4416c8653599af8713a4aa16 Author: Dongjoon Hyun AuthorDate: Fri Dec 1 18:29:42 2023 -0800 [SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer` ### What changes were proposed in this pull request? This PR aims to improve `PersistenceEngine` performance with `KryoSerializer` via introducing a new configuration, `spark.deploy.recoverySerializer`. ### Why are the changes needed? Allow users to choose a better serializer to get a better performance in their environment. Especially, `KryoSerializer` is about **3x faster** than `JavaSerializer` with `FileSystemPersistenceEngine`. ``` PersistenceEngineBenchmark OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure AMD EPYC 7763 64-Core Processor 1000 Workers:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative --- ZooKeeperPersistenceEngine with JavaSerializer1202 1298 138 0.0 1201614.2 1.0X ZooKeeperPersistenceEngine with KryoSerializer 951 1004 48 0.0 950559.0 1.3X FileSystemPersistenceEngine with JavaSerializer212 217 6 0.0 211623.2 5.7X FileSystemPersistenceEngine with KryoSerializer 79 81 2 0.0 79132.5 15.2X BlackHolePersistenceEngine 0 0 0 30.9 32.4 37109.8X ``` ### Does this PR introduce _any_ user-facing change? No. The default behavior is the same. ### How was this patch tested? Pass the CIs with the new added test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44113 from dongjoon-hyun/SPARK-46205. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- .../PersistenceEngineBenchmark-jdk21-results.txt | 12 ++- .../PersistenceEngineBenchmark-results.txt | 12 ++- .../master/FileSystemPersistenceEngine.scala | 2 +- .../org/apache/spark/deploy/master/Master.scala| 7 +-- .../org/apache/spark/internal/config/Deploy.scala | 16 +++ .../apache/spark/deploy/master/MasterSuite.scala | 22 .../deploy/master/PersistenceEngineBenchmark.scala | 24 +++--- .../deploy/master/PersistenceEngineSuite.scala | 14 - 8 files changed, 92 insertions(+), 17 deletions(-) diff --git a/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt b/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt index 3312d6feff88..65dbfd0990d3 100644 --- a/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt +++ b/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt @@ -4,10 +4,12 @@ PersistenceEngineBenchmark OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1051-azure AMD EPYC 7763 64-Core Processor -1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative - -ZooKeeperPersistenceEngine 1183 1266 129 0.0 1183158.2 1.0X -FileSystemPersistenceEngine 218222 4 0.0 218005.2 5.4X -BlackHolePersistenceEngine0 0 0 29.5 34.0 34846.9X +1000 Workers:Best Time(ms) Avg Time(ms) Stdev(ms)Rate(M/s) Per Row(ns) Relative +--- +ZooKeeperPersistenceEngine with JavaSerializer1100 1255 150 0.0 1099532.9 1.0X +ZooKeeperPersistenceEngine with KryoSerializer 946967 20 0.0 946367.3 1.2X +FileSy
