This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 335599e4b5 [spark] Publish paimon-spark-4.0 and add doc (#5752)
335599e4b5 is described below
commit 335599e4b549b7476db65214b5d3d007a47fc420
Author: Zouxxyy <[email protected]>
AuthorDate: Tue Jun 17 15:14:47 2025 +0800
[spark] Publish paimon-spark-4.0 and add doc (#5752)
---
.github/workflows/publish_snapshot-jdk17.yml | 68 ++++++++++++++++++++++++++++
docs/content/ecosystem/overview.md | 2 +-
docs/content/spark/quick-start.md | 6 ++-
3 files changed, 74 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/publish_snapshot-jdk17.yml
b/.github/workflows/publish_snapshot-jdk17.yml
new file mode 100644
index 0000000000..54660cfa3c
--- /dev/null
+++ b/.github/workflows/publish_snapshot-jdk17.yml
@@ -0,0 +1,68 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: Publish Snapshot on JDK 17
+
+on:
+ schedule:
+ # At the end of every day
+ - cron: '0 0 * * *'
+ workflow_dispatch:
+
+env:
+ JDK_VERSION: 17
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ publish-snapshot:
+ if: github.repository == 'apache/paimon'
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v4
+ - name: Set up JDK ${{ env.JDK_VERSION }}
+ uses: actions/setup-java@v4
+ with:
+ java-version: ${{ env.JDK_VERSION }}
+ distribution: 'temurin'
+ - name: Cache local Maven repository
+ uses: actions/cache@v4
+ with:
+ path: ~/.m2/repository
+ key: snapshot-maven-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ snapshot-maven-
+ - name: Publish snapshot
+ env:
+ ASF_USERNAME: ${{ secrets.NEXUS_USER }}
+ ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
+ MAVEN_OPTS: -Xmx4096m
+ run: |
+ tmp_settings="tmp-settings.xml"
+ echo "<settings><servers><server>" > $tmp_settings
+ echo
"<id>apache.snapshots.https</id><username>$ASF_USERNAME</username>" >>
$tmp_settings
+ echo "<password>$ASF_PASSWORD</password>" >> $tmp_settings
+ echo "</server></servers></settings>" >> $tmp_settings
+
+ mvn --settings $tmp_settings clean install -Dgpg.skip -Drat.skip
-DskipTests -Papache-release,spark4,flink1 -pl
org.apache.paimon:paimon-spark-4.0 -am
+ mvn --settings $tmp_settings clean deploy -Dgpg.skip -Drat.skip
-DskipTests -Papache-release,spark4,flink1 -pl
org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common,org.apache.paimon:paimon-spark-4.0
+
+ rm $tmp_settings
diff --git a/docs/content/ecosystem/overview.md
b/docs/content/ecosystem/overview.md
index 309628c814..7c10207d21 100644
--- a/docs/content/ecosystem/overview.md
+++ b/docs/content/ecosystem/overview.md
@@ -31,7 +31,7 @@ under the License.
| Engine
| Version | Batch Read | Batch Write | Create Table | Alter Table
| Streaming Write | Streaming Read | Batch Overwrite | DELETE & UPDATE |
MERGE INTO | Time Travel |
|:-------------------------------------------------------------------------------:|:-------------:|:-----------:|:-----------:|:-------------:|:-------------:|:----------------:|:----------------:|:---------------:|:---------------:|:----------:|:-----------:|
| Flink
| 1.15 - 1.20 | ✅ | ✅ | ✅ | ✅(1.17+) |
✅ | ✅ | ✅ | ✅(1.17+) | ❌
| ✅ |
-| Spark
| 3.2 - 3.5 | ✅ | ✅ | ✅ | ✅ |
✅(3.3+) | ✅(3.3+) | ✅ | ✅ | ✅
| ✅(3.3+) |
+| Spark
| 3.2 - 4.0 | ✅ | ✅ | ✅ | ✅ |
✅(3.3+) | ✅(3.3+) | ✅ | ✅ | ✅
| ✅(3.3+) |
| Hive
| 2.1 - 3.1 | ✅ | ✅ | ✅ | ❌ |
❌ | ❌ | ❌ | ❌ | ❌
| ✅ |
| Trino
| 420 - 440 | ✅ | ✅(427+) | ✅(427+) | ✅(427+) |
❌ | ❌ | ❌ | ❌ | ❌
| ✅ |
| Presto
| 0.236 - 0.280 | ✅ | ❌ | ✅ | ✅ |
❌ | ❌ | ❌ | ❌ | ❌
| ❌ |
diff --git a/docs/content/spark/quick-start.md
b/docs/content/spark/quick-start.md
index dea69636f5..6764797f9f 100644
--- a/docs/content/spark/quick-start.md
+++ b/docs/content/spark/quick-start.md
@@ -28,7 +28,9 @@ under the License.
## Preparation
-Paimon currently supports Spark 3.5, 3.4, 3.3, and 3.2. We recommend the
latest Spark version for a better experience.
+Paimon currently supports Spark versions 4.x (including 4.0) and 3.x
(including 3.5, 3.4, 3.3, and 3.2). We recommend using the latest Spark version
for a better experience.
+
+Note that Spark 4.x is pre-built with Java 17 and Scala 2.13. Spark 3.x is
pre-built with Java 8 and Scala 2.12.
Download the jar file with corresponding version.
@@ -36,6 +38,7 @@ Download the jar file with corresponding version.
| Version | Jar
|
|-----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Spark 4.0 | [paimon-spark-4.0-{{< version
>}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.0/{{<
version >}}/paimon-spark-4.0-{{< version >}}.jar) |
| Spark 3.5 | [paimon-spark-3.5-{{< version
>}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5/{{<
version >}}/paimon-spark-3.5-{{< version >}}.jar) |
| Spark 3.4 | [paimon-spark-3.4-{{< version
>}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.4/{{<
version >}}/paimon-spark-3.4-{{< version >}}.jar) |
| Spark 3.3 | [paimon-spark-3.3-{{< version
>}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.3/{{<
version >}}/paimon-spark-3.3-{{< version >}}.jar) |
@@ -47,6 +50,7 @@ Download the jar file with corresponding version.
| Version | Jar
|
|-----------|-------------------------------------------------------------------------------------------------------------------------------------|
+| Spark 4.0 | [paimon-spark-4.0-{{< version
>}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.0/{{<
version >}}/) |
| Spark 3.5 | [paimon-spark-3.5-{{< version
>}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5/{{<
version >}}/) |
| Spark 3.4 | [paimon-spark-3.4-{{< version
>}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.4/{{<
version >}}/) |
| Spark 3.3 | [paimon-spark-3.3-{{< version
>}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.3/{{<
version >}}/) |
