[spark] branch master updated: [SPARK-45546][BUILD][INFRA] Make `publish-snapshot` support `package` first then `deploy`

[yangjie01]

This is an automated email from the ASF dual-hosted git repository.

yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3ef18e2d00f [SPARK-45546][BUILD][INFRA] Make `publish-snapshot` 
support `package` first then `deploy`
3ef18e2d00f is described below

commit 3ef18e2d00f386196292f0c768816626bc903d47
Author: yangjie01 <yangji...@baidu.com>
AuthorDate: Wed Oct 18 10:15:27 2023 +0800

    [SPARK-45546][BUILD][INFRA] Make `publish-snapshot` support `package` first 
then `deploy`
    
    ### What changes were proposed in this pull request?
    This pr adds an environment variable `PACKAGE_BEFORE_DEPLOY` to the 
`publish-snapshot` process. When `PACKAGE_BEFORE_DEPLOY` is true, the publish 
process will be split into two steps: the first step is to package with mvn 
package, and the second step is to deploy the packaged jar.
    
    At the same time, this PR sets `PACKAGE_BEFORE_DEPLOY` to true in the 
`publish_snapshot.yml` configuration.
    
    ### Why are the changes needed?
    Make the `publish-snapshot` task in GitHub Action to be divided into two 
steps, which can alleviate the resource pressure brought by direct deploy.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Pass GitHub Actions
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #43378 from LuciferYang/no-doc-deploy.
    
    Lead-authored-by: yangjie01 <yangji...@baidu.com>
    Co-authored-by: YangJie <yangji...@baidu.com>
    Signed-off-by: yangjie01 <yangji...@baidu.com>
---
 .github/workflows/publish_snapshot.yml |  4 ++++
 dev/create-release/release-build.sh    | 14 ++++++++++++--
 pom.xml                                |  7 +++++++
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/publish_snapshot.yml 
b/.github/workflows/publish_snapshot.yml
index 7ed836f016b..476d41d0cf1 100644
--- a/.github/workflows/publish_snapshot.yml
+++ b/.github/workflows/publish_snapshot.yml
@@ -66,4 +66,8 @@ jobs:
         GPG_KEY: "not_used"
         GPG_PASSPHRASE: "not_used"
         GIT_REF: ${{ matrix.branch }}
+        # SPARK-45546 adds this environment variable to split the publish 
snapshot process into two steps:
+        # first package, then deploy. This is intended to reduce the resource 
pressure of deploy.
+        # When PACKAGE_BEFORE_DEPLOY is not set to true, it will revert to the 
one-step deploy method.
+        PACKAGE_BEFORE_DEPLOY: true
       run: ./dev/create-release/release-build.sh publish-snapshot
diff --git a/dev/create-release/release-build.sh 
b/dev/create-release/release-build.sh
index f3571c4e48c..3776c64e31e 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -432,14 +432,24 @@ if [[ "$1" == "publish-snapshot" ]]; then
   echo "</server></servers></settings>" >> $tmp_settings
 
   if [[ $PUBLISH_SCALA_2_12 = 1 ]]; then
-    $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES 
$PUBLISH_PROFILES clean deploy
+    if [ "$PACKAGE_BEFORE_DEPLOY" = "true" ]; then
+      $MVN -DskipTests $SCALA_2_12_PROFILES $PUBLISH_PROFILES clean package
+      $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES 
$PUBLISH_PROFILES deploy
+    else
+      $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES 
$PUBLISH_PROFILES clean deploy
+    fi
   fi
 
   if [[ $PUBLISH_SCALA_2_13 = 1 ]]; then
     if [[ $SPARK_VERSION < "4.0" ]]; then
       ./dev/change-scala-version.sh 2.13
     fi
-    $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES 
$PUBLISH_PROFILES clean deploy
+    if [ "$PACKAGE_BEFORE_DEPLOY" = "true" ]; then
+      $MVN -DskipTests $SCALA_2_13_PROFILES $PUBLISH_PROFILES clean package
+      $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES 
$PUBLISH_PROFILES deploy
+    else
+      $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES 
$PUBLISH_PROFILES clean deploy
+    fi
   fi
 
   rm $tmp_settings
diff --git a/pom.xml b/pom.xml
index 824ae49f6da..ade6537c2a1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3840,4 +3840,11 @@
       </build>
     </profile>
   </profiles>
+  <distributionManagement>
+    <snapshotRepository>
+      <id>internal.snapshot</id>
+      <name>Internal Snapshot Repository</name>
+      <url>http://localhost:8081/repository/maven-snapshots/</url>
+    </snapshotRepository>
+  </distributionManagement>
 </project>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org