This is an automated email from the ASF dual-hosted git repository. yangjie01 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 3ef18e2d00f [SPARK-45546][BUILD][INFRA] Make `publish-snapshot` support `package` first then `deploy` 3ef18e2d00f is described below commit 3ef18e2d00f386196292f0c768816626bc903d47 Author: yangjie01 <yangji...@baidu.com> AuthorDate: Wed Oct 18 10:15:27 2023 +0800 [SPARK-45546][BUILD][INFRA] Make `publish-snapshot` support `package` first then `deploy` ### What changes were proposed in this pull request? This pr adds an environment variable `PACKAGE_BEFORE_DEPLOY` to the `publish-snapshot` process. When `PACKAGE_BEFORE_DEPLOY` is true, the publish process will be split into two steps: the first step is to package with mvn package, and the second step is to deploy the packaged jar. At the same time, this PR sets `PACKAGE_BEFORE_DEPLOY` to true in the `publish_snapshot.yml` configuration. ### Why are the changes needed? Make the `publish-snapshot` task in GitHub Action to be divided into two steps, which can alleviate the resource pressure brought by direct deploy. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #43378 from LuciferYang/no-doc-deploy. Lead-authored-by: yangjie01 <yangji...@baidu.com> Co-authored-by: YangJie <yangji...@baidu.com> Signed-off-by: yangjie01 <yangji...@baidu.com> --- .github/workflows/publish_snapshot.yml | 4 ++++ dev/create-release/release-build.sh | 14 ++++++++++++-- pom.xml | 7 +++++++ 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish_snapshot.yml b/.github/workflows/publish_snapshot.yml index 7ed836f016b..476d41d0cf1 100644 --- a/.github/workflows/publish_snapshot.yml +++ b/.github/workflows/publish_snapshot.yml @@ -66,4 +66,8 @@ jobs: GPG_KEY: "not_used" GPG_PASSPHRASE: "not_used" GIT_REF: ${{ matrix.branch }} + # SPARK-45546 adds this environment variable to split the publish snapshot process into two steps: + # first package, then deploy. This is intended to reduce the resource pressure of deploy. + # When PACKAGE_BEFORE_DEPLOY is not set to true, it will revert to the one-step deploy method. + PACKAGE_BEFORE_DEPLOY: true run: ./dev/create-release/release-build.sh publish-snapshot diff --git a/dev/create-release/release-build.sh b/dev/create-release/release-build.sh index f3571c4e48c..3776c64e31e 100755 --- a/dev/create-release/release-build.sh +++ b/dev/create-release/release-build.sh @@ -432,14 +432,24 @@ if [[ "$1" == "publish-snapshot" ]]; then echo "</server></servers></settings>" >> $tmp_settings if [[ $PUBLISH_SCALA_2_12 = 1 ]]; then - $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES $PUBLISH_PROFILES clean deploy + if [ "$PACKAGE_BEFORE_DEPLOY" = "true" ]; then + $MVN -DskipTests $SCALA_2_12_PROFILES $PUBLISH_PROFILES clean package + $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES $PUBLISH_PROFILES deploy + else + $MVN --settings $tmp_settings -DskipTests $SCALA_2_12_PROFILES $PUBLISH_PROFILES clean deploy + fi fi if [[ $PUBLISH_SCALA_2_13 = 1 ]]; then if [[ $SPARK_VERSION < "4.0" ]]; then ./dev/change-scala-version.sh 2.13 fi - $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES $PUBLISH_PROFILES clean deploy + if [ "$PACKAGE_BEFORE_DEPLOY" = "true" ]; then + $MVN -DskipTests $SCALA_2_13_PROFILES $PUBLISH_PROFILES clean package + $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES $PUBLISH_PROFILES deploy + else + $MVN --settings $tmp_settings -DskipTests $SCALA_2_13_PROFILES $PUBLISH_PROFILES clean deploy + fi fi rm $tmp_settings diff --git a/pom.xml b/pom.xml index 824ae49f6da..ade6537c2a1 100644 --- a/pom.xml +++ b/pom.xml @@ -3840,4 +3840,11 @@ </build> </profile> </profiles> + <distributionManagement> + <snapshotRepository> + <id>internal.snapshot</id> + <name>Internal Snapshot Repository</name> + <url>http://localhost:8081/repository/maven-snapshots/</url> + </snapshotRepository> + </distributionManagement> </project> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org