This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d0b0b10d2eed [SPARK-52241][BUILD] Upload to repository in parallel in
release script
d0b0b10d2eed is described below
commit d0b0b10d2eed2214ca348a68cf4af30d148a8ba3
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed May 21 14:23:58 2025 +0900
[SPARK-52241][BUILD] Upload to repository in parallel in release script
### What changes were proposed in this pull request?
This PR proposes to execute `curl` in parallel when uploading the artifacts
to repository.
### Why are the changes needed?
When the network isn't that great, it takes too long.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50965 from HyukjinKwon/SPARK-52241.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/create-release/release-build.sh | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index d2a7e2b845f6..9fae3d58bcd6 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -563,14 +563,18 @@ if [[ "$1" == "publish-release" ]]; then
if ! is_dry_run; then
nexus_upload=$NEXUS_ROOT/deployByRepositoryId/$staged_repo_id
echo "Uploading files to $nexus_upload"
- for file in $(find . -type f)
- do
- # strip leading ./
- file_short=$(echo $file | sed -e "s/\.\///")
- dest_url="$nexus_upload/org/apache/spark/$file_short"
- echo " Uploading $file_short"
- curl --retry 3 --retry-all-errors -u $ASF_USERNAME:$ASF_PASSWORD
--upload-file $file_short $dest_url
- done
+ find . -type f | sed -e 's|^\./||' | \
+ xargs -P 8 -n 1 -I {} bash -c '
+ file_short="{}"
+
dest_url="'$NEXUS_ROOT'/deployByRepositoryId/'$staged_repo_id'/org/apache/spark/$file_short"
+ echo "[START] $file_short"
+ if curl --retry 3 --retry-all-errors -sS -u
"$ASF_USERNAME:$ASF_PASSWORD" \
+ --upload-file "$file_short" "$dest_url"; then
+ echo "[ OK ] $file_short"
+ else
+ echo "[FAIL ] $file_short"
+ fi
+ '
echo "Closing nexus staging repository"
repo_request="<promoteRequest><data><stagedRepositoryId>$staged_repo_id</stagedRepositoryId><description>Apache
Spark $SPARK_VERSION (commit $git_hash)</description></data></promoteRequest>"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
