This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new fa41fe19b5a7 [SPARK-52241][BUILD] Upload to repository in parallel in
release script
fa41fe19b5a7 is described below
commit fa41fe19b5a7e9eea1b987dbacc0f91a9afdaedf
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed May 21 14:23:58 2025 +0900
[SPARK-52241][BUILD] Upload to repository in parallel in release script
This PR proposes to execute `curl` in parallel when uploading the artifacts
to repository.
When the network isn't that great, it takes too long.
No, dev-only.
Manually.
No.
Closes #50965 from HyukjinKwon/SPARK-52241.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit d0b0b10d2eed2214ca348a68cf4af30d148a8ba3)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
dev/create-release/release-build.sh | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index 7728e84bf906..6b25ee8da68d 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -499,14 +499,18 @@ if [[ "$1" == "publish-release" ]]; then
if ! is_dry_run; then
nexus_upload=$NEXUS_ROOT/deployByRepositoryId/$staged_repo_id
echo "Uploading files to $nexus_upload"
- for file in $(find . -type f)
- do
- # strip leading ./
- file_short=$(echo $file | sed -e "s/\.\///")
- dest_url="$nexus_upload/org/apache/spark/$file_short"
- echo " Uploading $file_short"
- curl --retry 3 -u $ASF_USERNAME:$ASF_PASSWORD --upload-file $file_short
$dest_url
- done
+ find . -type f | sed -e 's|^\./||' | \
+ xargs -P 8 -n 1 -I {} bash -c '
+ file_short="{}"
+
dest_url="'$NEXUS_ROOT'/deployByRepositoryId/'$staged_repo_id'/org/apache/spark/$file_short"
+ echo "[START] $file_short"
+ if curl --retry 3 --retry-all-errors -sS -u
"$ASF_USERNAME:$ASF_PASSWORD" \
+ --upload-file "$file_short" "$dest_url"; then
+ echo "[ OK ] $file_short"
+ else
+ echo "[FAIL ] $file_short"
+ fi
+ '
echo "Closing nexus staging repository"
repo_request="<promoteRequest><data><stagedRepositoryId>$staged_repo_id</stagedRepositoryId><description>Apache
Spark $SPARK_VERSION (commit $git_hash)</description></data></promoteRequest>"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
