[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742361#comment-16742361 ]
shane knapp commented on SPARK-26565: ------------------------------------- [~felixcheung] sounds great. my goal here is to re-create the release process in an atomic build that does everything except to sign and push the release artifacts. the big question i have now, especially given how these scripts "interact", is what exactly is produced when a release is run... for example, {{make-distribution.sh}} deletes a bunch of stuff that {{release-build.sh}} expects to be there: {noformat} cp: cannot stat `spark-3.0.0-SNAPSHOT-bin-without-hadoop-scala-2.12/R/SparkR_3.0.0-SNAPSHOT.tar.gz': No such file or directory Skipping signing of R source package cp: cannot stat `spark-3.0.0-SNAPSHOT-bin-without-hadoop-scala-2.12/python/dist/pyspark-3.0.0.dev0.tar.gz': No such file or directory {noformat} this stuff hasn't been audited in probably ~5 years, and i'm finding a lot of cruft and bugs. in all honesty, i'm surprised it all still works. > modify dev/create-release/release-build.sh to let jenkins build packages w/o > publishing > --------------------------------------------------------------------------------------- > > Key: SPARK-26565 > URL: https://issues.apache.org/jira/browse/SPARK-26565 > Project: Spark > Issue Type: Bug > Components: Build > Affects Versions: 2.2.3, 2.3.3, 2.4.1, 3.0.0 > Reporter: shane knapp > Assignee: shane knapp > Priority: Major > Attachments: fine.png, no-idea.jpg > > > about a year+ ago, we stopped publishing releases directly from jenkins... > this means that the spark-\{branch}-packaging builds are failing due to gpg > signing failures, and i would like to update these builds to *just* perform > packaging. > example: > [https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/2183/console] > i propose to change dev/create-release/release-build.sh... > when the script is called w/the 'package' option, add an {{if}} statement to > skip the following sections when run on jenkins: > 1) gpg signing of the source tarball (lines 184-187) > 2) gpg signing of the sparkR dist (lines 243-248) > 3) gpg signing of the python dist (lines 256-261) > 4) gpg signing of the regular binary dist (lines 264-271) > 5) the svn push of the signed dists (lines 317-332) > > -another, and probably much better option, is to nuke the > spark-\{branch}-packaging builds and create new ones that just build things > w/o touching this incredible fragile shell scripting nightmare.- -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org