[ https://issues.apache.org/jira/browse/SPARK-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660374#comment-14660374 ]
Ryan Williams commented on SPARK-1517: -------------------------------------- Hey [~pwendell], thanks for continuing to push on this. A workflow I'd like to see supported (and maybe it already is; please let me know if so) is to more easily fetch these artifacts (both [Maven snapshots|https://repository.apache.org/content/repositories/snapshots/org/apache/spark/] and [bundled release {{.tgz}} files|https://people.apache.org/~pwendell/spark-nightly/]) by their git SHAs. For the Maven snapshots, I'd like to be able to just change the Spark version in a downstream project's POM to a git SHA and have Maven fetch the Spark JARs for that SHA (assuming it's one that has been built by the tools here); I'm fine with the (presumably necessary) step on my end of adding a Maven repository to make this work, either per-project or globally. Today, the Maven snapshots at e.g. https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-core_2.10/1.5.0-SNAPSHOT/ all seem to be uniquely ID'd by timestamps that I don't know how to get useful information out of, which has precluded my using them. On the bundled releases front, I see that the git SHA is being added to the folders at https://people.apache.org/~pwendell/spark-nightly/spark-master-bin/: !http://cl.ly/image/0o111a1o0U2N/Screen%20Shot%202015-08-06%20at%201.08.18%20PM.png! but those don't seem to stick around more than a day or so? Additionally, as that screenshot shows, there are 3 copies of one SHA there right now, and only 2 SHAs total. I rolled some of my own scripts for cloning, building, and selecting specific Spark versions locally at [ryan-williams/spark-helpers|https://github.com/ryan-williams/spark-helpers], which currently fetches release {{.tgz}} files for released Spark versions, but for arbitrary Spark SHAs there doesn't seem to be an easy way to download a pre-built Spark, so I am just cloning them and running {{mvn package}}. Let me know if you have thoughts about exposing built artifacts for more SHAs, the workflows I've described here, etc. Thanks again! > Publish nightly snapshots of documentation, maven artifacts, and binary builds > ------------------------------------------------------------------------------ > > Key: SPARK-1517 > URL: https://issues.apache.org/jira/browse/SPARK-1517 > Project: Spark > Issue Type: Improvement > Components: Build, Project Infra > Reporter: Patrick Wendell > Assignee: Patrick Wendell > Priority: Critical > > Should be pretty easy to do with Jenkins. The only thing I can think of that > would be tricky is to set up credentials so that jenkins can publish this > stuff somewhere on apache infra. > Ideally we don't want to have to put a private key on every jenkins box > (since they are otherwise pretty stateless). One idea is to encrypt these > credentials with a passphrase and post them somewhere publicly visible. Then > the jenkins build can download the credentials provided we set a passphrase > in an environment variable in jenkins. There may be simpler solutions as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org