And I think 195 is sufficient to build something that works; but I haven't personally tested it.
On Sun, Dec 8, 2013 at 8:41 PM, Mark Hamstra <m...@clearstorydata.com>wrote: > Well, what I've already done for ClearStory is very close to how Debian > packaging should be done for Apache Spark. That much can be put into a > pull request quickly. The only real issues are exactly how the packages > should be named, checking that the metadata of the packages are exactly > correct for an Apache release, and deciding whether we should be producing > a spark-examples package, spark-tools package, separate the Java and Python > APIs into their own packages, create a source package, etc. What we've > been doing up to now is essentially just the minimal packaging of a fat jar > that can be deployed by Chef or something similar. It's never really been > put together in a way appropriate to go into a Debian or Ubuntu > distribution, for instance. > > > On Sun, Dec 8, 2013 at 8:25 PM, Patrick Wendell <pwend...@gmail.com>wrote: > >> Hey Mark, >> >> Okay if 195 gets this in working order in the branch 0.8 let's just >> merge that to keep it consistent with our docs and the way this is >> done in 0.8.0 >> >> We can do a broader refactoring in 0.9. Would be great if you could >> kick off a JIRA discussion or submit a PR relating to that. >> >> - Patrick >> >> On Sun, Dec 8, 2013 at 8:07 PM, Mark Hamstra <m...@clearstorydata.com> >> wrote: >> > Well, 195 is sufficient to give you something that runs, but it doesn't >> run >> > the same way as Spark built/distributed by other means -- e.g., after >> 195 >> > the package still uses something equivalent to the old `run` script >> instead >> > of the current `spark-class` way. >> > >> > >> > On Sun, Dec 8, 2013 at 8:02 PM, Patrick Wendell <pwend...@gmail.com> >> wrote: >> > >> >> Hey Mark, >> >> >> >> What I'm asking is whether this patch is sufficient to have a working >> >> debian build in 0.8.1, or are there other outstanding issues to make >> >> it work? By working I mean, within the initial design that was >> >> contributed (with repl-bin) it works according to that approach. >> >> >> >> We can redesign this packaging in 0.9. That will require having a PR >> >> against Apache Spark, discussing, etc. But it doesn't need to be on >> >> the critical path for this release. >> >> >> >> - Patrick >> >> >> >> On Sun, Dec 8, 2013 at 7:54 PM, Mark Hamstra <m...@clearstorydata.com> >> >> wrote: >> >> > Whatever Debian package gets built has to work, so that's the first >> >> > requirement. I don't know how to decide whether a change is >> acceptable >> >> in >> >> > 0.8 or has to wait until 0.9, but the 0.9 packaging should definitely >> >> > leverage the assembly sub-project, making repl-bin unnecessary. >> >> > >> >> > >> >> > On Sun, Dec 8, 2013 at 7:46 PM, Patrick Wendell <pwend...@gmail.com> >> >> wrote: >> >> > >> >> >> Looked into this a bit more - I think removing repl-bin is something >> >> >> we should wait until 0.9 to do, because we've published it to maven >> in >> >> >> 0.8.0 and people might expect it to be there in 0.8.1. >> >> >> >> >> >> Merging the directly referenced pull request (195) seems like a good >> >> >> idea though since it fixes a bug in the script. >> >> >> >> >> >> Is that what you are suggesting? >> >> >> >> >> >> - Patrick >> >> >> >> >> >> On Sun, Dec 8, 2013 at 7:30 PM, Patrick Wendell <pwend...@gmail.com >> > >> >> >> wrote: >> >> >> > Hey Mark - ya this would be good to get in. >> >> >> > >> >> >> > Does merging that particular PR put this in sufficient shape for >> the >> >> >> > 0.8.1 release or are there other open patches we need to look at? >> >> >> > >> >> >> > - Patrick >> >> >> > >> >> >> > On Sun, Dec 8, 2013 at 6:05 PM, Mark Hamstra < >> m...@clearstorydata.com >> >> > >> >> >> wrote: >> >> >> >> SPARK-962 should be resolved before release. See also: >> >> >> >> https://github.com/apache/incubator-spark/pull/195 >> >> >> >> >> >> >> >> With the references to the way I changed Debian packaging for >> >> >> ClearStory, >> >> >> >> we should be at least 90% of the way toward doing it right for >> >> Apache. >> >> >> >> >> >> >> >> >> >> >> >> On Sun, Dec 8, 2013 at 5:29 PM, Patrick Wendell < >> pwend...@gmail.com> >> >> >> wrote: >> >> >> >> >> >> >> >>> For my own part I'll give a +1 to this RC. >> >> >> >>> >> >> >> >>> On Sun, Dec 8, 2013 at 4:30 PM, Taka Shinagawa < >> >> taka.epsi...@gmail.com >> >> >> > >> >> >> >>> wrote: >> >> >> >>> > OK. I will post the entire output via separate email. I just >> >> upgraded >> >> >> >>> > Hadoop to 2.2.0 recently. So there might be something I need >> to >> >> >> >>> > remove/clean up. >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > On Sun, Dec 8, 2013 at 4:24 PM, Patrick Wendell < >> >> pwend...@gmail.com> >> >> >> >>> wrote: >> >> >> >>> > >> >> >> >>> >> Hey Take, >> >> >> >>> >> >> >> >> >>> >> Could you start a separate thread to debug your build issue? >> In >> >> that >> >> >> >>> >> thread, could you paste the exact build command and entire >> >> output? >> >> >> The >> >> >> >>> log >> >> >> >>> >> you posted here suggests the first build detected hadoop >> 1.0.4 >> >> not >> >> >> 2.2.0 >> >> >> >>> >> based on the assembly file name it is logging. >> >> >> >>> >> >> >> >> >>> >> --- >> >> >> >>> >> sent from my phone >> >> >> >>> >> On Dec 8, 2013 4:13 PM, "Taka Shinagawa" < >> taka.epsi...@gmail.com >> >> > >> >> >> >>> wrote: >> >> >> >>> >> >> >> >> >>> >> > With Hadoop 2.2.0 (& Java 1.7.0_45) installed, I'm having >> >> trouble >> >> >> >>> >> > completing the build process (sbt/sbt assembly) on >> Macbook. The >> >> >> sbt >> >> >> >>> >> command >> >> >> >>> >> > hangs at the last step. >> >> >> >>> >> > >> >> >> >>> >> > ... >> >> >> >>> >> > ... >> >> >> >>> >> > [info] SHA-1: ce8275f5841002164c4305c912a2892ec7c1d395 >> >> >> >>> >> > [info] Packaging >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> >> >> >> >>> >> >> >> >> >> >> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/tools/target/scala-2.9.3/spark-tools-assembly-0.8.1-incubating.jar >> >> >> >>> >> > ... >> >> >> >>> >> > [info] SHA-1: 0657a347240266230247693f265a5797d40c326a >> >> >> >>> >> > [info] Packaging >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> >> >> >> >>> >> >> >> >> >> >> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop1.0.4.jar >> >> >> >>> >> > ... >> >> >> >>> >> > (hangs here) >> >> >> >>> >> > -------------------------- >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> > On another Macbook with Hadoop 1.1.1 (& Java 1.7.0_45) >> >> installed, >> >> >> I >> >> >> >>> was >> >> >> >>> >> > able to build it successfully. >> >> >> >>> >> > .. >> >> >> >>> >> > .. >> >> >> >>> >> > [info] SHA-1: 77109cd085bd4f0d2b601b3451b35b961d357534 >> >> >> >>> >> > [info] Packaging >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> >> >> >> >>> >> >> >> >> >> >> /Users/tshinagawa/Documents/Spark/RCs/spark-0.8.1-incubating/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar >> >> >> >>> >> > ... >> >> >> >>> >> > [info] Done packaging. >> >> >> >>> >> > [success] Total time: 266 s, completed Dec 8, 2013 3:03:10 >> PM >> >> >> >>> >> > -------------------------- >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> > On Sun, Dec 8, 2013 at 12:41 PM, Patrick Wendell < >> >> >> pwend...@gmail.com> >> >> >> >>> >> > wrote: >> >> >> >>> >> > >> >> >> >>> >> > > Please vote on releasing the following candidate as >> Apache >> >> Spark >> >> >> >>> >> > > (incubating) version 0.8.1. >> >> >> >>> >> > > >> >> >> >>> >> > > The tag to be voted on is v0.8.1-incubating (commit >> >> bf23794a): >> >> >> >>> >> > > >> >> >> >>> >> > > >> >> >> >>> >> > >> >> >> >>> >> >> >> >> >>> >> >> >> >> >> >> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=tag;h=e6ba91b5a7527316202797fc3dce469ff86cf203 >> >> >> >>> >> > > >> >> >> >>> >> > > The release files, including signatures, digests, etc >> can be >> >> >> found >> >> >> >>> at: >> >> >> >>> >> > > >> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2/ >> >> >> >>> >> > > >> >> >> >>> >> > > Release artifacts are signed with the following key: >> >> >> >>> >> > > https://people.apache.org/keys/committer/pwendell.asc >> >> >> >>> >> > > >> >> >> >>> >> > > The staging repository for this release can be found at: >> >> >> >>> >> > > >> >> >> >>> >> >> https://repository.apache.org/content/repositories/orgapachespark-024/ >> >> >> >>> >> > > >> >> >> >>> >> > > The documentation corresponding to this release can be >> found >> >> at: >> >> >> >>> >> > > >> >> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2-docs/ >> >> >> >>> >> > > >> >> >> >>> >> > > For information about the contents of this release see: >> >> >> >>> >> > > <attached> draft of release notes >> >> >> >>> >> > > <attached> draft of release credits >> >> >> >>> >> > > >> >> >> >>> >> >> https://github.com/apache/incubator-spark/blob/branch-0.8/CHANGES.txt >> >> >> >>> >> > > >> >> >> >>> >> > > Please vote on releasing this package as Apache Spark >> >> >> >>> 0.8.1-incubating! >> >> >> >>> >> > > >> >> >> >>> >> > > The vote is open until Wednesday, December 11th at 21:00 >> UTC >> >> and >> >> >> >>> >> > > passes if a majority of at least 3 +1 PPMC votes are >> cast. >> >> >> >>> >> > > >> >> >> >>> >> > > [ ] +1 Release this package as Apache Spark >> 0.8.1-incubating >> >> >> >>> >> > > [ ] -1 Do not release this package because ... >> >> >> >>> >> > > >> >> >> >>> >> > > To learn more about Apache Spark, please see >> >> >> >>> >> > > http://spark.incubator.apache.org/ >> >> >> >>> >> > > >> >> >> >>> >> > >> >> >> >>> >> >> >> >> >>> >> >> >> >> >> >> > >
