Hey Mark, Okay if 195 gets this in working order in the branch 0.8 let's just merge that to keep it consistent with our docs and the way this is done in 0.8.0
We can do a broader refactoring in 0.9. Would be great if you could kick off a JIRA discussion or submit a PR relating to that. - Patrick On Sun, Dec 8, 2013 at 8:07 PM, Mark Hamstra <m...@clearstorydata.com> wrote: > Well, 195 is sufficient to give you something that runs, but it doesn't run > the same way as Spark built/distributed by other means -- e.g., after 195 > the package still uses something equivalent to the old `run` script instead > of the current `spark-class` way. > > > On Sun, Dec 8, 2013 at 8:02 PM, Patrick Wendell <pwend...@gmail.com> wrote: > >> Hey Mark, >> >> What I'm asking is whether this patch is sufficient to have a working >> debian build in 0.8.1, or are there other outstanding issues to make >> it work? By working I mean, within the initial design that was >> contributed (with repl-bin) it works according to that approach. >> >> We can redesign this packaging in 0.9. That will require having a PR >> against Apache Spark, discussing, etc. But it doesn't need to be on >> the critical path for this release. >> >> - Patrick >> >> On Sun, Dec 8, 2013 at 7:54 PM, Mark Hamstra <m...@clearstorydata.com> >> wrote: >> > Whatever Debian package gets built has to work, so that's the first >> > requirement. I don't know how to decide whether a change is acceptable >> in >> > 0.8 or has to wait until 0.9, but the 0.9 packaging should definitely >> > leverage the assembly sub-project, making repl-bin unnecessary. >> > >> > >> > On Sun, Dec 8, 2013 at 7:46 PM, Patrick Wendell <pwend...@gmail.com> >> wrote: >> > >> >> Looked into this a bit more - I think removing repl-bin is something >> >> we should wait until 0.9 to do, because we've published it to maven in >> >> 0.8.0 and people might expect it to be there in 0.8.1. >> >> >> >> Merging the directly referenced pull request (195) seems like a good >> >> idea though since it fixes a bug in the script. >> >> >> >> Is that what you are suggesting? >> >> >> >> - Patrick >> >> >> >> On Sun, Dec 8, 2013 at 7:30 PM, Patrick Wendell <pwend...@gmail.com> >> >> wrote: >> >> > Hey Mark - ya this would be good to get in. >> >> > >> >> > Does merging that particular PR put this in sufficient shape for the >> >> > 0.8.1 release or are there other open patches we need to look at? >> >> > >> >> > - Patrick >> >> > >> >> > On Sun, Dec 8, 2013 at 6:05 PM, Mark Hamstra <m...@clearstorydata.com >> > >> >> wrote: >> >> >> SPARK-962 should be resolved before release. See also: >> >> >> https://github.com/apache/incubator-spark/pull/195 >> >> >> >> >> >> With the references to the way I changed Debian packaging for >> >> ClearStory, >> >> >> we should be at least 90% of the way toward doing it right for >> Apache. >> >> >> >> >> >> >> >> >> On Sun, Dec 8, 2013 at 5:29 PM, Patrick Wendell <pwend...@gmail.com> >> >> wrote: >> >> >> >> >> >>> For my own part I'll give a +1 to this RC. >> >> >>> >> >> >>> On Sun, Dec 8, 2013 at 4:30 PM, Taka Shinagawa < >> taka.epsi...@gmail.com >> >> > >> >> >>> wrote: >> >> >>> > OK. I will post the entire output via separate email. I just >> upgraded >> >> >>> > Hadoop to 2.2.0 recently. So there might be something I need to >> >> >>> > remove/clean up. >> >> >>> > >> >> >>> > >> >> >>> > On Sun, Dec 8, 2013 at 4:24 PM, Patrick Wendell < >> pwend...@gmail.com> >> >> >>> wrote: >> >> >>> > >> >> >>> >> Hey Take, >> >> >>> >> >> >> >>> >> Could you start a separate thread to debug your build issue? In >> that >> >> >>> >> thread, could you paste the exact build command and entire >> output? >> >> The >> >> >>> log >> >> >>> >> you posted here suggests the first build detected hadoop 1.0.4 >> not >> >> 2.2.0 >> >> >>> >> based on the assembly file name it is logging. >> >> >>> >> >> >> >>> >> --- >> >> >>> >> sent from my phone >> >> >>> >> On Dec 8, 2013 4:13 PM, "Taka Shinagawa" <taka.epsi...@gmail.com >> > >> >> >>> wrote: >> >> >>> >> >> >> >>> >> > With Hadoop 2.2.0 (& Java 1.7.0_45) installed, I'm having >> trouble >> >> >>> >> > completing the build process (sbt/sbt assembly) on Macbook. The >> >> sbt >> >> >>> >> command >> >> >>> >> > hangs at the last step. >> >> >>> >> > >> >> >>> >> > ... >> >> >>> >> > ... >> >> >>> >> > [info] SHA-1: ce8275f5841002164c4305c912a2892ec7c1d395 >> >> >>> >> > [info] Packaging >> >> >>> >> > >> >> >>> >> > >> >> >>> >> >> >> >>> >> >> >> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/tools/target/scala-2.9.3/spark-tools-assembly-0.8.1-incubating.jar >> >> >>> >> > ... >> >> >>> >> > [info] SHA-1: 0657a347240266230247693f265a5797d40c326a >> >> >>> >> > [info] Packaging >> >> >>> >> > >> >> >>> >> > >> >> >>> >> >> >> >>> >> >> >> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop1.0.4.jar >> >> >>> >> > ... >> >> >>> >> > (hangs here) >> >> >>> >> > -------------------------- >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > On another Macbook with Hadoop 1.1.1 (& Java 1.7.0_45) >> installed, >> >> I >> >> >>> was >> >> >>> >> > able to build it successfully. >> >> >>> >> > .. >> >> >>> >> > .. >> >> >>> >> > [info] SHA-1: 77109cd085bd4f0d2b601b3451b35b961d357534 >> >> >>> >> > [info] Packaging >> >> >>> >> > >> >> >>> >> > >> >> >>> >> >> >> >>> >> >> >> /Users/tshinagawa/Documents/Spark/RCs/spark-0.8.1-incubating/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar >> >> >>> >> > ... >> >> >>> >> > [info] Done packaging. >> >> >>> >> > [success] Total time: 266 s, completed Dec 8, 2013 3:03:10 PM >> >> >>> >> > -------------------------- >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > On Sun, Dec 8, 2013 at 12:41 PM, Patrick Wendell < >> >> pwend...@gmail.com> >> >> >>> >> > wrote: >> >> >>> >> > >> >> >>> >> > > Please vote on releasing the following candidate as Apache >> Spark >> >> >>> >> > > (incubating) version 0.8.1. >> >> >>> >> > > >> >> >>> >> > > The tag to be voted on is v0.8.1-incubating (commit >> bf23794a): >> >> >>> >> > > >> >> >>> >> > > >> >> >>> >> > >> >> >>> >> >> >> >>> >> >> >> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=tag;h=e6ba91b5a7527316202797fc3dce469ff86cf203 >> >> >>> >> > > >> >> >>> >> > > The release files, including signatures, digests, etc can be >> >> found >> >> >>> at: >> >> >>> >> > > >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2/ >> >> >>> >> > > >> >> >>> >> > > Release artifacts are signed with the following key: >> >> >>> >> > > https://people.apache.org/keys/committer/pwendell.asc >> >> >>> >> > > >> >> >>> >> > > The staging repository for this release can be found at: >> >> >>> >> > > >> >> >>> >> https://repository.apache.org/content/repositories/orgapachespark-024/ >> >> >>> >> > > >> >> >>> >> > > The documentation corresponding to this release can be found >> at: >> >> >>> >> > > >> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2-docs/ >> >> >>> >> > > >> >> >>> >> > > For information about the contents of this release see: >> >> >>> >> > > <attached> draft of release notes >> >> >>> >> > > <attached> draft of release credits >> >> >>> >> > > >> >> >>> >> https://github.com/apache/incubator-spark/blob/branch-0.8/CHANGES.txt >> >> >>> >> > > >> >> >>> >> > > Please vote on releasing this package as Apache Spark >> >> >>> 0.8.1-incubating! >> >> >>> >> > > >> >> >>> >> > > The vote is open until Wednesday, December 11th at 21:00 UTC >> and >> >> >>> >> > > passes if a majority of at least 3 +1 PPMC votes are cast. >> >> >>> >> > > >> >> >>> >> > > [ ] +1 Release this package as Apache Spark 0.8.1-incubating >> >> >>> >> > > [ ] -1 Do not release this package because ... >> >> >>> >> > > >> >> >>> >> > > To learn more about Apache Spark, please see >> >> >>> >> > > http://spark.incubator.apache.org/ >> >> >>> >> > > >> >> >>> >> > >> >> >>> >> >> >> >>> >> >> >>
