Okay Mark thanks for bringing this up, I'm going to cut a new RC with this fix.
On Sun, Dec 8, 2013 at 8:42 PM, Mark Hamstra <m...@clearstorydata.com> wrote: > And I think 195 is sufficient to build something that works; but I haven't > personally tested it. > > > On Sun, Dec 8, 2013 at 8:41 PM, Mark Hamstra <m...@clearstorydata.com>wrote: > >> Well, what I've already done for ClearStory is very close to how Debian >> packaging should be done for Apache Spark. That much can be put into a >> pull request quickly. The only real issues are exactly how the packages >> should be named, checking that the metadata of the packages are exactly >> correct for an Apache release, and deciding whether we should be producing >> a spark-examples package, spark-tools package, separate the Java and Python >> APIs into their own packages, create a source package, etc. What we've >> been doing up to now is essentially just the minimal packaging of a fat jar >> that can be deployed by Chef or something similar. It's never really been >> put together in a way appropriate to go into a Debian or Ubuntu >> distribution, for instance. >> >> >> On Sun, Dec 8, 2013 at 8:25 PM, Patrick Wendell <pwend...@gmail.com>wrote: >> >>> Hey Mark, >>> >>> Okay if 195 gets this in working order in the branch 0.8 let's just >>> merge that to keep it consistent with our docs and the way this is >>> done in 0.8.0 >>> >>> We can do a broader refactoring in 0.9. Would be great if you could >>> kick off a JIRA discussion or submit a PR relating to that. >>> >>> - Patrick >>> >>> On Sun, Dec 8, 2013 at 8:07 PM, Mark Hamstra <m...@clearstorydata.com> >>> wrote: >>> > Well, 195 is sufficient to give you something that runs, but it doesn't >>> run >>> > the same way as Spark built/distributed by other means -- e.g., after >>> 195 >>> > the package still uses something equivalent to the old `run` script >>> instead >>> > of the current `spark-class` way. >>> > >>> > >>> > On Sun, Dec 8, 2013 at 8:02 PM, Patrick Wendell <pwend...@gmail.com> >>> wrote: >>> > >>> >> Hey Mark, >>> >> >>> >> What I'm asking is whether this patch is sufficient to have a working >>> >> debian build in 0.8.1, or are there other outstanding issues to make >>> >> it work? By working I mean, within the initial design that was >>> >> contributed (with repl-bin) it works according to that approach. >>> >> >>> >> We can redesign this packaging in 0.9. That will require having a PR >>> >> against Apache Spark, discussing, etc. But it doesn't need to be on >>> >> the critical path for this release. >>> >> >>> >> - Patrick >>> >> >>> >> On Sun, Dec 8, 2013 at 7:54 PM, Mark Hamstra <m...@clearstorydata.com> >>> >> wrote: >>> >> > Whatever Debian package gets built has to work, so that's the first >>> >> > requirement. I don't know how to decide whether a change is >>> acceptable >>> >> in >>> >> > 0.8 or has to wait until 0.9, but the 0.9 packaging should definitely >>> >> > leverage the assembly sub-project, making repl-bin unnecessary. >>> >> > >>> >> > >>> >> > On Sun, Dec 8, 2013 at 7:46 PM, Patrick Wendell <pwend...@gmail.com> >>> >> wrote: >>> >> > >>> >> >> Looked into this a bit more - I think removing repl-bin is something >>> >> >> we should wait until 0.9 to do, because we've published it to maven >>> in >>> >> >> 0.8.0 and people might expect it to be there in 0.8.1. >>> >> >> >>> >> >> Merging the directly referenced pull request (195) seems like a good >>> >> >> idea though since it fixes a bug in the script. >>> >> >> >>> >> >> Is that what you are suggesting? >>> >> >> >>> >> >> - Patrick >>> >> >> >>> >> >> On Sun, Dec 8, 2013 at 7:30 PM, Patrick Wendell <pwend...@gmail.com >>> > >>> >> >> wrote: >>> >> >> > Hey Mark - ya this would be good to get in. >>> >> >> > >>> >> >> > Does merging that particular PR put this in sufficient shape for >>> the >>> >> >> > 0.8.1 release or are there other open patches we need to look at? >>> >> >> > >>> >> >> > - Patrick >>> >> >> > >>> >> >> > On Sun, Dec 8, 2013 at 6:05 PM, Mark Hamstra < >>> m...@clearstorydata.com >>> >> > >>> >> >> wrote: >>> >> >> >> SPARK-962 should be resolved before release. See also: >>> >> >> >> https://github.com/apache/incubator-spark/pull/195 >>> >> >> >> >>> >> >> >> With the references to the way I changed Debian packaging for >>> >> >> ClearStory, >>> >> >> >> we should be at least 90% of the way toward doing it right for >>> >> Apache. >>> >> >> >> >>> >> >> >> >>> >> >> >> On Sun, Dec 8, 2013 at 5:29 PM, Patrick Wendell < >>> pwend...@gmail.com> >>> >> >> wrote: >>> >> >> >> >>> >> >> >>> For my own part I'll give a +1 to this RC. >>> >> >> >>> >>> >> >> >>> On Sun, Dec 8, 2013 at 4:30 PM, Taka Shinagawa < >>> >> taka.epsi...@gmail.com >>> >> >> > >>> >> >> >>> wrote: >>> >> >> >>> > OK. I will post the entire output via separate email. I just >>> >> upgraded >>> >> >> >>> > Hadoop to 2.2.0 recently. So there might be something I need >>> to >>> >> >> >>> > remove/clean up. >>> >> >> >>> > >>> >> >> >>> > >>> >> >> >>> > On Sun, Dec 8, 2013 at 4:24 PM, Patrick Wendell < >>> >> pwend...@gmail.com> >>> >> >> >>> wrote: >>> >> >> >>> > >>> >> >> >>> >> Hey Take, >>> >> >> >>> >> >>> >> >> >>> >> Could you start a separate thread to debug your build issue? >>> In >>> >> that >>> >> >> >>> >> thread, could you paste the exact build command and entire >>> >> output? >>> >> >> The >>> >> >> >>> log >>> >> >> >>> >> you posted here suggests the first build detected hadoop >>> 1.0.4 >>> >> not >>> >> >> 2.2.0 >>> >> >> >>> >> based on the assembly file name it is logging. >>> >> >> >>> >> >>> >> >> >>> >> --- >>> >> >> >>> >> sent from my phone >>> >> >> >>> >> On Dec 8, 2013 4:13 PM, "Taka Shinagawa" < >>> taka.epsi...@gmail.com >>> >> > >>> >> >> >>> wrote: >>> >> >> >>> >> >>> >> >> >>> >> > With Hadoop 2.2.0 (& Java 1.7.0_45) installed, I'm having >>> >> trouble >>> >> >> >>> >> > completing the build process (sbt/sbt assembly) on >>> Macbook. The >>> >> >> sbt >>> >> >> >>> >> command >>> >> >> >>> >> > hangs at the last step. >>> >> >> >>> >> > >>> >> >> >>> >> > ... >>> >> >> >>> >> > ... >>> >> >> >>> >> > [info] SHA-1: ce8275f5841002164c4305c912a2892ec7c1d395 >>> >> >> >>> >> > [info] Packaging >>> >> >> >>> >> > >>> >> >> >>> >> > >>> >> >> >>> >> >>> >> >> >>> >>> >> >> >>> >> >>> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/tools/target/scala-2.9.3/spark-tools-assembly-0.8.1-incubating.jar >>> >> >> >>> >> > ... >>> >> >> >>> >> > [info] SHA-1: 0657a347240266230247693f265a5797d40c326a >>> >> >> >>> >> > [info] Packaging >>> >> >> >>> >> > >>> >> >> >>> >> > >>> >> >> >>> >> >>> >> >> >>> >>> >> >> >>> >> >>> /Users/taka/Documents/Spark/Releases/spark-0.8.1-incubating-rc2/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop1.0.4.jar >>> >> >> >>> >> > ... >>> >> >> >>> >> > (hangs here) >>> >> >> >>> >> > -------------------------- >>> >> >> >>> >> > >>> >> >> >>> >> > >>> >> >> >>> >> > On another Macbook with Hadoop 1.1.1 (& Java 1.7.0_45) >>> >> installed, >>> >> >> I >>> >> >> >>> was >>> >> >> >>> >> > able to build it successfully. >>> >> >> >>> >> > .. >>> >> >> >>> >> > .. >>> >> >> >>> >> > [info] SHA-1: 77109cd085bd4f0d2b601b3451b35b961d357534 >>> >> >> >>> >> > [info] Packaging >>> >> >> >>> >> > >>> >> >> >>> >> > >>> >> >> >>> >> >>> >> >> >>> >>> >> >> >>> >> >>> /Users/tshinagawa/Documents/Spark/RCs/spark-0.8.1-incubating/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar >>> >> >> >>> >> > ... >>> >> >> >>> >> > [info] Done packaging. >>> >> >> >>> >> > [success] Total time: 266 s, completed Dec 8, 2013 3:03:10 >>> PM >>> >> >> >>> >> > -------------------------- >>> >> >> >>> >> > >>> >> >> >>> >> > >>> >> >> >>> >> > >>> >> >> >>> >> > On Sun, Dec 8, 2013 at 12:41 PM, Patrick Wendell < >>> >> >> pwend...@gmail.com> >>> >> >> >>> >> > wrote: >>> >> >> >>> >> > >>> >> >> >>> >> > > Please vote on releasing the following candidate as >>> Apache >>> >> Spark >>> >> >> >>> >> > > (incubating) version 0.8.1. >>> >> >> >>> >> > > >>> >> >> >>> >> > > The tag to be voted on is v0.8.1-incubating (commit >>> >> bf23794a): >>> >> >> >>> >> > > >>> >> >> >>> >> > > >>> >> >> >>> >> > >>> >> >> >>> >> >>> >> >> >>> >>> >> >> >>> >> >>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=tag;h=e6ba91b5a7527316202797fc3dce469ff86cf203 >>> >> >> >>> >> > > >>> >> >> >>> >> > > The release files, including signatures, digests, etc >>> can be >>> >> >> found >>> >> >> >>> at: >>> >> >> >>> >> > > >>> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2/ >>> >> >> >>> >> > > >>> >> >> >>> >> > > Release artifacts are signed with the following key: >>> >> >> >>> >> > > https://people.apache.org/keys/committer/pwendell.asc >>> >> >> >>> >> > > >>> >> >> >>> >> > > The staging repository for this release can be found at: >>> >> >> >>> >> > > >>> >> >> >>> >>> >> https://repository.apache.org/content/repositories/orgapachespark-024/ >>> >> >> >>> >> > > >>> >> >> >>> >> > > The documentation corresponding to this release can be >>> found >>> >> at: >>> >> >> >>> >> > > >>> >> >> http://people.apache.org/~pwendell/spark-0.8.1-incubating-rc2-docs/ >>> >> >> >>> >> > > >>> >> >> >>> >> > > For information about the contents of this release see: >>> >> >> >>> >> > > <attached> draft of release notes >>> >> >> >>> >> > > <attached> draft of release credits >>> >> >> >>> >> > > >>> >> >> >>> >>> >> https://github.com/apache/incubator-spark/blob/branch-0.8/CHANGES.txt >>> >> >> >>> >> > > >>> >> >> >>> >> > > Please vote on releasing this package as Apache Spark >>> >> >> >>> 0.8.1-incubating! >>> >> >> >>> >> > > >>> >> >> >>> >> > > The vote is open until Wednesday, December 11th at 21:00 >>> UTC >>> >> and >>> >> >> >>> >> > > passes if a majority of at least 3 +1 PPMC votes are >>> cast. >>> >> >> >>> >> > > >>> >> >> >>> >> > > [ ] +1 Release this package as Apache Spark >>> 0.8.1-incubating >>> >> >> >>> >> > > [ ] -1 Do not release this package because ... >>> >> >> >>> >> > > >>> >> >> >>> >> > > To learn more about Apache Spark, please see >>> >> >> >>> >> > > http://spark.incubator.apache.org/ >>> >> >> >>> >> > > >>> >> >> >>> >> > >>> >> >> >>> >> >>> >> >> >>> >>> >> >> >>> >> >>> >> >>
