sbt-package-bin
Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala
Re: sbt-package-bin
Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala
Re: sbt-package-bin
A basic Debian package can already be created from the Maven build: mvn -Pdeb ... On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala
Re: sbt-package-bin
And there is a deb target as well - ah didn't see Mark's email. On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: Ya there is already some fragmentation here. Maven has some dist targets and there is also ./make-distribution.sh. On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra m...@clearstorydata.comwrote: A basic Debian package can already be created from the Maven build: mvn -Pdeb ... On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala
Re: sbt-package-bin
Ya there is already some fragmentation here. Maven has some dist targets and there is also ./make-distribution.sh. On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra m...@clearstorydata.comwrote: A basic Debian package can already be created from the Maven build: mvn -Pdeb ... On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala
Re: sbt-package-bin
Mark - sorry, would you mind expanding what the is? Something like mvn -Pdeb package ? I get: [ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find artifact org.apache:apache:pom:13 - [Help 1] On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: And there is a deb target as well - ah didn't see Mark's email. On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: Ya there is already some fragmentation here. Maven has some dist targets and there is also ./make-distribution.sh. On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra m...@clearstorydata.com wrote: A basic Debian package can already be created from the Maven build: mvn -Pdeb ... On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyalahttp://www.twitter.com/ooyala
Re: sbt-package-bin
What the ... is kind of depends on what you're trying to accomplish. You could be setting Hadoop version and other stuff in there, but if you go too much beyond a pretty basic build, you're probably also going to have to modify the configuration of the jdeb plugin in assembly/pom.xml to include/exclude just what you want/don't want in the Debian package. Anyway, a typical build would look something like 'mvn -U -Pdeb -DskipTests clean package', after which you should be able to find your .deb in assembly/target. On Tue, Apr 1, 2014 at 3:01 PM, Evan Chan e...@ooyala.com wrote: Mark - sorry, would you mind expanding what the is? Something like mvn -Pdeb package ? I get: [ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find artifact org.apache:apache:pom:13 - [Help 1] On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: And there is a deb target as well - ah didn't see Mark's email. On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: Ya there is already some fragmentation here. Maven has some dist targets and there is also ./make-distribution.sh. On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra m...@clearstorydata.com wrote: A basic Debian package can already be created from the Maven build: mvn -Pdeb ... On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala
Re: sbt-package-bin
...or at least you could do that if the Maven build wasn't broken right now. On Tue, Apr 1, 2014 at 6:01 PM, Mark Hamstra m...@clearstorydata.comwrote: What the ... is kind of depends on what you're trying to accomplish. You could be setting Hadoop version and other stuff in there, but if you go too much beyond a pretty basic build, you're probably also going to have to modify the configuration of the jdeb plugin in assembly/pom.xml to include/exclude just what you want/don't want in the Debian package. Anyway, a typical build would look something like 'mvn -U -Pdeb -DskipTests clean package', after which you should be able to find your .deb in assembly/target. On Tue, Apr 1, 2014 at 3:01 PM, Evan Chan e...@ooyala.com wrote: Mark - sorry, would you mind expanding what the is? Something like mvn -Pdeb package ? I get: [ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find artifact org.apache:apache:pom:13 - [Help 1] On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: And there is a deb target as well - ah didn't see Mark's email. On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: Ya there is already some fragmentation here. Maven has some dist targets and there is also ./make-distribution.sh. On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra m...@clearstorydata.com wrote: A basic Debian package can already be created from the Maven build: mvn -Pdeb ... On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala
Re: sbt-package-bin
Whoops! Looks like it was just my brain that was broken. On Tue, Apr 1, 2014 at 6:03 PM, Mark Hamstra m...@clearstorydata.comwrote: ...or at least you could do that if the Maven build wasn't broken right now. On Tue, Apr 1, 2014 at 6:01 PM, Mark Hamstra m...@clearstorydata.comwrote: What the ... is kind of depends on what you're trying to accomplish. You could be setting Hadoop version and other stuff in there, but if you go too much beyond a pretty basic build, you're probably also going to have to modify the configuration of the jdeb plugin in assembly/pom.xml to include/exclude just what you want/don't want in the Debian package. Anyway, a typical build would look something like 'mvn -U -Pdeb -DskipTests clean package', after which you should be able to find your .deb in assembly/target. On Tue, Apr 1, 2014 at 3:01 PM, Evan Chan e...@ooyala.com wrote: Mark - sorry, would you mind expanding what the is? Something like mvn -Pdeb package ? I get: [ERROR] Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-compiler-plugin:jar:3.1: Could not find artifact org.apache:apache:pom:13 - [Help 1] On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: And there is a deb target as well - ah didn't see Mark's email. On Tue, Apr 1, 2014 at 11:36 AM, Patrick Wendell pwend...@gmail.com wrote: Ya there is already some fragmentation here. Maven has some dist targets and there is also ./make-distribution.sh. On Tue, Apr 1, 2014 at 11:31 AM, Mark Hamstra m...@clearstorydata.com wrote: A basic Debian package can already be created from the Maven build: mvn -Pdeb ... On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Also, I understand this is the last week / merge window for 1.0, so if folks are interested I'd like to get in a PR quickly. thanks, Evan On Tue, Apr 1, 2014 at 11:24 AM, Evan Chan e...@ooyala.com wrote: Hey folks, We are in the middle of creating a Chef recipe for Spark. As part of that we want to create a Debian package for Spark. What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. Also I'm not sure if there's an equivalent plugin for Maven. thanks, Evan -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyala http://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala -- -- Evan Chan Staff Engineer e...@ooyala.com | http://www.ooyala.com/ http://www.facebook.com/ooyalahttp://www.linkedin.com/company/ooyala http://www.twitter.com/ooyala
Re: sbt-package-bin
What do folks think of adding the sbt-package-bin plugin to allow easy creation of a Spark .deb file? I believe it adds all dependency jars into a single lib/ folder, so in some ways it's even easier to manage than the assembly. You might also check out the sbt-native-packagerhttps://github.com/sbt/sbt-native-packager. Cheers, Lee
