Re: Build Changes for SBT Users
So I've put together a pull request which depends on the typesafe URLs (they have been stable for a long time) https://github.com/apache/incubator-spark/pull/331 and uses the system sbt if it is present. How do people feel about this? On Sat, Jan 4, 2014 at 9:07 PM, Patrick Wendell wrote: > I agree TD - I was just saying that Reynold's proposal that we could > update the release post-hoc is unfortunately not possible. > > On Sat, Jan 4, 2014 at 7:13 PM, Tathagata Das > wrote: > > Patrick, that is right. All we are trying to ensure is to make a > > "best-effort" attempt to make it smooth for a new user. The script will > try > > its best to automatically install / download sbt for the user. The > fallback > > will be that the user will have to install sbt on their own. If the URL > > happens to change and our script fails to automatically download, then we > > are *no worse* than not providing the script at all. > > > > TD > > > > > > On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell > wrote: > > > >> Reynold the issue is releases are immutable and we expect them to be > >> downloaded for several years after the release date. > >> > >> On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu wrote: > >> > Sound reasonable. But I think few installed sbt even it is easy to > >> install. I think can provide this tricky script in online document, > user > >> could download this script to install sbt independence. Sound like a yet > >> another brew install sbt? > >> > :) > >> > > >> > Yours, Xuefeng Wu 吴雪峰 敬上 > >> > > >> >> On 2014年1月5日, at 上午2:56, Patrick Wendell wrote: > >> >> > >> >> We thought about this but elected not to do this for a few reasons. > >> >> > >> >> 1. Some people build from machines that do not have internet access > >> >> for security reasons and retrieve dependency from internal nexus > >> >> repositories. So having a build dependency that relies on internet > >> >> downloads is not desirable. > >> >> > >> >> 2. It's a hard to ensure stability of a particular URL in perpetuity. > >> >> This is why maven central and other mirror networks exist. Keep in > >> >> mind that we can't change the release code ever once we release it, > >> >> and if something changed about the particular URL it could break the > >> >> build. > >> >> > >> >> - Patrick > >> >> > >> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash > >> wrote: > >> >>> +1 on bundling a script similar to that one > >> >>> > >> >>> > >> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau > > >> wrote: > >> > >> Could we ship a shell script which downloads the sbt jar if not > >> present > >> (like for example > https://github.com/holdenk/slashem/blob/master/sbt)? > >> > >> > >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell < > pwend...@gmail.com> > >> wrote: > >> > >> > Hey All, > >> > > >> > Due to an ASF requirement, we recently merged a patch which > removes > >> > the sbt jar from the build. This is necessary because we aren't > >> > allowed to distributed binary artifacts with our source packages. > >> > > >> > This means that instead of building Spark with "sbt/sbt XXX", > you'll > >> > need to have sbt yourself and just run "sbt XXX" from within the > >> Spark > >> > directory. This is similar to the maven build, where we expect > users > >> > already have maven installed. > >> > > >> > You can download sbt at http://www.scala-sbt.org/. It's okay to > just > >> > download the most recent version of sbt, since sbt knows how to > fetch > >> > other versions of itself and will always use the one we specify in > >> our > >> > build file to compile spark. > >> > > >> > - Patrick > >> > >> > >> > >> -- > >> Cell : 425-233-8271 > >> > >> > -- Cell : 425-233-8271
Re: Build Changes for SBT Users
I agree TD - I was just saying that Reynold's proposal that we could update the release post-hoc is unfortunately not possible. On Sat, Jan 4, 2014 at 7:13 PM, Tathagata Das wrote: > Patrick, that is right. All we are trying to ensure is to make a > "best-effort" attempt to make it smooth for a new user. The script will try > its best to automatically install / download sbt for the user. The fallback > will be that the user will have to install sbt on their own. If the URL > happens to change and our script fails to automatically download, then we > are *no worse* than not providing the script at all. > > TD > > > On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell wrote: > >> Reynold the issue is releases are immutable and we expect them to be >> downloaded for several years after the release date. >> >> On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu wrote: >> > Sound reasonable. But I think few installed sbt even it is easy to >> install. I think can provide this tricky script in online document, user >> could download this script to install sbt independence. Sound like a yet >> another brew install sbt? >> > :) >> > >> > Yours, Xuefeng Wu 吴雪峰 敬上 >> > >> >> On 2014年1月5日, at 上午2:56, Patrick Wendell wrote: >> >> >> >> We thought about this but elected not to do this for a few reasons. >> >> >> >> 1. Some people build from machines that do not have internet access >> >> for security reasons and retrieve dependency from internal nexus >> >> repositories. So having a build dependency that relies on internet >> >> downloads is not desirable. >> >> >> >> 2. It's a hard to ensure stability of a particular URL in perpetuity. >> >> This is why maven central and other mirror networks exist. Keep in >> >> mind that we can't change the release code ever once we release it, >> >> and if something changed about the particular URL it could break the >> >> build. >> >> >> >> - Patrick >> >> >> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash >> wrote: >> >>> +1 on bundling a script similar to that one >> >>> >> >>> >> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau >> wrote: >> >> Could we ship a shell script which downloads the sbt jar if not >> present >> (like for example https://github.com/holdenk/slashem/blob/master/sbt)? >> >> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell >> wrote: >> >> > Hey All, >> > >> > Due to an ASF requirement, we recently merged a patch which removes >> > the sbt jar from the build. This is necessary because we aren't >> > allowed to distributed binary artifacts with our source packages. >> > >> > This means that instead of building Spark with "sbt/sbt XXX", you'll >> > need to have sbt yourself and just run "sbt XXX" from within the >> Spark >> > directory. This is similar to the maven build, where we expect users >> > already have maven installed. >> > >> > You can download sbt at http://www.scala-sbt.org/. It's okay to just >> > download the most recent version of sbt, since sbt knows how to fetch >> > other versions of itself and will always use the one we specify in >> our >> > build file to compile spark. >> > >> > - Patrick >> >> >> >> -- >> Cell : 425-233-8271 >> >>
Re: Build Changes for SBT Users
Patrick, that is right. All we are trying to ensure is to make a "best-effort" attempt to make it smooth for a new user. The script will try its best to automatically install / download sbt for the user. The fallback will be that the user will have to install sbt on their own. If the URL happens to change and our script fails to automatically download, then we are *no worse* than not providing the script at all. TD On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell wrote: > Reynold the issue is releases are immutable and we expect them to be > downloaded for several years after the release date. > > On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu wrote: > > Sound reasonable. But I think few installed sbt even it is easy to > install. I think can provide this tricky script in online document, user > could download this script to install sbt independence. Sound like a yet > another brew install sbt? > > :) > > > > Yours, Xuefeng Wu 吴雪峰 敬上 > > > >> On 2014年1月5日, at 上午2:56, Patrick Wendell wrote: > >> > >> We thought about this but elected not to do this for a few reasons. > >> > >> 1. Some people build from machines that do not have internet access > >> for security reasons and retrieve dependency from internal nexus > >> repositories. So having a build dependency that relies on internet > >> downloads is not desirable. > >> > >> 2. It's a hard to ensure stability of a particular URL in perpetuity. > >> This is why maven central and other mirror networks exist. Keep in > >> mind that we can't change the release code ever once we release it, > >> and if something changed about the particular URL it could break the > >> build. > >> > >> - Patrick > >> > >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash > wrote: > >>> +1 on bundling a script similar to that one > >>> > >>> > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau > wrote: > > Could we ship a shell script which downloads the sbt jar if not > present > (like for example https://github.com/holdenk/slashem/blob/master/sbt)? > > > On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell > wrote: > > > Hey All, > > > > Due to an ASF requirement, we recently merged a patch which removes > > the sbt jar from the build. This is necessary because we aren't > > allowed to distributed binary artifacts with our source packages. > > > > This means that instead of building Spark with "sbt/sbt XXX", you'll > > need to have sbt yourself and just run "sbt XXX" from within the > Spark > > directory. This is similar to the maven build, where we expect users > > already have maven installed. > > > > You can download sbt at http://www.scala-sbt.org/. It's okay to just > > download the most recent version of sbt, since sbt knows how to fetch > > other versions of itself and will always use the one we specify in > our > > build file to compile spark. > > > > - Patrick > > > > -- > Cell : 425-233-8271 > >
Re: Build Changes for SBT Users
Reynold the issue is releases are immutable and we expect them to be downloaded for several years after the release date. On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu wrote: > Sound reasonable. But I think few installed sbt even it is easy to install. > I think can provide this tricky script in online document, user could > download this script to install sbt independence. Sound like a yet another > brew install sbt? > :) > > Yours, Xuefeng Wu 吴雪峰 敬上 > >> On 2014年1月5日, at 上午2:56, Patrick Wendell wrote: >> >> We thought about this but elected not to do this for a few reasons. >> >> 1. Some people build from machines that do not have internet access >> for security reasons and retrieve dependency from internal nexus >> repositories. So having a build dependency that relies on internet >> downloads is not desirable. >> >> 2. It's a hard to ensure stability of a particular URL in perpetuity. >> This is why maven central and other mirror networks exist. Keep in >> mind that we can't change the release code ever once we release it, >> and if something changed about the particular URL it could break the >> build. >> >> - Patrick >> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash wrote: >>> +1 on bundling a script similar to that one >>> >>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau wrote: Could we ship a shell script which downloads the sbt jar if not present (like for example https://github.com/holdenk/slashem/blob/master/sbt )? On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell wrote: > Hey All, > > Due to an ASF requirement, we recently merged a patch which removes > the sbt jar from the build. This is necessary because we aren't > allowed to distributed binary artifacts with our source packages. > > This means that instead of building Spark with "sbt/sbt XXX", you'll > need to have sbt yourself and just run "sbt XXX" from within the Spark > directory. This is similar to the maven build, where we expect users > already have maven installed. > > You can download sbt at http://www.scala-sbt.org/. It's okay to just > download the most recent version of sbt, since sbt knows how to fetch > other versions of itself and will always use the one we specify in our > build file to compile spark. > > - Patrick -- Cell : 425-233-8271
Re: Build Changes for SBT Users
Sound reasonable. But I think few installed sbt even it is easy to install. I think can provide this tricky script in online document, user could download this script to install sbt independence. Sound like a yet another brew install sbt? :) Yours, Xuefeng Wu 吴雪峰 敬上 > On 2014年1月5日, at 上午2:56, Patrick Wendell wrote: > > We thought about this but elected not to do this for a few reasons. > > 1. Some people build from machines that do not have internet access > for security reasons and retrieve dependency from internal nexus > repositories. So having a build dependency that relies on internet > downloads is not desirable. > > 2. It's a hard to ensure stability of a particular URL in perpetuity. > This is why maven central and other mirror networks exist. Keep in > mind that we can't change the release code ever once we release it, > and if something changed about the particular URL it could break the > build. > > - Patrick > >> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash wrote: >> +1 on bundling a script similar to that one >> >> >>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau wrote: >>> >>> Could we ship a shell script which downloads the sbt jar if not present >>> (like for example https://github.com/holdenk/slashem/blob/master/sbt )? >>> >>> >>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell >>> wrote: >>> Hey All, Due to an ASF requirement, we recently merged a patch which removes the sbt jar from the build. This is necessary because we aren't allowed to distributed binary artifacts with our source packages. This means that instead of building Spark with "sbt/sbt XXX", you'll need to have sbt yourself and just run "sbt XXX" from within the Spark directory. This is similar to the maven build, where we expect users already have maven installed. You can download sbt at http://www.scala-sbt.org/. It's okay to just download the most recent version of sbt, since sbt knows how to fetch other versions of itself and will always use the one we specify in our build file to compile spark. - Patrick >>> >>> >>> >>> -- >>> Cell : 425-233-8271 >>>
Re: Terminology: "worker" vs "slave"
+1 for worker =) On Thu, Jan 2, 2014 at 10:45 PM, Patrick Wendell wrote: > Ya we've been trying to standardize on the terminology here (see glossary): > > http://spark.incubator.apache.org/docs/latest/cluster-overview.html > > I think "slave" actually isn't mentioned here at all - but references > to slave in the codebase are synonymous with "worker". > > - Patrick > > On Thu, Jan 2, 2014 at 10:42 PM, Reynold Xin wrote: >> It is historic. >> >> I think we are converging towards >> >> worker: the "slave" daemon in the standalone cluster manager >> >> executor: the jvm process that is launched by the worker that executes tasks >> >> >> >> On Thu, Jan 2, 2014 at 10:39 PM, Andrew Ash wrote: >> >>> The terms worker and slave seem to be used interchangeably. Are they the >>> same? >>> >>> Worker is used more frequently in the codebase: >>> >>> aash@aash-mbp ~/git/spark$ git grep -i worker | wc -l >>> 981 >>> aash@aash-mbp ~/git/spark$ git grep -i slave | wc -l >>> 348 >>> aash@aash-mbp ~/git/spark$ >>> >>> Does it make sense to unify on one or the other? >>>
Re: Build Changes for SBT Users
Doesn't Apache do redirection from incubation. to the normal website also? By the time that happens, we can also update the URL in the script? On Sat, Jan 4, 2014 at 4:13 PM, Patrick Wendell wrote: > Hey Holden, > > That sounds reasonable to me. Where would we get a url we can control > though? Right now the project has web space is at incubator.apache... > but later this will change to a full apache domain. Is there somewhere > in maven central these jars are hosted... that would be the nicest > because things like repo1.maven.org basically never changes. > > - Patrick > > On Sat, Jan 4, 2014 at 1:20 PM, Holden Karau wrote: > > That makes sense, I think we could structure a script in such a way that > it > > would overcome these problems though and probably provide a fair a mount > of > > benefit for people who just want to get started quickly. > > > > The easiest would be to have it use the system sbt if present and then > fall > > back to downloading the sbt jar. As far as stability of the URL goes we > > could solve this by either having it point at a domain we control, or > just > > with an clear error message indicating it failed to download sbt and the > > user needs to install sbt. > > > > If a restructured script in that manner would be useful I could whip up a > > pull request :) > > > > > > On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell > wrote: > > > >> We thought about this but elected not to do this for a few reasons. > >> > >> 1. Some people build from machines that do not have internet access > >> for security reasons and retrieve dependency from internal nexus > >> repositories. So having a build dependency that relies on internet > >> downloads is not desirable. > >> > >> 2. It's a hard to ensure stability of a particular URL in perpetuity. > >> This is why maven central and other mirror networks exist. Keep in > >> mind that we can't change the release code ever once we release it, > >> and if something changed about the particular URL it could break the > >> build. > >> > >> - Patrick > >> > >> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash > wrote: > >> > +1 on bundling a script similar to that one > >> > > >> > > >> > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau > >> wrote: > >> > > >> >> Could we ship a shell script which downloads the sbt jar if not > present > >> >> (like for example https://github.com/holdenk/slashem/blob/master/sbt)? > >> >> > >> >> > >> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell > > >> >> wrote: > >> >> > >> >> > Hey All, > >> >> > > >> >> > Due to an ASF requirement, we recently merged a patch which removes > >> >> > the sbt jar from the build. This is necessary because we aren't > >> >> > allowed to distributed binary artifacts with our source packages. > >> >> > > >> >> > This means that instead of building Spark with "sbt/sbt XXX", > you'll > >> >> > need to have sbt yourself and just run "sbt XXX" from within the > Spark > >> >> > directory. This is similar to the maven build, where we expect > users > >> >> > already have maven installed. > >> >> > > >> >> > You can download sbt at http://www.scala-sbt.org/. It's okay to > just > >> >> > download the most recent version of sbt, since sbt knows how to > fetch > >> >> > other versions of itself and will always use the one we specify in > our > >> >> > build file to compile spark. > >> >> > > >> >> > - Patrick > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Cell : 425-233-8271 > >> >> > >> > > > > > > > > -- > > Cell : 425-233-8271 >
Re: Build Changes for SBT Users
Hey Holden, That sounds reasonable to me. Where would we get a url we can control though? Right now the project has web space is at incubator.apache... but later this will change to a full apache domain. Is there somewhere in maven central these jars are hosted... that would be the nicest because things like repo1.maven.org basically never changes. - Patrick On Sat, Jan 4, 2014 at 1:20 PM, Holden Karau wrote: > That makes sense, I think we could structure a script in such a way that it > would overcome these problems though and probably provide a fair a mount of > benefit for people who just want to get started quickly. > > The easiest would be to have it use the system sbt if present and then fall > back to downloading the sbt jar. As far as stability of the URL goes we > could solve this by either having it point at a domain we control, or just > with an clear error message indicating it failed to download sbt and the > user needs to install sbt. > > If a restructured script in that manner would be useful I could whip up a > pull request :) > > > On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell wrote: > >> We thought about this but elected not to do this for a few reasons. >> >> 1. Some people build from machines that do not have internet access >> for security reasons and retrieve dependency from internal nexus >> repositories. So having a build dependency that relies on internet >> downloads is not desirable. >> >> 2. It's a hard to ensure stability of a particular URL in perpetuity. >> This is why maven central and other mirror networks exist. Keep in >> mind that we can't change the release code ever once we release it, >> and if something changed about the particular URL it could break the >> build. >> >> - Patrick >> >> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash wrote: >> > +1 on bundling a script similar to that one >> > >> > >> > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau >> wrote: >> > >> >> Could we ship a shell script which downloads the sbt jar if not present >> >> (like for example https://github.com/holdenk/slashem/blob/master/sbt )? >> >> >> >> >> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell >> >> wrote: >> >> >> >> > Hey All, >> >> > >> >> > Due to an ASF requirement, we recently merged a patch which removes >> >> > the sbt jar from the build. This is necessary because we aren't >> >> > allowed to distributed binary artifacts with our source packages. >> >> > >> >> > This means that instead of building Spark with "sbt/sbt XXX", you'll >> >> > need to have sbt yourself and just run "sbt XXX" from within the Spark >> >> > directory. This is similar to the maven build, where we expect users >> >> > already have maven installed. >> >> > >> >> > You can download sbt at http://www.scala-sbt.org/. It's okay to just >> >> > download the most recent version of sbt, since sbt knows how to fetch >> >> > other versions of itself and will always use the one we specify in our >> >> > build file to compile spark. >> >> > >> >> > - Patrick >> >> > >> >> >> >> >> >> >> >> -- >> >> Cell : 425-233-8271 >> >> >> > > > > -- > Cell : 425-233-8271
Re: Build Changes for SBT Users
I'm in full agreement with Holden. We should provide a smooth out of the box experience while also not getting in the way of those who provide their own sbt installs. On Saturday, January 4, 2014, Holden Karau wrote: > That makes sense, I think we could structure a script in such a way that it > would overcome these problems though and probably provide a fair a mount of > benefit for people who just want to get started quickly. > > The easiest would be to have it use the system sbt if present and then fall > back to downloading the sbt jar. As far as stability of the URL goes we > could solve this by either having it point at a domain we control, or just > with an clear error message indicating it failed to download sbt and the > user needs to install sbt. > > If a restructured script in that manner would be useful I could whip up a > pull request :) > > > On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell > > > wrote: > > > We thought about this but elected not to do this for a few reasons. > > > > 1. Some people build from machines that do not have internet access > > for security reasons and retrieve dependency from internal nexus > > repositories. So having a build dependency that relies on internet > > downloads is not desirable. > > > > 2. It's a hard to ensure stability of a particular URL in perpetuity. > > This is why maven central and other mirror networks exist. Keep in > > mind that we can't change the release code ever once we release it, > > and if something changed about the particular URL it could break the > > build. > > > > - Patrick > > > > On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash > > > > wrote: > > > +1 on bundling a script similar to that one > > > > > > > > > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau > > > > > > > wrote: > > > > > >> Could we ship a shell script which downloads the sbt jar if not > present > > >> (like for example https://github.com/holdenk/slashem/blob/master/sbt)? > > >> > > >> > > >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell > > >> > > > > >> wrote: > > >> > > >> > Hey All, > > >> > > > >> > Due to an ASF requirement, we recently merged a patch which removes > > >> > the sbt jar from the build. This is necessary because we aren't > > >> > allowed to distributed binary artifacts with our source packages. > > >> > > > >> > This means that instead of building Spark with "sbt/sbt XXX", you'll > > >> > need to have sbt yourself and just run "sbt XXX" from within the > Spark > > >> > directory. This is similar to the maven build, where we expect users > > >> > already have maven installed. > > >> > > > >> > You can download sbt at http://www.scala-sbt.org/. It's okay to > just > > >> > download the most recent version of sbt, since sbt knows how to > fetch > > >> > other versions of itself and will always use the one we specify in > our > > >> > build file to compile spark. > > >> > > > >> > - Patrick > > >> > > > >> > > >> > > >> > > >> -- > > >> Cell : 425-233-8271 > > >> > > > > > > -- > Cell : 425-233-8271 >
Re: Build Changes for SBT Users
That makes sense, I think we could structure a script in such a way that it would overcome these problems though and probably provide a fair a mount of benefit for people who just want to get started quickly. The easiest would be to have it use the system sbt if present and then fall back to downloading the sbt jar. As far as stability of the URL goes we could solve this by either having it point at a domain we control, or just with an clear error message indicating it failed to download sbt and the user needs to install sbt. If a restructured script in that manner would be useful I could whip up a pull request :) On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell wrote: > We thought about this but elected not to do this for a few reasons. > > 1. Some people build from machines that do not have internet access > for security reasons and retrieve dependency from internal nexus > repositories. So having a build dependency that relies on internet > downloads is not desirable. > > 2. It's a hard to ensure stability of a particular URL in perpetuity. > This is why maven central and other mirror networks exist. Keep in > mind that we can't change the release code ever once we release it, > and if something changed about the particular URL it could break the > build. > > - Patrick > > On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash wrote: > > +1 on bundling a script similar to that one > > > > > > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau > wrote: > > > >> Could we ship a shell script which downloads the sbt jar if not present > >> (like for example https://github.com/holdenk/slashem/blob/master/sbt )? > >> > >> > >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell > >> wrote: > >> > >> > Hey All, > >> > > >> > Due to an ASF requirement, we recently merged a patch which removes > >> > the sbt jar from the build. This is necessary because we aren't > >> > allowed to distributed binary artifacts with our source packages. > >> > > >> > This means that instead of building Spark with "sbt/sbt XXX", you'll > >> > need to have sbt yourself and just run "sbt XXX" from within the Spark > >> > directory. This is similar to the maven build, where we expect users > >> > already have maven installed. > >> > > >> > You can download sbt at http://www.scala-sbt.org/. It's okay to just > >> > download the most recent version of sbt, since sbt knows how to fetch > >> > other versions of itself and will always use the one we specify in our > >> > build file to compile spark. > >> > > >> > - Patrick > >> > > >> > >> > >> > >> -- > >> Cell : 425-233-8271 > >> > -- Cell : 425-233-8271
Re: Build Changes for SBT Users
We thought about this but elected not to do this for a few reasons. 1. Some people build from machines that do not have internet access for security reasons and retrieve dependency from internal nexus repositories. So having a build dependency that relies on internet downloads is not desirable. 2. It's a hard to ensure stability of a particular URL in perpetuity. This is why maven central and other mirror networks exist. Keep in mind that we can't change the release code ever once we release it, and if something changed about the particular URL it could break the build. - Patrick On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash wrote: > +1 on bundling a script similar to that one > > > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau wrote: > >> Could we ship a shell script which downloads the sbt jar if not present >> (like for example https://github.com/holdenk/slashem/blob/master/sbt )? >> >> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell >> wrote: >> >> > Hey All, >> > >> > Due to an ASF requirement, we recently merged a patch which removes >> > the sbt jar from the build. This is necessary because we aren't >> > allowed to distributed binary artifacts with our source packages. >> > >> > This means that instead of building Spark with "sbt/sbt XXX", you'll >> > need to have sbt yourself and just run "sbt XXX" from within the Spark >> > directory. This is similar to the maven build, where we expect users >> > already have maven installed. >> > >> > You can download sbt at http://www.scala-sbt.org/. It's okay to just >> > download the most recent version of sbt, since sbt knows how to fetch >> > other versions of itself and will always use the one we specify in our >> > build file to compile spark. >> > >> > - Patrick >> > >> >> >> >> -- >> Cell : 425-233-8271 >>
Re: Build Changes for SBT Users
+1 on bundling a script similar to that one On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau wrote: > Could we ship a shell script which downloads the sbt jar if not present > (like for example https://github.com/holdenk/slashem/blob/master/sbt )? > > > On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell > wrote: > > > Hey All, > > > > Due to an ASF requirement, we recently merged a patch which removes > > the sbt jar from the build. This is necessary because we aren't > > allowed to distributed binary artifacts with our source packages. > > > > This means that instead of building Spark with "sbt/sbt XXX", you'll > > need to have sbt yourself and just run "sbt XXX" from within the Spark > > directory. This is similar to the maven build, where we expect users > > already have maven installed. > > > > You can download sbt at http://www.scala-sbt.org/. It's okay to just > > download the most recent version of sbt, since sbt knows how to fetch > > other versions of itself and will always use the one we specify in our > > build file to compile spark. > > > > - Patrick > > > > > > -- > Cell : 425-233-8271 >
Re: Build Changes for SBT Users
Could we ship a shell script which downloads the sbt jar if not present (like for example https://github.com/holdenk/slashem/blob/master/sbt )? On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell wrote: > Hey All, > > Due to an ASF requirement, we recently merged a patch which removes > the sbt jar from the build. This is necessary because we aren't > allowed to distributed binary artifacts with our source packages. > > This means that instead of building Spark with "sbt/sbt XXX", you'll > need to have sbt yourself and just run "sbt XXX" from within the Spark > directory. This is similar to the maven build, where we expect users > already have maven installed. > > You can download sbt at http://www.scala-sbt.org/. It's okay to just > download the most recent version of sbt, since sbt knows how to fetch > other versions of itself and will always use the one we specify in our > build file to compile spark. > > - Patrick > -- Cell : 425-233-8271
Build Changes for SBT Users
Hey All, Due to an ASF requirement, we recently merged a patch which removes the sbt jar from the build. This is necessary because we aren't allowed to distributed binary artifacts with our source packages. This means that instead of building Spark with "sbt/sbt XXX", you'll need to have sbt yourself and just run "sbt XXX" from within the Spark directory. This is similar to the maven build, where we expect users already have maven installed. You can download sbt at http://www.scala-sbt.org/. It's okay to just download the most recent version of sbt, since sbt knows how to fetch other versions of itself and will always use the one we specify in our build file to compile spark. - Patrick
