Can somebody give me Contributor permissions in the Bigtop JIRA? I can not comment to issues anymore because of the JIRA spam lockdown :(
On Thu, Apr 21, 2016 at 11:16 AM, Robert Metzger <[email protected]> wrote: > Thank you for your responses. We use shading for two main purposes: > - Relocating dependencies > - building fat-jars / uber jars > > I understand that it doesn't make sense to deploy fat-jars in a controlled > environment like bigtop. The dist jar of Flink is currently ~80MB, so its > not as bad as with the others you've mentioned. > Sadly, the build process of Flink is pretty complicated and has a lot of > implications, so changing this is certainly not trivial. > But its definitively on our TODO list to address the issue. > > Regarding the dependency relocation: We sadly have to do this because of > projects like Guava, which do not provide stable interfaces across releases. > Flink is shading away Hadoop's guava so that it doesn't conflict with our > guava. We in turn also shade away our guava dependency so that our users > can use whatever guava version they want to use. > > I'll keep building Flink from source for the bigtop packaging. > > > > On Wed, Apr 20, 2016 at 11:32 PM, Konstantin Boudnik <[email protected]> > wrote: > >> +1 what Olaf said! We already see monstrosities that Spark builds and >> provides >> (close to 400MB binary blobs) or Zeppelin, which goes over half a gig >> (sic!) >> >> We are producing the stack for very exact specifications, so can afford >> removing a bunch of redistributed bits and pieces, as well to avoid the >> shading nonsense. Build time is a very small price to pay to get clean >> deployment. >> >> Cos >> >> On Wed, Apr 20, 2016 at 08:16PM, Olaf Flebbe wrote: >> > Hi, >> > >> > Yep, we like to rebuild from source. AFAIK shading dependencies has to >> do >> > with uber jars. Since we do not use uber jars, can we please have simple >> > jars, since we use a packaging system to do the deployment? >> > >> > Regards, >> > Olaf >> > >> > >> > > Am 20.04.2016 um 16:24 schrieb Robert Metzger <[email protected]>: >> > > >> > > Hi, >> > > >> > > I'm currently looking into the existing pull request for the Flink >> > > integration [1]. >> > > While doing that, I wondered if its a requirement from Bigtop to build >> > > Flink from source when building the rpm / deb packages? >> > > Not building from source would speed up the build a lot and it would >> ensure >> > > that the binary build is verified by the Flink community (there are >> some >> > > issues with maven dependency shading when building Flink with maven >> 3.3.x) >> > > >> > > >> > > >> > > [1] https://github.com/apache/bigtop/pull/93/files >> > > >> > > On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <[email protected]> >> wrote: >> > > >> > >> Hi Konstantin, hi Jay, >> > >> >> > >> Thanks for the replies and the prompt progress on the pull request. >> It >> > >> looks like we will have a Flink RPM package in Bigtop very soon. For >> > >> the remaining integration we can open follow-up pull requests. >> > >> >> > >> If anything comes up while finishing up the work, please ping me! >> > >> >> > >> Cheers, >> > >> Max >> > >> >> > >> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas < >> [email protected]> >> > >> wrote: >> > >>> Yup , we're getting there ! I tend to this sometimes in after work >> hours, >> > >>> I'll call the grad students tonite, and see if they want to make a >> final >> > >>> push this wknd on it, i can help them. >> > >>> >> > >>> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <[email protected] >> > >> > >> wrote: >> > >>> >> > >>>> Hi Maximilian. >> > >>>> >> > >>>> Indeed. the saga of getting Flink into the Apache Bigdata stack >> has a >> > >> long >> > >>>> history ;) It's good to see it's finally converges. Your proposal >> of >> > >>>> breaking >> > >>>> the existing PR in peces certainly makes sense! That's how we >> prefer to >> > >> do >> > >>>> things as well - smaller changes are easier to check, fix, and even >> > >> revert >> > >>>> if >> > >>>> needed. >> > >>>> >> > >>>> Another thing: we normally would expect to have a single commit for >> > >> JIRA, >> > >>>> so >> > >>>> it would make sense to squash (rebase) the existing PR into smaller >> > >> number >> > >>>> of >> > >>>> commits. Otherwise, it is pretty hard to navigate through all of >> them. >> > >>>> >> > >>>> Please don't hesitate to ping the list shall you need any >> assistance. >> > >>>> Regards, >> > >>>> Cos >> > >>>> >> > >>>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote: >> > >>>>> Hi, >> > >>>>> >> > >>>>> I'm an Apache Flink committer and I'd be very happy to see Flink >> enter >> > >>>>> Bigtop. We have seen quite some interest in Bigtop in the Flink >> > >>>>> community. I've been checking out Bhupendra Singh's pull request >> which >> > >>>>> followed this thread: https://github.com/apache/bigtop/pull/93 >> > >>>>> >> > >>>>> The packaging of Flink remains one of the biggest hurdles for >> people >> > >>>>> who want to install and run Flink on a cluster. Thus, that was my >> main >> > >>>>> focus when reviewing the PR. Apart from a few issues I found, the >> pull >> > >>>>> request looks good. It would be great if we could bring it into a >> > >>>>> mergeable state. >> > >>>>> >> > >>>>> I wonder if it makes sense to break this pull request into several >> > >>>>> pull requests? For example, one for the packaging, one for the >> puppet >> > >>>>> scripts, and another one for the smoke tests. That could make >> > >>>>> reviewing of the changes easier and people could already try out >> > >>>>> incremental changes. I'd be happy to help out with the packaging >> and >> > >>>>> scripting. >> > >>>>> >> > >>>>> What do you think about that? >> > >>>>> >> > >>>>> Thanks, >> > >>>>> Max >> > >>>> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> jay vyas >> > >> >> > >> >> >> >
