Thank you for your responses. We use shading for two main purposes: - Relocating dependencies - building fat-jars / uber jars
I understand that it doesn't make sense to deploy fat-jars in a controlled environment like bigtop. The dist jar of Flink is currently ~80MB, so its not as bad as with the others you've mentioned. Sadly, the build process of Flink is pretty complicated and has a lot of implications, so changing this is certainly not trivial. But its definitively on our TODO list to address the issue. Regarding the dependency relocation: We sadly have to do this because of projects like Guava, which do not provide stable interfaces across releases. Flink is shading away Hadoop's guava so that it doesn't conflict with our guava. We in turn also shade away our guava dependency so that our users can use whatever guava version they want to use. I'll keep building Flink from source for the bigtop packaging. On Wed, Apr 20, 2016 at 11:32 PM, Konstantin Boudnik <[email protected]> wrote: > +1 what Olaf said! We already see monstrosities that Spark builds and > provides > (close to 400MB binary blobs) or Zeppelin, which goes over half a gig > (sic!) > > We are producing the stack for very exact specifications, so can afford > removing a bunch of redistributed bits and pieces, as well to avoid the > shading nonsense. Build time is a very small price to pay to get clean > deployment. > > Cos > > On Wed, Apr 20, 2016 at 08:16PM, Olaf Flebbe wrote: > > Hi, > > > > Yep, we like to rebuild from source. AFAIK shading dependencies has to do > > with uber jars. Since we do not use uber jars, can we please have simple > > jars, since we use a packaging system to do the deployment? > > > > Regards, > > Olaf > > > > > > > Am 20.04.2016 um 16:24 schrieb Robert Metzger <[email protected]>: > > > > > > Hi, > > > > > > I'm currently looking into the existing pull request for the Flink > > > integration [1]. > > > While doing that, I wondered if its a requirement from Bigtop to build > > > Flink from source when building the rpm / deb packages? > > > Not building from source would speed up the build a lot and it would > ensure > > > that the binary build is verified by the Flink community (there are > some > > > issues with maven dependency shading when building Flink with maven > 3.3.x) > > > > > > > > > > > > [1] https://github.com/apache/bigtop/pull/93/files > > > > > > On Wed, Apr 6, 2016 at 11:16 AM, Maximilian Michels <[email protected]> > wrote: > > > > > >> Hi Konstantin, hi Jay, > > >> > > >> Thanks for the replies and the prompt progress on the pull request. It > > >> looks like we will have a Flink RPM package in Bigtop very soon. For > > >> the remaining integration we can open follow-up pull requests. > > >> > > >> If anything comes up while finishing up the work, please ping me! > > >> > > >> Cheers, > > >> Max > > >> > > >> On Tue, Apr 5, 2016 at 6:01 PM, jay vyas <[email protected] > > > > >> wrote: > > >>> Yup , we're getting there ! I tend to this sometimes in after work > hours, > > >>> I'll call the grad students tonite, and see if they want to make a > final > > >>> push this wknd on it, i can help them. > > >>> > > >>> On Tue, Apr 5, 2016 at 11:56 AM, Konstantin Boudnik <[email protected]> > > >> wrote: > > >>> > > >>>> Hi Maximilian. > > >>>> > > >>>> Indeed. the saga of getting Flink into the Apache Bigdata stack has > a > > >> long > > >>>> history ;) It's good to see it's finally converges. Your proposal of > > >>>> breaking > > >>>> the existing PR in peces certainly makes sense! That's how we > prefer to > > >> do > > >>>> things as well - smaller changes are easier to check, fix, and even > > >> revert > > >>>> if > > >>>> needed. > > >>>> > > >>>> Another thing: we normally would expect to have a single commit for > > >> JIRA, > > >>>> so > > >>>> it would make sense to squash (rebase) the existing PR into smaller > > >> number > > >>>> of > > >>>> commits. Otherwise, it is pretty hard to navigate through all of > them. > > >>>> > > >>>> Please don't hesitate to ping the list shall you need any > assistance. > > >>>> Regards, > > >>>> Cos > > >>>> > > >>>> On Tue, Apr 05, 2016 at 11:25AM, Maximilian Michels wrote: > > >>>>> Hi, > > >>>>> > > >>>>> I'm an Apache Flink committer and I'd be very happy to see Flink > enter > > >>>>> Bigtop. We have seen quite some interest in Bigtop in the Flink > > >>>>> community. I've been checking out Bhupendra Singh's pull request > which > > >>>>> followed this thread: https://github.com/apache/bigtop/pull/93 > > >>>>> > > >>>>> The packaging of Flink remains one of the biggest hurdles for > people > > >>>>> who want to install and run Flink on a cluster. Thus, that was my > main > > >>>>> focus when reviewing the PR. Apart from a few issues I found, the > pull > > >>>>> request looks good. It would be great if we could bring it into a > > >>>>> mergeable state. > > >>>>> > > >>>>> I wonder if it makes sense to break this pull request into several > > >>>>> pull requests? For example, one for the packaging, one for the > puppet > > >>>>> scripts, and another one for the smoke tests. That could make > > >>>>> reviewing of the changes easier and people could already try out > > >>>>> incremental changes. I'd be happy to help out with the packaging > and > > >>>>> scripting. > > >>>>> > > >>>>> What do you think about that? > > >>>>> > > >>>>> Thanks, > > >>>>> Max > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> jay vyas > > >> > > > > >
