The advantages you've listed sound really compelling to me. - Do you have time to implement these changes or do we need a volunteer? ;)
- I assume that republishing the artifacts as you propose doesn't have any new legal implications since we already publish them with our JARs, right? - We might think about adding Netty to the list of shaded artifacts since some dependency conflicts were reported recently. Would have to double check the reported issues before doing that though. ;-) – Ufuk On Wed, May 10, 2017 at 8:45 PM, Stephan Ewen <se...@apache.org> wrote: > @chesnay: I used ASM as an example in the proposal. Maybe I did not say > that clearly. > > If we like that approach, we should deal with the other libraries (at least > the frequently used ones) in the same way. > > > I would imagine to have a project layout like that: > > flink-shaded-deps > - flink-shaded-asm > - flink-shaded-guava > - flink-shaded-curator > - flink-shaded-hadoop > > > "flink-shaded-deps" would not be built every time (and not be released > every time), but only when needed. > > > > > > > On Wed, May 10, 2017 at 7:28 PM, Chesnay Schepler <ches...@apache.org> > wrote: > >> I like the idea, thank you for bringing it up. >> >> Given that the raised problems aren't really ASM specific would it make >> sense to create one flink-shaded module that contains all frequently shaded >> libraries? (or maybe even all shaded dependencies by core modules) The >> proposal limits the scope of this to ASM and i was wondering why. >> >> I also remember that there was a discussion recently about why we shade >> things at all, and the idea of working against the shaded namespaces was >> brought up. Back then i was expressing doubts as to whether IDE's would >> properly support this; what's the state on that? >> >> On 10.05.2017 18:18, Stephan Ewen wrote: >> >>> Hi! >>> >>> This is a discussion about altering the way we handle dependencies and >>> shading in Flink. >>> I ran into quite a view problems trying to adjust / fix some shading >>> issues >>> during release validation. >>> >>> The issue is tracked under: https://issues.apache.org/jira >>> /browse/FLINK-6529 >>> Bring this discussion thread up because it is a bigger issue >>> >>> *Problem* >>> >>> Currently, Flink shades dependencies like ASM and Guava into all jars of >>> projects that reference it and relocate the classes. >>> >>> There are some drawbacks to that approach, let's discuss them at the >>> example of ASM: >>> >>> - The ASM classes are for example in flink-core, flink-java, >>> flink-scala, >>> flink-runtime, etc. >>> >>> - Users that reference these dependencies have the classes multiple >>> times >>> in the classpath. That is unclean (works, through, because the classes are >>> identical). The same happens when building the final dist. jar. >>> >>> - Some of these dependencies require to include license files in the >>> shaded jar. It is hard to impossible to build a good automatic solution >>> for >>> that, partly due to Maven's very poor cross-project path support >>> >>> - Most importantly: Scala does not support shading really well. Scala >>> classes have references to classes in more places than just the class >>> names >>> (apparently for Scala reflect support). Referencing a Scala project with >>> shaded ASM still requires to add a reference to unshaded ASM (at least as >>> a >>> compile dependency). >>> >>> *Proposal* >>> >>> I propose that we build and deploy a asm-flink-shaded version of ASM and >>> directly program against the relocated namespaces. Since we never use >>> classes that we relocate in public interfaces, Flink users will never see >>> the relocated class names. Internally, it does not hurt to use them. >>> >>> - Proper maven dependency management, no hidden (shaded) dependencies >>> >>> - One copy of each class for shaded dependencies >>> >>> - Proper Scala interoperability >>> >>> - Natural License management (license is part of deployed >>> asm-flink-shaded jar) >>> >>> >>> Happy to hear thoughts! >>> >>> Stephan >>> >>> >>
