@chesnay: I used ASM as an example in the proposal. Maybe I did not say that clearly.
If we like that approach, we should deal with the other libraries (at least the frequently used ones) in the same way. I would imagine to have a project layout like that: flink-shaded-deps - flink-shaded-asm - flink-shaded-guava - flink-shaded-curator - flink-shaded-hadoop "flink-shaded-deps" would not be built every time (and not be released every time), but only when needed. On Wed, May 10, 2017 at 7:28 PM, Chesnay Schepler <ches...@apache.org> wrote: > I like the idea, thank you for bringing it up. > > Given that the raised problems aren't really ASM specific would it make > sense to create one flink-shaded module that contains all frequently shaded > libraries? (or maybe even all shaded dependencies by core modules) The > proposal limits the scope of this to ASM and i was wondering why. > > I also remember that there was a discussion recently about why we shade > things at all, and the idea of working against the shaded namespaces was > brought up. Back then i was expressing doubts as to whether IDE's would > properly support this; what's the state on that? > > On 10.05.2017 18:18, Stephan Ewen wrote: > >> Hi! >> >> This is a discussion about altering the way we handle dependencies and >> shading in Flink. >> I ran into quite a view problems trying to adjust / fix some shading >> issues >> during release validation. >> >> The issue is tracked under: https://issues.apache.org/jira >> /browse/FLINK-6529 >> Bring this discussion thread up because it is a bigger issue >> >> *Problem* >> >> Currently, Flink shades dependencies like ASM and Guava into all jars of >> projects that reference it and relocate the classes. >> >> There are some drawbacks to that approach, let's discuss them at the >> example of ASM: >> >> - The ASM classes are for example in flink-core, flink-java, >> flink-scala, >> flink-runtime, etc. >> >> - Users that reference these dependencies have the classes multiple >> times >> in the classpath. That is unclean (works, through, because the classes are >> identical). The same happens when building the final dist. jar. >> >> - Some of these dependencies require to include license files in the >> shaded jar. It is hard to impossible to build a good automatic solution >> for >> that, partly due to Maven's very poor cross-project path support >> >> - Most importantly: Scala does not support shading really well. Scala >> classes have references to classes in more places than just the class >> names >> (apparently for Scala reflect support). Referencing a Scala project with >> shaded ASM still requires to add a reference to unshaded ASM (at least as >> a >> compile dependency). >> >> *Proposal* >> >> I propose that we build and deploy a asm-flink-shaded version of ASM and >> directly program against the relocated namespaces. Since we never use >> classes that we relocate in public interfaces, Flink users will never see >> the relocated class names. Internally, it does not hurt to use them. >> >> - Proper maven dependency management, no hidden (shaded) dependencies >> >> - One copy of each class for shaded dependencies >> >> - Proper Scala interoperability >> >> - Natural License management (license is part of deployed >> asm-flink-shaded jar) >> >> >> Happy to hear thoughts! >> >> Stephan >> >> >
