One case where this is tricky is if any jars cause native libraries to be loaded. The problem is that you can't load these libraries more than once. Furthermore, if you load the library in one non-root class loader, then the other class loaders can access the native library. This make things really complicated.
On Thu, Nov 26, 2015 at 10:47 AM, Thomas Weise <[email protected]> wrote: > I referred to the problem at Apex level that the platform developer faces. > > The same repeats, as you say, at the application level. Some operators > depend on Hadoop directly, others share little or no dependencies. > > > On Wed, Nov 25, 2015 at 3:28 PM, Chetan Narsude (cnarsude) < > [email protected]> wrote: > > > > > >But here, we depend on Hadoop so we cannot isolate it. > > > > User does not necessarily depend on Hadoop. The platform pulls the Hadoop > > dependency into application code irrespective of user's influence. The > > platform also pulls in other non-hadoop related N random dependencies > from > > application's perspective. > > > > The complexity comes from the effort needed to refactor parts of the code > > to isolate the UDF and its dependencies from platform dependencies. It¹s > a > > very large initiative (that¹s where the motivation part comes) but not an > > impossible one. > > > > Shading is a shady approach but it¹s an easy way out at smaller scale. At > > larger scale it starts foraying into the pain around application > > jar/platform jar deployment discussion and JVM PermGen memory issues. > > > > ‹ > > Chetan > > > > >
