On Mon, Dec 15, 2014 at 2:42 AM, Adam Taft <[email protected]> wrote:
> I definitely agree that we can use guidance for any consideration or > implementation of OSGi. I am definitely no OSGi expert, but at least for > the past couple of years, I have spent a reasonable amount of time > exploring the use of OSGi within/around the existing NiFi framework. Here > are some observations (hopefully as apple-to-apple as possible), for what > they're worth: > > - Both NiFi and OSGi are modular application frameworks, defining a plugin > component model realized by the NAR (NiFi Archive) format and the OSGi > bundle format, respectively. > > - Both frameworks are able to find, load and run components at application > startup. OSGi can additionally support dynamic loading and unloading at > any point during runtime. > > - Both provide module independence via classloader isolation. > > - Both implement a service layer architecture, allowing modules to > interact outside of their isolation container. OSGi has an extremely mature > service registration, discovery and binding architecture, given this was > one of its primary target areas. NiFi's service architecture approach is > limited and immature in comparison; several opportunities for growth can be > easily recognized. > > - The NAR format allows for a single parentage, effectively required (but > limiting) for service discovery. OSGi does not define any restriction > between modules and services, since any registered service can be > discovered by any module (many-to-many). > > - The NAR format, being very similar to WAR, includes all dependencies > inside of the archive. Multiple copies of the same base library (for > example, any Apache Commons jar) are loaded into each isolated classloader > space; NAR and WAR are very analogous in this way. > > In contrast, OSGi bundles do not traditionally include dependencies as part > of the bundle archive. Instead with OSGi, only a single copy of any common > library is maintained for the entire application framework. i.e. one and > only one copy of any commons jar is ever loaded. Additionally, OSGi > provides version conflict resolution, somewhat similar to maven conflict > resolution. OSGi benefits from reduced permgen usage loading due to these > features. > > - NiFi requires all NAR modules to be available locally at startup. OSGi > containers generally support dynamic loading of modules and services from > remote repositories (OBR or OSGi Bundle Repository). OSGi bundles can be > easily added or updated from an OBR, including all transitive dependencies. > > Note that usage of an OBR for dependency resolution, however, can be an > impediment in environments without network connectivity to such external > repositories. Manual download and local install would still likely be > required on any restricted network environment. (Nexus can act as an OBR, > though.) > > - Both provide module lifecycle support. However, OSGi lifecycle support > is geared more towards module or service loading/unloading; whereas, NAR > lifecycle support in NiFi is geared to processor level lifecycle events. > Limited lifecycle control is provided at the NAR level itself. > > - OSGi defines dependencies and service interfaces via bundle declarations > in the MANIFEST.MF file. NAR modules are self-contained and require no > META-INF information. NARs are arguably simpler to build, though OSGi > related maven plugins can help. > > In summary, the NiFi framework has a lot of features that OSGi provides, > particularly around application startup, module loading, classloader > isolation, service discovery, etc. OSGi gets "right" many of the concepts > that NiFi has grown into. > > While I don't believe any near-term release of NiFi could utilize OSGi, it > is something that could likely be considered for a major (definitely > non-backwards compatible) release in the future. It won't come without > significant investment in time, as many parts in the "guts" of NiFi will be > affected. Allowing OSGi compatible NARs (and the processors contained) to > be loaded/unloaded at anytime during runtime needs to be thought through > carefully. > > Obviously, NiFi should refrain from "reinventing the wheel" and "not > invented here" in places where a well recognized platform is already > available. Therefore it's possible that OSGi could be used for many of the > problems which NiFi addresses today. Most of the NiFi concepts mentioned > above can be implemented on top of OSGi. This would free NiFi to instead > focus on its true value proposition, namely data flow management. > > Food for thought. Hope this was useful. > > Adam > > I'm somewhat sorry that I started this rabbit running. I think Adam's summary is quite comprehensive. I am not on a mission to convince this group to trade in NAR for OSGi; this started as a discussion of howcome there's a maven plugin, and I think that question is now fully answered. If those of you close to the requirements want to explore some of the alternative trade-offs enabled by OSGi, I'd be happy to help, but I think that this is a rather large 'if'. > > > > > On Sun, Dec 14, 2014 at 6:08 PM, Joe Witt <[email protected]> wrote: > > > > I think ultimately with Nar or OSGi you can achieve the same ends (most > > likely). Where the differences come in would be how easily a developer > can > > build, access, and reason over a bundle and the associated > overhead/baggage > > that comes with it at runtime. > > > > Nars (the nifi kind) are purpose built for NiFi and so the overhead and > > class loading model is very well understood and battle tested. But the > > truth is our plugin reflects our novice (copy and paste) level of > knowledge > > of the maven internals specifically as it relates to building plugins. > > > > Nars did, and perhaps at the moment still do, have the right tradeoff. > But > > OSGi by design should be the right choice and so we need to gain > > familiarity or better yet do a good job of solving our other priorities > and > > attracting folks with expertise here who can help us get it right (though > > Benson's notes are encouraging too). > > > > I'm glad to hear folks aren't balking at the notion of deploying nar > > bundles to maven repos. That means we can prioritize other issues higher > > (like the docs and having a build process that is more common and > > approachable). > > > > Thanks > > Joe > > > > On Sun, Dec 14, 2014 at 7:34 PM, Mark Payne <[email protected]> > wrote: > > > > > > I have very little experience with OSGi (I toyed with it at home when > we > > > discussed the introduction of NARs several years ago but that's about > > it). > > > But I have quite a lot of knowledge about NARs, so I thought I'd point > > out > > > the key features/design points. OSGi may or may not cover all of these > > > concepts, but if we are thinking about OSGi as a possible alternative > > (or a > > > parallel), I think it's good to go ahead and call out these concepts. > > > NARs map directly to a Java ClassLoader. Whatever your NAR depends on > in > > > its pom is what will be loaded into the Java ClassLoader. This makes > > > understanding ClassLoading issues much easier, I think. > > > A NAR can declare a "Parent NAR." That means that the parent of its > > > ClassLoader will be the ClassLoader of the parent NAR. > > > NARs can also provide a 'native' folder that houses native libraries. > > This > > > provides native library isolation as well as class loading isolation. > > > You are able to define a Controller Service interface in a > 'service-api' > > > nar and then have that NAR be the parent of say my-processor-nar and > > > my-service-nar. The Processor can then call methods in the Controller > > > Service via the interface. Even though the implementation lives in > > > my-service-nar, we can cross that NAR boundary safely since we only > > > interact with the interface (which is defined in the common parent). > > > > > > > Date: Sun, 14 Dec 2014 18:44:05 -0500 > > > > Subject: Re: nar plugin > > > > From: [email protected] > > > > To: [email protected] > > > > > > > > I wouldn't reject Maven Central off-hand. > > > > > > > > And I wouldn't reject keeping your own maven plugin and packaging, > even > > > if > > > > it's a nearly complete fork of the maven-war-plugin. > > > > > > > > I think that Maven's handling of 'packaging' and such is less than > > delux, > > > > and leads in this direction. You'd like to be able to take advantage > of > > > the > > > > underlying function of the war plugin (or the assembly plugin) and > > still > > > > get DRY by avoiding having your devs need to copy a complex assembly > > > > descriptor or do something funny-looking with the war plugin, but I > > don't > > > > know a way. So if you want a nice, clean, pom for 'build me a NiFi > > > plugin', > > > > you end up, I suspect, with .. a maven plugin. > > > > > > > > I've recently stuck my toes into the OSGi waters at work; I have a > > theory > > > > of how to have an OSGi system while still cleaving closely to the > Maven > > > > dependency universe so as to avoid ending up adrift in OBR. The > theory > > is > > > > neither completely worked out nor proven in production, but I could > > write > > > > it up such as it is. I'm still digesting some ideas from the Felix > user > > > > list. > > > > > > > > Basically, I think of OSGi as a generalization of 'war', and then I > am > > > > trying to work out a low-labor way to keep track of all the > > dependencies. > > > > > > > > > > > > On Sun, Dec 14, 2014 at 6:30 PM, Tony Kurc <[email protected]> wrote: > > > > > > > > > there seem to be plenty of wars which include dependencies in > > > > > repository.apache.org > > > > > > > > > > On Sun, Dec 14, 2014 at 5:59 PM, Sean Busbey <[email protected]> > > > wrote: > > > > > > > > > > > > On Dec 14, 2014 4:25 PM, "Joe Witt" <[email protected]> wrote: > > > > > > > > > > > > > > OSGi was the most obvious choice at the time but we stayed away > > > due to > > > > > > the > > > > > > > tooling/cost/benefit. > > > > > > > > > > > > > > > > > > > Might be worth revisiting. I know the specter of OSGi is raised > > every > > > > > time > > > > > > classpath woes surface in Hadoop-land. > > > > > > > > > > > > > Our simple goal was for those building extensions to easily > > produce > > > > > > bundles > > > > > > > of their code and associated dependencies. > > > > > > > > > > > > > > Building uber jars was generally considered non-desirable > because > > > we > > > > > did > > > > > > > not want to change the structure of the extension or dependent > > > jars. > > > > > > > > > > > > > > > > > > > +1 for continuing to avoid uber jars. > > > > > > > > > > > > > The assembly plugin might in fact be totally sufficient for our > > > > > purposes. > > > > > > > It requires some extra steps for the client but those are > > probably > > > > > > > completely automated by maven archetypes. > > > > > > > > > > > > > > > > > > > Reworking things so that nar becomes a kind of assembly is an > > > interesting > > > > > > idea. Do we have docs on the expectations the runtime has for a > > nar? > > > > > > > > > > > > > In any event we recognize the current nar concept is > problematic > > > > > because > > > > > > > nobody is going to want us publishing nar artifact bundles to > > maven > > > > > > repos. > > > > > > > So we're looking at what we should do. > > > > > > > > > > > > > > > > > > > I don't see why publishing nars would be a problem. Other > projects > > > > > publish > > > > > > tarballs (source and binary) and things go well. > > > > > > > > > > > > > > > > > > > >
