Could we rewind a little bit please? What is the point of using the bundle symbolic name in the first place?
In OSGi, it is bad practice to refer to bundles directly. A bundle is a packaging unit, that's all, but it should not have any further consideration. It is perfectly feasible that a given package is exported by bundle A in a container, and by bundle B in another. One should not make any assumptions in that regard. A simple example of this is an enterprise-wide deployment of Ignite, where multiple apps in different OSGi containers share cache entities. Why would one expect these entities to be hosted in the same bundle across all apps? Referring to a bundle in OSGi world is like referring to a JAR in a normal Java environment. One would not always expect to resolve class A.B.C from bundle xyz-1.2.3.jar, right? You would resolve from the classpath. There are other mechanisms to reach into the OSGi environment and resolve a class arbitrarily from another bundle. Some of which I've pointed out. But relying on a bundle symbolic name receives an -1 from me. Can't elaborate now, I'm travelling. Will get back to my office tomorrow and I'll dig deeper into possibilities. Regards, Raúl. On 4 Nov 2015 03:07, "Dmitriy Setrakyan" <[email protected]> wrote: > Andrey, et al, > > Can you provide a way on how to get a required Bundle object based on, say, > bundle symbolic name and version? > > I have reached the end of the internet trying to find a way in OSGI to look > up a Bundle based on something other than an actual Class belonging to that > bundle, but no luck. > > D. > > On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <[email protected]> > wrote: > > > Dmitriy, > > > > I think your approach will work, but I let Romain respond. > > > > Also, in terms of the implementation, please keep in mind that the > > resolver must be called for each non-JDK and non-Ignite core class (it > > would probably make sense to eschew storing class loaders for such > classes > > in favor of compactness of the serialized representation -- see below). > > Also, it's worth keeping a cache of already resolved class loaders per > > marshaller invocation (this is probably the context that Romain has > > mentioned in his previous posting) to minimize the number of the resolver > > calls. > > > > In terms of the resolver's implementation, the simplest way to serialize > > the class loader would be by capturing two pieces of information (both > > strings): the bundle symbolic name and the bundle version. This approach > > however may result in bloating of the serialized representation: I'd > > roughly estimate the overhead per element to be at least 20-30 bytes (the > > length of the symbolic name string, plus the length of the version > string). > > There are way to reduce the overhead (like serializing the hash code of > the > > bundle id string rather than the string itself, and then come up with a > > clever way of resolving the hash collisions), but they all come at cost > of > > increased complexity... > > > > An alternative approach would be rely on the special bundle id which is > an > > integer and is generated by the OSGi container. But in this case, all > nodes > > must ensure that all the bundles have consistent ids (bundle A with id 42 > > on node N1, has the same id on every other node) which is not easy -- > while > > not entirely impossible -- to guarantee. As long as the nodes are > > homogeneous (have the same set of bundles deployed) the OSGi container is > > guaranteed to assign to the bundles the same ids. > > > > Thanks > > Andrey > > > > > From: [email protected] > > > Date: Tue, 3 Nov 2015 16:29:41 -0800 > > > Subject: Re: OSGi migration may require a special marshaller > > > To: [email protected] > > > > > > Romain, > > > > > > In the upcoming release we will be deprecating the OptimizedMarshaller > > and > > > will be switching to a default internal marshaller (which is based on > the > > > new PortableMarshaller donated by GridGain). > > > > > > Having said that, we may be able to pass BinaryWriter and BinaryReader > > > instead of byte arrays. This will be pretty close to passing the > stream, > > as > > > suggested by Andrey. > > > > > > Also, I still think that we should only resolve class loaders and not > the > > > class itself. The main reason is that Ignite will encode class names > into > > > an integer hash code and will store the integer->class-fqn mapping in > > > internal replicated cache. I doubt users will get more efficient than > an > > > integer (4 bytes) for a class name. > > > > > > On the receiving side, once we are able to get the right class loader, > we > > > can easily get the proper class by calling > > ClassLoader.findClass(class-fqn). > > > > > > Thoughts? > > > > > > D. > > > > > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <[email protected] > > > > > wrote: > > > > > > > Hi, > > > > Maybe a missing point. I think but I'm not sure that in the > > > > OptimizedMarshaller there is a caching of already serialized classes. > > Due > > > > to the dynamic nature of OSGi this may lead to memory leak. In fact > if > > a > > > > bundle is refreshed, it will produce a new BundleRevision and > > therefore a > > > > new classloader. And if you don't release the class from the previous > > > > BundleRevision then you endup with memory leak. So maybe the > Marshaller > > > > interface or somewhere should provide a way to free those classes / > > > > classloaders. > > > > > > > > Regards, > > > > > > > > Romain > > > > > > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <[email protected] > > > > a > > > > écrit : > > > > > > > > > Romain/Dmitriy, > > > > > > > > > > I prefer Romain's approach, but just curious, in the API you guys > are > > > > > proposing why use a byte[] rather than OutputStream/InputStream? > > With a > > > > > byte[], one would inevitably end up wrapping it into a byte stream > > class. > > > > > Also, the stream-based interface would be more in line with the > > > > Marshaller > > > > > API. > > > > > > > > > > Also for symmetry with the readClass() method, I suggest the > > writeClass() > > > > > take a Class<?> rather than an object. > > > > > > > > > > Regards > > > > > Andrey > > > > > > > > > > > From: [email protected] > > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000 > > > > > > Subject: Re: OSGi migration may require a special marshaller > > > > > > To: [email protected] > > > > > > > > > > > > Hi Dmitriy, > > > > > > I think your solution is good. Maybe I will change it a little > > bit... > > > > :P > > > > > > I think you should delegate the Class resolution to the resolver. > > > > Because > > > > > > for example with your current solution the marshaller may before > > (or > > > > > after) > > > > > > store the fqn of the class (maybe only the first time it > encounter > > it) > > > > > but > > > > > > in order to save the classloader context resolution the > > implementation > > > > of > > > > > > the resolver may have to save the package name of the given > object > > and > > > > > some > > > > > > extra info therefore the content package name will be serialized > > two > > > > > times. > > > > > > Ok, it's not a big deal. > > > > > > But now if the resolver identify that it can reuse the same class > > > > loader > > > > > > for a couple of classes. It will be interesting for it to have a > > > > > > serialization context in order to save, let say, classloader/id > > mapping > > > > > in > > > > > > order to save the id instead of a more longer representation. > > > > > > So I propose something like that: > > > > > > *public interface ClassResolver {* > > > > > > * // This method is invoked on the sender side to * > > > > > > * // marshal the information about the class.* > > > > > > * // where the context is a map style object that is reset/new > > for > > > > > each > > > > > > object graph serialization.* > > > > > > * public byte[] writeClass(Object o, Context context) throws > > > > > > IgniteException;* > > > > > > > > > > > > * // This method is invoked on the receiving side to* > > > > > > * // retrieve the class based on provided information.* > > > > > > * // where the context is a map style object that is reset/new > > for > > > > > each > > > > > > object graph serialization.* > > > > > > * public Class<?> readClass(byte[], Context context) throws > > > > > > IgniteException;* > > > > > > *}* > > > > > > > > > > > > I think your proposal will solve our issue and maybe also open a > > door > > > > for > > > > > > the osgi development. > > > > > > Let me know what do you think about me proposal? :) > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Romain > > > > > > > > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan < > > [email protected]> > > > > a > > > > > > écrit : > > > > > > > > > > > > > Romain, > > > > > > > > > > > > > > Can you comment on the ClassLoaderResolver suggestion that I > > proposed > > > > > > > earlier? Will it solve your problem? > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Raul, > > > > > > > > - Do you plan to use the TCCL when marshalling in OSGi env? > > If yes > > > > > how? > > > > > > > > - I like the point 2. Maybe for a graph of object that come > > from > > > > > > > different > > > > > > > > packages / bundles you may have to recursively capture the > > package > > > > > > > version > > > > > > > > of the individual element of your graph. > > > > > > > > - For the point 3 I wonder if it will not over-complexify > the > > > > > solution. > > > > > > > As > > > > > > > > a developer you will have to think about it. And it is not > > obvious > > > > > in the > > > > > > > > code where things are serialized or not. You may use lambda > in > > your > > > > > > > > application code therefore the current package become what > you > > call > > > > > a DTO > > > > > > > > package. The current state of ignite does not enforce you to > > > > specify > > > > > it > > > > > > > for > > > > > > > > "classical" classloading application. If you introduce this > > extra > > > > > step > > > > > > > for > > > > > > > > OSGi ready application you will maybe discourage people to > use > > > > OSGi. > > > > > > > > > > > > > > > > To comeback to our solution. We start we a strong assumption: > > our > > > > > cluster > > > > > > > > is homogeneous in term of code so, of course, it simplify our > > life > > > > > :). > > > > > > > > > > > > > > > > BTW if you can introduce an extension point in the class > > resolution > > > > > > > > mechanism it can be interesting for end user in order to > allow > > them > > > > > to > > > > > > > > optimize it based on there specific use cases. > > > > > > > > > > > > > > > > Romain. > > > > > > > > > > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani < > [email protected]> > > a > > > > > écrit : > > > > > > > > > > > > > > > > > Hi Andrey, > > > > > > > > > > > > > > > > > > Thanks for the participation in this topic. > > > > > > > > > > > > > > > > > > I don't like the solution to incorporate the bundle > symbolic > > name > > > > > in > > > > > > > the > > > > > > > > > serialised form. Nothing guarantees that the classdef will > be > > > > > located > > > > > > > > under > > > > > > > > > the same bundle in both source and target machines. We also > > have > > > > to > > > > > > > take > > > > > > > > > into account that OSGi allows for multiple versions of the > > same > > > > > > > > > bundle/packages to co-exist in the same container. So it > > becomes > > > > > more > > > > > > > > > complex. > > > > > > > > > > > > > > > > > > Using the TCCL should work when serialising, but it'll > > probably > > > > be > > > > > of > > > > > > > no > > > > > > > > > use when deserialising on the other end. > > > > > > > > > > > > > > > > > > I need to enhance Ignite to: > > > > > > > > > > > > > > > > > > 1. Use the TCCL when marshalling on one end. > > > > > > > > > 2. Incorporate the package version of the class in the > > serialised > > > > > form > > > > > > > if > > > > > > > > > Ignite is running in an OSGi environment. > > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in > all > > > > > bundles > > > > > > > > > through a custom OSGi manifest header or the like, as I > > explained > > > > > > > before. > > > > > > > > > Package version must be taken into account. > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > Raúl. > > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" < > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > Raul, > > > > > > > > > > > > > > > > > > > > The fundamental hurdle we need to jump over to make > Ignite > > > > > > > OSGi-enabled > > > > > > > > > is > > > > > > > > > > the marshalling. More specifically the issue is with > > > > > deserialization > > > > > > > of > > > > > > > > > the > > > > > > > > > > classes that are provided by the bundles other than the > > Ignite > > > > > bundle > > > > > > > > > > itself. > > > > > > > > > > > > > > > > > > > > When the Ignite transport layer receives a message it > > needs to > > > > > figure > > > > > > > > out > > > > > > > > > > how to deserialize the bytes and for that it needs to > know > > the > > > > > bundle > > > > > > > > > that > > > > > > > > > > provides the class to be deserailized. At this point TCCL > > is of > > > > > no > > > > > > > use. > > > > > > > > > To > > > > > > > > > > make things more complex, the class may contain other > > classes > > > > > that > > > > > > > come > > > > > > > > > > from other bundles, and so on recursively. This means > that > > each > > > > > > > object > > > > > > > > in > > > > > > > > > > the hierarchy must be serialized with its bundle name (or > > > > bundle > > > > > id), > > > > > > > > so > > > > > > > > > > that the deserializer will then be able to correctly > > resolve > > > > the > > > > > > > class > > > > > > > > > > while traversing the object hierarchy during > > deserialization. > > > > > > > > > > > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking > the > > > > > ability to > > > > > > > > > plug > > > > > > > > > > in a custom class resolver. Romain's solution was to use > > Kryo > > > > > that > > > > > > > does > > > > > > > > > > provide a way to customize class resolution. It has > solved > > the > > > > > > > problem > > > > > > > > of > > > > > > > > > > capturing the bundle info and he was able to successfully > > run > > > > > Ignite > > > > > > > > as a > > > > > > > > > > bundle in an OSGi container (after some repackaging and > > > > > inclusion of > > > > > > > > the > > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot of > > > > > complexity > > > > > > > to > > > > > > > > > the > > > > > > > > > > code and incorrect use of Kryo's numerous serializers > > caused > > > > some > > > > > > > weird > > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate > > cache > > > > > entries > > > > > > > > due > > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap > > class -- > > > > > go > > > > > > > > > > figure!). Overall the Kryo-based solution is brittle and > > hard > > > > to > > > > > > > > > maintain. > > > > > > > > > > > > > > > > > > > > I feel the correct solution to OSGi problem would be to > > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom class > > > > > resolution. > > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in > > addition to > > > > > the > > > > > > > > > > original one) to be used in OSGi environment. > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000 > > > > > > > > > > > Subject: Re: OSGi migration may require a special > > marshaller > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > > > > > Hi Romain, > > > > > > > > > > > > > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I > > appreciate > > > > > your > > > > > > > > > input. > > > > > > > > > > > > > > > > > > > > > > I'm thinking about the situation you describe and I > > wonder if > > > > > > > you're > > > > > > > > > > > exporting Ignite as an OSGi service which is then > > consumed > > > > from > > > > > > > other > > > > > > > > > > > bundles. Under this situation, it would be quite easy > to > > > > > reproduce > > > > > > > > the > > > > > > > > > > > behaviour you describe if Ignite is not resolving > > classes via > > > > > the > > > > > > > > TCCL. > > > > > > > > > > > Need to dig deeper into that. > > > > > > > > > > > > > > > > > > > > > > Off the top of my head, there are two alternatives to > > solve > > > > it: > > > > > > > > > > > > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not > > already > > > > > > > used) – > > > > > > > > > we > > > > > > > > > > > gotta be wary of possible regressions. > > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package > so > > that > > > > > > > bundles > > > > > > > > > > > containing DTOs can expose packages to Ignite's > > marshallers. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > *Raúl Kripalani* > > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | > > Integration, > > > > > Big > > > > > > > Data > > > > > > > > > and > > > > > > > > > > > Messaging Engineer > > > > > > > > > > > http://about.me/raulkripalani | > > > > > > > > > http://www.linkedin.com/in/raulkripalani > > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain < > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > I'm really interested in this issue: > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . > We > > > > some > > > > > > > stuff > > > > > > > > to > > > > > > > > > > make > > > > > > > > > > > > it work in our osgi environment. The main issue for > us > > now > > > > > is the > > > > > > > > > > > > serialization. I think it you will have to rework the > > > > > > > > > > OptimizedMarshaller > > > > > > > > > > > > or any other marshaller that works with object that > > come > > > > from > > > > > > > > outside > > > > > > > > > > your > > > > > > > > > > > > class space. > > > > > > > > > > > > > > > > > > > > > > > > We have try kryo that works. Kryo provide an > extension > > > > point > > > > > in > > > > > > > > order > > > > > > > > > > to > > > > > > > > > > > > resolve the classes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java > > > > > > > > > > > > . With this extension we are able to solve the > problem > > of > > > > > > > external > > > > > > > > > > classes. > > > > > > > > > > > > The only issue with kryo is that some classes need a > > > > certain > > > > > care > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > serialization process and therefore a specialized > > > > serializer. > > > > > > > > > > > > > > > > > > > > > > > > So I would like to know from the community what do > > think of > > > > > > > > changing > > > > > > > > > > the > > > > > > > > > > > > way the optimized marshaller works or introducing the > > > > > support of > > > > > > > > yet > > > > > > > > > > > > another marshaller based on a kryo like technology? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Romain. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > PS: I'm ready to help in the both cases. > > > > > > > > > > > > "Misys" is the trade name of the Misys group of > > companies. > > > > > This > > > > > > > > email > > > > > > > > > > and > > > > > > > > > > > > any attachments have been scanned for known viruses > > using > > > > > > > multiple > > > > > > > > > > > > scanners. This email message is intended for the > named > > > > > recipient > > > > > > > > > only. > > > > > > > > > > It > > > > > > > > > > > > may be privileged and/or confidential. If you are not > > the > > > > > named > > > > > > > > > > recipient > > > > > > > > > > > > of this email please notify us immediately and do not > > copy > > > > > it or > > > > > > > > use > > > > > > > > > > it for > > > > > > > > > > > > any purpose, nor disclose its contents to any other > > person. > > > > > This > > > > > > > > > email > > > > > > > > > > does > > > > > > > > > > > > not constitute the commencement of legal relations > > between > > > > > you > > > > > > > and > > > > > > > > > > Misys. > > > > > > > > > > > > Please refer to the executed contract between you and > > the > > > > > > > relevant > > > > > > > > > > member > > > > > > > > > > > > of the Misys group for the identity of the > contracting > > > > party > > > > > with > > > > > > > > > > which you > > > > > > > > > > > > are dealing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
