On Wed, Nov 4, 2015 at 9:07 AM, Romain Gilles <[email protected]> wrote:
> Hi Dmitriy, > I found this post that explain how to find a bundle based on its bundle > name and version. > This post explain for the past to now how to do it in the standard way with > "pull" approach: https://www.eclipse.org/forums/index.php/t/205719/ > Regarding the version of OSGi you want to support then some solutions will > works some others will not. > There is an other way to do this stuff without using those "pull" style > approach based on BundleTracker. If you want I can give you the code to do > it with BundlTracker. I think with this solution you will support a wider > range of OSGi version. > Romain, if you can provide a generic code sample to look up a ClassLoader in OSGI based on manifest properties, would be great. > > Le mer. 4 nov. 2015 à 04:07, Dmitriy Setrakyan <[email protected]> a > écrit : > > > Andrey, et al, > > > > Can you provide a way on how to get a required Bundle object based on, > say, > > bundle symbolic name and version? > > > > I have reached the end of the internet trying to find a way in OSGI to > look > > up a Bundle based on something other than an actual Class belonging to > that > > bundle, but no luck. > > > > D. > > > > On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <[email protected]> > > wrote: > > > > > Dmitriy, > > > > > > I think your approach will work, but I let Romain respond. > > > > > > Also, in terms of the implementation, please keep in mind that the > > > resolver must be called for each non-JDK and non-Ignite core class (it > > > would probably make sense to eschew storing class loaders for such > > classes > > > in favor of compactness of the serialized representation -- see below). > > > Also, it's worth keeping a cache of already resolved class loaders per > > > marshaller invocation (this is probably the context that Romain has > > > mentioned in his previous posting) to minimize the number of the > resolver > > > calls. > > > > > > In terms of the resolver's implementation, the simplest way to > serialize > > > the class loader would be by capturing two pieces of information (both > > > strings): the bundle symbolic name and the bundle version. This > approach > > > however may result in bloating of the serialized representation: I'd > > > roughly estimate the overhead per element to be at least 20-30 bytes > (the > > > length of the symbolic name string, plus the length of the version > > string). > > > There are way to reduce the overhead (like serializing the hash code of > > the > > > bundle id string rather than the string itself, and then come up with a > > > clever way of resolving the hash collisions), but they all come at cost > > of > > > increased complexity... > > > > > > An alternative approach would be rely on the special bundle id which is > > an > > > integer and is generated by the OSGi container. But in this case, all > > nodes > > > must ensure that all the bundles have consistent ids (bundle A with id > 42 > > > on node N1, has the same id on every other node) which is not easy -- > > while > > > not entirely impossible -- to guarantee. As long as the nodes are > > > homogeneous (have the same set of bundles deployed) the OSGi container > is > > > guaranteed to assign to the bundles the same ids. > > > > > > Thanks > > > Andrey > > > > > > > From: [email protected] > > > > Date: Tue, 3 Nov 2015 16:29:41 -0800 > > > > Subject: Re: OSGi migration may require a special marshaller > > > > To: [email protected] > > > > > > > > Romain, > > > > > > > > In the upcoming release we will be deprecating the > OptimizedMarshaller > > > and > > > > will be switching to a default internal marshaller (which is based on > > the > > > > new PortableMarshaller donated by GridGain). > > > > > > > > Having said that, we may be able to pass BinaryWriter and > BinaryReader > > > > instead of byte arrays. This will be pretty close to passing the > > stream, > > > as > > > > suggested by Andrey. > > > > > > > > Also, I still think that we should only resolve class loaders and not > > the > > > > class itself. The main reason is that Ignite will encode class names > > into > > > > an integer hash code and will store the integer->class-fqn mapping in > > > > internal replicated cache. I doubt users will get more efficient than > > an > > > > integer (4 bytes) for a class name. > > > > > > > > On the receiving side, once we are able to get the right class > loader, > > we > > > > can easily get the proper class by calling > > > ClassLoader.findClass(class-fqn). > > > > > > > > Thoughts? > > > > > > > > D. > > > > > > > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles < > [email protected] > > > > > > > wrote: > > > > > > > > > Hi, > > > > > Maybe a missing point. I think but I'm not sure that in the > > > > > OptimizedMarshaller there is a caching of already serialized > classes. > > > Due > > > > > to the dynamic nature of OSGi this may lead to memory leak. In fact > > if > > > a > > > > > bundle is refreshed, it will produce a new BundleRevision and > > > therefore a > > > > > new classloader. And if you don't release the class from the > previous > > > > > BundleRevision then you endup with memory leak. So maybe the > > Marshaller > > > > > interface or somewhere should provide a way to free those classes / > > > > > classloaders. > > > > > > > > > > Regards, > > > > > > > > > > Romain > > > > > > > > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev < > [email protected] > > > > > > a > > > > > écrit : > > > > > > > > > > > Romain/Dmitriy, > > > > > > > > > > > > I prefer Romain's approach, but just curious, in the API you guys > > are > > > > > > proposing why use a byte[] rather than OutputStream/InputStream? > > > With a > > > > > > byte[], one would inevitably end up wrapping it into a byte > stream > > > class. > > > > > > Also, the stream-based interface would be more in line with the > > > > > Marshaller > > > > > > API. > > > > > > > > > > > > Also for symmetry with the readClass() method, I suggest the > > > writeClass() > > > > > > take a Class<?> rather than an object. > > > > > > > > > > > > Regards > > > > > > Andrey > > > > > > > > > > > > > From: [email protected] > > > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000 > > > > > > > Subject: Re: OSGi migration may require a special marshaller > > > > > > > To: [email protected] > > > > > > > > > > > > > > Hi Dmitriy, > > > > > > > I think your solution is good. Maybe I will change it a little > > > bit... > > > > > :P > > > > > > > I think you should delegate the Class resolution to the > resolver. > > > > > Because > > > > > > > for example with your current solution the marshaller may > before > > > (or > > > > > > after) > > > > > > > store the fqn of the class (maybe only the first time it > > encounter > > > it) > > > > > > but > > > > > > > in order to save the classloader context resolution the > > > implementation > > > > > of > > > > > > > the resolver may have to save the package name of the given > > object > > > and > > > > > > some > > > > > > > extra info therefore the content package name will be > serialized > > > two > > > > > > times. > > > > > > > Ok, it's not a big deal. > > > > > > > But now if the resolver identify that it can reuse the same > class > > > > > loader > > > > > > > for a couple of classes. It will be interesting for it to have > a > > > > > > > serialization context in order to save, let say, classloader/id > > > mapping > > > > > > in > > > > > > > order to save the id instead of a more longer representation. > > > > > > > So I propose something like that: > > > > > > > *public interface ClassResolver {* > > > > > > > * // This method is invoked on the sender side to * > > > > > > > * // marshal the information about the class.* > > > > > > > * // where the context is a map style object that is > reset/new > > > for > > > > > > each > > > > > > > object graph serialization.* > > > > > > > * public byte[] writeClass(Object o, Context context) throws > > > > > > > IgniteException;* > > > > > > > > > > > > > > * // This method is invoked on the receiving side to* > > > > > > > * // retrieve the class based on provided information.* > > > > > > > * // where the context is a map style object that is > reset/new > > > for > > > > > > each > > > > > > > object graph serialization.* > > > > > > > * public Class<?> readClass(byte[], Context context) throws > > > > > > > IgniteException;* > > > > > > > *}* > > > > > > > > > > > > > > I think your proposal will solve our issue and maybe also open > a > > > door > > > > > for > > > > > > > the osgi development. > > > > > > > Let me know what do you think about me proposal? :) > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Romain > > > > > > > > > > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan < > > > [email protected]> > > > > > a > > > > > > > écrit : > > > > > > > > > > > > > > > Romain, > > > > > > > > > > > > > > > > Can you comment on the ClassLoaderResolver suggestion that I > > > proposed > > > > > > > > earlier? Will it solve your problem? > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Raul, > > > > > > > > > - Do you plan to use the TCCL when marshalling in OSGi > env? > > > If yes > > > > > > how? > > > > > > > > > - I like the point 2. Maybe for a graph of object that > come > > > from > > > > > > > > different > > > > > > > > > packages / bundles you may have to recursively capture the > > > package > > > > > > > > version > > > > > > > > > of the individual element of your graph. > > > > > > > > > - For the point 3 I wonder if it will not over-complexify > > the > > > > > > solution. > > > > > > > > As > > > > > > > > > a developer you will have to think about it. And it is not > > > obvious > > > > > > in the > > > > > > > > > code where things are serialized or not. You may use lambda > > in > > > your > > > > > > > > > application code therefore the current package become what > > you > > > call > > > > > > a DTO > > > > > > > > > package. The current state of ignite does not enforce you > to > > > > > specify > > > > > > it > > > > > > > > for > > > > > > > > > "classical" classloading application. If you introduce this > > > extra > > > > > > step > > > > > > > > for > > > > > > > > > OSGi ready application you will maybe discourage people to > > use > > > > > OSGi. > > > > > > > > > > > > > > > > > > To comeback to our solution. We start we a strong > assumption: > > > our > > > > > > cluster > > > > > > > > > is homogeneous in term of code so, of course, it simplify > our > > > life > > > > > > :). > > > > > > > > > > > > > > > > > > BTW if you can introduce an extension point in the class > > > resolution > > > > > > > > > mechanism it can be interesting for end user in order to > > allow > > > them > > > > > > to > > > > > > > > > optimize it based on there specific use cases. > > > > > > > > > > > > > > > > > > Romain. > > > > > > > > > > > > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani < > > [email protected]> > > > a > > > > > > écrit : > > > > > > > > > > > > > > > > > > > Hi Andrey, > > > > > > > > > > > > > > > > > > > > Thanks for the participation in this topic. > > > > > > > > > > > > > > > > > > > > I don't like the solution to incorporate the bundle > > symbolic > > > name > > > > > > in > > > > > > > > the > > > > > > > > > > serialised form. Nothing guarantees that the classdef > will > > be > > > > > > located > > > > > > > > > under > > > > > > > > > > the same bundle in both source and target machines. We > also > > > have > > > > > to > > > > > > > > take > > > > > > > > > > into account that OSGi allows for multiple versions of > the > > > same > > > > > > > > > > bundle/packages to co-exist in the same container. So it > > > becomes > > > > > > more > > > > > > > > > > complex. > > > > > > > > > > > > > > > > > > > > Using the TCCL should work when serialising, but it'll > > > probably > > > > > be > > > > > > of > > > > > > > > no > > > > > > > > > > use when deserialising on the other end. > > > > > > > > > > > > > > > > > > > > I need to enhance Ignite to: > > > > > > > > > > > > > > > > > > > > 1. Use the TCCL when marshalling on one end. > > > > > > > > > > 2. Incorporate the package version of the class in the > > > serialised > > > > > > form > > > > > > > > if > > > > > > > > > > Ignite is running in an OSGi environment. > > > > > > > > > > 3. On the receiver end, discover cache entities / DTOs in > > all > > > > > > bundles > > > > > > > > > > through a custom OSGi manifest header or the like, as I > > > explained > > > > > > > > before. > > > > > > > > > > Package version must be taken into account. > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > Raúl. > > > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Raul, > > > > > > > > > > > > > > > > > > > > > > The fundamental hurdle we need to jump over to make > > Ignite > > > > > > > > OSGi-enabled > > > > > > > > > > is > > > > > > > > > > > the marshalling. More specifically the issue is with > > > > > > deserialization > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > > classes that are provided by the bundles other than the > > > Ignite > > > > > > bundle > > > > > > > > > > > itself. > > > > > > > > > > > > > > > > > > > > > > When the Ignite transport layer receives a message it > > > needs to > > > > > > figure > > > > > > > > > out > > > > > > > > > > > how to deserialize the bytes and for that it needs to > > know > > > the > > > > > > bundle > > > > > > > > > > that > > > > > > > > > > > provides the class to be deserailized. At this point > TCCL > > > is of > > > > > > no > > > > > > > > use. > > > > > > > > > > To > > > > > > > > > > > make things more complex, the class may contain other > > > classes > > > > > > that > > > > > > > > come > > > > > > > > > > > from other bundles, and so on recursively. This means > > that > > > each > > > > > > > > object > > > > > > > > > in > > > > > > > > > > > the hierarchy must be serialized with its bundle name > (or > > > > > bundle > > > > > > id), > > > > > > > > > so > > > > > > > > > > > that the deserializer will then be able to correctly > > > resolve > > > > > the > > > > > > > > class > > > > > > > > > > > while traversing the object hierarchy during > > > deserialization. > > > > > > > > > > > > > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking > > the > > > > > > ability to > > > > > > > > > > plug > > > > > > > > > > > in a custom class resolver. Romain's solution was to > use > > > Kryo > > > > > > that > > > > > > > > does > > > > > > > > > > > provide a way to customize class resolution. It has > > solved > > > the > > > > > > > > problem > > > > > > > > > of > > > > > > > > > > > capturing the bundle info and he was able to > successfully > > > run > > > > > > Ignite > > > > > > > > > as a > > > > > > > > > > > bundle in an OSGi container (after some repackaging and > > > > > > inclusion of > > > > > > > > > the > > > > > > > > > > > manifest). But Kryo-based marshalling introduced a lot > of > > > > > > complexity > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > code and incorrect use of Kryo's numerous serializers > > > caused > > > > > some > > > > > > > > weird > > > > > > > > > > > hard-to-debug issues in the Ignite core (like duplicate > > > cache > > > > > > entries > > > > > > > > > due > > > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap > > > class -- > > > > > > go > > > > > > > > > > > figure!). Overall the Kryo-based solution is brittle > and > > > hard > > > > > to > > > > > > > > > > maintain. > > > > > > > > > > > > > > > > > > > > > > I feel the correct solution to OSGi problem would be to > > > > > > > > > > > 1) enhance the OptimizedMarshaller to allow custom > class > > > > > > resolution. > > > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in > > > addition to > > > > > > the > > > > > > > > > > > original one) to be used in OSGi environment. > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > > > > From: [email protected] > > > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000 > > > > > > > > > > > > Subject: Re: OSGi migration may require a special > > > marshaller > > > > > > > > > > > > To: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > Hi Romain, > > > > > > > > > > > > > > > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I > > > appreciate > > > > > > your > > > > > > > > > > input. > > > > > > > > > > > > > > > > > > > > > > > > I'm thinking about the situation you describe and I > > > wonder if > > > > > > > > you're > > > > > > > > > > > > exporting Ignite as an OSGi service which is then > > > consumed > > > > > from > > > > > > > > other > > > > > > > > > > > > bundles. Under this situation, it would be quite easy > > to > > > > > > reproduce > > > > > > > > > the > > > > > > > > > > > > behaviour you describe if Ignite is not resolving > > > classes via > > > > > > the > > > > > > > > > TCCL. > > > > > > > > > > > > Need to dig deeper into that. > > > > > > > > > > > > > > > > > > > > > > > > Off the top of my head, there are two alternatives to > > > solve > > > > > it: > > > > > > > > > > > > > > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not > > > already > > > > > > > > used) – > > > > > > > > > > we > > > > > > > > > > > > gotta be wary of possible regressions. > > > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package > > so > > > that > > > > > > > > bundles > > > > > > > > > > > > containing DTOs can expose packages to Ignite's > > > marshallers. > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > > > *Raúl Kripalani* > > > > > > > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | > > > Integration, > > > > > > Big > > > > > > > > Data > > > > > > > > > > and > > > > > > > > > > > > Messaging Engineer > > > > > > > > > > > > http://about.me/raulkripalani | > > > > > > > > > > http://www.linkedin.com/in/raulkripalani > > > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain < > > > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > I'm really interested in this issue: > > > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 > . > > We > > > > > some > > > > > > > > stuff > > > > > > > > > to > > > > > > > > > > > make > > > > > > > > > > > > > it work in our osgi environment. The main issue for > > us > > > now > > > > > > is the > > > > > > > > > > > > > serialization. I think it you will have to rework > the > > > > > > > > > > > OptimizedMarshaller > > > > > > > > > > > > > or any other marshaller that works with object that > > > come > > > > > from > > > > > > > > > outside > > > > > > > > > > > your > > > > > > > > > > > > > class space. > > > > > > > > > > > > > > > > > > > > > > > > > > We have try kryo that works. Kryo provide an > > extension > > > > > point > > > > > > in > > > > > > > > > order > > > > > > > > > > > to > > > > > > > > > > > > > resolve the classes: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java > > > > > > > > > > > > > . With this extension we are able to solve the > > problem > > > of > > > > > > > > external > > > > > > > > > > > classes. > > > > > > > > > > > > > The only issue with kryo is that some classes need > a > > > > > certain > > > > > > care > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > serialization process and therefore a specialized > > > > > serializer. > > > > > > > > > > > > > > > > > > > > > > > > > > So I would like to know from the community what do > > > think of > > > > > > > > > changing > > > > > > > > > > > the > > > > > > > > > > > > > way the optimized marshaller works or introducing > the > > > > > > support of > > > > > > > > > yet > > > > > > > > > > > > > another marshaller based on a kryo like technology? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Romain. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > PS: I'm ready to help in the both cases. > > > > > > > > > > > > > "Misys" is the trade name of the Misys group of > > > companies. > > > > > > This > > > > > > > > > email > > > > > > > > > > > and > > > > > > > > > > > > > any attachments have been scanned for known viruses > > > using > > > > > > > > multiple > > > > > > > > > > > > > scanners. This email message is intended for the > > named > > > > > > recipient > > > > > > > > > > only. > > > > > > > > > > > It > > > > > > > > > > > > > may be privileged and/or confidential. If you are > not > > > the > > > > > > named > > > > > > > > > > > recipient > > > > > > > > > > > > > of this email please notify us immediately and do > not > > > copy > > > > > > it or > > > > > > > > > use > > > > > > > > > > > it for > > > > > > > > > > > > > any purpose, nor disclose its contents to any other > > > person. > > > > > > This > > > > > > > > > > email > > > > > > > > > > > does > > > > > > > > > > > > > not constitute the commencement of legal relations > > > between > > > > > > you > > > > > > > > and > > > > > > > > > > > Misys. > > > > > > > > > > > > > Please refer to the executed contract between you > and > > > the > > > > > > > > relevant > > > > > > > > > > > member > > > > > > > > > > > > > of the Misys group for the identity of the > > contracting > > > > > party > > > > > > with > > > > > > > > > > > which you > > > > > > > > > > > > > are dealing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
