Romain/Dmitriy, I prefer Romain's approach, but just curious, in the API you guys are proposing why use a byte[] rather than OutputStream/InputStream? With a byte[], one would inevitably end up wrapping it into a byte stream class. Also, the stream-based interface would be more in line with the Marshaller API.
Also for symmetry with the readClass() method, I suggest the writeClass() take a Class<?> rather than an object. Regards Andrey > From: [email protected] > Date: Tue, 3 Nov 2015 21:24:01 +0000 > Subject: Re: OSGi migration may require a special marshaller > To: [email protected] > > Hi Dmitriy, > I think your solution is good. Maybe I will change it a little bit... :P > I think you should delegate the Class resolution to the resolver. Because > for example with your current solution the marshaller may before (or after) > store the fqn of the class (maybe only the first time it encounter it) but > in order to save the classloader context resolution the implementation of > the resolver may have to save the package name of the given object and some > extra info therefore the content package name will be serialized two times. > Ok, it's not a big deal. > But now if the resolver identify that it can reuse the same class loader > for a couple of classes. It will be interesting for it to have a > serialization context in order to save, let say, classloader/id mapping in > order to save the id instead of a more longer representation. > So I propose something like that: > *public interface ClassResolver {* > * // This method is invoked on the sender side to * > * // marshal the information about the class.* > * // where the context is a map style object that is reset/new for each > object graph serialization.* > * public byte[] writeClass(Object o, Context context) throws > IgniteException;* > > * // This method is invoked on the receiving side to* > * // retrieve the class based on provided information.* > * // where the context is a map style object that is reset/new for each > object graph serialization.* > * public Class<?> readClass(byte[], Context context) throws > IgniteException;* > *}* > > I think your proposal will solve our issue and maybe also open a door for > the osgi development. > Let me know what do you think about me proposal? :) > > Thanks, > > Romain > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <[email protected]> a > écrit : > > > Romain, > > > > Can you comment on the ClassLoaderResolver suggestion that I proposed > > earlier? Will it solve your problem? > > > > D. > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <[email protected]> > > wrote: > > > > > Hi Raul, > > > - Do you plan to use the TCCL when marshalling in OSGi env? If yes how? > > > - I like the point 2. Maybe for a graph of object that come from > > different > > > packages / bundles you may have to recursively capture the package > > version > > > of the individual element of your graph. > > > - For the point 3 I wonder if it will not over-complexify the solution. > > As > > > a developer you will have to think about it. And it is not obvious in the > > > code where things are serialized or not. You may use lambda in your > > > application code therefore the current package become what you call a DTO > > > package. The current state of ignite does not enforce you to specify it > > for > > > "classical" classloading application. If you introduce this extra step > > for > > > OSGi ready application you will maybe discourage people to use OSGi. > > > > > > To comeback to our solution. We start we a strong assumption: our cluster > > > is homogeneous in term of code so, of course, it simplify our life :). > > > > > > BTW if you can introduce an extension point in the class resolution > > > mechanism it can be interesting for end user in order to allow them to > > > optimize it based on there specific use cases. > > > > > > Romain. > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <[email protected]> a écrit : > > > > > > > Hi Andrey, > > > > > > > > Thanks for the participation in this topic. > > > > > > > > I don't like the solution to incorporate the bundle symbolic name in > > the > > > > serialised form. Nothing guarantees that the classdef will be located > > > under > > > > the same bundle in both source and target machines. We also have to > > take > > > > into account that OSGi allows for multiple versions of the same > > > > bundle/packages to co-exist in the same container. So it becomes more > > > > complex. > > > > > > > > Using the TCCL should work when serialising, but it'll probably be of > > no > > > > use when deserialising on the other end. > > > > > > > > I need to enhance Ignite to: > > > > > > > > 1. Use the TCCL when marshalling on one end. > > > > 2. Incorporate the package version of the class in the serialised form > > if > > > > Ignite is running in an OSGi environment. > > > > 3. On the receiver end, discover cache entities / DTOs in all bundles > > > > through a custom OSGi manifest header or the like, as I explained > > before. > > > > Package version must be taken into account. > > > > > > > > What do you think? > > > > > > > > Raúl. > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <[email protected]> wrote: > > > > > > > > > Raul, > > > > > > > > > > The fundamental hurdle we need to jump over to make Ignite > > OSGi-enabled > > > > is > > > > > the marshalling. More specifically the issue is with deserialization > > of > > > > the > > > > > classes that are provided by the bundles other than the Ignite bundle > > > > > itself. > > > > > > > > > > When the Ignite transport layer receives a message it needs to figure > > > out > > > > > how to deserialize the bytes and for that it needs to know the bundle > > > > that > > > > > provides the class to be deserailized. At this point TCCL is of no > > use. > > > > To > > > > > make things more complex, the class may contain other classes that > > come > > > > > from other bundles, and so on recursively. This means that each > > object > > > in > > > > > the hierarchy must be serialized with its bundle name (or bundle id), > > > so > > > > > that the deserializer will then be able to correctly resolve the > > class > > > > > while traversing the object hierarchy during deserialization. > > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller is lacking the ability to > > > > plug > > > > > in a custom class resolver. Romain's solution was to use Kryo that > > does > > > > > provide a way to customize class resolution. It has solved the > > problem > > > of > > > > > capturing the bundle info and he was able to successfully run Ignite > > > as a > > > > > bundle in an OSGi container (after some repackaging and inclusion of > > > the > > > > > manifest). But Kryo-based marshalling introduced a lot of complexity > > to > > > > the > > > > > code and incorrect use of Kryo's numerous serializers caused some > > weird > > > > > hard-to-debug issues in the Ignite core (like duplicate cache entries > > > due > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap class -- go > > > > > figure!). Overall the Kryo-based solution is brittle and hard to > > > > maintain. > > > > > > > > > > I feel the correct solution to OSGi problem would be to > > > > > 1) enhance the OptimizedMarshaller to allow custom class resolution. > > > > > 2) provide an OSGi-enabled OptimizedMarshaller (in addition to the > > > > > original one) to be used in OSGi environment. > > > > > > > > > > Regards > > > > > Andrey > > > > > > > > > > > From: [email protected] > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000 > > > > > > Subject: Re: OSGi migration may require a special marshaller > > > > > > To: [email protected] > > > > > > > > > > > > Hi Romain, > > > > > > > > > > > > I'm working on the OSGi compatibility of Ignite. I appreciate your > > > > input. > > > > > > > > > > > > I'm thinking about the situation you describe and I wonder if > > you're > > > > > > exporting Ignite as an OSGi service which is then consumed from > > other > > > > > > bundles. Under this situation, it would be quite easy to reproduce > > > the > > > > > > behaviour you describe if Ignite is not resolving classes via the > > > TCCL. > > > > > > Need to dig deeper into that. > > > > > > > > > > > > Off the top of my head, there are two alternatives to solve it: > > > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling (if not already > > used) – > > > > we > > > > > > gotta be wary of possible regressions. > > > > > > 2. Create a special OSGi header Ignite-Export-Package so that > > bundles > > > > > > containing DTOs can expose packages to Ignite's marshallers. > > > > > > > > > > > > Regards, > > > > > > > > > > > > *Raúl Kripalani* > > > > > > PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big > > Data > > > > and > > > > > > Messaging Engineer > > > > > > http://about.me/raulkripalani | > > > > http://www.linkedin.com/in/raulkripalani > > > > > > http://blog.raulkr.net | twitter: @raulvk > > > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles, Romain < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'm really interested in this issue: > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270 . We some > > stuff > > > to > > > > > make > > > > > > > it work in our osgi environment. The main issue for us now is the > > > > > > > serialization. I think it you will have to rework the > > > > > OptimizedMarshaller > > > > > > > or any other marshaller that works with object that come from > > > outside > > > > > your > > > > > > > class space. > > > > > > > > > > > > > > We have try kryo that works. Kryo provide an extension point in > > > order > > > > > to > > > > > > > resolve the classes: > > > > > > > > > > > > > > > > > > > > > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java > > > > > > > . With this extension we are able to solve the problem of > > external > > > > > classes. > > > > > > > The only issue with kryo is that some classes need a certain care > > > in > > > > > the > > > > > > > serialization process and therefore a specialized serializer. > > > > > > > > > > > > > > So I would like to know from the community what do think of > > > changing > > > > > the > > > > > > > way the optimized marshaller works or introducing the support of > > > yet > > > > > > > another marshaller based on a kryo like technology? > > > > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > Romain. > > > > > > > > > > > > > > > > > > > > > PS: I'm ready to help in the both cases. > > > > > > > "Misys" is the trade name of the Misys group of companies. This > > > email > > > > > and > > > > > > > any attachments have been scanned for known viruses using > > multiple > > > > > > > scanners. This email message is intended for the named recipient > > > > only. > > > > > It > > > > > > > may be privileged and/or confidential. If you are not the named > > > > > recipient > > > > > > > of this email please notify us immediately and do not copy it or > > > use > > > > > it for > > > > > > > any purpose, nor disclose its contents to any other person. This > > > > email > > > > > does > > > > > > > not constitute the commencement of legal relations between you > > and > > > > > Misys. > > > > > > > Please refer to the executed contract between you and the > > relevant > > > > > member > > > > > > > of the Misys group for the identity of the contracting party with > > > > > which you > > > > > > > are dealing. > > > > > > > > > > > > > > > > > > > > >
