If anyone else is interested, looks like the code is at https://github.com/JulesGosnell/dada/tree/master/dada-core/src/main/java/clojure/lang
Pretty simple modifications actually. Thanks to Jules for figuring it out. On Tue, Apr 26, 2011 at 2:16 PM, kovas boguta <kovas.bog...@gmail.com> wrote: > Hi, > > This sounds very interesting and useful. > > Is the source code available somewhere? Or a more detailed description? > > Thanks > > On Fri, Mar 4, 2011 at 1:48 AM, Jules <jules.gosn...@gmail.com> wrote: >> >> So, >> >> Just a quick update. >> >> I gave the client-side ClassLoader encapsulation idea some more >> thought and rejeted it (sorry) for the reasons that : >> >> - I've lived through J2EE ClassLoader hell and don't want to repeat >> the experience >> >> - I'm looking towards an architecture where I can define new types on >> any node of my Clojure data-grid and have them accessible in any other >> node. This might be done by keeping my class-name:bytecode table in a >> clustered cache and inserting a single ClusteredClassLoader (NYI) into >> the client's ClassLoader hierarchy. Each client would be able to mix >> classes from multiple server JVMs without having to be concerned about >> keeping them all segregated. >> >> So, I introduced the concept of a per-jvm id and hacked it into RT, >> Compiler and LispReader. There were not too many places that needed to >> be changed. >> >> The result is, I can now create a type in one jvm, create an instance >> of the type, serialise it, ship the bytes to another jvm and >> deserialise it, at which point the instance's class is pulled from the >> original jvm. No more nasty exceptions because of name collisions etc. >> >> I have plumbed this infrastructure in underneath my app and, with the >> exception of some unrelated bugs that I am chasing down, everything >> seems to work beautifully :-) >> >> There are some limitations with this technique that I haven't looked >> into too deeply yet - the most obvious one is that whilst remote types >> can be made available within a local jvm, this is only at the java >> level, so extra things that defrecord might do for you like intern the >> classname as a symbol in your current namespace are not being done - >> which, I can currently live with, but I can see this being a tripping >> point to a completely transparent solution. >> >> For me, this is a great leap forward as I have found each Clojure JVM >> an island until I got this working, but now I should be able to build >> a distributed app in Clojure without having to sacrifice Clojure's >> dynamicity in the process - I can have my cake and eat it :-) >> >> I'm going to play with it for a few days, iron out all my other issues >> and investigate exactly what I have achieved, then I will post back a >> proper description in case anyone else is trying to build distributed >> Clojure apps in a similar manner. >> >> cheers >> >> >> Jules >> >> On Mar 3, 2:30 pm, Jules <jules.gosn...@gmail.com> wrote: >>> On Mar 3, 1:22 pm, Alessio Stalla <alessiosta...@gmail.com> wrote: >>> >>> >>> >>> >>> >>> > On Thursday, March 3, 2011 11:46:03 AM UTC+1, Jules wrote: >>> >>> > > Thanks, Alessio, >>> >>> > > I did know this, but it is a welcome addition to the thread. >>> >>> > Ok. Classloaders are a tricky matter and many people don't have clear >>> > ideas >>> > about them - sorry for assuming you were one of those people :) >>> >>> np :-) >>> >>> > > I reread my last posting and it was a bit confused - I've done a >>> > > little research in the Compiler class and can now clarify what I think >>> > > is happening. >>> >>> > > Clojure 1.3.0-alpha4 >>> > > user=> (type (fn [])) >>> > > user$eval1$fn__2 >>> > > user=> >>> >>> > > Here I have created a new function and asked it for its class. Clojure >>> > > has emitted bytecode on the fly to implement this and packed this into >>> > > a Java Class. A Class has a name unique - as you pointed out - within >>> > > its ClassLoader, so to avoid name collisions, Clojure attempts to >>> > > create a unique name for the class. This is composed, amongst other >>> > > things from its namespace and RT.nextID() (I'm assuming RT = RunTime). >>> >>> > > RT.nextID seems to start at 0 and allocate ids sequentially - lets >>> > > create another fn : >>> >>> > > user=> (type (fn [])) >>> > > user$eval5$fn__6 >>> > > user=> >>> >>> > > I suspect that the gap between 2 and 6 is due to other processes going >>> > > on behind the scenes which require ids, rather than some intentional >>> > > design - but haven't checked. >>> >>> > > This all works fine in a single JVM - no two functions will end up >>> > > creating types with the same name - however if I take the class for a >>> > > function from [a ClassLoader in] one clojure runtime and copy it >>> > > across to another clojure runtime that has already allocated this >>> > > classes id to another [in the receiving ClassLoader] then I would see >>> > > a collision. >>> >>> > > I think that this is what is happening above. I am serialising an >>> > > object in one jvm, putting it into a message and sending the message >>> > > to another jvm. Upon receipt I attempt to deserialise the object >>> > > contained in the message. This involves looking up its class by name >>> > > [within the scope of a ClassLoader] - but the name has already been >>> > > allocated to a local class [within the same ClassLoader] which is >>> > > mistakenly then used to try to deserialise the object. The object's >>> > > data is not presented in the format that the local class expects for >>> > > one of its own instances - hence the Exception in my posting above. >>> >>> > > This is currently only theory. >>> >>> > Makes sense. >>> >>> > > I think that the easiest way to fix it is to use not just a jvm-local >>> > > uid as part of the class name, but also an id which uniquely >>> > > identifies this jvm amongst all the jvms involved in my cluster. It >>> > > looks like not too many classes make use of RT.nextID(), so I think my >>> > > next move will be to add an e.g. RT.jvmID and use this in conjunction >>> > > with RT.nextID(). If this resolves my problem then there is a good >>> > > chance that my supposition was correct - otherwise it is back to the >>> > > drawing board. >>> >>> > I think a better option is to make sure that the classloader you use for >>> > deserializing is completely separated from the classloader Clojure uses to >>> > load compiled functions. Then, no possibility of collision exists, >>> > regardless of how you generate class names.- Hide quoted text - >>> >>> > - Show quoted text - >>> >>> That's an interesting idea, that I haven't really pursued, I think, >>> OTOH, because client-side, I think that I want to be able to mix >>> objects that herald from both local and remote jvms... - but this is >>> just a gut feeling, I haven't really looked for any concrete usecases >>> in any detail, so I'll give your suggestion consideration and see >>> whether it would fit the bill. >>> >>> Using your approach would also be a good way of checking that my >>> assumption about the name collision was correct, so I may see if I can >>> set up my code to use a different ClassLoader in which to deserialise >>> to see if that solves my problem. >>> >>> Thanks for this idea, it may help me a lot :-) >>> >>> Jules- Hide quoted text - >>> >>> - Show quoted text - >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with your >> first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
