Re: custom serialization seems to work...
What do you mean wint traverse parents? The only thing i don't do is the traversing and call the read/writeobject when they are there. But for the rest all the fields are stored that are not static and not transient of the complete object. Cool. I tested a little bit and saw your stuff works for anon/ local classes. because there is nothing special about that. an inner class differences is only that i under the hood generates a field to the parent. And that is just a reference already saved so it is just a handle. Take a look at SerializableChecker$HandleTable, which I basically copied from ObjectOutputStream. That's what you should use instead of the hasmap you're using as you're looking for system hash rather then what the object thinks it's identity should be like. It's not only way cheaper to do that, but it also shields against problems that can arise when objects have a hashcode implementation that e.g. depends on a hibernate session being available and current. But anyway, the speed argument alone is more than worth it. ahh yes i already thought about that.. :) what i do now is even better compression because if an object says it is content equal then i only do one. The problem is that that is the case for immutable objects especially string. I really don't want to print Eelco and Eelco twice into the buffer There i really want content equals (The same goes for all the Number classes, but those are handled already specially) So i will make it identity equal for all objects except String JBossSer does exactly the same. And i have now idea how anybody that writes custom serialization can do it any other way. Thats just not possible. Because thats the only way to set final fields (reflection can't do that since 1.2) And we in wicket uses final fields all the time. I am very curious how for example xml serialization or other kinds are really doing this.. I've searched around for it, and it doesn't seem to be a license violation. And as you said, many others, including JBoss, use Unsafe. Related read: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6379948 hmm there is a workaround, but for that you need really All Permissions to be set for the wicket application. thats not always workable for wicket.. johan
Re: custom serialization seems to work...
nope that doesn't work String a=new String(a); String b=new String(ab).substring(0,1); b.inter()==a.intern() that works. On 2/13/07, Igor Vaynberg [EMAIL PROTECTED] wrote: On 2/13/07, Johan Compagner [EMAIL PROTECTED] wrote: So i will make it identity equal for all objects except String but isnt string identity guaranteed for equal strings because the jvm pools them? so String a=new String(a); String b=new String(ab).substring(0,1); b==a -igor
Re: custom serialization seems to work...
On 2/13/07, Igor Vaynberg [EMAIL PROTECTED] wrote: but isnt string identity guaranteed for equal strings because the jvm pools them? so String a=new String(a); String b=new String(ab).substring(0,1); b==a Don't know for 1.5, but this type of code was a sure way to get a flogging in the 1.4 days. Martijn -- Vote for Wicket at the http://www.thebeststuffintheworld.com/vote_for/wicket Wicket 1.2.4 is as easy as 1-2-4. Download Wicket now! http://wicketframework.org
Re: custom serialization seems to work...
Take a look at SerializableChecker$HandleTable, which I basically copied from ObjectOutputStream. That's what you should use instead of the hasmap you're using as you're looking for system hash rather then what the object thinks it's identity should be like. It's not only way cheaper to do that, but it also shields against problems that can arise when objects have a hashcode implementation that e.g. depends on a hibernate session being available and current. But anyway, the speed argument alone is more than worth it. ahh yes i already thought about that.. :) what i do now is even better compression because if an object says it is content equal then i only do one. At the cost of invoking a more expensive function (hashCode compared to System.identityHashCode) which you should also guard with try/ catch as it may fail and when it does you don't want to fail the whole serialization because of it (you probably should fall back on indentity hashcode in that case). Also, the benefit of using object identity proper would be mostly for 'domain objects' that properly provide a hashCode and equals function. And only if they implemented it efficiently. The problem is that that is the case for immutable objects especially string. I really don't want to print Eelco and Eelco twice into the buffer There i really want content equals (The same goes for all the Number classes, but those are handled already specially) So i will make it identity equal for all objects Good idea. except String Measure whether it makes a difference and whether the slightly more expensive call to string justifies it. Eelco
Re: custom serialization seems to work...
But it should be tested, if we disable it then it will never be tested... for a release (beta) we can disable it but i like to keep it enabled in svn. johan On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: I've turned that custom streaming off by default. Until we are absolutely, entirely, utmost sure it works without exceptions we should not commit it (leaving that line commented is fine by me though). Eelco On 2/11/07, Eelco Hillenius [EMAIL PROTECTED] wrote: Not entirely I'm afraid: 2007-02-11 16:25:47,595 ERROR wicket.protocol.http.FilePageStore - Error in page save thread java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at wicket.util.io.ClassStreamHandler.invokeWriteMethod( ClassStreamHandler.java:523) at wicket.util.io.WicketObjectOutputStream.writeObjectOverride( WicketObjectOutputStream.java:124) at java.io.ObjectOutputStream.writeObject( ObjectOutputStream.java:287) at wicket.util.io.ClassStreamHandler.writeFields( ClassStreamHandler.java:273) at wicket.util.io.WicketObjectOutputStream.writeObjectOverride( WicketObjectOutputStream.java:126) at java.io.ObjectOutputStream.writeObject( ObjectOutputStream.java:287) at wicket.util.lang.Objects.objectToByteArray(Objects.java:1037) at wicket.protocol.http.FilePageStore.serializePage( FilePageStore.java:414) at wicket.protocol.http.FilePageStore.access$4( FilePageStore.java:407) at wicket.protocol.http.FilePageStore$PageSavingThread.run( FilePageStore.java:601) at java.lang.Thread.run(Thread.java:613) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at wicket.util.io.ClassStreamHandler.invokeWriteMethod( ClassStreamHandler.java:511) ... 10 more Caused by: java.lang.NullPointerException at java.io.ObjectOutputStream.writeUTF(ObjectOutputStream.java :805) at wicket.protocol.http.request.urlcompressing.URLCompressor.writeObject( URLCompressor.java:105) ... 15 more Eelco
Re: custom serialization seems to work...
This is fixed. I didn't overwrite all the read and write methods. johan On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: Not entirely I'm afraid: 2007-02-11 16:25:47,595 ERROR wicket.protocol.http.FilePageStore - Error in page save thread java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at wicket.util.io.ClassStreamHandler.invokeWriteMethod( ClassStreamHandler.java:523) at wicket.util.io.WicketObjectOutputStream.writeObjectOverride( WicketObjectOutputStream.java:124) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java :287) at wicket.util.io.ClassStreamHandler.writeFields( ClassStreamHandler.java:273) at wicket.util.io.WicketObjectOutputStream.writeObjectOverride( WicketObjectOutputStream.java:126) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java :287) at wicket.util.lang.Objects.objectToByteArray(Objects.java:1037) at wicket.protocol.http.FilePageStore.serializePage( FilePageStore.java:414) at wicket.protocol.http.FilePageStore.access$4(FilePageStore.java :407) at wicket.protocol.http.FilePageStore$PageSavingThread.run( FilePageStore.java:601) at java.lang.Thread.run(Thread.java:613) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at wicket.util.io.ClassStreamHandler.invokeWriteMethod( ClassStreamHandler.java:511) ... 10 more Caused by: java.lang.NullPointerException at java.io.ObjectOutputStream.writeUTF(ObjectOutputStream.java :805) at wicket.protocol.http.request.urlcompressing.URLCompressor.writeObject( URLCompressor.java:105) ... 15 more Eelco
Re: custom serialization seems to work...
hmm are you sure you test with the latest code? i just committed some fixes. Can you test with the latest code? Also look what kind of files are created (what is the filename of a page) I don't get this at all anymore on my machine. I did get it before i made the change to a stricter syncing. for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days johan On 2/12/07, Matej Knopp [EMAIL PROTECTED] wrote: Sorry, no luck either, This time i got three expired pages and after that the test just stopped.
Re: custom serialization seems to work...
hmm i get on my machine a bigger leap. 115 seconds agains 90 seconds. but i think that the synching gets to much time. johan On 2/12/07, Matej Knopp [EMAIL PROTECTED] wrote: 2nd level with byteoutputstream 53906 2nd level with wicketbyteoutputstream 50350 http session store 37714 Johan Compagner wrote: yes that i know.. I first wanted it to be reliable! Now the tweaking. But what happens if you switch in Objects.byteToObject and objectToByteArray to the default ObjectOut en InputStreams? johan On 2/12/07, Matej Knopp [EMAIL PROTECTED] wrote: Okay, with current svn version it works. That's the good news. Bad news is, that second level session store is now quite slower. cca 50 seconds vs 38 seconds http session store. -Matej Johan Compagner wrote: hmm are you sure you test with the latest code? i just committed some fixes. Can you test with the latest code? Also look what kind of files are created (what is the filename of a page) I don't get this at all anymore on my machine. I did get it before i made the change to a stricter syncing. for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days johan On 2/12/07, Matej Knopp [EMAIL PROTECTED] wrote: Sorry, no luck either, This time i got three expired pages and after that the test just stopped.
Re: custom serialization seems to work...
for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? * Externalizable is not supported yet? * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. * The code depends on SUN code directly. I'm wondering if we can even do that considering our license, but I'm also wondering how quick that'll fail. The diagnostics class depends on some quasi internals - quasi because they are package private but at least they are part of the normal JDK and seem unlikely to chance - and has a fall back when it recognizes it is not available. It also seems that if for whatever reason our custom serialization wouldn't be available, that's currently bad luck for the client as it just won't work then. * It needs a lot of improvement for error reporting * Document soon please. Or it doesn't get done at all. Eelco
Re: custom serialization seems to work...
before we start doing all this have you guys tried the jboss serialization thing yet? the problem, like johan mentioned, is that this wont work across jvms because he keeps some kind of cache? but then this makes it useless for clustering. i think whatever solution we come up with needs to work across jvms because i can see the page store saving pages to a nas for fail over -igor On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? * Externalizable is not supported yet? * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. * The code depends on SUN code directly. I'm wondering if we can even do that considering our license, but I'm also wondering how quick that'll fail. The diagnostics class depends on some quasi internals - quasi because they are package private but at least they are part of the normal JDK and seem unlikely to chance - and has a fall back when it recognizes it is not available. It also seems that if for whatever reason our custom serialization wouldn't be available, that's currently bad luck for the client as it just won't work then. * It needs a lot of improvement for error reporting * Document soon please. Or it doesn't get done at all. Eelco
Re: custom serialization seems to work...
On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: before we start doing all this have you guys tried the jboss serialization thing yet? Yes, and it didn't even remotely work for the project I'm working on. Furthermore, maybe I'm wrong, but it seems that after doing two initial releases this project got forgotten/ abandoned whatever. It also is LGPL licensed. the problem, like johan mentioned, is that this wont work across jvms because he keeps some kind of cache? but then this makes it useless for clustering. i think whatever solution we come up with needs to work across jvms because i can see the page store saving pages to a nas for fail over That too. Eelco
Re: custom serialization seems to work...
There probably are a couple of cases where our code could benefit from either custom read/writeObject methods or implementing Externalizable. What are your ideas about that? Eelco
Re: custom serialization seems to work...
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: before we start doing all this have you guys tried the jboss serialization thing yet? Yes, and it didn't even remotely work for the project I'm working on. Furthermore, maybe I'm wrong, but it seems that after doing two initial releases this project got forgotten/ abandoned whatever. It also is LGPL licensed. Also, if I remember correctly, Johan said he did get it to work, but experienced that it was actually slower than normal JDK serialization. Eelco
Re: custom serialization seems to work...
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. Actually, we should abstract it so that the handling is plug-gable. It would be great if clients can decide whether they want to use JDK's default, jboss, xstream or maybe our custom scheme. Support for XStream might be a bit difficult actually. The easiest thing for us to do would to have a setting or factory method that produced the type of ObjectOutputStream/ObjectInputStream that should be used. WDYT? Eelco
Re: custom serialization seems to work...
well the fact that it is LGPL is ok, we can rewrite the pieces we need. i was mainly interested in the ideas they used, they did claim it was 30% faster? hmm -igor On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: before we start doing all this have you guys tried the jboss serialization thing yet? Yes, and it didn't even remotely work for the project I'm working on. Furthermore, maybe I'm wrong, but it seems that after doing two initial releases this project got forgotten/ abandoned whatever. It also is LGPL licensed. Also, if I remember correctly, Johan said he did get it to work, but experienced that it was actually slower than normal JDK serialization. Eelco
Re: custom serialization seems to work...
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? * Externalizable is not supported yet? * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. * The code depends on SUN code directly. I'm wondering if we can even do that considering our license, but I'm also wondering how quick that'll fail. The diagnostics class depends on some quasi internals - quasi because they are package private but at least they are part of the normal JDK and seem unlikely to chance - and has a fall back when it recognizes it is not available. It also seems that if for whatever reason our custom serialization wouldn't be available, that's currently bad luck for the client as it just won't work then. * It needs a lot of improvement for error reporting * Document soon please. Or it doesn't get done at all. Oh, did I mention unit testing? If there's ever been a case where some aggressive unit testing would help, this would be it. :) Eelco
Re: custom serialization seems to work...
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
for eelco: you had to disable both methods! (objectToByte and byteToObject)
For now i enabled both so that it gets tested as much as possible the coming
few days
Ok, we can keep it in for a few days, but it has to improve quite a
bit before it's ready for real world use.
Actually, we should abstract it so that the handling is plug-gable. It
would be great if clients can decide whether they want to use JDK's
default, jboss, xstream or maybe our custom scheme.
Support for XStream might be a bit difficult actually. The easiest
thing for us to do would to have a setting or factory method that
produced the type of ObjectOutputStream/ObjectInputStream that should
be used.
Just implemented that:
public interface IObjectStreamFactory
{
/**
* Gets a new instance of an [EMAIL PROTECTED] ObjectInputStream} with
the provided
* [EMAIL PROTECTED] InputStream}.
*
* @param in
*The inpu stream that should be used for the reading
* @return a new object input stream instance
* @throws IOException
* if an I/O error occurs while reading stream header
*/
ObjectInputStream newObjectInputStream(InputStream in) throws
IOException;
/**
* Gets a new instance of an [EMAIL PROTECTED] ObjectOutputStream} with
the provided
* [EMAIL PROTECTED] OutputStream}.
*
* @param out
*The output stream that should be used for the writing
* @return a new object output stream instance
* @throws IOException
* if an I/O error occurs while writing stream header
*/
ObjectOutputStream newObjectOutputStream(OutputStream out) throws
IOException;
}
It's set as a static instance on the Objects class, and you can set it
for your application using Objects#setObjectStreamFactory.
I choose - as an exception - for a static var as this the Application
often will not be available as a thread local when
Objects#byteArrayToObject and Objects#objectToByteArray is used, and
it seems very unlikely to me that people will want to vary this
between instances of applications anyway (though the door is still
open to do that, they would just have to program a few lines
themselves).
Eelco
Re: custom serialization seems to work...
I tried jboss didn't get me anything no speed and size improvement. when i used JBossOut en In instead of the normal out en in. I can make it work for clusterings but the bytes will be much greater then i guess because i need to write a class name instead of a short. Of course we could do that or have some method where we type all our classes and most used classes (ArrayList or something) Then it can be stable. But even clustering works fine what doesn't work. If 1 server drops out and all the sessions transfer to another. then it still works fine. Except when they directly use the back button at that time same time So if i set it up. Then i don't think i would use a NAS server anyway. Because that overhead you have with that for only catching a failover and then directly a backbutton. I don't think i would use that. johan On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: before we start doing all this have you guys tried the jboss serialization thing yet? the problem, like johan mentioned, is that this wont work across jvms because he keeps some kind of cache? but then this makes it useless for clustering. i think whatever solution we come up with needs to work across jvms because i can see the page store saving pages to a nas for fail over -igor On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? * Externalizable is not supported yet? * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. * The code depends on SUN code directly. I'm wondering if we can even do that considering our license, but I'm also wondering how quick that'll fail. The diagnostics class depends on some quasi internals - quasi because they are package private but at least they are part of the normal JDK and seem unlikely to chance - and has a fall back when it recognizes it is not available. It also seems that if for whatever reason our custom serialization wouldn't be available, that's currently bad luck for the client as it just won't work then. * It needs a lot of improvement for error reporting * Document soon please. Or it doesn't get done at all. Eelco
Re: custom serialization seems to work...
well you know, im just playing a devils advocate :) i guess you are kinda using the codec ideas we discussed - cutting out the long class header here is what i would try i guess right now you are keeping an application map:classname-byte but what if we do this: keep an application map:classname-byte for often used classes but only use lower 7 bits then you do what you do now - build a toc map:classname-byte if its a classname that is not in the registry yet so if a class is in a registry you represent it as 0xxx and if its not and is in the toc you represent it as 1000 the last 8 bits being the byte in the serialization-local toc map so then you can represent the serialization as [local toc][data] that way we get great compression by avoiding a lot of class headers and it is stable across jvms -igor On 2/12/07, Johan Compagner [EMAIL PROTECTED] wrote: I tried jboss didn't get me anything no speed and size improvement. when i used JBossOut en In instead of the normal out en in. I can make it work for clusterings but the bytes will be much greater then i guess because i need to write a class name instead of a short. Of course we could do that or have some method where we type all our classes and most used classes (ArrayList or something) Then it can be stable. But even clustering works fine what doesn't work. If 1 server drops out and all the sessions transfer to another. then it still works fine. Except when they directly use the back button at that time same time So if i set it up. Then i don't think i would use a NAS server anyway. Because that overhead you have with that for only catching a failover and then directly a backbutton. I don't think i would use that. johan On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: before we start doing all this have you guys tried the jboss serialization thing yet? the problem, like johan mentioned, is that this wont work across jvms because he keeps some kind of cache? but then this makes it useless for clustering. i think whatever solution we come up with needs to work across jvms because i can see the page store saving pages to a nas for fail over -igor On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? * Externalizable is not supported yet? * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. * The code depends on SUN code directly. I'm wondering if we can even do that considering our license, but I'm also wondering how quick that'll fail. The diagnostics class depends on some quasi internals - quasi because they are package private but at least they are part of the normal JDK and seem unlikely to chance - and has a fall back when it recognizes it is not available. It also seems that if for whatever reason our custom serialization wouldn't be available, that's currently bad luck for the client as it just won't work then. * It needs a lot of improvement for error reporting * Document soon please. Or it doesn't get done at all. Eelco
Re: custom serialization seems to work...
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. the only thing i need to improve as far as i see now is that i need to traverse the class super classes and try to find the private readObject and writeObject methods i now only do it for the given class now (i was wondering that but i didn't see how they really did that in the normal In/Out) But i have to do it. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? What do you mean wint traverse parents? The only thing i don't do is the traversing and call the read/writeobject when they are there. But for the rest all the fields are stored that are not static and not transient of the complete object. * Externalizable is not supported yet? no that is not yet supported. read and write object is. * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. Error reporting should improve a bit yes. i can do that perfectly because i can really build up the exact stack. But what do you mean with put objects in a hashset/map? what kind of object do a put in what kind of map? I guess i need now and then a bit more check if it is serializeable because it could be that i serialize now pretty much everything ... :) * The code depends on SUN code directly. I'm wondering if we can even do that considering our license, but I'm also wondering how quick that'll fail. The diagnostics class depends on some quasi internals - quasi because they are package private but at least they are part of the normal JDK and seem unlikely to chance - and has a fall back when it recognizes it is not available. It also seems that if for whatever reason our custom serialization wouldn't be available, that's currently bad luck for the client as it just won't work then. JBossSer does exactly the same. And i have now idea how anybody that writes custom serialization can do it any other way. Thats just not possible. Because thats the only way to set final fields (reflection can't do that since 1.2) And we in wicket uses final fields all the time. I am very curious how for example xml serialization or other kinds are really doing this.. But it is a very very very interesting class that Unsave!! do you see for example the reallocateMemory method..?? That looks very very cool! :) * It needs a lot of improvement for error reporting yes as i said above that is one of the improvements that should be done. But not directly critical for the usage. * Document soon please. Or it doesn't get done at all. will do but most methods are just ObjectOutput/InStream things. Except the simple Class id generator. (and fields updater class) johan Eelco
Re: custom serialization seems to work...
So you want to add a registry of classnames-id and also save that? (as a TOC prepended to the beginning of the the stream is very hard because you write as you go further) But why do that? then you still write the classnames and if we write a classname it is a string. So that classname will be written only once in the stream! Because the same class after that is just an handle (short i think 65K objects should be enough i think?) So i guess having a build in map of the first 254 mostly used classes and only write a byte and let the rest be just as the classname (only once, the second time it is a handle) johan On 2/13/07, Igor Vaynberg [EMAIL PROTECTED] wrote: well you know, im just playing a devils advocate :) i guess you are kinda using the codec ideas we discussed - cutting out the long class header here is what i would try i guess right now you are keeping an application map:classname-byte but what if we do this: keep an application map:classname-byte for often used classes but only use lower 7 bits then you do what you do now - build a toc map:classname-byte if its a classname that is not in the registry yet so if a class is in a registry you represent it as 0xxx and if its not and is in the toc you represent it as 1000 the last 8 bits being the byte in the serialization-local toc map so then you can represent the serialization as [local toc][data] that way we get great compression by avoiding a lot of class headers and it is stable across jvms -igor On 2/12/07, Johan Compagner [EMAIL PROTECTED] wrote: I tried jboss didn't get me anything no speed and size improvement. when i used JBossOut en In instead of the normal out en in. I can make it work for clusterings but the bytes will be much greater then i guess because i need to write a class name instead of a short. Of course we could do that or have some method where we type all our classes and most used classes (ArrayList or something) Then it can be stable. But even clustering works fine what doesn't work. If 1 server drops out and all the sessions transfer to another. then it still works fine. Except when they directly use the back button at that time same time So if i set it up. Then i don't think i would use a NAS server anyway. Because that overhead you have with that for only catching a failover and then directly a backbutton. I don't think i would use that. johan On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: before we start doing all this have you guys tried the jboss serialization thing yet? the problem, like johan mentioned, is that this wont work across jvms because he keeps some kind of cache? but then this makes it useless for clustering. i think whatever solution we come up with needs to work across jvms because i can see the page store saving pages to a nas for fail over -igor On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? * Externalizable is not supported yet? * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. * The code depends on SUN code directly. I'm wondering if we can even do that considering our license, but I'm also wondering how quick that'll fail. The diagnostics class depends on some quasi internals - quasi because they are package private but at least they are part of the normal JDK and seem unlikely to chance - and has a fall back when it recognizes it is not available. It also seems that if for whatever reason our custom serialization wouldn't be available, that's currently bad luck for the client as it just won't work then. * It needs a lot of improvement for error reporting * Document soon please. Or it doesn't get done at all. Eelco
Re: custom serialization seems to work...
custom read/write is supported (only have to fix one bug) I have no idea what Externalizeable brings you exactly compared to the read/write objects.. But checking if it implements that interface and then just call those methods is pretty easy. johan On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: There probably are a couple of cases where our code could benefit from either custom read/writeObject methods or implementing Externalizable. What are your ideas about that? Eelco
Re: custom serialization seems to work...
Thats fine. Already thought about it where we let the Objects class look at a setting where you can choose what every you want. But i don't think that is really a choice many people will use. But if they can come up with a better way for serialization and deserialization thats fine with me We just have to make sure that those 2 methods are called always when we do the clone or save... For example the sizeOf methods should also be altered! johan On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. Actually, we should abstract it so that the handling is plug-gable. It would be great if clients can decide whether they want to use JDK's default, jboss, xstream or maybe our custom scheme. Support for XStream might be a bit difficult actually. The easiest thing for us to do would to have a setting or factory method that produced the type of ObjectOutputStream/ObjectInputStream that should be used. WDYT? Eelco
Re: custom serialization seems to work...
What do you think i have been doing the past weekend!! :) This is really as fast and as small as we can get. Its almost no overhead and we only store as less bytes as possible. Its really almost done (for a few bugs of course) The only thing that is open now is that new GregorianCalendar() doesn't work quite right And i should improve the primitive arrays. Those are done wrong at the moment. johan On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: well the fact that it is LGPL is ok, we can rewrite the pieces we need. i was mainly interested in the ideas they used, they did claim it was 30% faster? hmm -igor On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote: before we start doing all this have you guys tried the jboss serialization thing yet? Yes, and it didn't even remotely work for the project I'm working on. Furthermore, maybe I'm wrong, but it seems that after doing two initial releases this project got forgotten/ abandoned whatever. It also is LGPL licensed. Also, if I remember correctly, Johan said he did get it to work, but experienced that it was actually slower than normal JDK serialization. Eelco
Re: custom serialization seems to work...
hmm the application should be available
because if we start writing classnames and we have to resolve them
then i want to be able to get the right classloader.
But i gues this can also be done in the constructor of
WicketObjectInputStream (give the classloader)
johan
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
for eelco: you had to disable both methods! (objectToByte and
byteToObject)
For now i enabled both so that it gets tested as much as possible
the coming
few days
Ok, we can keep it in for a few days, but it has to improve quite a
bit before it's ready for real world use.
Actually, we should abstract it so that the handling is plug-gable. It
would be great if clients can decide whether they want to use JDK's
default, jboss, xstream or maybe our custom scheme.
Support for XStream might be a bit difficult actually. The easiest
thing for us to do would to have a setting or factory method that
produced the type of ObjectOutputStream/ObjectInputStream that should
be used.
Just implemented that:
public interface IObjectStreamFactory
{
/**
* Gets a new instance of an [EMAIL PROTECTED] ObjectInputStream} with
the
provided
* [EMAIL PROTECTED] InputStream}.
*
* @param in
*The inpu stream that should be used for the reading
* @return a new object input stream instance
* @throws IOException
* if an I/O error occurs while reading stream header
*/
ObjectInputStream newObjectInputStream(InputStream in) throws
IOException;
/**
* Gets a new instance of an [EMAIL PROTECTED] ObjectOutputStream} with
the
provided
* [EMAIL PROTECTED] OutputStream}.
*
* @param out
*The output stream that should be used for the
writing
* @return a new object output stream instance
* @throws IOException
* if an I/O error occurs while writing stream header
*/
ObjectOutputStream newObjectOutputStream(OutputStream out) throws
IOException;
}
It's set as a static instance on the Objects class, and you can set it
for your application using Objects#setObjectStreamFactory.
I choose - as an exception - for a static var as this the Application
often will not be available as a thread local when
Objects#byteArrayToObject and Objects#objectToByteArray is used, and
it seems very unlikely to me that people will want to vary this
between instances of applications anyway (though the door is still
open to do that, they would just have to program a few lines
themselves).
Eelco
Re: custom serialization seems to work...
hmmm
so if you have something like this
class A {}
class B { private A a; private A aprime; }
when you serialize B does it write the class header for A once or twice?
because i think that header has the classname so it would be output twice
no? last time i checked the header was a bit over 100bytes. so if it does
write it twice and you keep a local toc then you save yourself that second
100+ byte class header
-igor
On 2/12/07, Johan Compagner [EMAIL PROTECTED] wrote:
So you want to add a registry of classnames-id and also save that?
(as a TOC prepended to the beginning of the the stream is very hard
because
you write as you go further)
But why do that? then you still write the classnames and if we write a
classname
it is a string. So that classname will be written only once in the stream!
Because the same class after that is just an handle (short i think 65K
objects should be enough i think?)
So i guess having a build in map of the first 254 mostly used classes and
only write a byte
and let the rest be just as the classname (only once, the second time it
is
a handle)
johan
On 2/13/07, Igor Vaynberg [EMAIL PROTECTED] wrote:
well you know, im just playing a devils advocate :)
i guess you are kinda using the codec ideas we discussed - cutting out
the
long class header
here is what i would try
i guess right now you are keeping an application map:classname-byte
but what if we do this:
keep an application map:classname-byte for often used classes but only
use
lower 7 bits
then you do what you do now - build a toc map:classname-byte if its a
classname that is not in the registry yet
so if a class is in a registry you represent it as 0xxx and if its
not
and is in the toc you represent it as 1000 the last 8 bits
being
the byte in the serialization-local toc map
so then you can represent the serialization as [local toc][data]
that way we get great compression by avoiding a lot of class headers and
it
is stable across jvms
-igor
On 2/12/07, Johan Compagner [EMAIL PROTECTED] wrote:
I tried jboss didn't get me anything no speed and size improvement.
when i used JBossOut en In instead of the normal out en in.
I can make it work for clusterings but the bytes will be much greater
then
i
guess
because i need to write a class name instead of a short.
Of course we could do that or have some method where we type all our
classes
and most used classes (ArrayList or something)
Then it can be stable.
But even clustering works fine what doesn't work. If 1 server drops
out
and all the sessions transfer to another. then it still works fine.
Except when they directly use the back button at that time same time
So if i set it up. Then i don't think i would use a NAS server anyway.
Because that overhead you have with that for only catching a failover
and
then
directly a backbutton. I don't think i would use that.
johan
On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote:
before we start doing all this have you guys tried the jboss
serialization
thing yet?
the problem, like johan mentioned, is that this wont work across
jvms
because he keeps some kind of cache? but then this makes it useless
for
clustering. i think whatever solution we come up with needs to work
across
jvms because i can see the page store saving pages to a nas for fail
over
-igor
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
for eelco: you had to disable both methods! (objectToByte and
byteToObject)
For now i enabled both so that it gets tested as much as
possible
the
coming
few days
Ok, we can keep it in for a few days, but it has to improve quite
a
bit before it's ready for real world use.
* Does your code anonymous and local class instances and traverse
parents? Not from what I can see as you're basically doing normal
introspection, right?
* Externalizable is not supported yet?
* Whenever you put objects in a hashSet/Map you'll need to be
ready
to
catch exceptions. Wicket was trying to serializale some Hibernate
objects in my app (which it shouldn't, but that's exactly what I'm
trying to diagnose) and they throw exceptions in some occasions
when
they are trying e.g. to use a lazy connection. If fact, we (me for
the
diagnostics thinghy as well) probably should just use identity
directly.
* The code depends on SUN code directly. I'm wondering if we can
even
do that considering our license, but I'm also wondering how quick
that'll fail. The diagnostics class depends on some quasi
internals
-
quasi because they are package private but at least they are part
of
the normal JDK and seem unlikely to chance - and has a fall back
when
it recognizes it is not available. It also seems that if for
whatever
reason our custom serialization wouldn't be available, that's
currently bad luck
Re: custom serialization seems to work...
On 2/12/07, Johan Compagner [EMAIL PROTECTED] wrote: On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote: for eelco: you had to disable both methods! (objectToByte and byteToObject) For now i enabled both so that it gets tested as much as possible the coming few days Ok, we can keep it in for a few days, but it has to improve quite a bit before it's ready for real world use. the only thing i need to improve as far as i see now is that i need to traverse the class super classes and try to find the private readObject and writeObject methods i now only do it for the given class now (i was wondering that but i didn't see how they really did that in the normal In/Out) But i have to do it. Yeah, I have to do that as well I just found out. * Does your code anonymous and local class instances and traverse parents? Not from what I can see as you're basically doing normal introspection, right? What do you mean wint traverse parents? The only thing i don't do is the traversing and call the read/writeobject when they are there. But for the rest all the fields are stored that are not static and not transient of the complete object. Cool. I tested a little bit and saw your stuff works for anon/ local classes. * Whenever you put objects in a hashSet/Map you'll need to be ready to catch exceptions. Wicket was trying to serializale some Hibernate objects in my app (which it shouldn't, but that's exactly what I'm trying to diagnose) and they throw exceptions in some occasions when they are trying e.g. to use a lazy connection. If fact, we (me for the diagnostics thinghy as well) probably should just use identity directly. Error reporting should improve a bit yes. i can do that perfectly because i can really build up the exact stack. But what do you mean with put objects in a hashset/map? what kind of object do a put in what kind of map? Take a look at SerializableChecker$HandleTable, which I basically copied from ObjectOutputStream. That's what you should use instead of the hasmap you're using as you're looking for system hash rather then what the object thinks it's identity should be like. It's not only way cheaper to do that, but it also shields against problems that can arise when objects have a hashcode implementation that e.g. depends on a hibernate session being available and current. But anyway, the speed argument alone is more than worth it. I guess i need now and then a bit more check if it is serializeable because it could be that i serialize now pretty much everything ... :) Yeah. Error reporting isn't very helpful atm. E.g. this is a stacktrace: Exception in thread main java.lang.RuntimeException: Failed to get the constructor from clz: class TestWithError$1Foo at wicket.util.io.ClassStreamHandler.init(ClassStreamHandler.java:171) at wicket.util.io.ClassStreamHandler.lookup(ClassStreamHandler.java:124) at wicket.util.io.WicketObjectOutputStream.writeObjectOverride(WicketObjectOutputStream.java:112) ... That's a test where an object is not serializable. JBossSer does exactly the same. And i have now idea how anybody that writes custom serialization can do it any other way. Thats just not possible. Because thats the only way to set final fields (reflection can't do that since 1.2) And we in wicket uses final fields all the time. I am very curious how for example xml serialization or other kinds are really doing this.. I've searched around for it, and it doesn't seem to be a license violation. And as you said, many others, including JBoss, use Unsafe. Related read: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6379948 But it is a very very very interesting class that Unsave!! do you see for example the reallocateMemory method..?? That looks very very cool! :) But I wouldn't get carried away. :) It's an internal class and people are officially encouraged to never depend on sun.* classes. And that 10 times more important for a framework too. Eelco
Re: custom serialization seems to work...
On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote:
hmmm
so if you have something like this
class A {}
class B { private A a; private A aprime; }
when you serialize B does it write the class header for A once or twice?
because i think that header has the classname so it would be output twice
no? last time i checked the header was a bit over 100bytes. so if it does
write it twice and you keep a local toc then you save yourself that second
100+ byte class header
AFAIK, JDK's serialization writes headers once and then references.
Eelco
Re: custom serialization seems to work...
On 2/12/07, Johan Compagner [EMAIL PROTECTED] wrote: hmm the application should be available because if we start writing classnames and we have to resolve them then i want to be able to get the right classloader. But i gues this can also be done in the constructor of WicketObjectInputStream (give the classloader) You could set it in the IObjectStreamFactory implementation. The file save thread will not have the application as a thread local available for instance. Eelco
Re: custom serialization seems to work...
On 2/12/07, Johan Compagner [EMAIL PROTECTED] wrote: Thats fine. Already thought about it where we let the Objects class look at a setting where you can choose what every you want. But i don't think that is really a choice many people will use. But if they can come up with a better way for serialization and deserialization thats fine with me I just want to make sure we have a fallback for people to use in case we have some unforseen bug in our mechanism or their security environment doesn't allow for some of the things we do or they have some other reason to prefer another (i.e. Java's default) serialization mechanism. We just have to make sure that those 2 methods are called always when we do the clone or save... For example the sizeOf methods should also be altered! I wouldn't have a problem with sizeOf depending on it more directly if there isn't a practical way around it (though I'm sure there is); that's merely a utility whereas serialization for versions is a central piece of the framework. Eelco
Re: custom serialization seems to work...
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
On 2/12/07, Igor Vaynberg [EMAIL PROTECTED] wrote:
hmmm
so if you have something like this
class A {}
class B { private A a; private A aprime; }
when you serialize B does it write the class header for A once or twice?
because i think that header has the classname so it would be output
twice
no? last time i checked the header was a bit over 100bytes. so if it
does
write it twice and you keep a local toc then you save yourself that
second
100+ byte class header
AFAIK, JDK's serialization writes headers once and then references.
no, this is different
B b=new B(); b.a=new A(); b.aprime=new A();
they are different instances, i am talking about class headers not
references
the way jdk serialization works is that for every class it does something
like this
[class-header classname,etc][fields]
so what i want to know and dont really have time to look into is
when you serialize B is it
[B-header][b-data [A-header][B.a data][A-header][B.aprime data]]
or does it also do what we do and keep some kind of toc so that it looks
like
[B-header][b-data [A-header][B.a data][A-header-pointer][B.aprime data]]
because that is kinda what johan is doing, creating pointers to class
headers instead of writing them out all the time.
-igor
Eelco
Re: custom serialization seems to work...
no, this is different
B b=new B(); b.a=new A(); b.aprime=new A();
they are different instances, i am talking about class headers not
references
the way jdk serialization works is that for every class it does something
like this
[class-header classname,etc][fields]
so what i want to know and dont really have time to look into is
This:
static class A implements Serializable {
}
static class B implements Serializable {
private A first;
private A second;
private A third;
private A fourth;
}
and this
B b = new B();
b.first = new A();
b.second = new A();
b.third = new A();
b.fourth = new A();
Serializing b will first write out the class header for B, then, when
it writes out B's fields, the class header for A the first time it
encounters it, and then 3 times a just reference to the header (and
bye for tag and an int for the reference.
Maybe we're talking about different things here? But I'm sure Johan
will be able to take care of it :)
Eelco
Re: custom serialization seems to work...
hrm, so the default jdk serialization is also building a toc? interesting,
thats what i wanted to know.
-igor
On 2/12/07, Eelco Hillenius [EMAIL PROTECTED] wrote:
no, this is different
B b=new B(); b.a=new A(); b.aprime=new A();
they are different instances, i am talking about class headers not
references
the way jdk serialization works is that for every class it does
something
like this
[class-header classname,etc][fields]
so what i want to know and dont really have time to look into is
This:
static class A implements Serializable {
}
static class B implements Serializable {
private A first;
private A second;
private A third;
private A fourth;
}
and this
B b = new B();
b.first = new A();
b.second = new A();
b.third = new A();
b.fourth = new A();
Serializing b will first write out the class header for B, then, when
it writes out B's fields, the class header for A the first time it
encounters it, and then 3 times a just reference to the header (and
bye for tag and an int for the reference.
Maybe we're talking about different things here? But I'm sure Johan
will be able to take care of it :)
Eelco
Re: custom serialization seems to work...
I've turned that custom streaming off by default. Until we are absolutely, entirely, utmost sure it works without exceptions we should not commit it (leaving that line commented is fine by me though). Eelco On 2/11/07, Eelco Hillenius [EMAIL PROTECTED] wrote: Not entirely I'm afraid: 2007-02-11 16:25:47,595 ERROR wicket.protocol.http.FilePageStore - Error in page save thread java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at wicket.util.io.ClassStreamHandler.invokeWriteMethod(ClassStreamHandler.java:523) at wicket.util.io.WicketObjectOutputStream.writeObjectOverride(WicketObjectOutputStream.java:124) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:287) at wicket.util.io.ClassStreamHandler.writeFields(ClassStreamHandler.java:273) at wicket.util.io.WicketObjectOutputStream.writeObjectOverride(WicketObjectOutputStream.java:126) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:287) at wicket.util.lang.Objects.objectToByteArray(Objects.java:1037) at wicket.protocol.http.FilePageStore.serializePage(FilePageStore.java:414) at wicket.protocol.http.FilePageStore.access$4(FilePageStore.java:407) at wicket.protocol.http.FilePageStore$PageSavingThread.run(FilePageStore.java:601) at java.lang.Thread.run(Thread.java:613) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at wicket.util.io.ClassStreamHandler.invokeWriteMethod(ClassStreamHandler.java:511) ... 10 more Caused by: java.lang.NullPointerException at java.io.ObjectOutputStream.writeUTF(ObjectOutputStream.java:805) at wicket.protocol.http.request.urlcompressing.URLCompressor.writeObject(URLCompressor.java:105) ... 15 more Eelco
