Re: [Chicken-users] s11n among different chicken versions
Daishi, A while ago we had a discussion on how portable s11n is, and how portable it should strive to be. Felix gave a technical explanation of the implementation here: http://lists.gnu.org/archive/html/chicken-users/2005-11/msg00105.html. The eggdoc also mentions that serialized data is endian- and word-size-dependent, and when procedures/continuations are serialized this dependency extends to identical binaries. The egg is not intended to be terribly robust as S-exprs are (sometimes) a good alternative. I believe if you do not serialize procedures or continuations, it should deserialize across Chicken versions, but I have not tried. As an aside, I -think- endianness is not a factor any more, I had provided a simple patch which removed this limitation. This was only to make testing easier for me on a heterogeneous network. Further robustness was deemed outside the scope of the egg. I hope this helps. On 7/20/06, Daishi Kato <[EMAIL PROTECTED]> wrote: Hi, Is it well known that a serialization and a deserialization are not compatible between different chicken versions? Any way for convertion? any workaround? Thanks, Daishi ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
Oops, sorry about that. I'll check the past discussion. Daishi At Thu, 20 Jul 2006 21:50:05 -0500, Zbigniew wrote: > > Daishi, > > A while ago we had a discussion on how portable s11n is, and how > portable it should strive to be. Felix gave a technical explanation > of the implementation here: > http://lists.gnu.org/archive/html/chicken-users/2005-11/msg00105.html. > The eggdoc also mentions that serialized data is endian- and > word-size-dependent, and when procedures/continuations are serialized > this dependency extends to identical binaries. The egg is not > intended to be terribly robust as S-exprs are (sometimes) a good > alternative. > > I believe if you do not serialize procedures or continuations, it > should deserialize across Chicken versions, but I have not tried. > > As an aside, I -think- endianness is not a factor any more, I had > provided a simple patch which removed this limitation. This was only > to make testing easier for me on a heterogeneous network. Further > robustness was deemed outside the scope of the egg. > > I hope this helps. > > On 7/20/06, Daishi Kato <[EMAIL PROTECTED]> wrote: > > Hi, > > > > Is it well known that a serialization and a deserialization > > are not compatible between different chicken versions? > > Any way for convertion? any workaround? > > > > Thanks, > > Daishi > > > > > > ___ > > Chicken-users mailing list > > Chicken-users@nongnu.org > > http://lists.nongnu.org/mailman/listinfo/chicken-users > > ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
Hm, maybe I don't need s11n for my purpose. I simply would like to store hash-tables to a storage. These hash tables are normal, meaning the procedures are default of "hash" and "equal?". So, the question is if there is an easy way to "write" a hash-table. Converting a hash-table into an alist looks the way to go, except that my hash-tables are nested. Any idea would be welcome. Daishi At Fri, 21 Jul 2006 11:56:04 +0900, Daishi Kato wrote: > > Oops, sorry about that. > I'll check the past discussion. > Daishi > > At Thu, 20 Jul 2006 21:50:05 -0500, > Zbigniew wrote: > > > > Daishi, > > > > A while ago we had a discussion on how portable s11n is, and how > > portable it should strive to be. Felix gave a technical explanation > > of the implementation here: > > http://lists.gnu.org/archive/html/chicken-users/2005-11/msg00105.html. > > The eggdoc also mentions that serialized data is endian- and > > word-size-dependent, and when procedures/continuations are serialized > > this dependency extends to identical binaries. The egg is not > > intended to be terribly robust as S-exprs are (sometimes) a good > > alternative. > > > > I believe if you do not serialize procedures or continuations, it > > should deserialize across Chicken versions, but I have not tried. > > > > As an aside, I -think- endianness is not a factor any more, I had > > provided a simple patch which removed this limitation. This was only > > to make testing easier for me on a heterogeneous network. Further > > robustness was deemed outside the scope of the egg. > > > > I hope this helps. > > > > On 7/20/06, Daishi Kato <[EMAIL PROTECTED]> wrote: > > > Hi, > > > > > > Is it well known that a serialization and a deserialization > > > are not compatible between different chicken versions? > > > Any way for convertion? any workaround? > > > > > > Thanks, > > > Daishi > > > > > > > > > ___ > > > Chicken-users mailing list > > > Chicken-users@nongnu.org > > > http://lists.nongnu.org/mailman/listinfo/chicken-users > > > ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 7/21/06, Daishi Kato <[EMAIL PROTECTED]> wrote: Hi, Is it well known that a serialization and a deserialization are not compatible between different chicken versions? Any way for convertion? any workaround? The remark about versions means, that it *can* be incompatible among chicken versions, for example if object layout changes. If you have control over the installed chicken version, then this is not a problem. s11n is *not* intended as a general, platform-independent information interchange format - it is intended as an efficient binary storage format for arbitrary, even cyclic data. cheers, felix -- http://galinha.ucpel.tche.br:8081/blog/blog.ssp ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 7/21/06, Daishi Kato <[EMAIL PROTECTED]> wrote:
So, the question is if there is an easy way to "write" a hash-table.
Converting a hash-table into an alist looks the way to go,
except that my hash-tables are nested.
Any idea would be welcome.
Well, I would use s11n ;-) Otherwise, using your own
mapping from hash-tables to a-lists would be the way
to go. You might also use SRFI-10 ("#,...") syntax to integrate
this into the reader.
cheers,
felix
--
http://galinha.ucpel.tche.br:8081/blog/blog.ssp
___
Chicken-users mailing list
Chicken-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
At Fri, 21 Jul 2006 08:58:50 +0200,
felix winkelmann wrote:
>
> On 7/21/06, Daishi Kato <[EMAIL PROTECTED]> wrote:
>
> > So, the question is if there is an easy way to "write" a hash-table.
> > Converting a hash-table into an alist looks the way to go,
> > except that my hash-tables are nested.
> > Any idea would be welcome.
>
>
> Well, I would use s11n ;-) Otherwise, using your own
> mapping from hash-tables to a-lists would be the way
> to go. You might also use SRFI-10 ("#,...") syntax to integrate
> this into the reader.
Does SRFI-10 keep identity?
In other words, how can it handle this: (let ([x (cons 1 1)]) (cons x x))
OK, it seems s11n is easier for my purpose.
Although I still do not understand the internals of s11n,
my proposal is to allow deserialize to have an optional
fallback deserializer, just like serialize has an optional
fallbak serializer; why not?
Thanks,
Daishi
___
Chicken-users mailing list
Chicken-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 7/21/06, Daishi Kato <[EMAIL PROTECTED]> wrote: Does SRFI-10 keep identity? In other words, how can it handle this: (let ([x (cons 1 1)]) (cons x x)) No, it's just a means of extending the reader. OK, it seems s11n is easier for my purpose. Although I still do not understand the internals of s11n, my proposal is to allow deserialize to have an optional fallback deserializer, just like serialize has an optional fallbak serializer; why not? Good point. I'll take a look. cheers, felix -- http://galinha.ucpel.tche.br:8081/blog/blog.ssp ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
Looks like the s11n egg is updated. So I tried, only to see I'm missing something. Here is my fallback func: (define (deserialize-fallback id) (cond ((string=? id "f_3497library.scm") ; chicken-2.315 equal?) ((string=? id "f_6914extras.scm") ; chicken-2.315 hash) (else (error "deserialize-fallback unknown id" id What is missing? Daishi On 7/21/06, felix winkelmann <[EMAIL PROTECTED]> wrote: On 7/21/06, Daishi Kato <[EMAIL PROTECTED]> wrote: > > Does SRFI-10 keep identity? > In other words, how can it handle this: (let ([x (cons 1 1)]) (cons x x)) No, it's just a means of extending the reader. > > OK, it seems s11n is easier for my purpose. > Although I still do not understand the internals of s11n, > my proposal is to allow deserialize to have an optional > fallback deserializer, just like serialize has an optional > fallbak serializer; why not? Good point. I'll take a look. cheers, felix -- http://galinha.ucpel.tche.br:8081/blog/blog.ssp ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 8/1/06, Daishi Kato <[EMAIL PROTECTED]> wrote: Looks like the s11n egg is updated. So I tried, only to see I'm missing something. Here is my fallback func: (define (deserialize-fallback id) (cond ((string=? id "f_3497library.scm") ; chicken-2.315 equal?) ((string=? id "f_6914extras.scm") ; chicken-2.315 hash) (else (error "deserialize-fallback unknown id" id What is missing? I haven't seen your error-message yet. (But note that this is likely to fail, since the ID will probably be re-used and now name different procedures. This approach is not safe). cheers, felix -- http://galinha.ucpel.tche.br:8081/blog/blog.ssp ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
So, this actually is a correct way of using fallback, isn't it? OK, I'll look for my real problem. About the safety, don't worry too much, since this is just a temporal solution, and I already confirmed with chicken-dump that procedures in my file are only equal? and hash. Daishi On 8/1/06, felix winkelmann <[EMAIL PROTECTED]> wrote: On 8/1/06, Daishi Kato <[EMAIL PROTECTED]> wrote: > Looks like the s11n egg is updated. So I tried, only to see I'm > missing something. > > Here is my fallback func: > > (define (deserialize-fallback id) > (cond ((string=? id "f_3497library.scm") ; chicken-2.315 > equal?) > ((string=? id "f_6914extras.scm") ; chicken-2.315 > hash) > (else > (error "deserialize-fallback unknown id" id > > What is missing? > I haven't seen your error-message yet. (But note that this is likely to fail, since the ID will probably be re-used and now name different procedures. This approach is not safe). cheers, felix -- http://galinha.ucpel.tche.br:8081/blog/blog.ssp ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
Now, I can reproduce the problem. % cat dump.scm (use s11n) (define x (make-hash-table)) (hash-table-set! x 'aaa (make-hash-table)) (hash-table-set! x 'bbb (make-hash-table)) (with-output-to-file "dump" (lambda () (serialize x))) % cat undump.scm (use s11n) (define (deserialize-fallback id) (cond ((string=? id "f_3497library.scm") ; chicken-2.315 equal?) ((string=? id "f_6914extras.scm") ; chicken-2.315 hash) (else (error "deserialize-fallback unknown id" id (with-input-from-file "dump" (lambda () (deserialize (current-input-port) deserialize-fallback))) % /usr/local/chicken-2.315/bin/csi -q -s dump.scm % /usr/local/chicken-2.41/bin/csi -q -s undump.scm Error: (for-each) argument is not a proper list: ((aaa . #) . #) Call history: (##sys#require (quote s11n)) (with-input-from-file "dump" (lambda () (deserialize (current-input-port) deserialize-fallback))) (deserialize (current-input-port) deserialize-fallback) (current-input-port) [deserialize-fallback] (string=? id "f_3497library.scm") [deserialize-fallback] (string=? id "f_3497library.scm") [deserialize-fallback] (string=? id "f_6914extras.scm")<-- % chicken-dump dump File: dump #0: hash table: #1:(items) pair: #2:(car) pair: #3:(car) symbol: #4:(name) string "aaa" #5:(cdr) hash table: (items) () #6:(comparison procedure) procedure: #7:(id) string "f_3497library.scm" #8:[1] lambda info # #9:(hash function) procedure: #10:(id) string "f_6914extras.scm" #11:[1] lambda info # #12:(cdr) pair: #13:(car) pair: #14:(car) symbol: #15:(name) string "bbb" #16:(cdr) hash table: (items) () (comparison procedure) back reference -> #6 (hash function) back reference -> #9 (cdr) () (comparison procedure) back reference -> #6 (hash function) back reference -> #9 How's it? Daishi On 8/1/06, Daishi Kato <[EMAIL PROTECTED]> wrote: So, this actually is a correct way of using fallback, isn't it? OK, I'll look for my real problem. About the safety, don't worry too much, since this is just a temporal solution, and I already confirmed with chicken-dump that procedures in my file are only equal? and hash. Daishi On 8/1/06, felix winkelmann <[EMAIL PROTECTED]> wrote: > On 8/1/06, Daishi Kato <[EMAIL PROTECTED]> wrote: > > Looks like the s11n egg is updated. So I tried, only to see I'm > > missing something. > > > > Here is my fallback func: > > > > (define (deserialize-fallback id) > > (cond ((string=? id "f_3497library.scm") ; chicken-2.315 > > equal?) > > ((string=? id "f_6914extras.scm") ; chicken-2.315 > > hash) > > (else > > (error "deserialize-fallback unknown id" id > > > > What is missing? > > > > I haven't seen your error-message yet. > (But note that this is likely to fail, since the ID will probably be re-used > and now name different procedures. This approach is not safe). > > > cheers, > felix > -- > http://galinha.ucpel.tche.br:8081/blog/blog.ssp > ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 8/3/06, Daishi Kato <[EMAIL PROTECTED]> wrote: Now, I can reproduce the problem. % cat dump.scm (use s11n) (define x (make-hash-table)) (hash-table-set! x 'aaa (make-hash-table)) (hash-table-set! x 'bbb (make-hash-table)) (with-output-to-file "dump" (lambda () (serialize x))) % cat undump.scm (use s11n) (define (deserialize-fallback id) (cond ((string=? id "f_3497library.scm") ; chicken-2.315 equal?) ((string=? id "f_6914extras.scm") ; chicken-2.315 hash) (else (error "deserialize-fallback unknown id" id (with-input-from-file "dump" (lambda () (deserialize (current-input-port) deserialize-fallback))) % /usr/local/chicken-2.315/bin/csi -q -s dump.scm % /usr/local/chicken-2.41/bin/csi -q -s undump.scm Error: (for-each) argument is not a proper list: ((aaa . #) . #) Call history: (##sys#require (quote s11n)) (with-input-from-file "dump" (lambda () (deserialize (current-input-port) deserialize-fallback))) (deserialize (current-input-port) deserialize-fallback) (current-input-port) [deserialize-fallback] (string=? id "f_3497library.scm") [deserialize-fallback] (string=? id "f_3497library.scm") [deserialize-fallback] (string=? id "f_6914extras.scm")<-- I can't say much about this problem. As I already said, the procedure-IDs might be re-used, so this approach is definitely not safe. cheers, felix -- http://galinha.ucpel.tche.br:8081/blog/blog.ssp ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 8/7/06, felix winkelmann <[EMAIL PROTECTED]> wrote: On 8/3/06, Daishi Kato <[EMAIL PROTECTED]> wrote: > Now, I can reproduce the problem. > > % cat dump.scm > (use s11n) > (define x (make-hash-table)) > (hash-table-set! x 'aaa (make-hash-table)) > (hash-table-set! x 'bbb (make-hash-table)) > (with-output-to-file "dump" > (lambda () > (serialize x))) > % cat undump.scm > (use s11n) > (define (deserialize-fallback id) > (cond ((string=? id "f_3497library.scm") ; chicken-2.315 > equal?) > ((string=? id "f_6914extras.scm") ; chicken-2.315 > hash) > (else > (error "deserialize-fallback unknown id" id > (with-input-from-file "dump" > (lambda () > (deserialize (current-input-port) deserialize-fallback))) > % /usr/local/chicken-2.315/bin/csi -q -s dump.scm > % /usr/local/chicken-2.41/bin/csi -q -s undump.scm > Error: (for-each) argument is not a proper list: ((aaa . > #) . #) > > Call history: > > (##sys#require (quote s11n)) > (with-input-from-file "dump" (lambda () > (deserialize (current-input-port) deserialize-fallback))) > (deserialize (current-input-port) deserialize-fallback) > (current-input-port) > [deserialize-fallback] (string=? id "f_3497library.scm") > [deserialize-fallback] (string=? id "f_3497library.scm") > [deserialize-fallback] (string=? id > "f_6914extras.scm")<-- I can't say much about this problem. As I already said, the procedure-IDs might be re-used, so this approach is definitely not safe. If the procedure ID is really re-used, the fallback procedure is never called, isn't it? If I could not find any solution to this, my final resort would be editting the binary dump file to change procedure IDs. Daishi ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 8/7/06, Daishi Kato <[EMAIL PROTECTED]> wrote: If the procedure ID is really re-used, the fallback procedure is never called, isn't it? That's correct, unfortunately. If I could not find any solution to this, my final resort would be editting the binary dump file to change procedure IDs. I think it would be better to write a custom version of "deserialize", the last few lines look like this: (switch tag ... (procedure-tag (let ((id (fetch))) ;; >> do some custom processing here <<< (cond ((##core#inline "set_procedure_ptr" x (##sys#make-c-string id)) (fetchslots x 1) ) (fallback (let ((proc (fallback id))) (vector-set! backrefs r proc) proc) ) (else (error 'deserialize "unable to deserialize procedure - no table entry found" id))) ) ) (else (error 'deserialize "invalid tag" tag)) ) ) ) ) ) ) (fetch) ) ) It shouldn't be to hard to add some hook there to do any custom handling of specific IDs. But, again, it's slippy terrain you're moving on. cheers, felix -- http://galinha.ucpel.tche.br:8081/blog/blog.ssp ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
On 8/7/06, felix winkelmann <[EMAIL PROTECTED]> wrote: On 8/7/06, Daishi Kato <[EMAIL PROTECTED]> wrote: > > If the procedure ID is really re-used, the fallback procedure is never > called, isn't it? That's correct, unfortunately. OK, I found another weird behavior. Procedures of hash-tables are referenced in a dump file, but when I run deserialize (w/ different chicken version), the fallback is called more then once for the same ID. I'm afraid there is a bug in backref handling with the procedure returned by the fallback. > > If I could not find any solution to this, my final resort would be > editting the binary dump file to change procedure IDs. I think it would be better to write a custom version of "deserialize", the last few lines look like this: (switch tag ... (procedure-tag (let ((id (fetch))) ;; >> do some custom processing here <<< (cond ((##core#inline "set_procedure_ptr" x (##sys#make-c-string id)) (fetchslots x 1) ) (fallback (let ((proc (fallback id))) (vector-set! backrefs r proc) proc) ) (else (error 'deserialize "unable to deserialize procedure - no table entry found" id))) ) ) (else (error 'deserialize "invalid tag" tag)) ) ) ) ) ) ) (fetch) ) ) It shouldn't be to hard to add some hook there to do any custom handling of specific IDs. But, again, it's slippy terrain you're moving on. I understand it's slippy, but I also think it's useful. Isn't it possible to add a trap at "do some custom processing here" that calls an optional procedure, just like the fallback procedure, so that I don't need to modify the s11n code? In this case, it is desired to pass not only a procedure ID but also the lambda info that is shown when you run chicken-dump. I hope this makes sense. Daishi ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] s11n among different chicken versions
felix, Should this be a bug? Daishi Index: s11n/s11n.scm === --- s11n/s11n.scm (revision 1371) +++ s11n/s11n.scm (working copy) @@ -237,6 +237,7 @@ (cond ((##core#inline "set_procedure_ptr" x (##sys#make-c-string id)) (fetchslots x 1) ) (fallback +(fetch) (let ((proc (fallback id))) (vector-set! backrefs r proc) proc) ) ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
