Lisandro,
One more question: what did you mean by
> you will need to implement a custom way for data persistence. Hope you
> do not have general Python object attributes in your base classes!.
I've got cython base classes and I want to pickle subclasses of those
cython base classes. But those subclasses could have just about anything
as an attribute. Is that still going to screw me?
Thanks again,
...Eric
>
>
> > On Wed, 2008-06-11 at 12:35 -0300, Lisandro Dalcin wrote:
> > > Eric, I've tried hard in the past to speedup pickling for
> > > communication general Python object via MPI (for my project mpi4py).
> > > I'm now reimplementing mpi4py from scratch using Cython, but I've not
> > > yet reimplemented my pickle machinery.
> > >
> > > Could you try to output your pickles to a cStringIO instance (instead
> > > of a normal file instance) and tell me if you can get some speedup? If
> > > you get some speedup, then perhaps we can hack a bit to make it even
> > > faster for your use case...
> > >
> > > Of course, the other way would be to define from scratch a custom
> > > 'format' for saving your data, but I anticipate that that would be
> > > really a lot of work, and I even doubt you can get more speedup that
> > > with cPickle if you still want to be have to serialize arbitrary
> > > Python data...
> > >
> > >
> > > On 6/11/08, Eric Jonas <[EMAIL PROTECTED]> wrote:
> > > > I assume most of us are using Cython because normal python is too slow
> > > > for our particular operation. I've been writing a compiler in python
> > > > which generates large graphs with ~600k nodes, and I'd like to
> > > > serialize/checkpoint these to disk for later stages of the compiler
> > > > pipeline to use. However, at the moment, it takes ~60 seconds to
> > > > serialize the resulting graph to disk with cPickle.
> > > >
> > > > Have any Cython users found better/faster ways of serializing to
> > disk? I
> > > > could potentially use the "marshal" module, but my understanding is
> > that
> > > > it only works for built-in types, and of course, that the root of my
> > > > class hierarchy I have four cython classes. I'd love to avoid writing
> > my
> > > > own serialize/deserialize methods, if at all possible.
> > > >
> > > > I can't be the only person in this position,
> > > >
> > > > Thanks,
> > > > ...Eric
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Cython-dev mailing list
> > > > [email protected]
> > > > http://codespeak.net/mailman/listinfo/cython-dev
> > > >
> > >
> > >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > [email protected]
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
>
>
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
