On 6/11/08, Eric Jonas <[EMAIL PROTECTED]> wrote: > I've got cython base classes and I want to pickle subclasses of those > cython base classes. But those subclasses could have just about anything > as an attribute. Is that still going to screw me?
Yes, if your 'nodes' can have arbitrary Python attibutes, then implementing yourself a general mechanism ala pickle protocol is going to be a nightmare for you. Perhaps you can take a look at lxml, as Stefan pointed out? Perhaps you can borrow some good ideas. Have you your code available somewhere to take a look at your base classes you want to pickle? Or perhaps you can send me a private mail with some file attached? Perhaps the only way to make a clever suggestion is to look at the actual code ;-) > > ...Eric > > > > > > > > > > On Wed, 2008-06-11 at 12:35 -0300, Lisandro Dalcin wrote: > > > > Eric, I've tried hard in the past to speedup pickling for > > > > communication general Python object via MPI (for my project mpi4py). > > > > I'm now reimplementing mpi4py from scratch using Cython, but I've not > > > > yet reimplemented my pickle machinery. > > > > > > > > Could you try to output your pickles to a cStringIO instance (instead > > > > of a normal file instance) and tell me if you can get some speedup? If > > > > you get some speedup, then perhaps we can hack a bit to make it even > > > > faster for your use case... > > > > > > > > Of course, the other way would be to define from scratch a custom > > > > 'format' for saving your data, but I anticipate that that would be > > > > really a lot of work, and I even doubt you can get more speedup that > > > > with cPickle if you still want to be have to serialize arbitrary > > > > Python data... > > > > > > > > > > > > On 6/11/08, Eric Jonas <[EMAIL PROTECTED]> wrote: > > > > > I assume most of us are using Cython because normal python is too > slow > > > > > for our particular operation. I've been writing a compiler in > python > > > > > which generates large graphs with ~600k nodes, and I'd like to > > > > > serialize/checkpoint these to disk for later stages of the compiler > > > > > pipeline to use. However, at the moment, it takes ~60 seconds to > > > > > serialize the resulting graph to disk with cPickle. > > > > > > > > > > Have any Cython users found better/faster ways of serializing to > disk? I > > > > > could potentially use the "marshal" module, but my understanding > is that > > > > > it only works for built-in types, and of course, that the root of > my > > > > > class hierarchy I have four cython classes. I'd love to avoid > writing my > > > > > own serialize/deserialize methods, if at all possible. > > > > > > > > > > I can't be the only person in this position, > > > > > > > > > > Thanks, > > > > > ...Eric > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Cython-dev mailing list > > > > > [email protected] > > > > > http://codespeak.net/mailman/listinfo/cython-dev > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Cython-dev mailing list > > > [email protected] > > > http://codespeak.net/mailman/listinfo/cython-dev > > > > > > > > > _______________________________________________ > Cython-dev mailing list > [email protected] > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
