Pickled objects over the network

I'm working on a distributed computing program and need to send Python objects over a TCP socket. Specifically, the objects that I'm working with subclass the builtin list type (I don't know whether or not that matters), but also include other data fields. These objects are put into dictionaries a

Pickled objects over the network

Irmen de Jong wrote > In what way would Pyro be overkill where Yaml (also a module that you need > to install separately) wouldn't be? Sure they are the same to install and sure pyro can do the job (pyro is a nice package). But I got the impression that the questioner wanted to do the networking

Re: Pickled objects over the network

Walker Lindley wrote: 8< complaint about pickle error on receiving side - Google for netstrings. It looks to me like you are trying to unpickle something that is not a whole pickle... Append the length of the pickle to the front of it, and on the receiving side, first receive th

Re: Pickled objects over the network

On Tue, 17 Jul 2007 14:57:16 -0700, Walker Lindley <[EMAIL PROTECTED]> wrote: >I'm working on a distributed computing program and need to send Python >objects over a TCP socket. Specifically, the objects that I'm working with >subclass the builtin list type (I don't know whether or not that matters

Re: Pickled objects over the network

Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > On Tue, 17 Jul 2007 14:57:16 -0700, Walker Lindley <[EMAIL PROTECTED]> wrote: > >I'm working on a distributed computing program and need to send Python > >objects over a TCP socket. [snip] > >Hopefully I'm doing something obviously wrong, but if any

Re: Pickled objects over the network

Thanks for all the help, I tried sending the length and then the string and that appears to work, so I'll take a look at Pyro, too. -Walker On 7/18/07, Nick Craig-Wood <[EMAIL PROTECTED]> wrote: Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > On Tue, 17 Jul 2007 14:57:16 -0700, Walker Lindl

Re: Pickled objects over the network

Sure pyro may be the solution but it may also be overkill Why not use safe_load from the yaml module? -- http://mail.python.org/mailman/listinfo/python-list

Re: Pickled objects over the network

On 7/18/07, Jean-Paul Calderone <[EMAIL PROTECTED]> wrote: > On Tue, 17 Jul 2007 14:57:16 -0700, Walker Lindley <[EMAIL PROTECTED]> wrote: [...] > The obvious thing you're doing wrong is using pickle over a network. ;) > > http://jcalderone.livejournal.com/15864.html Ok, maybe not the best tools

Re: Pickled objects over the network

I tried implementing the sending the length and then the pickle string method and that worked most of the time. The problem we ran into was if the string got split up into multiple packets and you read the first one and tried to unpickle it, you'd get an error. So a while loop that keeps calling r

Re: Pickled objects over the network

Rustom Mody wrote: > Sure pyro may be the solution but it may also be overkill > Why not use safe_load from the yaml module? In what way would Pyro be overkill where Yaml (also a module that you need to install separately) wouldn't be? -irmen -- http://mail.python.org/mailman/listinfo/python-lis

Re: Pickled objects over the network

Right, I could use Pyro, but I don't need RPC, I just wanted an easy way to send objects across the network. I'm sure both Pyro and Yami can do that and I may end up using one of them. For the initial version pickle will work because we have the networking issues figured out with it, just not the

Re: Pickled objects over the network

Walker Lindley wrote: >Right, I could use Pyro, but I don't need RPC, I just wanted an easy way to send objects across the network. I'm sure >both Pyro and Yami can do that and I may end up using one of them. For the initial version pickle will work because we >have the networking issues figured

Re: Pickled objects over the network

Thats just what I was saying: If the IPC issues have been solved on your own terms but the data-serialization -- what you call making strings -- are not so satisfactory, you could use yaml (with safe_load) as a replacement for pickle and continue to use networking and IPC as you are currently doin

Re: Pickled objects over the network

On Fri, 20 Jul 2007 09:32:17 +0200, Hendrik van Rooyen <[EMAIL PROTECTED]> wrote: > Walker Lindley wrote: > >>Right, I could use Pyro, but I don't need RPC, I just wanted an easy way to >send objects across the network. I'm sure >both Pyro and Yami can do that and I >may end up using one of them.

Re: Pickled objects over the network

Rustom Mody <[EMAIL PROTECTED]> wrote: >Since pickle has problems >-- does not interface well with networking In what way does pickle not interface well with networking? Other than, possibly, security issues which you list as a separate problem with it. I've taken a working XML-RPC system and repl

Re: Pickled objects over the network

Hendrik van Rooyen wrote: > Walker Lindley wrote: > >> Right, I could use Pyro, but I don't need RPC, I just wanted an easy way to > send objects across the network. I'm sure >both Pyro and Yami can do that and > I > may end up using one of them. For the initial version pickle will work because

Re: Pickled objects over the network

It is feasible to an extent since loading each builtin object type is handled by a different function. However, as others have pointed out it makes more sense to use a more robust protocol than try to patch pickle. -Walker On 7/20/07, Hendrik van Rooyen <[EMAIL PROTECTED]> wrote: Walker Lindl

Re: Pickled objects over the network

It doesn't interface well because the string you end up with often doesn't fit into a single packet. Therefore you have to add a layer of protocol on top of it that allows you to check to make sure you have the whole string received before trying to unpickle it. This the case even if you use socke

Re: Pickled objects over the network

On Fri, 20 Jul 2007 08:27:13 -0700, Walker Lindley <[EMAIL PROTECTED]> wrote: >It doesn't interface well because the string you end up with often doesn't >fit into a single packet. Therefore you have to add a layer of protocol on >top of it that allows you to check to make sure you have the whole s

Re: Pickled objects over the network

Right, it's just a big problem with pickle because if you do a recv at the wrong time and try to load it with pickle, you'll start getting weird errors down in the pickle module as opposed to just not getting the full string you expected if you were using plaintext strings. This is probably me bei

Re: Pickled objects over the network

"Steve Holden" <[EMAIL PROTECTED]> wrote: > Hmm, I suspect I detect the sounds of the square wheel being reinvented. Very helpful, thank you, Steve - Now how about pointing out in which direction the round wheels are kept, and what their monikers are? - Hendrik -- http://mail.python.org/mailma

Re: Pickled objects over the network

Hendrik van Rooyen wrote: > "Steve Holden" <[EMAIL PROTECTED]> wrote: > >> Hmm, I suspect I detect the sounds of the square wheel being reinvented. > > Very helpful, thank you, Steve - Now how about pointing out in which > direction the round wheels are kept, and what their monikers are? > I thi

Re: Pickled objects over the network

"Steve Holden" <[EMAIL PROTECTED]> wrote: > I think someone has already pointed out netstrings, which will allow you > to send arbitrary strings over network connections deterministically. Yes I brought it up > I'm afraid for the rest it's just a matter of encoding your information > in a way

Re: Pickled objects over the network

Hendrik van Rooyen wrote: > "Steve Holden" <[EMAIL PROTECTED]> wrote: > >> I think someone has already pointed out netstrings, which will allow you >> to send arbitrary strings over network connections deterministically. > > Yes I brought it up > >> I'm afraid for the rest it's just a matter o

Re: Pickled objects over the network

"Steve Holden" <[EMAIL PROTECTED]> wrote: > Yes. Why? - Hendrik -- http://mail.python.org/mailman/listinfo/python-list

Re: Pickled objects over the network

Hendrik van Rooyen wrote: > "Steve Holden" <[EMAIL PROTECTED]> wrote: > >> Yes. > > Why? > It's difficult to establish, and then correctly implement, almost any security protocol without leaving cracks that attackers can lever open and use to inject code into your process's memory space. By a

Re: Pickled objects over the network

"Steve Holden" wrote: > It's difficult to establish, and then correctly implement, almost any > security protocol without leaving cracks that attackers can lever open > and use to inject code into your process's memory space. I can accept this - its difficult enough to write a receiver that sync

Re: Pickled objects over the network

Hendrik van Rooyen ha scritto: > But more seriously - is there any need for a simple serialiser that will > be able to be used to transfer a subset of the built in types over an > open network in a safe manner, for the transfer of things like lists of > parameters? Yes, there seems to be a need f

Re: Pickled objects over the network

"Marco Mariani" wrote: > Hendrik van Rooyen ha scritto: > > > But more seriously - is there any need for a simple serialiser that will > > be able to be used to transfer a subset of the built in types over an > > open network in a safe manner, for the transfer of things like lists of > > parame

Re: Pickled objects over the network

"Hendrik van Rooyen" <[EMAIL PROTECTED]> writes: > But more seriously - is there any need for a simple serialiser that will > be able to be used to transfer a subset of the built in types over an > open network in a safe manner, for the transfer of things like lists of > parameters? > > Or am I th

Re: Pickled objects over the network

"Paul Rubin" wrote: > "Hendrik van Rooyen" writes: > > But more seriously - is there any need for a simple serialiser that will > > be able to be used to transfer a subset of the built in types over an > > open network in a safe manner, for the transfer of things like lis