Re: [Twisted-Python] is twisted compatible with pickle?
While everything the people above me have said is correct, I would suggest
AMP as an alternative. It would allow you to send most basic Python data
types in your messages.
On Mar 29, 2013 2:32 AM, "succer...@tiscali.it"
wrote:
> I have made 2 application:
> The client extract data from a sql server (10k lines), and send every line
> pickled to a "collector" server via socket.
> The server uses twisted and receive every line, unpikle it and store the
> data in another sql server.
>
> Everytime i start sending data from client to server, in the first 200
> line (everytime a different line) **the server** throws an exception:
> SOMETIMES it something like:
>
> Traceback (most recent call last):
> File "collector2.py", line 81, in dataReceived
> self.count,account = pickle.loads(data)
> File "/usr/lib/python2.6/pickle.py", line 1374, in loads
> return Unpickler(file).load()
> File "/usr/lib/python2.6/pickle.py", line 858, in load
> dispatch[key](self)
> File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop
> del self.stack[-1]
> IndexError: list assignment index out of range
>
> But it's NOT every time the same. Printing my exception i red:
> Exception: pop from empty list
> Exception: list index out of range
> Exception: "'"
> Exception: list assignment index out of range
>
> Another strange errors is:
> File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
> __import__(module)
> exceptions.ImportError: No module named ond'
>
> for i in listaSAI:
> crm={}
> try:
> crm['uid']=i[0]
> except:
> crm['uid']=None
> try:
> crm['type_cond']=i[01]
> except:
> crm['type_cond']=None
> try:
> crm['population_id']=i[2]
> except:
> crm['population_id']=None
> try:
> crm['product_id']=i[3]
> except:
> crm['product_id']=None
> try:
> crm['a_id']=i[4]
> except:
> crm['a_id']=None
> try:
> crm['status']=i[5]
> except:
> crm['status']=None
> #time.sleep(0.001)
> serialized = pickle.dumps((count,crm))
> #print "sent num", count, crm
> s.sendall(serialized)
> count += 1
>
>
> And my server:
>
> def dataReceived(self, data):
> try:
> self.count,account = pickle.loads(data)
> except Exception as e:
> print "Eccezione:", e
> print self.count+1
> print data
> print traceback.print_exc()
>
>
>
> Printing the data in my client tells me that everything it's ok.
> *If i try to slow down the process of sending using time.sleep(0.01) in my
> client, EVERYTHING IS FINE, and no exception are raised.*
>
> What can i do to debug my code?
>
> p.s.
> I suspect that exceptions.ImportError: No module named ond' refers to
> "type_cond" key in crm.
>
>
> Invita i tuoi amici e Tiscali ti premia! Il consiglio di un amico vale più
> di uno spot in TV. Per ogni nuovo abbonato 30 € di premio per te e per lui!
> Un amico al mese e parli e navighi sempre gratis:
> http://freelosophy.tiscali.it/
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] is twisted compatible with pickle?
> > On Thu, Mar 28, 2013 at 6:24 PM, succer...@tiscali.it < > succer...@tiscali.it> wrote: > >> I have made 2 application: >> The client extract data from a sql server (10k lines), and send every >> line pickled to a "collector" server via socket. >> The server uses twisted and receive every line, unpikle it and store the >> data in another sql server. >> >> Everytime i start sending data from client to server, in the first 200 >> line (everytime a different line) **the server** throws an exception: >> SOMETIMES it something like: >> >> Traceback (most recent call last): >> File "collector2.py", line 81, in dataReceived >> self.count,account = pickle.loads(data) >> File "/usr/lib/python2.6/pickle.py", line 1374, in loads >> return Unpickler(file).load() >> File "/usr/lib/python2.6/pickle.py", line 858, in load >> dispatch[key](self) >> File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop >> del self.stack[-1] >> IndexError: list assignment index out of range >> >> > And my server: >> >> def dataReceived(self, data): >> try: >> self.count,account = pickle.loads(data) >> except Exception as e: >> print "Eccezione:", e >> print self.count+1 >> print data >> print traceback.print_exc() >> >> >> On Thu, Mar 28, 2013 at 10:07 PM, David Reid wrote: > dataReceived gets called with any data that is available on the socket. > That might not be all data you sent on the other side. To ensure complete > "messages" are delivered your application has to specify some framing, such > as Netstrings. > > See: > http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.NetstringReceiver.html > and > the original specification of netstrings > http://cr.yp.to/proto/netstrings.txt > > That being said, it's a very bad idea to send pickles over the network > because unpickling can result in arbitrary code execution. > > Peruse some of the results of > https://www.google.com/search?q=pickle+execute+arbitrary+code for > examples of these dangers. > > -David > > We also have a FAQ entry about this: http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whyisprotocol.dataReceivedcalledwithonlypartofthedataIcalledtransport.writewith -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] is twisted compatible with pickle?
dataReceived gets called with any data that is available on the socket.
That might not be all data you sent on the other side. To ensure complete
"messages" are delivered your application has to specify some framing, such
as Netstrings.
See:
http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.NetstringReceiver.htmland
the original specification of netstrings
http://cr.yp.to/proto/netstrings.txt
That being said, it's a very bad idea to send pickles over the network
because unpickling can result in arbitrary code execution.
Peruse some of the results of
https://www.google.com/search?q=pickle+execute+arbitrary+code for examples
of these dangers.
-David
On Thu, Mar 28, 2013 at 6:24 PM, succer...@tiscali.it
wrote:
> I have made 2 application:
> The client extract data from a sql server (10k lines), and send every line
> pickled to a "collector" server via socket.
> The server uses twisted and receive every line, unpikle it and store the
> data in another sql server.
>
> Everytime i start sending data from client to server, in the first 200
> line (everytime a different line) **the server** throws an exception:
> SOMETIMES it something like:
>
> Traceback (most recent call last):
> File "collector2.py", line 81, in dataReceived
> self.count,account = pickle.loads(data)
> File "/usr/lib/python2.6/pickle.py", line 1374, in loads
> return Unpickler(file).load()
> File "/usr/lib/python2.6/pickle.py", line 858, in load
> dispatch[key](self)
> File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop
> del self.stack[-1]
> IndexError: list assignment index out of range
>
> But it's NOT every time the same. Printing my exception i red:
> Exception: pop from empty list
> Exception: list index out of range
> Exception: "'"
> Exception: list assignment index out of range
>
> Another strange errors is:
> File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
> __import__(module)
> exceptions.ImportError: No module named ond'
>
> for i in listaSAI:
> crm={}
> try:
> crm['uid']=i[0]
> except:
> crm['uid']=None
> try:
> crm['type_cond']=i[01]
> except:
> crm['type_cond']=None
> try:
> crm['population_id']=i[2]
> except:
> crm['population_id']=None
> try:
> crm['product_id']=i[3]
> except:
> crm['product_id']=None
> try:
> crm['a_id']=i[4]
> except:
> crm['a_id']=None
> try:
> crm['status']=i[5]
> except:
> crm['status']=None
> #time.sleep(0.001)
> serialized = pickle.dumps((count,crm))
> #print "sent num", count, crm
> s.sendall(serialized)
> count += 1
>
>
> And my server:
>
> def dataReceived(self, data):
> try:
> self.count,account = pickle.loads(data)
> except Exception as e:
> print "Eccezione:", e
> print self.count+1
> print data
> print traceback.print_exc()
>
>
>
> Printing the data in my client tells me that everything it's ok.
> *If i try to slow down the process of sending using time.sleep(0.01) in my
> client, EVERYTHING IS FINE, and no exception are raised.*
>
> What can i do to debug my code?
>
> p.s.
> I suspect that exceptions.ImportError: No module named ond' refers to
> "type_cond" key in crm.
>
>
> Invita i tuoi amici e Tiscali ti premia! Il consiglio di un amico vale più
> di uno spot in TV. Per ogni nuovo abbonato 30 € di premio per te e per lui!
> Un amico al mese e parli e navighi sempre gratis:
> http://freelosophy.tiscali.it/
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] is twisted compatible with pickle?
I have made 2 application: The client extract data from a sql server (10k
lines), and send every line pickled to a "collector" server via socket.The
server uses twisted and receive every line, unpikle it and store the data in
another sql server.
Everytime i start sending data from client to server, in the first 200 line
(everytime a different line) **the server** throws an exception:SOMETIMES it
something like:
Traceback (most recent call last): File "collector2.py", line 81, in
dataReceivedself.count,account= pickle.loads(data) File
"/usr/lib/python2.6/pickle.py", line 1374, in loadsreturn
Unpickler(file).load() File "/usr/lib/python2.6/pickle.py", line 858, in
loaddispatch[key](self) File "/usr/lib/python2.6/pickle.py", line
1138, in load_popdel self.stack[-1]IndexError: list assignment
index out of range
But it's NOT every time the same. Printing my exception i red: Exception: pop
from empty listException: list index out of rangeException: "'"Exception: list
assignment index out of range
Another strange errors is:File "/usr/lib/python2.6/pickle.py", line 1124, in
find_class__import__(module)exceptions.ImportError: No module named ond'
for i in listaSAI: crm={}try: crm['uid']=i[0]
except: crm['uid']=None try:
crm['type_cond']=i[01] except: crm['type_cond']=None
try:crm['population_id']=i[2] except:
crm['population_id']=None try:
crm['product_id']=i[3] except: crm['product_id']=None
try:crm['a_id']=i[4]except:
crm['a_id']=Nonetry:crm['status']=i[5] except:
crm['status']=None #time.sleep(0.001) serialized =
pickle.dumps((count,crm)) #print "sent num", count, crm
s.sendall(serialized) count += 1
And my server:
def dataReceived(self, data): try:
self.count,account = pickle.loads(data)except
Exception as e: print "Eccezione:", e print
self.count+1 print data print traceback.print_exc()
Printing the data in my client tells me that everything it's ok.*If i try to
slow down the process of sending using time.sleep(0.01) in my client,
EVERYTHING IS FINE, and no exception are raised.*
What can i do to debug my code?
p.s. I suspect that exceptions.ImportError: No module named ond' refers to
"type_cond" key in crm.
Invita i tuoi amici e Tiscali ti premia! Il consiglio di un amico vale più di
uno spot in TV. Per ogni nuovo abbonato 30 € di premio per te e per lui! Un
amico al mese e parli e navighi sempre gratis: http://freelosophy.tiscali.it/
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
