RE: [OT] Off-Topic Posts and Threads on the Python Mailing List
Lol you all sound like google's angry birds with their feathers ruffled by a comment. You guys should open up another mailing list to extinguish your virtually bruised egos. . . . On Sep 30, 2011 10:27 PM, Prasad, Ramit ramit.pra...@jpmorgan.com wrote: -- http://mail.python.org/mailman/listinfo/python-list
Re: [OT] Benefit and belief
Hi, I am looking for the python mailing list. . ? Have you guys seen it somewhere? I think I accidently reached the cry-me-a-river list? Regards, Nav On Sep 30, 2011 1:03 AM, Petite Abeille petite.abei...@gmail.com wrote: On Sep 29, 2011, at 8:49 PM, Devin Jeanpierre wrote: It could certainly be _interpreted_ as an attack (and was interpreted that way), and that's really all that's necessary for a hostile environment. In other news: http://alt.textdrive.com/assets/public/non/nq050616.gif -- Tout le monde il est beau, tout le monde il est gentil -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Decision on python technologies
Hi Guys, I have been a python developer for a bit now and for the life of me I am not being able to decide something. I am trying to develop a web based application in python. I am torn between using python 2 or 3. All the good frameworks are still in 2.x. Now, cherrypy, sqlalchemy and jinja2 support python 3. But do I really want to do all the boilerplate work again? I have this strong urge to use python 3 and call it my indecisiveness , I am somehow not wanting to use 2.x, though it has everything I need to build my app. Hence, I finally decided to turn to the community for helping me make this decision. Please help. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
How to handle application level errors
Hi Guys, Not sure if this is the place to ask, but I am trying find out a way to handle application level errors from a global config. Any help would be really appreciated. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Sqlalchemy + Cherrypy
Hi Guys, I am not sure if this is the right place to put this question. I am trying to figure out what the proper/clean way is to integrate cherrypy and sqlalchemy? I am currently trying to do this cherrypy 3 and sqlalchemy .7 in python 3.2 a) Cherrypy tools b) Integrate it directly into the app Any help will be really appreciated. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Inheriting Object
Hi Guys, I have been wondering for a while now as to why some classes inherit Object? And what does it really do for the class? Can anyone shed some light on this? Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Inheriting Object
On Fri, May 20, 2011 at 8:25 AM, MRAB pyt...@mrabarnett.plus.com wrote: On 20/05/2011 03:13, Navkirat Singh wrote: Hi Guys, I have been wondering for a while now as to why some classes inherit Object? And what does it really do for the class? Can anyone shed some light on this? Read section 3.3 New-style and classic classes in the Python docs. -- http://mail.python.org/mailman/listinfo/python-list Thanks Guys...I will look deeper into this. I thought I read somewhere that it was required in older python releases, but in newer releases it is not. I might be wrong though. Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Memcached in python 3
On Tue, May 17, 2011 at 4:07 AM, Dan Stromberg drsali...@gmail.com wrote: Do you need something shared across multiple hosts? Across multiple CPU's of the same host? Or single process? By Python Dictionary Manager, do you mean the manager stuff in the multiprocessing module? On Sun, May 15, 2011 at 9:52 PM, Navkirat Singh n4vpyt...@gmail.comwrote: Hi Guys, How can I used memcached with python 3? Are there any other good alternatives to memcached? What about python dictionary manager, would it compare to memcached if I were to use it for storing in-memory information? Any light on this matter will be appreciated. Regards, Navkirat -- http://mail.python.org/mailman/listinfo/python-list Hi, I need something that can be scalable. Something that if needed, can scale to multiple hosts. Yes, I mean the manager stuff in the multiprocessing module. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Python 3 vs Python 2.7 dilemma
Hi Guys, I have been trying to fight this issue for sometime now. I know that a large part of the python 3rd party software base has not been ported to python 3 yet. I am trying to build a web-based enterprise solution for my client. Most of reputed frameworks like Django and Turbo gears are yet in the 2.x stage. I know that these frameworks are extremely good. But I wanted to build my base with python 3 as that is what is going to prevail in the future. I have built my own little architecture using python3. Here is what I have accomplished till now: a) A multiprocessing webserver built directly using low level sockets for maximum control, conforming to RFC 2616 (not completely right now). b) A HTTP message parser for parsing HTTP/1.1 requests and generating response messages c) A session control base using python multiprocessing dictionary manager d) A partially build MVC model, without a templating engine at the moment. I am planning to put Jinja 3 there. I have spent months of free time doing this. I have learnt a lot, but well I am not sure if the path I am on is the right one. My question to everyone is whether I should go ahead with this approach, or should I just use 2.x technology? I am not sure if I will be able to port all the code to python3 later. I will really appreciate any input. Thanks and regards, Navkirat -- http://mail.python.org/mailman/listinfo/python-list
recvall()
Hi All, I am trying to program an HTTP webserver, I am a little confused about the best way to program a recvall function. There are a couple of ways to do this? But performance wise which one is better? a) recvall using a timeout? b) recvall using a condition that nothing was received? c) recvall using a condition for it to wait for a CRLF or a timeout whichever occurs first? Any help would be appreciated. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Memcached in python 3
Hi Guys, How can I used memcached with python 3? Are there any other good alternatives to memcached? What about python dictionary manager, would it compare to memcached if I were to use it for storing in-memory information? Any light on this matter will be appreciated. Regards, Navkirat -- http://mail.python.org/mailman/listinfo/python-list
Persistent Database Cursors
Hi Guys, I am having trouble with database cursors. I am using py-postgresql cursors to fetch data over HTTP. I want to be able to refer to the cursor over multiple HTTP requests. Any light on this matter would be of great help. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Pickling a database cursor?
Hi Guys, Is there any way to pickle a database cursor? I would like a persistent cursor over multiple HTTP requests. Any help would be awesome ! Nav -- http://mail.python.org/mailman/listinfo/python-list
Webcam support
Hi Guys, I am programming a web centric app in python for customer, which needs to click a snap of the customer and forward the pic to the server via POST. I am not very familiar with how I can achieve this. Any direction would be much appreciated. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Writing byte stream as jpeg format to disk
Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Webcam support
On 26-Aug-2010, at 9:49 PM, garabik-news-2005...@kassiopeia.juls.savba.sk wrote:
Navkirat Singh navkir...@gmail.com wrote:
Hi Guys,
I am programming a web centric app in python for customer, which needs
to click a snap of the customer and forward the pic to the server via
POST. I am not very familiar with how I can achieve this. Any
direction would be much appreciated.
For something very similar, I used fswebcam, the crucial code looked
like this:
def capture_frame(filename, brightness=50):
os.system('fswebcam -S 1 -r %s -s brightness=%i%%
--font /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf
--jpeg 60 --save %s' % (RESOLUTION, brightness, filename))
--
---
| Radovan Garabík http://kassiopeia.juls.savba.sk/~garabik/ |
| __..--^^^--..__garabik @ kassiopeia.juls.savba.sk |
---
Antivirus alert: file .signature infected by signature virus.
Hi! I'm a signature virus! Copy me into your signature file to help me spread
--
http://mail.python.org/mailman/listinfo/python-list
Thanks guys, I stumbled upon jpegcam (javascript and flash library) which does
this. It works awesome !! :)
Regards
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 26-Aug-2010, at 11:01 PM, John Bokma wrote: Navkirat Singh navkir...@gmail.com writes: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. Why? -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list why? I am not quite sure what you have not understood. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
I am sorry, maybe I was not elaborate in what I was having trouble with. I am using a jpegcam library, which on my web page captures a webcam image and sends it to the server via the POST method. On the Server side (python 3), I receive this image as a part of header content in bytes (I know thats not how it should be done, but the author has some reason for it), so I first convert the headers to a string so I can separate them. From the separated headers I fish out the content of the image file (which is a string). This is where I get stuck, how am I supposed to convert it back to an image, so I can save as a jpeg on disk. Regards, Nav On 27-Aug-2010, at 12:07 AM, Stefan Schwarzer wrote: Hi Navkirat, On 2010-08-26 19:22, Navkirat Singh wrote: I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. I guess you mean you see a byte string in your server and want to write that to disk. Assuming the string you got is the correct image data in the first place, you can, in Python 2.x, write the string data to disk like this: fobj = open(some_image.jpg, wb) fobj.write(byte_string) fobj.close() Note that you should use wb as mode to write as binary. Otherwise you'll get automatic line ending conversion (at least on Windows) which will give the result you describe. If my answer doesn't help, you probably need to describe in more detail what you're doing, including showing some real code. Stefan -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:10 AM, Robert Kern wrote: On 8/26/10 1:25 PM, Navkirat Singh wrote: On 26-Aug-2010, at 11:01 PM, John Bokma wrote: Navkirat Singhnavkir...@gmail.com writes: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. Why? -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list why? I am not quite sure what you have not understood. Why decode the bytes to (presumably) unicode strings just to encode them back to bytes again? JPEG is not composed of unicode characters; you need to leave them as bytes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list The image bytes are a part of a HTTP header content ( not the message body ). To separate the header content from the image I have to first convert the bytes to string to perform parsing. The resultant string then needs to be converted back to the image. Hence, my problem. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 12:45 AM, MRAB wrote: On 26/08/2010 19:57, Navkirat Singh wrote: I am sorry, maybe I was not elaborate in what I was having trouble with. I am using a jpegcam library, which on my web page captures a webcam image and sends it to the server via the POST method. On the Server side (python 3), I receive this image as a part of header content in bytes (I know thats not how it should be done, but the author has some reason for it), so I first convert the headers to a string so I can separate them. From the separated headers I fish out the content of the image file (which is a string). This is where I get stuck, how am I supposed to convert it back to an image, so I can save as a jpeg on disk. [snip] What does that string look like? Try printing out repr(image[ : 100]). If it looks like plain bytes, then write it to file. If it looks like a series of hex digits, then decode to bytes before writing. -- http://mail.python.org/mailman/listinfo/python-list Thanks MRAB, your suggestions have always been very helpful to me. I shall let you know on what I see. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Nav-- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Also, my apologies for lack of knowledge of character encodings. You have pointed out correctly about unicode encoding. I was under the impression that a unicode will preserve the integrity of the message which has been encoded. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:57 AM, MRAB wrote: On 26/08/2010 21:14, Navkirat Singh wrote: On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Try: image = header.split(b'keep-alive\r\n\r\n', 1)[-1] open(image_path, 'wb').write(image) -- http://mail.python.org/mailman/listinfo/python-list I think I forgot to mention that the original is a stream of bytes decoded using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again). @MRAB - the split() method in python 3 works only on strings and throws an error if I try to use bytes -- http://mail.python.org/mailman/listinfo/python-list
Re: How far can stack [LIFO] solve do automatic garbage collection and prevent memory leak ?
On 27-Aug-2010, at 2:14 AM, Brad wrote: On Aug 25, 4:05 am, Alex McDonald b...@rivadpm.com wrote: Your example of writing code with memory leaks *and not caring because it's a waste of your time* makes me think that you've never been a programmer of any sort. Windows applications are immune from memory leaks since programmers can count on regular crashes to automatically release previously allocated RAM. -- http://mail.python.org/mailman/listinfo/python-list Sorry if I may sound rude, but I have to do this on the windows applications comment - hahahahaha -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:57 AM, MRAB wrote:
On 26/08/2010 21:14, Navkirat Singh wrote:
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote:
Navkirat Singh wrote:
Hey guys,
I am programming a webserver, I receive a jpeg file with the POST
method.The file (.jpeg) is encoded in bytes, I parse the bytes by
decoding them to a string. I wanted to know how i could write the
file (now a string) as a jpeg image on disk. When I try to encode the
same string to a bytes and write them in binary format to disk, the
file is not recognized as jpeg. I would be grateful if someone could
help me with this.
Regards,
Nav
If by decoding them to a string you mean converting to Unicode, then
you've already trashed the data. That's only valid if the bytes had
been encoded from valid Unicode characters, and then only if you use
the corresponding decoding technique.
If you mean some other decoding, then the question is meaningless
without telling us just what the decoding is, preferably with some code.
It also might be useful to know what version of Python you're using,
when you post the code.
DaveA
Dave,
I am using Python3 and I receive a byte stream with a jpeg attached sent
by the web browser over a socket, which looks like this:
b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f
From the above, I need to:
a) Split the header content from the image content, which comes after
the keep-alive\r\n\r\n part
b) Then write the image content to file for further use as a jpeg.
Try:
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
open(image_path, 'wb').write(image)
--
http://mail.python.org/mailman/listinfo/python-list
Yay !! I figured it outit was really very simple. And if I really feel
guilty if I have wasted your time - @MRAB, @DAVE.
Here is what I needed to do:
a) Separate image content from header content of the byte stream received from
the web browser.
b) Save the image content to disk for further use.
Here is what I did. Following is just a snippet:
#-HERE IS WHERE I RECEIVE THE DATA
while True:
buff = socket.recv(8192)
byteStr +=buff
if not buff: break
#--ENCODING/DECODING STARTS FROM HERE (since I want to use
split/partition functions to separate header content from the image content)
strMsg = byteStr.decode(ISO-8859-1)
listMsg = strMsg.split('\r\n')
#
# do some more processing to search the list for the image content, say
supposing index is 11
#---
imageStr = listMsg[11].encode(ISO-8859-1) #Transform the byte string
just the way I found it
f = open('received.jpg','w'b)
f.write(imageStr)
The resultant file is a jpg file.
Thanks,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 2:40 AM, Robert Kern wrote: On 8/26/10 3:47 PM, Navkirat Singh wrote: On 27-Aug-2010, at 1:57 AM, MRAB wrote: On 26/08/2010 21:14, Navkirat Singh wrote: On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Try: image = header.split(b'keep-alive\r\n\r\n', 1)[-1] open(image_path, 'wb').write(image) -- http://mail.python.org/mailman/listinfo/python-list I think I forgot to mention that the original is a stream of bytes decoded using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again). @MRAB - the split() method in python 3 works only on strings and throws an error if I try to use bytes This is incorrect. Python 3.1.2 (r312:79360M, Mar 24 2010, 01:33:18) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type help, copyright, credits or license for more information. bytes = b'Connection: keep-alive\r\n\r\nbody' bytes.split(b'\r\n\r\n', 1)[-1] b'body' FYI: the JPEG data is not in the header. The b'\r\n\r\n' sequence delimits the header from the body. Do not rely on Connection: keep-alive being the last header. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list Thanks Everyone, @Robert - Thanks a lot for your time :-) , I did know that the body starts after the occurrence two CRLF sequences, but I was following RFC2616 as a guide, which specifically mentions: The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer- Encoding header field in the request’s message-headers Which has not been done by the author of the library, hence I said what I did. Or I have misunderstood the RFC Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 2:48 AM, MRAB wrote:
On 26/08/2010 21:47, Navkirat Singh wrote:
On 27-Aug-2010, at 1:57 AM, MRAB wrote:
On 26/08/2010 21:14, Navkirat Singh wrote:
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote:
Navkirat Singh wrote:
Hey guys,
I am programming a webserver, I receive a jpeg file with the POST
method.The file (.jpeg) is encoded in bytes, I parse the bytes by
decoding them to a string. I wanted to know how i could write the
file (now a string) as a jpeg image on disk. When I try to encode the
same string to a bytes and write them in binary format to disk, the
file is not recognized as jpeg. I would be grateful if someone could
help me with this.
Regards,
Nav
If by decoding them to a string you mean converting to Unicode, then
you've already trashed the data. That's only valid if the bytes had
been encoded from valid Unicode characters, and then only if you use
the corresponding decoding technique.
If you mean some other decoding, then the question is meaningless
without telling us just what the decoding is, preferably with some code.
It also might be useful to know what version of Python you're using,
when you post the code.
DaveA
Dave,
I am using Python3 and I receive a byte stream with a jpeg attached sent
by the web browser over a socket, which looks like this:
b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f
From the above, I need to:
a) Split the header content from the image content, which comes after
the keep-alive\r\n\r\n part
b) Then write the image content to file for further use as a jpeg.
Try:
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
open(image_path, 'wb').write(image)
--
http://mail.python.org/mailman/listinfo/python-list
I think I forgot to mention that the original is a stream of bytes decoded
using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again).
@MRAB - the split() method in python 3 works only on strings and throws an
error if I try to use bytes
All i can say is that it works for me:
header = b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
image
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
What error did you get?
--
http://mail.python.org/mailman/listinfo/python-list
Hi MRAB,
Here is the error:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: Type str doesn't support the buffer API
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 2:58 AM, Navkirat Singh wrote:
On 27-Aug-2010, at 2:48 AM, MRAB wrote:
On 26/08/2010 21:47, Navkirat Singh wrote:
On 27-Aug-2010, at 1:57 AM, MRAB wrote:
On 26/08/2010 21:14, Navkirat Singh wrote:
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote:
Navkirat Singh wrote:
Hey guys,
I am programming a webserver, I receive a jpeg file with the POST
method.The file (.jpeg) is encoded in bytes, I parse the bytes by
decoding them to a string. I wanted to know how i could write the
file (now a string) as a jpeg image on disk. When I try to encode the
same string to a bytes and write them in binary format to disk, the
file is not recognized as jpeg. I would be grateful if someone could
help me with this.
Regards,
Nav
If by decoding them to a string you mean converting to Unicode, then
you've already trashed the data. That's only valid if the bytes had
been encoded from valid Unicode characters, and then only if you use
the corresponding decoding technique.
If you mean some other decoding, then the question is meaningless
without telling us just what the decoding is, preferably with some code.
It also might be useful to know what version of Python you're using,
when you post the code.
DaveA
Dave,
I am using Python3 and I receive a byte stream with a jpeg attached sent
by the web browser over a socket, which looks like this:
b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f
From the above, I need to:
a) Split the header content from the image content, which comes after
the keep-alive\r\n\r\n part
b) Then write the image content to file for further use as a jpeg.
Try:
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
open(image_path, 'wb').write(image)
--
http://mail.python.org/mailman/listinfo/python-list
I think I forgot to mention that the original is a stream of bytes decoded
using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again).
@MRAB - the split() method in python 3 works only on strings and throws an
error if I try to use bytes
All i can say is that it works for me:
header = b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
image
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
What error did you get?
--
http://mail.python.org/mailman/listinfo/python-list
Hi MRAB,
Here is the error:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: Type str doesn't support the buffer API
I got your point
The argument for the split I have give is a string, hence the error.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 3:02 AM, Robert Kern wrote:
On 8/26/10 4:17 PM, Navkirat Singh wrote:
Here is what I needed to do:
a) Separate image content from header content of the byte stream received
from the web browser.
b) Save the image content to disk for further use.
Here is what I did. Following is just a snippet:
#-HERE IS WHERE I RECEIVE THE DATA
while True:
buff = socket.recv(8192)
byteStr +=buff
if not buff: break
#--ENCODING/DECODING STARTS FROM HERE (since I want to use
split/partition functions to separate header content from the image content)
strMsg = byteStr.decode(ISO-8859-1)
listMsg = strMsg.split('\r\n')
#
# do some more processing to search the list for the image content, say
supposing index is 11
#---
imageStr = listMsg[11].encode(ISO-8859-1) #Transform the byte string
just the way I found it
If your JPEG happens to contain the bytes \r\n, then this will not work.
Please follow our advice. Split using b'\r\n\r\n' and use the maxsplit=1
argument to make sure that you do not split on spurious b'\r\n\r\n' sequences
inside the JPEG body. Do not decode the bytes.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Thanks Robert,
My method worked too, I was able to do the above and save the jpeg flawlessly,
but your method seems better as I will not have to take the extra step of
encoding/decoding.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 3:04 AM, Robert Kern wrote: On 8/26/10 4:25 PM, Navkirat Singh wrote: @Robert - Thanks a lot for your time :-) , I did know that the body starts after the occurrence two CRLF sequences, but I was following RFC2616 as a guide, which specifically mentions: The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer- Encoding header field in the request’s message-headers Which has not been done by the author of the library, hence I said what I did. Or I have misunderstood the RFC It certainly looks like the data has a Content-length header: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\n ... -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list Once again you opened my eyes, I think its been a tough night for me. Seeing too much and nothing at the same time. Regards -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 3:15 AM, Robert Kern wrote: On 8/26/10 4:17 PM, Navkirat Singh wrote: #-HERE IS WHERE I RECEIVE THE DATA while True: buff = socket.recv(8192) byteStr +=buff if not buff: break Also, you probably shouldn't bother writing an HTTP server using raw sockets. Use HTTPServer instead: http://docs.python.org/py3k/library/http.server.html or better, wsgiref: http://docs.python.org/py3k/library/wsgiref.html -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list Thanks a lot guys, you all have been a lot of help !! -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 4:23 AM, Terry Reedy wrote:
On 8/26/2010 5:28 PM, Navkirat Singh wrote:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
You are trying to split bytes with a string, which is impossible.
Split bytes with bytes, strings with strings.
Traceback (most recent call last):
File stdin, line 1, inmodule
TypeError: Type str doesn't support the buffer API
--
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
Thanks !
--
http://mail.python.org/mailman/listinfo/python-list
Re: Iterative vs. Recursive coding
On 20-Aug-2010, at 1:17 PM, News123 wrote: On 08/20/2010 02:26 AM, Steven D'Aprano wrote: On Thu, 19 Aug 2010 22:00:16 +, Martin Gregorie wrote: Recursion can be quite a trick to get your mind round at first Really? Do people actually find the *concept* of recursion to be tricky? Is this a sincere surprise or are you just boasting? If I remember correctly, my puzzlement about recursion lasted about 15 seconds. I remember thinking How does the function foo know that there is a function foo when foo doesn't fully exist yet?, but once I accepted the fact that it just does it all just seemed obvious. Getting recursion *right* is sometimes tricky, but the idea itself isn't. Well there's two things where I remember, that at least quite some people in our class (at least the ones who didn't do maths or programming in their spare time) had problems with. This were recursion and Mathematical induction. (quite the same though) The fact, that you didn't have the issue doens't mean it's easy for others. -- http://mail.python.org/mailman/listinfo/python-list Well I guess why you did not have a problem understanding recursion is because you took for granted that it is the way it is. Most people, like me, try to reason why something is the way it is ! Hence, the delay in understanding the concept fully. -- http://mail.python.org/mailman/listinfo/python-list
Re: Using the print command in Python3
On 10-Aug-2010, at 11:04 AM, Benjamin Kaplan wrote:
On Mon, Aug 9, 2010 at 10:17 PM, Grady Knotts gradykno...@gmail.com wrote:
In earlier versions of Python I can do:
print 'A',
print 'B'
to print everything on the same line: 'A B'
But I don't know how to do this with Python3
I've been trying things like:
print('A',)
print('B')
and it prints two different lines.
So, do I get two different print statements to output on the same line?
help(print)
Help on built-in function print in module builtins:
print(...)
print(value, ..., sep=' ', end='\n', file=sys.stdout)
Prints the values to a stream, or to sys.stdout by default.
Optional keyword arguments:
file: a file-like object (stream); defaults to the current sys.stdout.
sep: string inserted between values, default a space.
end: string appended after the last value, default a newline.
--
http://mail.python.org/mailman/listinfo/python-list
One method of doing this:
Use the join method of the string:
print(.join([ 'A' , 'space B' ] )
This will give you :
'AspaceB'
I have used extra spaces just for clarity. Hope this helps !
Nav
--
http://mail.python.org/mailman/listinfo/python-list
favicon.ico problem
Hi guys, I am having this strange problem. I have programmed a very basic multiprocessing webserver using low level sockets. Each time the server receives a request it spawns a new process to handle the request. Now when through a web browser I type http://localhost:8001/ it automatically creates two processes: One process to server the '/' path and another one to serve the '/favicon.ico' path. I have not programmed it to serve the latter. Infact I dont even know where that name '/favicon.ico' comes from. Any insight into this weird behavior would be greatly appreciated. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: favicon.ico problem
On 10-Aug-2010, at 10:57 AM, Xia, Zhen wrote: On Tue, 10 Aug 2010 10:02:49 +0530 Navkirat Singh navkir...@gmail.com wrote: Hi guys, I am having this strange problem. I have programmed a very basic multiprocessing webserver using low level sockets. Each time the server receives a request it spawns a new process to handle the request. Now when through a web browser I type http://localhost:8001/ it automatically creates two processes: One process to server the '/' path and another one to serve the '/favicon.ico' path. I have not programmed it to serve the latter. Infact I dont even know where that name '/favicon.ico' comes from. Any insight into this weird behavior would be greatly appreciated. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list Your server just need to return an HTTP 404 response for this 'favicon.ico' request. -- Xia, Zhen zh...@njnet.edu.cn -- http://mail.python.org/mailman/listinfo/python-list Thanks guys ! That explains it ! Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Import python modules from sub-directories
On 06-Aug-2010, at 1:13 PM, 夏震 wrote: Hi guys, I am new to python and would like to import certain classes in sub-directories of the working directory. I was wondering how will I be able to achieve this? Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list Please try touch __init__.py in sub-directories. -- http://mail.python.org/mailman/listinfo/python-list Thanks, that worked : ) -- http://mail.python.org/mailman/listinfo/python-list
Re: new to python - trouble calling a function from another function
I was looking at the code, I dont have much time to go through it, but I might have found a typo - yield (p.destination - self.currenteFloor) , I think it should be currentFloor.Maybe thats your problem. Will look into the code more later. Regards, Nav On 05-Aug-2010, at 12:55 PM, Brandon McCombs wrote: Hello, I'm building an elevator simulator for a class assignment. I recently ran into a roadblock and don't know how to fix it. For some reason, in my checkQueue function below, the call to self.goUp() is never executed. It is on the last line of code I pasted in. I can put print statements before and after the call and I have a print statement in goUp() itself. Only the print statements before and after the call are executed. The one inside goUp() is never executed because goUp() never seems to be executed. How can that be? I don't get any errors when the script executes. Surely this isn't some limitation I'm encountering? thanks sorry about the formatting - class Elevator(Process): def __init__(self,name): Process.__init__(self,name=name) self.numPassengers = 0 self.passengerList = [] self.passengerWaitQ = [] self.currentFloor = 1 self.idle = 1 self.newPassengers = 0 def goUp(self): print here bubbleSort(self.passengerList, len(self.passengerList)) self.currentFloor += 1 if len(self.passengerList) 0: for p in self.passengerList: if self.currentFloor == p.destination: yield (p.destination - self.currenteFloor) * TRAVELTIME, self reactivate(p) p.inBuilding() else: self.goUp() def checkQueue(self): if (len(self.passengerWaitQ)) 0 and len(self.passengerList) MAXCAPACITY: if len(self.passengerWaitQ) MAXCAPACITY: self.newPassengers = len(self.passengerWaitQ) else: self.newPassengers = MAXCAPACITY - len(self.passengerList) for i in range(0,self.newPassengers): self.passengerList.append(self.passengerWaitQ.pop()) self.goUp() -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Import python modules from sub-directories
Hi guys, I am new to python and would like to import certain classes in sub-directories of the working directory. I was wondering how will I be able to achieve this? Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Import python modules from sub-directories
On 06-Aug-2010, at 10:44 AM, Navkirat Singh wrote: Hi guys, I am new to python and would like to import certain classes in sub-directories of the working directory. I was wondering how will I be able to achieve this? Regards, Nav Thanks. I got it. I just put an empty __init__.py file in the subdirectories and it worked. -- http://mail.python.org/mailman/listinfo/python-list
Re: Global variables problem
Thanks a lot guys !!
I solved the problem:
In the lines:
new_process = process(target=newprocess)
new_process.start()
The target=newprocess is pointing towards a variable, instead of a function.
So, appending a () will make it goto that function, thereby changing the global
variable : )
Thanks,
Nav
On 04-Aug-2010, at 11:42 AM, Daniel da Silva wrote:
Your problem lies somewhere in the use of the Process class, not with global
variables.
If you replace your p = ... and p.start() lines with a direct call to
self.handle_connection(), your code works as expected. I don't know much
about the multiprocessing module, so I can't really comment on what you're
doing wrong, but I hope this points you in the right direction.
Sorry I couldn't be of more help,
Daniel
On Tue, Aug 3, 2010 at 9:48 PM, Navkirat Singh navkir...@gmail.com wrote:
On 04-Aug-2010, at 9:46 AM, Daniel da Silva wrote:
Please post approximate code that actually works and displays the problem.
On Tue, Aug 3, 2010 at 9:00 PM, Navkirat Singh navkir...@gmail.com wrote:
Hey guys,
I am using a multiprocessing program, where the new process is supposed to
change a variable in the main class that it branches out from. This is
somehow not working, following is an approximate code. Would really
appreciate any insight into this matter:
var = {}
class Something():
def set_var(self):
global var
var = somevalue
def get_var(self):
return var
def newprocess(self):
self.set_var()
def do_multiprocessing(self):
while true:
self.get_var()
new_process = process(target=newprocess)
new_process.start()
I am really confused here !
Any help would be awesome : )
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
This is a working code, streamlined, but it is where the problem is:
from multiprocessing import *
dicts = 0
print('global ', dicts)
class WebServer():
def set_sessionInfo(self):
global dicts
dicts = dicts + 1
def get_sessionInfo(self):
return dicts
def handle_connection(self):
self.set_sessionInfo()
def serve_forever(self):
for x in range(10):
p = Process(target=self.handle_connection)
p.start()
print(self.get_sessionInfo())
ws = WebServer()
ws.serve_forever()
print(dicts)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Global variables problem
: (
False alarm, the earlier solution breaks multiprocessing. Whats happening here
is the child needs to change a variable in the parent process, So I think I am
looking at shared memory (maybe). Any suggestions?
Regards,
Nav
On 04-Aug-2010, at 12:41 PM, Navkirat Singh wrote:
Thanks a lot guys !!
I solved the problem:
In the lines:
new_process = process(target=newprocess)
new_process.start()
The target=newprocess is pointing towards a variable, instead of a function.
So, appending a () will make it goto that function, thereby changing the
global variable : )
Thanks,
Nav
On 04-Aug-2010, at 11:42 AM, Daniel da Silva wrote:
Your problem lies somewhere in the use of the Process class, not with global
variables.
If you replace your p = ... and p.start() lines with a direct call to
self.handle_connection(), your code works as expected. I don't know much
about the multiprocessing module, so I can't really comment on what you're
doing wrong, but I hope this points you in the right direction.
Sorry I couldn't be of more help,
Daniel
On Tue, Aug 3, 2010 at 9:48 PM, Navkirat Singh navkir...@gmail.com wrote:
On 04-Aug-2010, at 9:46 AM, Daniel da Silva wrote:
Please post approximate code that actually works and displays the problem.
On Tue, Aug 3, 2010 at 9:00 PM, Navkirat Singh navkir...@gmail.com wrote:
Hey guys,
I am using a multiprocessing program, where the new process is supposed to
change a variable in the main class that it branches out from. This is
somehow not working, following is an approximate code. Would really
appreciate any insight into this matter:
var = {}
class Something():
def set_var(self):
global var
var = somevalue
def get_var(self):
return var
def newprocess(self):
self.set_var()
def do_multiprocessing(self):
while true:
self.get_var()
new_process = process(target=newprocess)
new_process.start()
I am really confused here !
Any help would be awesome : )
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
This is a working code, streamlined, but it is where the problem is:
from multiprocessing import *
dicts = 0
print('global ', dicts)
class WebServer():
def set_sessionInfo(self):
global dicts
dicts = dicts + 1
def get_sessionInfo(self):
return dicts
def handle_connection(self):
self.set_sessionInfo()
def serve_forever(self):
for x in range(10):
p = Process(target=self.handle_connection)
p.start()
print(self.get_sessionInfo())
ws = WebServer()
ws.serve_forever()
print(dicts)
--
http://mail.python.org/mailman/listinfo/python-list
Difference between queues and pipes in multiprocessing
Hi, I was wondering what are the differences between queues and pipes implemented using multiprocessing python module. Am I correct if I say, in pipes, if another process writes to one receiving end concurrently, then an error will be raised and in queues the later processes data will just queued up? Thanks Nav -- http://mail.python.org/mailman/listinfo/python-list
Global variables problem
Hey guys,
I am using a multiprocessing program, where the new process is supposed to
change a variable in the main class that it branches out from. This is somehow
not working, following is an approximate code. Would really appreciate any
insight into this matter:
var = {}
class Something():
def set_var(self):
global var
var = somevalue
def get_var(self):
return var
def newprocess(self):
self.set_var()
def do_multiprocessing(self):
while true:
self.get_var()
new_process = process(target=newprocess)
new_process.start()
I am really confused here !
Any help would be awesome : )
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
Re: Global variables problem
On 04-Aug-2010, at 9:46 AM, Daniel da Silva wrote:
Please post approximate code that actually works and displays the problem.
On Tue, Aug 3, 2010 at 9:00 PM, Navkirat Singh navkir...@gmail.com wrote:
Hey guys,
I am using a multiprocessing program, where the new process is supposed to
change a variable in the main class that it branches out from. This is
somehow not working, following is an approximate code. Would really
appreciate any insight into this matter:
var = {}
class Something():
def set_var(self):
global var
var = somevalue
def get_var(self):
return var
def newprocess(self):
self.set_var()
def do_multiprocessing(self):
while true:
self.get_var()
new_process = process(target=newprocess)
new_process.start()
I am really confused here !
Any help would be awesome : )
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
This is a working code, streamlined, but it is where the problem is:
from multiprocessing import *
dicts = 0
print('global ', dicts)
class WebServer():
def set_sessionInfo(self):
global dicts
dicts = dicts + 1
def get_sessionInfo(self):
return dicts
def handle_connection(self):
self.set_sessionInfo()
def serve_forever(self):
for x in range(10):
p = Process(target=self.handle_connection)
p.start()
print(self.get_sessionInfo())
ws = WebServer()
ws.serve_forever()
print(dicts)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Performance ordered dictionary vs normal dictionary
On 29-Jul-2010, at 11:41 AM, Chris Rebert wrote: On Wed, Jul 28, 2010 at 8:12 PM, Navkirat Singh navkir...@gmail.com wrote: Sorry, I might have been a bit vague: (Also, I am new to pythong) I am trying to do construct my own web session tracking algorithm for a web server (which also I have constructed). The book keeping is for the session information I track. The dictionary object will do this for me. I was initially using a plain dictionary object, but I read this in the documentation that made me think about the question I asked :- Does your program actually make use of the ordering? If so, then yes, using OrderedDict is obviously preferable (Why reimplement something needlessly?). If not, then why bother with the extra complication? (You are aware that the ordered in OrderedDict means that its keys are ordered, and not that, say, a list containing OrderedDicts can be sorted, right?) Cheers, Chris -- http://blog.rebertia.com I am still in the analysis phase/experimental phase and the need might arise. Hence, I just wanted to know my options. If a collections object will improve performance than an ordinary dictionary, then yes I would like to go with the OrderedDict. Thanks, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Performance ordered dictionary vs normal dictionary
On 29-Jul-2010, at 2:50 PM, Hrvoje Niksic wrote: sturlamolden sturlamol...@yahoo.no writes: On 29 Jul, 03:47, Navkirat Singh navkir...@gmail.com wrote: I was wondering what would be better to do some medium to heavy book keeping in memory - Ordered Dictionary or a plain simple Dictionary object?? It depends on the problem. A dictionary is a hash table. An ordered dictionary is a binary search tree (BST). The ordered dictionary shipped with Python is also a hash table, with an internal list to keep track of item order. The one thing not mentioned in the thread is that ordered dict's deletion is O(n), which might impact heavy bookkeeping. As Raymond said, where order doesn't matter, it's best to stick with dict. -- http://mail.python.org/mailman/listinfo/python-list Thanks that was an excellent point : ) -- http://mail.python.org/mailman/listinfo/python-list
Performance ordered dictionary vs normal dictionary
Hi guys, I was wondering what would be better to do some medium to heavy book keeping in memory - Ordered Dictionary or a plain simple Dictionary object?? Regards, N4v -- http://mail.python.org/mailman/listinfo/python-list
Function parameter scope
Hi, I had another question: What is the scope of a parameter passed to a function? I know its a very basic question, but I am just sharpening my basics :) def func_something(x) return print(x+1); Does x become a local variable or does it stay as a module scoped variable? Though I think its local, but just want to be sure. Thanks, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Performance ordered dictionary vs normal dictionary
On 29-Jul-2010, at 9:36 AM, sturlamolden wrote: On 29 Jul, 03:47, Navkirat Singh navkir...@gmail.com wrote: I was wondering what would be better to do some medium to heavy book keeping in memory - Ordered Dictionary or a plain simple Dictionary object?? It depends on the problem. A dictionary is a hash table. An ordered dictionary is a binary search tree (BST). Those are different data structures. The main things to note is that: - Access is best-case O(1) for the hash table and O(log n) for the BST. - Worst case behavior is for access is O(n) for both. The pathologic conditions are excessive collisions (hash) or an unbalanced tree (BST). In pathologic cases they both converge towards a linked list. - Searches are only meaningful for == and != for a hash table, but BST searches are also meaningful for , , =, and =. For this reason, databases are often implemented as BSTs. - A BST can be more friendly to cache than a hash table, particularly when they are large. (Remember that O(1) can be slower than O(log n). Big-O only says how run-time scales with n.) That said, I have not compared ordered dicts with normal dicts, as I still use 2.6. -- http://mail.python.org/mailman/listinfo/python-list Thanks for the insight. I am still in the thinking stage, so will let you know as and when I get down to implementation of my idea. Thanks for your time. : ) Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Socket performance
On 25-Jul-2010, at 5:52 PM, Roy Smith wrote: In article 4c4bd0b1$0$1624$742ec...@news.sonic.net, John Nagle na...@animats.com wrote: 1. When writing to a TCP socket, write everything you have to write with one send or write operation if at all possible. Don't write a little at a time. That results in sending small packets, because sockets are flushed after each write. There's nothing that guarantees that a single write won't be split into multiple packets, nor that multiple writes won't be coalesced into a single packet. Or any combination of splitting and coalescing that the kernel feels like. That being said, for any sane implementation, what John says is true most of the time, and is indeed a reasonable optimization. Just don't depend on it being true all the time. The most common case where it will not be true is if you're trying to send a large amount of data and exceed the MTU of the network. Then you are certain to get fragmentation. Depending on what you're doing, this can be a point of networking trivia, or it can be the difference between your application working and not working. If you're just streaming data from one place to another, you don't have to worry about it. But, if you're doing some sort of interactive protocol where you send a command, wait for a respond, send another command, etc, you really do need to be aware of how this works. Let's say you're writing something like a HTTP client. You send a bunch of headers, then expect to get back something like 200 OK\r\n, or 404 Not Found\r\n. You can't just do a read() on the socket and then examine the string to see if the first three characters are 200 or 404, because (regardless of how the server sent them), it is legal for your read() to return just a single character (i.e. 2), and then for the next read() to get 00 OK\r\n. You need to do buffering inside your application which keeps doing read() until you find the \r\n (and stops there, even if the read() returned more data beyond that). -- http://mail.python.org/mailman/listinfo/python-list Thanks John, Roy. I really appreciate your valuable input. I have made a note of what you have said and will implement keeping the same in mind : ) Nav -- http://mail.python.org/mailman/listinfo/python-list
Multiprocessing zombie processes
Hi,
I have been meddling around with forking and multiprocessing. Now both of them
spawn new processes from parent (atleast from what I have understood). I have
been able to reproduce a zombie state in a fork with:
import os,time
print('before fork',os.getpid())
pid = os.fork()
if pid:
print('child: ',pid)
time.sleep(120)
Now doing a ps ax | grep pid I can find a zombie child process, but I am not
being able to reproduce the same with multiprocessing. Am I missing something?
Or multiprocessing does not have the zombie problem?
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing zombie processes
OK I wanted zombie processes and have been able to regenerate them with
multiprocessing. Now lets see how I can handle them.
Nav
On 25-Jul-2010, at 4:37 AM, Navkirat Singh wrote:
Hi,
I have been meddling around with forking and multiprocessing. Now both of
them spawn new processes from parent (atleast from what I have understood). I
have been able to reproduce a zombie state in a fork with:
import os,time
print('before fork',os.getpid())
pid = os.fork()
if pid:
print('child: ',pid)
time.sleep(120)
Now doing a ps ax | grep pid I can find a zombie child process, but I am
not being able to reproduce the same with multiprocessing. Am I missing
something? Or multiprocessing does not have the zombie problem?
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing zombie processes
I want to kill Zombiesso first I have to create them...simple law of nature On 25-Jul-2010, at 5:08 AM, Chris Rebert wrote: On Sat, Jul 24, 2010 at 4:11 PM, Navkirat Singh navkir...@gmail.com wrote: OK I wanted zombie processes snip Now lets see how I can handle them. Paging Dr. Frankenstein. Dr. Frankenstein to the lab. Paging Dr. Frankenstein. Cheers, Chris -- Most people try to /avoid/ making zombies. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing zombie processes
On 25-Jul-2010, at 5:25 AM, Cameron Simpson wrote:
[ Please don't top post. Post below so that things read like a
conversation. (And trim excess quoted junk.) It doesn't take long and
makes things a lot easier for your readers. ]
On 25Jul2010 04:41, Navkirat Singh navkir...@gmail.com wrote:
| On 25-Jul-2010, at 4:37 AM, Navkirat Singh wrote:
| I have been meddling around with forking and multiprocessing. Now
| both of them spawn new processes from parent (atleast from what I have
| understood). I have been able to reproduce a zombie state in a fork with:
|
| import os,time
| print('before fork',os.getpid())
| pid = os.fork()
| if pid:
|print('child: ',pid)
|time.sleep(120)
|
| Now doing a ps ax | grep pid I can find a zombie child process,
| but I am not being able to reproduce the same with multiprocessing. Am
| I missing something? Or multiprocessing does not have the zombie problem?
|
| OK I wanted zombie processes and have been able to regenerate them with
multiprocessing. Now lets see how I can handle them.
A zombie process is a fairly common thing and nothing to panic about
usually.
When you have a normal, running process it has an entry in the
system process table (that ps shows) that points at it and shows its
state.
When a process exits, its exit status (exit code, whether it exited
normally or from a signal, and if so which signal, some stats) _remain_
in the process table as a record of the exit status. This is shown as a
zombie process in ps - a process which has exited and whose exit
status is hanging around waiting to be collected.
After the process' parent has issued a wait() call to collect the status
the process slot if released and the :zombie is gone.
In your fork() example above, if that's the whole program, the child
exits immediately and the parent has not waited for it.
If the parent exits without waiting for all its children, the init
process (process 1) inherits them, and it collects the statuses.
Having a few zombies lying around isn't a disaster provided your program
does get around to waiting for them at some point (or if your main
program is short lived, so they get reaped by init after it exits).
It's only a process table slot after all - little memory is needed for
it.
A program that spawns a lot of children and never waits for them _is_
bad in the long run. Process ids are often 2-byte integers on many
systems for it's quite feasible to fill the table, and depending on the
kernel's process table implementation a big process table might have other
performance side effects.
Cheers,
--
Cameron Simpson c...@zip.com.au DoD#743
http://www.cskk.ezoshosting.com/cs/
I'm behind a corporate Exchange Server which seems to have
changed recently to converting everything it sees to HTML.
How embarrassing. - Dan Espen des...@telcordia.com
Thanks a lot for all the information. It cleared a lot of my doubts.
Nav
--
http://mail.python.org/mailman/listinfo/python-list
Re: Socket performance
On 25-Jul-2010, at 6:45 AM, Lawrence D'Oliveiro wrote: In message mailman.1097.1279930004.1673.python-l...@python.org, Navkirat Singh wrote: I had a question, programming sockets, what are the things that would degrade performance and what steps could help in a performance boost? Remember the old saying, “premature optimization is the root of all evil”. Have you actually got some code working properly first, before worrying about how good or bad its performance is? -- Lawrence trying not to say “performant” :) -- http://mail.python.org/mailman/listinfo/python-list I agree with you, it was just for the sake of knowledge. Its always good to have a map right? -- http://mail.python.org/mailman/listinfo/python-list
Socket performance
Hey Everyone, I had a question, programming sockets, what are the things that would degrade performance and what steps could help in a performance boost? I would also appreciate being pointed to some formal documentation or article. I am new to this. Warm regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Socket performance
Thanks for the info : ). I will look into it ! Right now I am having a strange problem. I am trying to use cookies and the import function returns an error: I am using python 3: from http import cookies importError: No module named http Is it my configuration or has something changed since the documentation was written? Sorry I might be asking too many question, I am pretty new to this stuff and kinda feel lost here and there : ( Thanks, Nav On 24-Jul-2010, at 6:34 AM, MRAB wrote: Navkirat Singh wrote: Hey Everyone, I had a question, programming sockets, what are the things that would degrade performance and what steps could help in a performance boost? I would also appreciate being pointed to some formal documentation or article. I am new to this. Interleaving processing and sending/receiving might reduce throughput because when you're processing the socket is idle (depending on how much buffering there is). You could do the socket stuff in another thread so that it's not waiting for the processing to finish while there is data available, and use a queue to transfer the data between that thread and the processing thread. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Sending HTTP headers via plain sockets
Hi Guys,
I am very new to python and I am trying to send HTTP headers for
redirection using sockets in python 3, but in vain. If I use the meta
tag REFRESH method the redirection works. Please advise what I am
missing, below is the snippet of my code:
hostsock is the socket object
print('Redirecting client')
hostsock.send(HTTP/1.1 301 Moved Permanently
Location: http://www.example.com;)
I have been up all night trying to figure this one out : ( I would be
grateful if someone could point me in the right direction
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
Re: Sending HTTP headers via plain sockets
Aaah figured it out...!!
It was elementary, I was lacking the carriage return and line feed
characters at the end of the status and header line. Here is how I
solved it:
hostsock.send(b'HTTP/1.1 301 Moved Permanently\r\nLocation: http://www.example.com\r\n'
)
Regards,
Nav
On 23-Jul-2010, at 7:00 AM, Navkirat Singh wrote:
Hi Guys,
I am very new to python and I am trying to send HTTP headers for
redirection using sockets in python 3, but in vain. If I use the
meta tag REFRESH method the redirection works. Please advise what I
am missing, below is the snippet of my code:
hostsock is the socket object
print('Redirecting client')
hostsock.send(HTTP/1.1 301 Moved Permanently
Location: http://www.example.com;)
I have been up all night trying to figure this one out : ( I would
be grateful if someone could point me in the right direction
Regards,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
