Re: Writing byte stream as jpeg format to disk
Robert Kern wrote: Please follow our advice. Split using b'\r\n\r\n' and use the maxsplit=1 argument to make sure that you do not split on spurious b'\r\n\r\n' sequences inside the JPEG body. Do not decode the bytes. Correct, and I'll add that this is a case where we might want to be better than correct. BaseHTTPRequestHandler in the Python standard library accommodates clients that incorrectly omit the '\r' and end header lines with just '\n'. Such apps have been seen in the wild. Since bare '\n' never appears in correctly formed HTTP headers, interpreting it as equivalent to '\r\n' doesn't break anything. The re module offers a split that does what we want. import re boundary_re = re.compile(br'\r?\n\r?\n') then you can use: (headers, content) = boundary_re.split(rawdata, 1) I like Robert's suggestion to use the HTTP server or wsgiref in the Python library. There's significant arcane wisdom programmed in already. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On Thu, 26 Aug 2010 23:56:26 -0700, Bryan wrote: follow our advice. Split using b'\r\n\r\n' and use the maxsplit=1 argument to make sure that you do not split on spurious b'\r\n\r\n' sequences inside the JPEG body. Do not decode the bytes. Correct, and I'll add that this is a case where we might want to be better than correct. BaseHTTPRequestHandler in the Python standard library accommodates clients that incorrectly omit the '\r' and end header lines with just '\n'. Such apps have been seen in the wild. Since bare '\n' never appears in correctly formed HTTP headers, interpreting it as equivalent to '\r\n' doesn't break anything. Yes it does. It breaks upstream filtering rules which are intended to prohibit, remove or modify certain headers. This class of attack is known as HTTP request smuggling. By appending a header preceded by a bare '\r' or '\n' to the end of another header, the header can be smuggled past a filter which parses headers using the correct syntax, but will still be treated as a header by software which incorrectly parses headers using bare '\r' or '\n' as separators. The safest solution would be to simply reject any request (or response) which contains bare '\r' or '\n' characters within headers, at least by default. Force the programmer to read the documentation (where the risks would be described) if they want the fault tolerant behaviour. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
Nobody wrote: Bryan wrote: this is a case where we might want to be better than correct. BaseHTTPRequestHandler in the Python standard library accommodates clients that incorrectly omit the '\r' and end header lines with just '\n'. Such apps have been seen in the wild. Since bare '\n' never appears in correctly formed HTTP headers, interpreting it as equivalent to '\r\n' doesn't break anything. Yes it does. It breaks upstream filtering rules which are intended to prohibit, remove or modify certain headers. This class of attack is known as HTTP request smuggling. By appending a header preceded by a bare '\r' or '\n' to the end of another header, the header can be smuggled past a filter which parses headers using the correct syntax, How does a bare '\r' or '\n' get past a filter which parses headers using the correct syntax? I don't see where the correct syntax of the HTTP protocol allows that. but will still be treated as a header by software which incorrectly parses headers using bare '\r' or '\n' as separators. Why blame software that incorrectly accepts '\n' as a line break, and not the filter that incorrectly accepted '\n' in the middle of a header? Both are accepting incorrect syntax, but only the former has good reason to do so. The safest solution would be to simply reject any request (or response) which contains bare '\r' or '\n' characters within headers, at least by default. Force the programmer to read the documentation (where the risks would be described) if they want the fault tolerant behaviour. The Internet has a tradition of protocols above the transport level being readable by eye and writable by hand. The result has been quick development, but many mistakes that can induce unforeseen consequences. This case is somewhat subtle. Within a text entity-body, HTTP allows any one of the three end-of-line delimiters. That's just the body; the header portion is more rigid. In HTTP 1.0: This flexibility regarding line breaks applies only to text media in the Entity-Body; a bare CR or LF should not be substituted for CRLF within any of the HTTP control structures (such as header fields and multipart boundaries). -- RFC 1945 While in HTTP 1.1: This flexibility regarding line breaks applies only to text media in the entity-body; a bare CR or LF MUST NOT be substituted for CRLF within any of the HTTP control structures (such as header fields and multipart boundaries). -- RFC 2616 Note the change from should not to MUST NOT. In reality our code might be called upon to work with apps that botch the technically- correct HTTP end-of-line marker. Rejecting bare '\n' may be safe from a technical security perspective, but if our safe code breaks a previously working system, then it will appear in a bug database and not in production. 'Nobody' makes a fair point. I'd love to see Internet protocols defined with mechanical rigor. Our discipline commonly specifies programming language syntax formally, and Internet protocols are syntactically simpler than programming languages. For now, HTTP is a bit of a mess, so write it absolutely correctly but read it a bit flexibly. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
In message mailman.64.1282843346.29448.python-l...@python.org, Navkirat Singh wrote: I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. I assume the JPEG data is received along with other field values in the POST. You’ll be saving those other fields in a database, right? So why not save the JPEG image there as well? -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
In message mailman.71.1282852048.29448.python-l...@python.org, Navkirat Singh wrote: The image bytes are a part of a HTTP header content ( not the message body ). In which case, won’t they be in some encoding like Base-64? I don’t think you’re allowed arbitrary binary bytes in an HTTP header. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/27/10 5:58 PM, Lawrence D'Oliveiro wrote: In message mailman.64.1282843346.29448.python-l...@python.org, Navkirat Singh wrote: I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. I assume the JPEG data is received along with other field values in the POST. You’ll be saving those other fields in a database, right? So why not save the JPEG image there as well? No, the only thing in the body of the POST are the bytes of the JPEG. He was incorrect in thinking that the JPEG data was arriving in the header. See the later posts in the thread for complete answers to his problem. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Writing byte stream as jpeg format to disk
Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
Navkirat Singh navkir...@gmail.com writes: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. Why? -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 26-Aug-2010, at 11:01 PM, John Bokma wrote: Navkirat Singh navkir...@gmail.com writes: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. Why? -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list why? I am not quite sure what you have not understood. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 2010-08-26, Navkirat Singh navkir...@gmail.com wrote: On 26-Aug-2010, at 11:01 PM, John Bokma wrote: Navkirat Singh navkir...@gmail.com writes: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. Why? why? I am not quite sure what you have not understood. You're starting with JPEG data. If you want to write it to a file, then write it to a file. Whatever process you're describing as I parse the bytes by decoding them to a string is not needed and is apparently converting the JPEG data into something that's not JPEG data. -- Grant Edwards grant.b.edwardsYow! PARDON me, am I at speaking ENGLISH? gmail.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
Hi Navkirat, On 2010-08-26 19:22, Navkirat Singh wrote: I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. I guess you mean you see a byte string in your server and want to write that to disk. Assuming the string you got is the correct image data in the first place, you can, in Python 2.x, write the string data to disk like this: fobj = open(some_image.jpg, wb) fobj.write(byte_string) fobj.close() Note that you should use wb as mode to write as binary. Otherwise you'll get automatic line ending conversion (at least on Windows) which will give the result you describe. If my answer doesn't help, you probably need to describe in more detail what you're doing, including showing some real code. Stefan -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
I am sorry, maybe I was not elaborate in what I was having trouble with. I am using a jpegcam library, which on my web page captures a webcam image and sends it to the server via the POST method. On the Server side (python 3), I receive this image as a part of header content in bytes (I know thats not how it should be done, but the author has some reason for it), so I first convert the headers to a string so I can separate them. From the separated headers I fish out the content of the image file (which is a string). This is where I get stuck, how am I supposed to convert it back to an image, so I can save as a jpeg on disk. Regards, Nav On 27-Aug-2010, at 12:07 AM, Stefan Schwarzer wrote: Hi Navkirat, On 2010-08-26 19:22, Navkirat Singh wrote: I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. I guess you mean you see a byte string in your server and want to write that to disk. Assuming the string you got is the correct image data in the first place, you can, in Python 2.x, write the string data to disk like this: fobj = open(some_image.jpg, wb) fobj.write(byte_string) fobj.close() Note that you should use wb as mode to write as binary. Otherwise you'll get automatic line ending conversion (at least on Windows) which will give the result you describe. If my answer doesn't help, you probably need to describe in more detail what you're doing, including showing some real code. Stefan -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 26/08/2010 19:57, Navkirat Singh wrote: I am sorry, maybe I was not elaborate in what I was having trouble with. I am using a jpegcam library, which on my web page captures a webcam image and sends it to the server via the POST method. On the Server side (python 3), I receive this image as a part of header content in bytes (I know thats not how it should be done, but the author has some reason for it), so I first convert the headers to a string so I can separate them. From the separated headers I fish out the content of the image file (which is a string). This is where I get stuck, how am I supposed to convert it back to an image, so I can save as a jpeg on disk. [snip] What does that string look like? Try printing out repr(image[ : 100]). If it looks like plain bytes, then write it to file. If it looks like a series of hex digits, then decode to bytes before writing. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/26/10 1:25 PM, Navkirat Singh wrote: On 26-Aug-2010, at 11:01 PM, John Bokma wrote: Navkirat Singhnavkir...@gmail.com writes: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. Why? -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list why? I am not quite sure what you have not understood. Why decode the bytes to (presumably) unicode strings just to encode them back to bytes again? JPEG is not composed of unicode characters; you need to leave them as bytes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:10 AM, Robert Kern wrote: On 8/26/10 1:25 PM, Navkirat Singh wrote: On 26-Aug-2010, at 11:01 PM, John Bokma wrote: Navkirat Singhnavkir...@gmail.com writes: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. Why? -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list why? I am not quite sure what you have not understood. Why decode the bytes to (presumably) unicode strings just to encode them back to bytes again? JPEG is not composed of unicode characters; you need to leave them as bytes. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list The image bytes are a part of a HTTP header content ( not the message body ). To separate the header content from the image I have to first convert the bytes to string to perform parsing. The resultant string then needs to be converted back to the image. Hence, my problem. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 12:45 AM, MRAB wrote: On 26/08/2010 19:57, Navkirat Singh wrote: I am sorry, maybe I was not elaborate in what I was having trouble with. I am using a jpegcam library, which on my web page captures a webcam image and sends it to the server via the POST method. On the Server side (python 3), I receive this image as a part of header content in bytes (I know thats not how it should be done, but the author has some reason for it), so I first convert the headers to a string so I can separate them. From the separated headers I fish out the content of the image file (which is a string). This is where I get stuck, how am I supposed to convert it back to an image, so I can save as a jpeg on disk. [snip] What does that string look like? Try printing out repr(image[ : 100]). If it looks like plain bytes, then write it to file. If it looks like a series of hex digits, then decode to bytes before writing. -- http://mail.python.org/mailman/listinfo/python-list Thanks MRAB, your suggestions have always been very helpful to me. I shall let you know on what I see. Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Nav-- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Also, my apologies for lack of knowledge of character encodings. You have pointed out correctly about unicode encoding. I was under the impression that a unicode will preserve the integrity of the message which has been encoded. -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 26/08/2010 21:14, Navkirat Singh wrote: On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Try: image = header.split(b'keep-alive\r\n\r\n', 1)[-1] open(image_path, 'wb').write(image) -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
Navkirat Singh wrote: O snip I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Nav An arbitrary string of bytes is not necessarily valid utf-8, so I'm not sure why you haven't been getting errors during that decode. In any case, such a conversion is not reversible. I would parse that as bytes, perhaps by searching for 'keep-alive'. Then split the byte stream into the two parts, and only convert the first part to Unicode (Python 3 string). For safety, you could check to make sure the search pattern only appears once, and potentially decode it multiple times. It'll only make sense once. DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:57 AM, MRAB wrote: On 26/08/2010 21:14, Navkirat Singh wrote: On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Try: image = header.split(b'keep-alive\r\n\r\n', 1)[-1] open(image_path, 'wb').write(image) -- http://mail.python.org/mailman/listinfo/python-list I think I forgot to mention that the original is a stream of bytes decoded using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again). @MRAB - the split() method in python 3 works only on strings and throws an error if I try to use bytes -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
Navkirat Singh navkir...@gmail.com writes: I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f You're mistaken that the content is part of the headers, it's not. The \r\n\r\n separates headers from the content. Why don't you use urllib to save you from all this hassle? -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/26/10 3:47 PM, Navkirat Singh wrote: On 27-Aug-2010, at 1:57 AM, MRAB wrote: On 26/08/2010 21:14, Navkirat Singh wrote: On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Try: image = header.split(b'keep-alive\r\n\r\n', 1)[-1] open(image_path, 'wb').write(image) -- http://mail.python.org/mailman/listinfo/python-list I think I forgot to mention that the original is a stream of bytes decoded using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again). @MRAB - the split() method in python 3 works only on strings and throws an error if I try to use bytes This is incorrect. Python 3.1.2 (r312:79360M, Mar 24 2010, 01:33:18) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type help, copyright, credits or license for more information. bytes = b'Connection: keep-alive\r\n\r\nbody' bytes.split(b'\r\n\r\n', 1)[-1] b'body' FYI: the JPEG data is not in the header. The b'\r\n\r\n' sequence delimits the header from the body. Do not rely on Connection: keep-alive being the last header. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 1:57 AM, MRAB wrote:
On 26/08/2010 21:14, Navkirat Singh wrote:
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote:
Navkirat Singh wrote:
Hey guys,
I am programming a webserver, I receive a jpeg file with the POST
method.The file (.jpeg) is encoded in bytes, I parse the bytes by
decoding them to a string. I wanted to know how i could write the
file (now a string) as a jpeg image on disk. When I try to encode the
same string to a bytes and write them in binary format to disk, the
file is not recognized as jpeg. I would be grateful if someone could
help me with this.
Regards,
Nav
If by decoding them to a string you mean converting to Unicode, then
you've already trashed the data. That's only valid if the bytes had
been encoded from valid Unicode characters, and then only if you use
the corresponding decoding technique.
If you mean some other decoding, then the question is meaningless
without telling us just what the decoding is, preferably with some code.
It also might be useful to know what version of Python you're using,
when you post the code.
DaveA
Dave,
I am using Python3 and I receive a byte stream with a jpeg attached sent
by the web browser over a socket, which looks like this:
b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f
From the above, I need to:
a) Split the header content from the image content, which comes after
the keep-alive\r\n\r\n part
b) Then write the image content to file for further use as a jpeg.
Try:
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
open(image_path, 'wb').write(image)
--
http://mail.python.org/mailman/listinfo/python-list
Yay !! I figured it outit was really very simple. And if I really feel
guilty if I have wasted your time - @MRAB, @DAVE.
Here is what I needed to do:
a) Separate image content from header content of the byte stream received from
the web browser.
b) Save the image content to disk for further use.
Here is what I did. Following is just a snippet:
#-HERE IS WHERE I RECEIVE THE DATA
while True:
buff = socket.recv(8192)
byteStr +=buff
if not buff: break
#--ENCODING/DECODING STARTS FROM HERE (since I want to use
split/partition functions to separate header content from the image content)
strMsg = byteStr.decode(ISO-8859-1)
listMsg = strMsg.split('\r\n')
#
# do some more processing to search the list for the image content, say
supposing index is 11
#---
imageStr = listMsg[11].encode(ISO-8859-1) #Transform the byte string
just the way I found it
f = open('received.jpg','w'b)
f.write(imageStr)
The resultant file is a jpg file.
Thanks,
Nav
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 26/08/2010 21:47, Navkirat Singh wrote: On 27-Aug-2010, at 1:57 AM, MRAB wrote: On 26/08/2010 21:14, Navkirat Singh wrote: On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Try: image = header.split(b'keep-alive\r\n\r\n', 1)[-1] open(image_path, 'wb').write(image) -- http://mail.python.org/mailman/listinfo/python-list I think I forgot to mention that the original is a stream of bytes decoded using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again). @MRAB - the split() method in python 3 works only on strings and throws an error if I try to use bytes All i can say is that it works for me: header = b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f' image = header.split(b'keep-alive\r\n\r\n', 1)[-1] image b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f' What error did you get? -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 2:40 AM, Robert Kern wrote: On 8/26/10 3:47 PM, Navkirat Singh wrote: On 27-Aug-2010, at 1:57 AM, MRAB wrote: On 26/08/2010 21:14, Navkirat Singh wrote: On 27-Aug-2010, at 1:32 AM, Dave Angel wrote: Navkirat Singh wrote: Hey guys, I am programming a webserver, I receive a jpeg file with the POST method.The file (.jpeg) is encoded in bytes, I parse the bytes by decoding them to a string. I wanted to know how i could write the file (now a string) as a jpeg image on disk. When I try to encode the same string to a bytes and write them in binary format to disk, the file is not recognized as jpeg. I would be grateful if someone could help me with this. Regards, Nav If by decoding them to a string you mean converting to Unicode, then you've already trashed the data. That's only valid if the bytes had been encoded from valid Unicode characters, and then only if you use the corresponding decoding technique. If you mean some other decoding, then the question is meaningless without telling us just what the decoding is, preferably with some code. It also might be useful to know what version of Python you're using, when you post the code. DaveA Dave, I am using Python3 and I receive a byte stream with a jpeg attached sent by the web browser over a socket, which looks like this: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\nConnection: keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f From the above, I need to: a) Split the header content from the image content, which comes after the keep-alive\r\n\r\n part b) Then write the image content to file for further use as a jpeg. Try: image = header.split(b'keep-alive\r\n\r\n', 1)[-1] open(image_path, 'wb').write(image) -- http://mail.python.org/mailman/listinfo/python-list I think I forgot to mention that the original is a stream of bytes decoded using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again). @MRAB - the split() method in python 3 works only on strings and throws an error if I try to use bytes This is incorrect. Python 3.1.2 (r312:79360M, Mar 24 2010, 01:33:18) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type help, copyright, credits or license for more information. bytes = b'Connection: keep-alive\r\n\r\nbody' bytes.split(b'\r\n\r\n', 1)[-1] b'body' FYI: the JPEG data is not in the header. The b'\r\n\r\n' sequence delimits the header from the body. Do not rely on Connection: keep-alive being the last header. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list Thanks Everyone, @Robert - Thanks a lot for your time :-) , I did know that the body starts after the occurrence two CRLF sequences, but I was following RFC2616 as a guide, which specifically mentions: The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer- Encoding header field in the request’s message-headers Which has not been done by the author of the library, hence I said what I did. Or I have misunderstood the RFC Regards, Nav -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 2:48 AM, MRAB wrote:
On 26/08/2010 21:47, Navkirat Singh wrote:
On 27-Aug-2010, at 1:57 AM, MRAB wrote:
On 26/08/2010 21:14, Navkirat Singh wrote:
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote:
Navkirat Singh wrote:
Hey guys,
I am programming a webserver, I receive a jpeg file with the POST
method.The file (.jpeg) is encoded in bytes, I parse the bytes by
decoding them to a string. I wanted to know how i could write the
file (now a string) as a jpeg image on disk. When I try to encode the
same string to a bytes and write them in binary format to disk, the
file is not recognized as jpeg. I would be grateful if someone could
help me with this.
Regards,
Nav
If by decoding them to a string you mean converting to Unicode, then
you've already trashed the data. That's only valid if the bytes had
been encoded from valid Unicode characters, and then only if you use
the corresponding decoding technique.
If you mean some other decoding, then the question is meaningless
without telling us just what the decoding is, preferably with some code.
It also might be useful to know what version of Python you're using,
when you post the code.
DaveA
Dave,
I am using Python3 and I receive a byte stream with a jpeg attached sent
by the web browser over a socket, which looks like this:
b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f
From the above, I need to:
a) Split the header content from the image content, which comes after
the keep-alive\r\n\r\n part
b) Then write the image content to file for further use as a jpeg.
Try:
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
open(image_path, 'wb').write(image)
--
http://mail.python.org/mailman/listinfo/python-list
I think I forgot to mention that the original is a stream of bytes decoded
using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again).
@MRAB - the split() method in python 3 works only on strings and throws an
error if I try to use bytes
All i can say is that it works for me:
header = b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
image
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
What error did you get?
--
http://mail.python.org/mailman/listinfo/python-list
Hi MRAB,
Here is the error:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: Type str doesn't support the buffer API
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/26/10 4:17 PM, Navkirat Singh wrote:
Here is what I needed to do:
a) Separate image content from header content of the byte stream received from
the web browser.
b) Save the image content to disk for further use.
Here is what I did. Following is just a snippet:
#-HERE IS WHERE I RECEIVE THE DATA
while True:
buff = socket.recv(8192)
byteStr +=buff
if not buff: break
#--ENCODING/DECODING STARTS FROM HERE (since I want to use
split/partition functions to separate header content from the image content)
strMsg = byteStr.decode(ISO-8859-1)
listMsg = strMsg.split('\r\n')
#
# do some more processing to search the list for the image content, say
supposing index is 11
#---
imageStr = listMsg[11].encode(ISO-8859-1) #Transform the byte string
just the way I found it
If your JPEG happens to contain the bytes \r\n, then this will not work. Please
follow our advice. Split using b'\r\n\r\n' and use the maxsplit=1 argument to
make sure that you do not split on spurious b'\r\n\r\n' sequences inside the
JPEG body. Do not decode the bytes.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/26/10 4:25 PM, Navkirat Singh wrote: @Robert - Thanks a lot for your time :-) , I did know that the body starts after the occurrence two CRLF sequences, but I was following RFC2616 as a guide, which specifically mentions: The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer- Encoding header field in the request’s message-headers Which has not been done by the author of the library, hence I said what I did. Or I have misunderstood the RFC It certainly looks like the data has a Content-length header: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\n ... -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 2:58 AM, Navkirat Singh wrote:
On 27-Aug-2010, at 2:48 AM, MRAB wrote:
On 26/08/2010 21:47, Navkirat Singh wrote:
On 27-Aug-2010, at 1:57 AM, MRAB wrote:
On 26/08/2010 21:14, Navkirat Singh wrote:
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote:
Navkirat Singh wrote:
Hey guys,
I am programming a webserver, I receive a jpeg file with the POST
method.The file (.jpeg) is encoded in bytes, I parse the bytes by
decoding them to a string. I wanted to know how i could write the
file (now a string) as a jpeg image on disk. When I try to encode the
same string to a bytes and write them in binary format to disk, the
file is not recognized as jpeg. I would be grateful if someone could
help me with this.
Regards,
Nav
If by decoding them to a string you mean converting to Unicode, then
you've already trashed the data. That's only valid if the bytes had
been encoded from valid Unicode characters, and then only if you use
the corresponding decoding technique.
If you mean some other decoding, then the question is meaningless
without telling us just what the decoding is, preferably with some code.
It also might be useful to know what version of Python you're using,
when you post the code.
DaveA
Dave,
I am using Python3 and I receive a byte stream with a jpeg attached sent
by the web browser over a socket, which looks like this:
b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f
From the above, I need to:
a) Split the header content from the image content, which comes after
the keep-alive\r\n\r\n part
b) Then write the image content to file for further use as a jpeg.
Try:
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
open(image_path, 'wb').write(image)
--
http://mail.python.org/mailman/listinfo/python-list
I think I forgot to mention that the original is a stream of bytes decoded
using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again).
@MRAB - the split() method in python 3 works only on strings and throws an
error if I try to use bytes
All i can say is that it works for me:
header = b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
image
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
What error did you get?
--
http://mail.python.org/mailman/listinfo/python-list
Hi MRAB,
Here is the error:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
Traceback (most recent call last):
File stdin, line 1, in module
TypeError: Type str doesn't support the buffer API
I got your point
The argument for the split I have give is a string, hence the error.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/26/10 4:28 PM, Navkirat Singh wrote:
On 27-Aug-2010, at 2:48 AM, MRAB wrote:
On 26/08/2010 21:47, Navkirat Singh wrote:
On 27-Aug-2010, at 1:57 AM, MRAB wrote:
On 26/08/2010 21:14, Navkirat Singh wrote:
On 27-Aug-2010, at 1:32 AM, Dave Angel wrote:
Navkirat Singh wrote:
Hey guys,
I am programming a webserver, I receive a jpeg file with the POST
method.The file (.jpeg) is encoded in bytes, I parse the bytes by
decoding them to a string. I wanted to know how i could write the
file (now a string) as a jpeg image on disk. When I try to encode the
same string to a bytes and write them in binary format to disk, the
file is not recognized as jpeg. I would be grateful if someone could
help me with this.
Regards,
Nav
If by decoding them to a string you mean converting to Unicode, then
you've already trashed the data. That's only valid if the bytes had
been encoded from valid Unicode characters, and then only if you use
the corresponding decoding technique.
If you mean some other decoding, then the question is meaningless
without telling us just what the decoding is, preferably with some code.
It also might be useful to know what version of Python you're using,
when you post the code.
DaveA
Dave,
I am using Python3 and I receive a byte stream with a jpeg attached sent
by the web browser over a socket, which looks like this:
b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding:
gzip, deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f
From the above, I need to:
a) Split the header content from the image content, which comes after
the keep-alive\r\n\r\n part
b) Then write the image content to file for further use as a jpeg.
Try:
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
open(image_path, 'wb').write(image)
--
http://mail.python.org/mailman/listinfo/python-list
I think I forgot to mention that the original is a stream of bytes decoded
using ISO-8859-1 as utf-8 trhrew errors (lack of knowlegdge again).
@MRAB - the split() method in python 3 works only on strings and throws an
error if I try to use bytes
All i can say is that it works for me:
header = b': image/jpeg\r\nAccept: text/*\r\nReferer:
http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip,
deflate\r\nContent-Length: 91783\r\nConnection:
keep-alive\r\n\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
image = header.split(b'keep-alive\r\n\r\n', 1)[-1]
image
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00\x84\x00\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05\n\x07\x07\x06\x08\x0c\n\x0c\x0c\x0b\n\x0b\x0b\r\x0e\x12\x10\r\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f'
What error did you get?
--
http://mail.python.org/mailman/listinfo/python-list
Hi MRAB,
Here is the error:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
Traceback (most recent call last):
File stdin, line 1, inmodule
TypeError: Type str doesn't support the buffer API
Follow MRAB's example. You need to use a bytes object for the *argument*, too.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 3:02 AM, Robert Kern wrote:
On 8/26/10 4:17 PM, Navkirat Singh wrote:
Here is what I needed to do:
a) Separate image content from header content of the byte stream received
from the web browser.
b) Save the image content to disk for further use.
Here is what I did. Following is just a snippet:
#-HERE IS WHERE I RECEIVE THE DATA
while True:
buff = socket.recv(8192)
byteStr +=buff
if not buff: break
#--ENCODING/DECODING STARTS FROM HERE (since I want to use
split/partition functions to separate header content from the image content)
strMsg = byteStr.decode(ISO-8859-1)
listMsg = strMsg.split('\r\n')
#
# do some more processing to search the list for the image content, say
supposing index is 11
#---
imageStr = listMsg[11].encode(ISO-8859-1) #Transform the byte string
just the way I found it
If your JPEG happens to contain the bytes \r\n, then this will not work.
Please follow our advice. Split using b'\r\n\r\n' and use the maxsplit=1
argument to make sure that you do not split on spurious b'\r\n\r\n' sequences
inside the JPEG body. Do not decode the bytes.
--
Robert Kern
I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth.
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Thanks Robert,
My method worked too, I was able to do the above and save the jpeg flawlessly,
but your method seems better as I will not have to take the extra step of
encoding/decoding.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/26/10 4:17 PM, Navkirat Singh wrote: #-HERE IS WHERE I RECEIVE THE DATA while True: buff = socket.recv(8192) byteStr +=buff if not buff: break Also, you probably shouldn't bother writing an HTTP server using raw sockets. Use HTTPServer instead: http://docs.python.org/py3k/library/http.server.html or better, wsgiref: http://docs.python.org/py3k/library/wsgiref.html -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 3:04 AM, Robert Kern wrote: On 8/26/10 4:25 PM, Navkirat Singh wrote: @Robert - Thanks a lot for your time :-) , I did know that the body starts after the occurrence two CRLF sequences, but I was following RFC2616 as a guide, which specifically mentions: The presence of a message-body in a request is signaled by the inclusion of a Content-Length or Transfer- Encoding header field in the request’s message-headers Which has not been done by the author of the library, hence I said what I did. Or I have misunderstood the RFC It certainly looks like the data has a Content-length header: b': image/jpeg\r\nAccept: text/*\r\nReferer: http://127.0.0.1:8001/\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate\r\nContent-Length: 91783\r\n ... -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list Once again you opened my eyes, I think its been a tough night for me. Seeing too much and nothing at the same time. Regards -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 3:15 AM, Robert Kern wrote: On 8/26/10 4:17 PM, Navkirat Singh wrote: #-HERE IS WHERE I RECEIVE THE DATA while True: buff = socket.recv(8192) byteStr +=buff if not buff: break Also, you probably shouldn't bother writing an HTTP server using raw sockets. Use HTTPServer instead: http://docs.python.org/py3k/library/http.server.html or better, wsgiref: http://docs.python.org/py3k/library/wsgiref.html -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list Thanks a lot guys, you all have been a lot of help !! -- http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 8/26/2010 5:28 PM, Navkirat Singh wrote:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
You are trying to split bytes with a string, which is impossible.
Split bytes with bytes, strings with strings.
Traceback (most recent call last):
File stdin, line 1, inmodule
TypeError: Type str doesn't support the buffer API
--
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
Re: Writing byte stream as jpeg format to disk
On 27-Aug-2010, at 4:23 AM, Terry Reedy wrote:
On 8/26/2010 5:28 PM, Navkirat Singh wrote:
b = b'asdf'
type(b)
class 'bytes'
s = b.split(':')
You are trying to split bytes with a string, which is impossible.
Split bytes with bytes, strings with strings.
Traceback (most recent call last):
File stdin, line 1, inmodule
TypeError: Type str doesn't support the buffer API
--
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
Thanks !
--
http://mail.python.org/mailman/listinfo/python-list
