Re: Python recv loop
Hi Roy, On Feb 11, 2013, at 10:24 AM, Roy Smith r...@panix.com wrote: Is this server that you're talking to something that you have control over, i.e. are you stuck with this protocol? Given a choice, I'd go with something like JSON, for which pre-existing libraries for every language under the sun. I'm running JSON for my application messaging protocol but with JSON and python default unordered dict, there's no guarantee if I put in the length key in the JSON message, it will be placed on the first bytes hence why it was designed for a fixed 4-byte at the start of the message to indicate the message length. Beyond the 4-bytes it is all JSON. but if you have a better idea, i would certainly welcome it. Do you actually *know* what the value of nbuf is? Is it possible that (somehow) it's 0? You should print (log, whatever), the value of nbuf, just to make sure. nbuf is printing the right bytes amount, I removed the print statement before I made the first post. So to clarify, I added a print statement between the first recv and the second. {msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 2}} connected to misty:8080 sending data 138 bytes sent: 0x86{msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 2}} receiving data message length is 188 0 bytes received: So the subsequent recv() call will be readjusted with 188 bytes buffer size so theoretically, recv shouldn't return 0. The same logic that I used to send to the server from the python client that the server will readjust the second recv() call based on the length information. On this 2nd recv() call the server is able to obtain the rest of the messages. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
Hi Dave, On Feb 11, 2013, at 9:22 AM, Dave Angel da...@davea.name wrote: Exactly how are you sending hexadecimal ? If that 0xad (which is only one byte, what about the other 3 ?) is intended to be a C description, then it's certainly not hex, it's binary. And probably little-endian, to boot. That's a mistake, as network protocols almost always use big-endian. So what is the range of values for the length, and how are they actually encoded? Are they uint32 in native binary format? And are you permitted to change the encoding, to fix it? (If it were my choice, I'd make it a printable decimal value, first choice, or printable hex, second choice.) They are ASCII stream, actually JSON with the exception of the first 4 bytes. I avoided using struct class for this as it's overkill for my application. So the idea is that i code the length to a max of 0xff (max length of 256 bytes), I would only have to assign 4 bytes to it. If i need 1k length, i just need to increase it to 6 bytes or 4 if i decide to strip the 0x. the code for the function is as follows: def request_get_session(sock, jmsg): # append message length plen = hex(len(jmsg)) msg = '{0}{1}'.format(plen, jmsg) print 'sending data' n = sock.send(msg) str = '{0} bytes sent: {1}'.format(n, msg) print str # receive message length print 'receiving data' mlen = sock.recv(4) try: nbuf = int(mlen, 16) except ValueError as e: print 'invalid length type' return -1 print 'message length is {0}'.format(nbuf) while True: buf = sock.recv(nbuf) if not buf: break slen = len(buf) str = {0} bytes received: {1}.format(slen, buf) print str return 0 I've managed to receive and translate the message length until I reach my second recv which I readjusted the buffer size to include the new message length. However that failed and recv received 0 bytes. I implemented the same algorithm on the server side using C and it work so appreciate if you can help me on this. # receive message length print 'receiving data' mlen = sock.recv(4) try: nbuf = int(mlen, 16) That supposes the count is being sent as a printable string of hex digits. That's not what I concluded above. That is what I meant. it just an ascii string. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
Hi MRAB, My code now works thanks to your advice. {msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 2}} connected to misty:8080 sending data 138 bytes sent: 0x86{msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 2}} receiving data message length is 188 188 bytes received: { msgver: 1.00, appver: 1.00, appid: 1000, msgid: 10, subcode: 20, data: [ 110.159.183.16, 124.108.16.94, 2400:3700:50::2, 2400:3700:50::1, 2400:3700:51:: ] } Many thanks. On Feb 11, 2013, at 9:55 AM, MRAB pyt...@mrabarnett.plus.com wrote: You should keep reading until you get all need or the connection is closed: buf = b'' while len(buf) nbuf: chunk = sock.recv(nbuf - len(buf)) if not chunk: break buf += chunk -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On 2013-02-11 14:56, Ihsan Junaidi Ibrahim wrote: Hi Roy, On Feb 11, 2013, at 10:24 AM, Roy Smith r...@panix.com wrote: Is this server that you're talking to something that you have control over, i.e. are you stuck with this protocol? Given a choice, I'd go with something like JSON, for which pre-existing libraries for every language under the sun. I'm running JSON for my application messaging protocol but with JSON and python default unordered dict, there's no guarantee if I put in the length key in the JSON message, it will be placed on the first bytes hence why it was designed for a fixed 4-byte at the start of the message to indicate the message length. Beyond the 4-bytes it is all JSON. but if you have a better idea, i would certainly welcome it. I probably wouldn't make it fixed length. I'd have the length in decimal followed by, say, \n. Do you actually *know* what the value of nbuf is? Is it possible that (somehow) it's 0? You should print (log, whatever), the value of nbuf, just to make sure. nbuf is printing the right bytes amount, I removed the print statement before I made the first post. So to clarify, I added a print statement between the first recv and the second. {msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 2}} connected to misty:8080 sending data 138 bytes sent: 0x86{msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 2}} receiving data message length is 188 0 bytes received: So the subsequent recv() call will be readjusted with 188 bytes buffer size so theoretically, recv shouldn't return 0. The same logic that I used to send to the server from the python client that the server will readjust the second recv() call based on the length information. On this 2nd recv() call the server is able to obtain the rest of the messages. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote: I probably wouldn't make it fixed length. I'd have the length in decimal followed by, say, \n. Or even followed by any non-digit. Chances are your JSON data begins with a non-digit, so you'd just have to insert a space in the event that you're JSON-encoding a flat integer. (Which might not ever happen, if you know that your data will always be an object.) ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On 02/11/2013 10:02 AM, Ihsan Junaidi Ibrahim wrote: snip print 'message length is {0}'.format(nbuf) while True: buf = sock.recv(nbuf) if not buf: break This loop doesn't terminate till buf is zero length, which it will be eventually. At that point, you've overwritten the real data you may have gotten. So the loop is just plain wrong. Uwe MRAB's code, since there's no promise that all the data will be returned in a single call. Keep accumulating it as you loop. slen = len(buf) str = {0} bytes received: {1}.format(slen, buf) print str return 0 -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On Feb 11, 2013, at 11:24 PM, Chris Angelico ros...@gmail.com wrote: On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote: I probably wouldn't make it fixed length. I'd have the length in decimal followed by, say, \n. Or even followed by any non-digit. Chances are your JSON data begins with a non-digit, so you'd just have to insert a space in the event that you're JSON-encoding a flat integer. (Which might not ever happen, if you know that your data will always be an object.) ChrisA So on the first recv() call, I set the buffer at 1 character and I iterate over single character until a non-digit character is encountered? -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On Tue, Feb 12, 2013 at 12:41 PM, Ihsan Junaidi Ibrahim ih...@grep.my wrote: On Feb 11, 2013, at 11:24 PM, Chris Angelico ros...@gmail.com wrote: On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote: I probably wouldn't make it fixed length. I'd have the length in decimal followed by, say, \n. Or even followed by any non-digit. Chances are your JSON data begins with a non-digit, so you'd just have to insert a space in the event that you're JSON-encoding a flat integer. (Which might not ever happen, if you know that your data will always be an object.) ChrisA So on the first recv() call, I set the buffer at 1 character and I iterate over single character until a non-digit character is encountered? More efficient would be to guess that it'll be, say, 10 bytes, and then retain any excess for your JSON read loop. But you'd need to sort that out between the halves of your code. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
In article mailman.1655.1360594595.2939.python-l...@python.org, Ihsan Junaidi Ibrahim ih...@grep.my wrote: I'm running JSON for my application messaging protocol but with JSON and python default unordered dict, there's no guarantee if I put in the length key in the JSON message, it will be placed on the first bytes hence why it was designed for a fixed 4-byte at the start of the message to indicate the message length. Beyond the 4-bytes it is all JSON. I'm confused. It sounds like you're making things way more complicated than they have to be. Can you give us an example of an actual data message? -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On 2013-02-12 02:20, Chris Angelico wrote: On Tue, Feb 12, 2013 at 12:41 PM, Ihsan Junaidi Ibrahim ih...@grep.my wrote: On Feb 11, 2013, at 11:24 PM, Chris Angelico ros...@gmail.com wrote: On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote: I probably wouldn't make it fixed length. I'd have the length in decimal followed by, say, \n. Or even followed by any non-digit. Chances are your JSON data begins with a non-digit, so you'd just have to insert a space in the event that you're JSON-encoding a flat integer. (Which might not ever happen, if you know that your data will always be an object.) ChrisA So on the first recv() call, I set the buffer at 1 character and I iterate over single character until a non-digit character is encountered? More efficient would be to guess that it'll be, say, 10 bytes, and then retain any excess for your JSON read loop. But you'd need to sort that out between the halves of your code. If the length is always followed by a space then it's easier to split it off the input: buf = sock.recv(10) space_pos = buf.find(b ) nbuf = int(buf[ : space_pos]) buf = buf[space_pos+ 1 : ] while len(buf) nbuf: chunk = sock.recv(nbuf - len(buf)) if not chunk: break buf += chunk I'm assuming that: 1. The initial recv returns the length followed by a space. It could, of course, return fewer bytes (space_pos == -1), so you may need to recv some more bytes, like what's done later on. 2. At least 10 bytes were sent. Imagine what would happen if the sender sent b2 [] immediately followed by b2 []. The initial recv could return all of it. In that case you could save the excess until next time. Alternatively, the sender could guarantee that it would never send fewer than the 10 bytes, padding with several b if necessary. -- http://mail.python.org/mailman/listinfo/python-list
Python recv loop
Hi, I'm implementing a python client connecting to a C-backend server and am currently stuck to as to how to proceed with receiving variable-length byte stream coming in from the server. I have coded the first 4 bytes (in hexadecimal) of message coming in from the server to specify the length of the message payload i.e. 0xad{...} I've managed to receive and translate the message length until I reach my second recv which I readjusted the buffer size to include the new message length. However that failed and recv received 0 bytes. I implemented the same algorithm on the server side using C and it work so appreciate if you can help me on this. # receive message length print 'receiving data' mlen = sock.recv(4) try: nbuf = int(mlen, 16) except ValueError as e: print 'invalid length type' return -1 while True: buf = sock.recv(nbuf) if not buf: break slen = len(buf) str = {0} bytes received: {1}.format(slen, buf) print str -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On 02/10/2013 07:48 PM, Ihsan Junaidi Ibrahim wrote: Hi, I'm implementing a python client connecting to a C-backend server and am currently stuck to as to how to proceed with receiving variable-length byte stream coming in from the server. I have coded the first 4 bytes (in hexadecimal) of message coming in from the server to specify the length of the message payload i.e. 0xad{...} Exactly how are you sending hexadecimal ? If that 0xad (which is only one byte, what about the other 3 ?) is intended to be a C description, then it's certainly not hex, it's binary. And probably little-endian, to boot. That's a mistake, as network protocols almost always use big-endian. So what is the range of values for the length, and how are they actually encoded? Are they uint32 in native binary format? And are you permitted to change the encoding, to fix it? (If it were my choice, I'd make it a printable decimal value, first choice, or printable hex, second choice.) I've managed to receive and translate the message length until I reach my second recv which I readjusted the buffer size to include the new message length. However that failed and recv received 0 bytes. I implemented the same algorithm on the server side using C and it work so appreciate if you can help me on this. # receive message length print 'receiving data' mlen = sock.recv(4) try: nbuf = int(mlen, 16) That supposes the count is being sent as a printable string of hex digits. That's not what I concluded above. except ValueError as e: If the count starts as 0xad, I can't imagine why this exception wasn't thrown. print 'invalid length type' return -1 while True: buf = sock.recv(nbuf) if not buf: break slen = len(buf) str = {0} bytes received: {1}.format(slen, buf) print str You might need to play with hexlify, if you persist in sending the count in binary. -- DaveA -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
On 2013-02-11 00:48, Ihsan Junaidi Ibrahim wrote: Hi, I'm implementing a python client connecting to a C-backend server and am currently stuck to as to how to proceed with receiving variable-length byte stream coming in from the server. I have coded the first 4 bytes (in hexadecimal) of message coming in from the server to specify the length of the message payload i.e. 0xad{...} I've managed to receive and translate the message length until I reach my second recv which I readjusted the buffer size to include the new message length. However that failed and recv received 0 bytes. I implemented the same algorithm on the server side using C and it work so appreciate if you can help me on this. # receive message length print 'receiving data' mlen = sock.recv(4) try: nbuf = int(mlen, 16) except ValueError as e: print 'invalid length type' return -1 while True: buf = sock.recv(nbuf) if not buf: break slen = len(buf) str = {0} bytes received: {1}.format(slen, buf) print str You should keep reading until you get all need or the connection is closed: buf = b'' while len(buf) nbuf: chunk = sock.recv(nbuf - len(buf)) if not chunk: break buf += chunk -- http://mail.python.org/mailman/listinfo/python-list
Re: Python recv loop
In article mailman.1612.1360544258.2939.python-l...@python.org, Ihsan Junaidi Ibrahim ih...@grep.my wrote: I'm implementing a python client connecting to a C-backend server and am currently stuck to as to how to proceed with receiving variable-length byte stream coming in from the server. I have coded the first 4 bytes (in hexadecimal) of message coming in from the server to specify the length of the message payload i.e. 0xad{...} Is this server that you're talking to something that you have control over, i.e. are you stuck with this protocol? Given a choice, I'd go with something like JSON, for which pre-existing libraries for every language under the sun. But, let's assume for the moment that you're stuck with this length-value encoding. OK, but it's going to be more complicated than you think. [I assume we're talking TCP here?] Carefully read the documentation for socket.recv(): socket.recv(bufsize[, flags]) [...] The maximum amount of data to be received at once is specified by bufsize. Linger on the word maximum, and try to grok the fullness of of how annoying that can be. What it means is that if the other side sent 120 bytes (octets), recv() might return all 120 at once, or it might return them one at a time, or anything in between. So, what you need to do is call recv() repeatedly in a loop, each time passing it a value for bufsize which represents the amount left in the message (i.e. the original message length parsed earlier minus all the little bits and pieces that have been read so far). Keep in mind, you also need to do this when you recv() the first 4 octets, which make up the length field. What you've got, recv(4), will work MOST of the time, but it's perfectly legal for recv() to return a short read. You can't predict how fragmentation and retry timeouts and all sorts of low-level crud will cause your message boundaries to get scrambled. # receive message length print 'receiving data' mlen = sock.recv(4) try: nbuf = int(mlen, 16) except ValueError as e: print 'invalid length type' return -1 while True: buf = sock.recv(nbuf) if not buf: break slen = len(buf) str = {0} bytes received: {1}.format(slen, buf) print str Do you actually *know* what the value of nbuf is? Is it possible that (somehow) it's 0? You should print (log, whatever), the value of nbuf, just to make sure. And, once you'e got all this working, tear it all out and convert to using something sane like JSON. Let somebody else worry about all the horrible details. -- http://mail.python.org/mailman/listinfo/python-list