Re: Select hangs after some reads
Donn Cave wrote: In article [EMAIL PROTECTED], Steve Holden [EMAIL PROTECTED] wrote: The PSH flag indicates that the data stream must be flushed right through to the other end. This is essential for interactive protocols such as FTP: without it the server has no way to know that the client has sent a complete command, and vice versa. So you would expect to see this done explicitly in the source for an FTP client or server implementation -- something that sets this PSH flag or else the protocol won't work? Or am I misunderstanding what you mean? I don't see that specific string anywhere in a the particular FTP implementation whose source happened to be handy, basically a Berkeley 4.4 variant as I think most are on UNIX. Somewhere around here I have a pass-through FTP client/server application that adds GSSAPI-Kerberos5 authentication to the protocol traffic, and I don't remember needing to do any such thing there. I'd have to look harder at the details, but as I recall it, like any sane application the protocol is defined in terms of data, so you know if you have a complete command by looking at what you have. Nope, I wouldn't look any harder. You and I both know you won't find them. The client in this case was the client's transport layer, not the client application - I should have said sender, since I referred to the system at the other end as the receiver. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Love me, love my blog http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
Steve Holden wrote: Nope, I wouldn't look any harder. You and I both know you won't find them. The client in this case was the client's transport layer, not the client application - I should have said sender, since I referred to the system at the other end as the receiver. you know that it's a terminology issue when three experienced developers reply well, it already does that, you cannot do that, and I don't understand what you mean to the same question. (if you're playing Who Wants To Be A Millionaire, this is the right time to take the money and quit...) /F -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
Grant Edwards escreveu: On 2006-06-08, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Well, actually I´m using a very simple protocol wich sends only strings ended by newline. I need to send 3 chunks of information and a newline after them. On the reader side I make 3 readline(), this way I wouldn´t have to care about this problem, but maybe that´s where I´m falling. If that´s the case, I´ll have to use a more complex protocol. You can't use readline() with select(). Select tells you whether recv() called on the underlying socket will block or not. What's probably happening is that all of the data has been read from the underlying socket and is being held in a buffer waiting to be read by readline(). Yes, as I expected, its the buffers. In my opinion the problem is that the socket module doesn't provide a way of reading all its internal buffer. What internal buffer? readlines() just make subsequent calls to readline and readline may call recv, so we have a locked scene. I want to know if I will block anyway. I'm lost. Of course I can clean the buffer myself, but I think the socket module should provide a way of doing this. I'm afraid I don't understand what you mean. Since you talked about calling readline(), I assumed that you had called the socket object's makefile() method to create a file-object. The select system call can only tell you whether a recv on the socket will block or not. It knows nothing about the state of the file object on which you're calling readline(). Yes, that's right. The problem I was having and discussing is that, if you read one request at a time, using a readline, and used select to check for the incoming of new ones, you will block and the other unread requests will rest in the socket buffer (_rbuf class attribute, wich is a string). If you use several readline() (that's readlines()), to try to read all the buffer, and use select to check for the new ones, you will block because after the buffer is empty readline() will try a recv(). So the problem: readline() and readlines() can block and there's nothing (given by the module) to prevent you from it happening. There are solutions to this of course, but what I was arguing is that the module and the documentation don't warn and dont treat this case. -- Grant Edwards grante Yow! Is this an out-take at from the BRADY BUNCH? visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
Steve Holden wrote: [EMAIL PROTECTED] wrote: Hi, I'm building a multithreaded application and I encountered a tiny and annoying problem. I use a select to wait for data to be read from a socket, after some reads, the select simply blocks and stays that way until I close the connection on the other side of the socket. When the socket is closed on the writer end the select releases and then I get only empty strings from the socket. That's to be expected: the first return of a zero-length string from socket.read() indicates end of file. My question is this: Why did it block? The reading has never ended, every test I make I write 50 requests (wich are strings) to the socket and I have read the maximum of 34 requests. I'm using winPdb to take a closer look on what's happening and I see the threads blocked on this same select. If I send anything more through the socket, the select releases for a thread, despite the other data that is still unread. Are you sure that the received data is presenting in block the same size as are being sent? There's no guarantee this will be so on a TCP socket, and it may be that multiple sends are being coalesced into a single read. The important thing to focus on is the total number of bytes sent and received. Well, actually I´m using a very simple protocol wich sends only strings ended by newline. I need to send 3 chunks of information and a newline after them. On the reader side I make 3 readline(), this way I wouldn´t have to care about this problem, but maybe that´s where I´m falling. If that´s the case, I´ll have to use a more complex protocol. This is the select: rd,w,e = select.select([self.rfd],[],[]) self.rfd is the fileno of the file object returned by the makefile method from the socket object. It would be simpler to use the result of the socket's .fileno() method directly. Yes, you are right, this is something remanescent from old ideas. I know that's some buffer behavior that I'm missing but I don't know what it is. anything is helpfull, If I'm beeing stupid you can say it. thanks Without being able to see all your code it's hard to say whether or not you are doing something daft, but you give a general impression of competence. Thanks, the code has a significant size now, if I don´t solve it quickly I´ll post the most important parts. thanks again, Andre LS Meirelles regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Love me, love my blog http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
On 2006-06-08, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Well, actually I´m using a very simple protocol wich sends only strings ended by newline. I need to send 3 chunks of information and a newline after them. On the reader side I make 3 readline(), this way I wouldn´t have to care about this problem, but maybe that´s where I´m falling. If that´s the case, I´ll have to use a more complex protocol. You can't use readline() with select(). Select tells you whether recv() called on the underlying socket will block or not. What's probably happening is that all of the data has been read from the underlying socket and is being held in a buffer waiting to be read by readline(). The Select call has no way of knowing about that buffered data. As far as it's concerned there's no more data left to read, so it block until the socket is closed. -- Grant Edwards grante Yow! My mind is a potato at field... visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
Grant Edwards escreveu: On 2006-06-08, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Well, actually I´m using a very simple protocol wich sends only strings ended by newline. I need to send 3 chunks of information and a newline after them. On the reader side I make 3 readline(), this way I wouldn´t have to care about this problem, but maybe that´s where I´m falling. If that´s the case, I´ll have to use a more complex protocol. You can't use readline() with select(). Select tells you whether recv() called on the underlying socket will block or not. What's probably happening is that all of the data has been read from the underlying socket and is being held in a buffer waiting to be read by readline(). Yes, as I expected, its the buffers. In my opinion the problem is that the socket module doesn't provide a way of reading all its internal buffer. readlines() just make subsequent calls to readline and readline may call recv, so we have a locked scene. I want to know if I will block anyway. Of course I can clean the buffer myself, but I think the socket module should provide a way of doing this. It's not a big problem though. The Select call has no way of knowing about that buffered data. As far as it's concerned there's no more data left to read, so it block until the socket is closed. You're very right. thanks, Andre LS Meirelles -- Grant Edwards grante Yow! My mind is a potato at field... visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
[EMAIL PROTECTED] wrote: Grant Edwards escreveu: On 2006-06-08, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Well, actually I´m using a very simple protocol wich sends only strings ended by newline. I need to send 3 chunks of information and a newline after them. On the reader side I make 3 readline(), this way I wouldn´t have to care about this problem, but maybe that´s where I´m falling. If that´s the case, I´ll have to use a more complex protocol. You can't use readline() with select(). Select tells you whether recv() called on the underlying socket will block or not. What's probably happening is that all of the data has been read from the underlying socket and is being held in a buffer waiting to be read by readline(). Yes, as I expected, its the buffers. In my opinion the problem is that the socket module doesn't provide a way of reading all its internal buffer. readlines() just make subsequent calls to readline and readline may call recv, so we have a locked scene. I want to know if I will block anyway. Of course I can clean the buffer myself, but I think the socket module should provide a way of doing this. It's not a big problem though. The Select call has no way of knowing about that buffered data. As far as it's concerned there's no more data left to read, so it block until the socket is closed. You're very right. thanks, Of course, if the client forces the TCP PSH flag true then the receiver is guaranteed to debuffer the stream up to that point - this is how FTP clients work, for example. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Love me, love my blog http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
[EMAIL PROTECTED] wrote: Yes, as I expected, its the buffers. In my opinion the problem is that the socket module doesn't provide a way of reading all its internal buffer. umm. that's what recv() does, of course, of you pass in a large enough buffersize. for your use case, I suggest looking at asyncore/asynchat instead of trying to write your own asynchronous socket layer... /F -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
On 2006-06-08, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Well, actually I´m using a very simple protocol wich sends only strings ended by newline. I need to send 3 chunks of information and a newline after them. On the reader side I make 3 readline(), this way I wouldn´t have to care about this problem, but maybe that´s where I´m falling. If that´s the case, I´ll have to use a more complex protocol. You can't use readline() with select(). Select tells you whether recv() called on the underlying socket will block or not. What's probably happening is that all of the data has been read from the underlying socket and is being held in a buffer waiting to be read by readline(). Yes, as I expected, its the buffers. In my opinion the problem is that the socket module doesn't provide a way of reading all its internal buffer. What internal buffer? readlines() just make subsequent calls to readline and readline may call recv, so we have a locked scene. I want to know if I will block anyway. I'm lost. Of course I can clean the buffer myself, but I think the socket module should provide a way of doing this. I'm afraid I don't understand what you mean. Since you talked about calling readline(), I assumed that you had called the socket object's makefile() method to create a file-object. The select system call can only tell you whether a recv on the socket will block or not. It knows nothing about the state of the file object on which you're calling readline(). -- Grant Edwards grante Yow! Is this an out-take at from the BRADY BUNCH? visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
In article [EMAIL PROTECTED], Steve Holden wrote: Of course, if the client forces the TCP PSH flag true then the receiver is guaranteed to debuffer the stream up to that point - this is how FTP clients work, for example. I don't think that's right. You are confusing the PSH flag (which is basically unused in Unix networking I think) and the URG flag (which is extremely rarely used, but is indeed used by FTP to get abort requests to 'jump the queue' as it were). -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
Jon Ribbens wrote: In article [EMAIL PROTECTED], Steve Holden wrote: Of course, if the client forces the TCP PSH flag true then the receiver is guaranteed to debuffer the stream up to that point - this is how FTP clients work, for example. I don't think that's right. You are confusing the PSH flag (which is basically unused in Unix networking I think) and the URG flag (which is extremely rarely used, but is indeed used by FTP to get abort requests to 'jump the queue' as it were). Nope. The URG flag indicates that a packet contains out-of-band data, whihc is what you describe above. The PSH flag indicates that the data stream must be flushed right through to the other end. This is essential for interactive protocols such as FTP: without it the server has no way to know that the client has sent a complete command, and vice versa. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Love me, love my blog http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
In article [EMAIL PROTECTED], Steve Holden wrote: I don't think that's right. You are confusing the PSH flag (which is basically unused in Unix networking I think) and the URG flag (which is extremely rarely used, but is indeed used by FTP to get abort requests to 'jump the queue' as it were). Nope. I'm sorry, but you're definitely mistaken. The URG flag indicates that a packet contains out-of-band data, whihc is what you describe above. out-of-band is just way the sockets interface slightly mis-describes it. Either way, it causes (in Unix) the FTP server process to receive a SIGURG; the signal handler sets a flag and the data-transfer loop stops. The PSH flag indicates that the data stream must be flushed right through to the other end. That's sort-of true, but irrelevant. The PSH flag, when received, means that the networking layer should unbuffer any data and ensure it is released to the receiving application. *However*, BSD sockets never wait before allowing the application to receive incoming data, so the incoming PSH flag is unnecessary and ignored. In addition, the sockets interface simply doesn't provide a method for applications to set the PSH flag on output. The networking system simply sets the PSH flag by default whenever the output buffer empties. This is essential for interactive protocols such as FTP: without it the server has no way to know that the client has sent a complete command, and vice versa. I'm afraid that's completely wrong. The FTP server knows it has received a complete command because it sees the CRLF sequence in the data stream. I must admit it's refreshingly unusual to find someone who is more familiar with TCP/IP as described theoretically in the 1981 RFCs than the realities of the sockets interface ;-) -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
In article [EMAIL PROTECTED], Steve Holden [EMAIL PROTECTED] wrote: The PSH flag indicates that the data stream must be flushed right through to the other end. This is essential for interactive protocols such as FTP: without it the server has no way to know that the client has sent a complete command, and vice versa. So you would expect to see this done explicitly in the source for an FTP client or server implementation -- something that sets this PSH flag or else the protocol won't work? Or am I misunderstanding what you mean? I don't see that specific string anywhere in a the particular FTP implementation whose source happened to be handy, basically a Berkeley 4.4 variant as I think most are on UNIX. Somewhere around here I have a pass-through FTP client/server application that adds GSSAPI-Kerberos5 authentication to the protocol traffic, and I don't remember needing to do any such thing there. I'd have to look harder at the details, but as I recall it, like any sane application the protocol is defined in terms of data, so you know if you have a complete command by looking at what you have. Donn Cave, [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
Select hangs after some reads
Hi, I'm building a multithreaded application and I encountered a tiny and annoying problem. I use a select to wait for data to be read from a socket, after some reads, the select simply blocks and stays that way until I close the connection on the other side of the socket. When the socket is closed on the writer end the select releases and then I get only empty strings from the socket. My question is this: Why did it block? The reading has never ended, every test I make I write 50 requests (wich are strings) to the socket and I have read the maximum of 34 requests. I'm using winPdb to take a closer look on what's happening and I see the threads blocked on this same select. If I send anything more through the socket, the select releases for a thread, despite the other data that is still unread. This is the select: rd,w,e = select.select([self.rfd],[],[]) self.rfd is the fileno of the file object returned by the makefile method from the socket object. I know that's some buffer behavior that I'm missing but I don't know what it is. anything is helpfull, If I'm beeing stupid you can say it. thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: Select hangs after some reads
[EMAIL PROTECTED] wrote: Hi, I'm building a multithreaded application and I encountered a tiny and annoying problem. I use a select to wait for data to be read from a socket, after some reads, the select simply blocks and stays that way until I close the connection on the other side of the socket. When the socket is closed on the writer end the select releases and then I get only empty strings from the socket. That's to be expected: the first return of a zero-length string from socket.read() indicates end of file. My question is this: Why did it block? The reading has never ended, every test I make I write 50 requests (wich are strings) to the socket and I have read the maximum of 34 requests. I'm using winPdb to take a closer look on what's happening and I see the threads blocked on this same select. If I send anything more through the socket, the select releases for a thread, despite the other data that is still unread. Are you sure that the received data is presenting in block the same size as are being sent? There's no guarantee this will be so on a TCP socket, and it may be that multiple sends are being coalesced into a single read. The important thing to focus on is the total number of bytes sent and received. This is the select: rd,w,e = select.select([self.rfd],[],[]) self.rfd is the fileno of the file object returned by the makefile method from the socket object. It would be simpler to use the result of the socket's .fileno() method directly. I know that's some buffer behavior that I'm missing but I don't know what it is. anything is helpfull, If I'm beeing stupid you can say it. thanks Without being able to see all your code it's hard to say whether or not you are doing something daft, but you give a general impression of competence. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Love me, love my blog http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden -- http://mail.python.org/mailman/listinfo/python-list
