Re: Thread Question
On Thu, 20 Mar 2003 11:45:28 -0700, Verdon Walker wrote: >I knew I had not explained myself well enough given your last >response. >I did not mean to attempt to abort the "current chunck", but rather >to >be able to read the cancel between chunks so I could do just what >you >suggest. Of course, the definition of "current chunk" is part of my >problem. I was thinking of the fact that when I do an SSL_write on a >very large buffer, OpenSSL breaks it up into 16k pieces. Can I >somehow >do a read in between writing those 16k pieces or do I have to step >back >and do it between just between the "chunks" I sent to OpenSSL >through >SSL_write? Now I'm really puzzled. First you want OpenSSL to provide semantics like normal TCP sockets. Now you want more access to SSL internals such as the 16Kb chunks it uses internally. Which way do you want it? >BTW, we are using non-blocking sockets so I could clearly read from >the >socket between socket writes, but if I SSL_read from the socket >during a >long SSL_write operation, I am asking for trouble (with the internal >state) so I don't think I can do the synchronization down at the BIO >level unless I am still missing a big piece of the puzzle. If you're using non-blocking sockets, there are no "long SSL_write operation"s. If you have data to write and you can write, then write. If there's data to read, read it from the socket and see if it has application data. Basically, all you have to do is *not* call SSL_write with large chunks as a blocking operation. If you did this, you couldn't abort them no matter what anyway as you'd kill the protocol you're layering on top of SSL, so I don't even understand why there's an issue. >Thanks again for taking the time to respond to my questions. Your >feedback is very helpful. > >I apologize that this discussion has taken on a "user" tone rather >than >a "developer" tone. That was not my original intent. I have to admit I'm utterly baffled as to what you want to do. If all your writes are non-blocking, you can easily read between writes. For example, you can code a 'write' function that works as follows: 1) Acquire a single mutex for this connection. 2) Attempt a non-blocking SSL write. 3) Release the mutex. 4) If all data was written, return. 5) Otherwise, 'select' on the socket for writing, then jump to step 1 when 'select' returns. Then you can code a 'read' function that works as follow: 1) Acquire the single mutex for this connection. 2) Check if SSL_read has any processed data to be read (allowing it to read the socket but not block). 3) Release the mutex. If data was read, return. 4) 'select' on the socket for reading, optionally with a timeout. Then either return or jump to step 1. This will give you either non-blocking or blocking semantics and will allow you to call 'read' and 'write' at the same time without a problem. I don't understand exactly what semantics you want, but whatever they are, you can code them. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: Thread Question
I knew I had not explained myself well enough given your last response. I did not mean to attempt to abort the "current chunck", but rather to be able to read the cancel between chunks so I could do just what you suggest. Of course, the definition of "current chunk" is part of my problem. I was thinking of the fact that when I do an SSL_write on a very large buffer, OpenSSL breaks it up into 16k pieces. Can I somehow do a read in between writing those 16k pieces or do I have to step back and do it between just between the "chunks" I sent to OpenSSL through SSL_write? BTW, we are using non-blocking sockets so I could clearly read from the socket between socket writes, but if I SSL_read from the socket during a long SSL_write operation, I am asking for trouble (with the internal state) so I don't think I can do the synchronization down at the BIO level unless I am still missing a big piece of the puzzle. Thanks again for taking the time to respond to my questions. Your feedback is very helpful. I apologize that this discussion has taken on a "user" tone rather than a "developer" tone. That was not my original intent. Verdon >>> [EMAIL PROTECTED] 3/19/2003 5:54:04 PM >>> On Wed, 19 Mar 2003 16:53:32 -0700, Verdon Walker wrote: >First, thank you for your responses. I appreciate the feedback, but >I don't think I understand the points you are making in your last >email. >Perhaps, I did not explain myself well enough, but the idea of >allowing >long operations to be cancelled is hardly rare. Suppose for example, >you >want to search for a friend in the phone book. Whatever criteria you >specify you are going to want to stop excessive search results from >continuing once you have found the person you are looking for. >Supporting that in a clear text mode, but not in an SSL mode doesn't >make sense. You try to send 2Mb but then interrupt it. 1,503,257 bytes have been sent. Now what do you do? Do you go back retrospectively and say, "oh, I'm 39 bytes into a 512 byte protocol structure, I'll have to send 473 bytes to recover synchronization so I can start another block? I have written code layered on top of TCP for more than 16 years and I've never seen anything like this. Much more common is to stop feeding data to the socket by not sending additional chunks to the sender. In other words, you finish the current 'chunk' and don't send anymore chunks. This requires no special support from the socket code and will work fine with OpenSSL. >It would seem that the SSL structure could be shared between reads >and >writes if we could guarantee that they didn't use the same members >of >the structure. Does anyone know which members are shared between both >reads and writes. I know the "rwstate" is, but I'm not sure what >else. You are barking up the wrong tree. Sending on an SSL link can require receiving from the underlying TCP stream and vice versa. You can't easily make them independent and I don't think you should want to. If you must, you can write your own send and receive functions that acquire a single lock while they're waiting for the send or receive to be possible using non-blocking sockets or BIO pairs. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: Thread Question
On Wed, 19 Mar 2003 16:53:32 -0700, Verdon Walker wrote: >First, thank you for your responses. I appreciate the feedback, but >I don't think I understand the points you are making in your last >email. >Perhaps, I did not explain myself well enough, but the idea of >allowing >long operations to be cancelled is hardly rare. Suppose for example, >you >want to search for a friend in the phone book. Whatever criteria you >specify you are going to want to stop excessive search results from >continuing once you have found the person you are looking for. >Supporting that in a clear text mode, but not in an SSL mode doesn't >make sense. You try to send 2Mb but then interrupt it. 1,503,257 bytes have been sent. Now what do you do? Do you go back retrospectively and say, "oh, I'm 39 bytes into a 512 byte protocol structure, I'll have to send 473 bytes to recover synchronization so I can start another block? I have written code layered on top of TCP for more than 16 years and I've never seen anything like this. Much more common is to stop feeding data to the socket by not sending additional chunks to the sender. In other words, you finish the current 'chunk' and don't send anymore chunks. This requires no special support from the socket code and will work fine with OpenSSL. >It would seem that the SSL structure could be shared between reads >and >writes if we could guarantee that they didn't use the same members >of >the structure. Does anyone know which members are shared between both >reads and writes. I know the "rwstate" is, but I'm not sure what >else. You are barking up the wrong tree. Sending on an SSL link can require receiving from the underlying TCP stream and vice versa. You can't easily make them independent and I don't think you should want to. If you must, you can write your own send and receive functions that acquire a single lock while they're waiting for the send or receive to be possible using non-blocking sockets or BIO pairs. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: Thread Question
First, thank you for your responses. I appreciate the feedback, but I don't think I understand the points you are making in your last email. Perhaps, I did not explain myself well enough, but the idea of allowing long operations to be cancelled is hardly rare. Suppose for example, you want to search for a friend in the phone book. Whatever criteria you specify you are going to want to stop excessive search results from continuing once you have found the person you are looking for. Supporting that in a clear text mode, but not in an SSL mode doesn't make sense. It would seem that the SSL structure could be shared between reads and writes if we could guarantee that they didn't use the same members of the structure. Does anyone know which members are shared between both reads and writes. I know the "rwstate" is, but I'm not sure what else. Verdon >>> [EMAIL PROTECTED] 3/19/2003 12:41:37 PM >>> OpenSSL doesn't support cancelable operations simply because it is extremely rare that someone would want to use them. You may implement your own buffering with asynchronous semantics if you wish. You could also use OpenSSL BIO pairs for this purpose. OpenSSL just can't give you the same semantics TCP does without being integrated into the operating system. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: Thread Question
On Wed, 19 Mar 2003 09:51:20 -0700, Verdon Walker wrote: >We have one thread wanting to write a large amount of data (say 2 >Meg). >OpenSSL will break that data up into 16k chunks, SSLize them and >write >each separately. During the process, it is using the "rwstate" in >the >SSL structure to indicate the status of the write operation. How do >I >then process a "cancel" instruction that I might receive from an >impatient client who decides they don't want or need to see all 2 >Meg of >data? Is it safe to do a read operation between the writes of each >16k >chunk or do I need to wait until all 2 Meg have been written? With >clear >sockets, a separate thread for reading and writing works very well >to >handle this. With OpenSSL, the state of the write can easily be >stomped >on by reading the "cancel". OpenSSL doesn't support cancelable operations simply because it is extremely rare that someone would want to use them. You may implement your own buffering with asynchronous semantics if you wish. You could also use OpenSSL BIO pairs for this purpose. OpenSSL just can't give you the same semantics TCP does without being integrated into the operating system. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: Thread Question
While that (not simultaneously accessing the same structure from different threads) is certainly understandable, it is often the case that an SSL implementation will mimic a socket implementation. In fact, the clear socket implementation is always a candidate for SSL work and it is perfectly acceptable to have separate threads reading and writing from/to a socket. The socket itself will handle the contention this may cause. I am getting "beat up" because OpenSSL doesn't follow the "socket model" which seems unreasonable to some of my coworkers trying to use SSL sockets in more or less the same fashion as clear sockets (at least with the same basic architecture). I know from looking in the email archives that this issue has come up before so we are not the only ones struggling with it. The suggested solution was to create a mutex and use the ex_data for the SSL structure to tie it to the connection and then mutex read/write access. I am willing to do that, but even that is not terribly straight forward. Consider: We have one thread wanting to write a large amount of data (say 2 Meg). OpenSSL will break that data up into 16k chunks, SSLize them and write each separately. During the process, it is using the "rwstate" in the SSL structure to indicate the status of the write operation. How do I then process a "cancel" instruction that I might receive from an impatient client who decides they don't want or need to see all 2 Meg of data? Is it safe to do a read operation between the writes of each 16k chunk or do I need to wait until all 2 Meg have been written? With clear sockets, a separate thread for reading and writing works very well to handle this. With OpenSSL, the state of the write can easily be stomped on by reading the "cancel". Verdon >>> [EMAIL PROTECTED] 3/18/2003 7:10:23 PM >>> On Mon, 17 Mar 2003 11:26:46 -0700, Verdon Walker wrote: >I know from looking in the archives that this question has been >asked >before, but I am wondering if anything has been done in the 0.9.7 >branch >to address it. >We have an application that uses separate threads for its readers and >writers. Currently, the threads can stomp on each others state since >they are using the same SSL structure. In particular, the "rwstate" >flag >can get trashed and confuse one thread or the other about what is >happening. I know that OpenSSL does not support this in the 0.9.6 >branch, but was anything done to address it in 0.9.7 or is anything >planned in 0.9.8? No. Most libraries are thread safe in the sense that they work fine so long as two threads don't try to modify the same structure at the same time. Trying to change this generally makes things worse. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Re: Thread Question
On Mon, 17 Mar 2003 11:26:46 -0700, Verdon Walker wrote: >I know from looking in the archives that this question has been >asked >before, but I am wondering if anything has been done in the 0.9.7 >branch >to address it. >We have an application that uses separate threads for its readers and >writers. Currently, the threads can stomp on each others state since >they are using the same SSL structure. In particular, the "rwstate" >flag >can get trashed and confuse one thread or the other about what is >happening. I know that OpenSSL does not support this in the 0.9.6 >branch, but was anything done to address it in 0.9.7 or is anything >planned in 0.9.8? No. Most libraries are thread safe in the sense that they work fine so long as two threads don't try to modify the same structure at the same time. Trying to change this generally makes things worse. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
