Re: How to determine when data is finished on an SSL socket

[Matthew Donald]

When in doubt, use the source - or in this case RFC2616
§4.4.

TL;DR version - the length is indicated by one of three situations:

1. A Content-Length header exists
2. Transfer-Encoding is "chunked", in which case each chunk (there can be
several) begins with a length in hex.  A zero length chunk terminates the
grouping and the trailer is terminated with a CRLF.
3. Otherwise when the connection is closed.

Read the RFC for all the gory details.



On 11 January 2014 23:45, M. V.  wrote:

> Hi Matthew,
>
> but from what I know, not all HTTP responses have "Content-Length" field.
> responses with "chunked" payload are one of them. what can I do then?
>
> Thank you.
>
>
>
>
>   On Saturday, January 11, 2014 1:23 PM, Matthew Donald <
> matthew.b.don...@gmail.com> wrote:
>
> Sockets (SSL or otherwise) are layer-3 objects while content is a layer-7
> object.  To get an accurate end-of-content marker, you'll need to parse the
> output, buts since you are using the https protocol, that's going to be
> easy.
>
> Essentially, parse out the "Content-Length" header and read that many
> bytes past the end of the http headers.
>
>
>
>
> On 11 January 2014 19:46, M. V.  wrote:
>
>  Hi everybody,
>
> I'm writing an application that creates multiple non-blocking SSL
> connections to an https server, in each one I send a request and read the
> server's response. my problem is, whatever I do, I can't determine when the
> response data is finished. here's part of my code responsible for sending
> and receiving data:
>
> 
> fd_set connectionfds;struct timeval timeout2;
> FD_ZERO(&connectionfds);
> FD_SET(socket_server, &connectionfds);
> timeout2.tv_usec = 0;
> timeout2.tv_sec = 1;while(1){
> r=BIO_read(io,buf,BUFSIZZ-1);
> if (r>0){
> //gather data
> continue;
> }
> else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
> int ret = select(socket_server + 1, &connectionfds, NULL, NULL, 
> &timeout2);
> if (ret <= 0){
> break;
> }
> continue;
> }
> else{
> break;
> }}// use whole gathered data
>
> my problem with above code is, if I set select timeout to a small time, I
> can't guarantee that all data is received (because some servers are really
> slow), and if I set timeout to a long time (5-10 seconds), my socket is
> getting stucked in wait-state for a long time and I can't use the response
> before that. I tried to make this work by using "BIO_should_read()" or
> "BIO_pending()" functions, but none of them is giving me what I want. so,
> Is there a way to determine when exactly there is nothing else to read on
> the SSL socket?
>
> Thank you.
>
>
>
>
>

Re: How to determine when data is finished on an SSL socket

[M. V.]

Hi Matthew,

but from what I know, not all HTTP responses have "Content-Length" field. 
responses with "chunked" payload are one of them. what can I do then?

Thank you.





On Saturday, January 11, 2014 1:23 PM, Matthew Donald 
 wrote:
 
Sockets (SSL or otherwise) are layer-3 objects while content is a layer-7 
object.  To get an accurate end-of-content marker, you'll need to parse the 
output, buts since you are using the https protocol, that's going to be easy.
>
>
>Essentially, parse out the "Content-Length" header and read that many bytes 
>past the end of the http headers.
>
>
>
>
>
>
>
>On 11 January 2014 19:46, M. V.  wrote:
>
>Hi everybody,
>>
>>
>>
>>I'm writing an application that creates multiple non-blocking SSL connections 
>>to an https server, in each one I send a request and 
read the server's response. my problem is, whatever I do, I can't 
determine when the response data is finished. here's part of my code 
responsible for sending and receiving data:
>>fd_set connectionfds;structtimeval 
>>timeout2;FD_ZERO(&connectionfds);FD_SET(socket_server,&connectionfds);timeout2.tv_usec
>> =0;timeout2.tv_sec =1;while(1){r=BIO_read(io,buf,BUFSIZZ-1);if(r>0){//gather 
>>datacontinue;}elseif(SSL_get_error(ssl,r)==SSL_ERROR_WANT_READ){intret 
>>=select(socket_server +1,&connectionfds,NULL,NULL,&timeout2);if(ret 
>><=0){break;}continue;}else{break;}}// use whole gathered data
>>my problem with above code is, if I set select timeout to a small 
time, I can't guarantee that all data is received (because some servers 
are really slow), and if I set timeout to a long time (5-10 seconds), my socket 
is getting stucked in wait-state for a long time and I can't use the response 
before that.
I tried to make this work by using "BIO_should_read()" or 
"BIO_pending()" functions, but none of them is giving me what I want. 
so, Is there a way to determine when exactly there is nothing else to 
read on the SSL socket?
>>
>>
>>Thank you.
>>
>>
>>
>
>
>

Re: How to determine when data is finished on an SSL socket

[chris . gray]

Matthew Donald scripsit:

> Sockets (SSL or otherwise) are layer-3 objects while content is a
layer-7
> object.  To get an accurate end-of-content marker, you'll need to parse the
> output, buts since you are using the https protocol, that's going to be
easy.
> Essentially, parse out the "Content-Length" header and read that many bytes
> past the end of the http headers.

True, except that in various circumstances the "Content-Length" header may
be absent (e.g. special Transfer-Encoding, or a chunked response).  For
the full story see
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4

Regards

Chris Gray

> On 11 January 2014 19:46, M. V.  wrote:
>> Hi everybody,
>> I'm writing an application that creates multiple non-blocking SSL
connections to an https server, in each one I send a request and read
the
>> server's response. my problem is, whatever I do, I can't determine when
the
>> response data is finished. here's part of my code responsible for sending
>> and receiving data:
>> 
>> fd_set connectionfds;struct timeval timeout2;
>> FD_ZERO(&connectionfds);
>> FD_SET(socket_server, &connectionfds);
>> timeout2.tv_usec = 0;
>> timeout2.tv_sec = 1;while(1){
>> r=BIO_read(io,buf,BUFSIZZ-1);
>> if (r>0){
>> //gather data
>> continue;
>> }
>> else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
>> int ret = select(socket_server + 1, &connectionfds, NULL, NULL,
>> &timeout2);
>> if (ret <= 0){
>> break;
>> }
>> continue;
>> }
>> else{
>> break;
>> }}// use whole gathered data
>> my problem with above code is, if I set select timeout to a small time, I
>> can't guarantee that all data is received (because some servers are really
>> slow), and if I set timeout to a long time (5-10 seconds), my socket is
getting stucked in wait-state for a long time and I can't use the
response
>> before that. I tried to make this work by using "BIO_should_read()" or
"BIO_pending()" functions, but none of them is giving me what I want.
so,
>> Is there a way to determine when exactly there is nothing else to read on
>> the SSL socket?
>> Thank you.




__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager   majord...@openssl.org

Re: How to determine when data is finished on an SSL socket

[Matthew Donald]

Sockets (SSL or otherwise) are layer-3 objects while content is a layer-7
object.  To get an accurate end-of-content marker, you'll need to parse the
output, buts since you are using the https protocol, that's going to be
easy.

Essentially, parse out the "Content-Length" header and read that many bytes
past the end of the http headers.




On 11 January 2014 19:46, M. V.  wrote:

> Hi everybody,
>
> I'm writing an application that creates multiple non-blocking SSL
> connections to an https server, in each one I send a request and read the
> server's response. my problem is, whatever I do, I can't determine when the
> response data is finished. here's part of my code responsible for sending
> and receiving data:
>
> 
> fd_set connectionfds;struct timeval timeout2;
> FD_ZERO(&connectionfds);
> FD_SET(socket_server, &connectionfds);
> timeout2.tv_usec = 0;
> timeout2.tv_sec = 1;while(1){
> r=BIO_read(io,buf,BUFSIZZ-1);
> if (r>0){
> //gather data
> continue;
> }
> else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
> int ret = select(socket_server + 1, &connectionfds, NULL, NULL, 
> &timeout2);
> if (ret <= 0){
> break;
> }
> continue;
> }
> else{
> break;
> }}// use whole gathered data
>
> my problem with above code is, if I set select timeout to a small time, I
> can't guarantee that all data is received (because some servers are really
> slow), and if I set timeout to a long time (5-10 seconds), my socket is
> getting stucked in wait-state for a long time and I can't use the response
> before that. I tried to make this work by using "BIO_should_read()" or
> "BIO_pending()" functions, but none of them is giving me what I want. so,
> Is there a way to determine when exactly there is nothing else to read on
> the SSL socket?
>
> Thank you.
>
>

How to determine when data is finished on an SSL socket

[M. V.]

Hi everybody,


I'm writing an application that creates multiple non-blocking SSL connections 
to an https server, in each one I send a request and 
read the server's response. my problem is, whatever I do, I can't 
determine when the response data is finished. here's part of my code 
responsible for sending and receiving data:
fd_set connectionfds;structtimeval 
timeout2;FD_ZERO(&connectionfds);FD_SET(socket_server,&connectionfds);timeout2.tv_usec
 =0;timeout2.tv_sec =1;while(1){r=BIO_read(io,buf,BUFSIZZ-1);if(r>0){//gather 
datacontinue;}elseif(SSL_get_error(ssl,r)==SSL_ERROR_WANT_READ){intret 
=select(socket_server +1,&connectionfds,NULL,NULL,&timeout2);if(ret 
<=0){break;}continue;}else{break;}}// use whole gathered data
my problem with above code is, if I set select timeout to a small 
time, I can't guarantee that all data is received (because some servers 
are really slow), and if I set timeout to a long time (5-10 seconds), my socket 
is getting stucked in wait-state for a long time and I can't use the response 
before that.
I tried to make this work by using "BIO_should_read()" or 
"BIO_pending()" functions, but none of them is giving me what I want. 
so, Is there a way to determine when exactly there is nothing else to 
read on the SSL socket?

Thank you.