RE: The smallest minimal example of an HTTPS GET request with openssl

[Porter, Andrew]

You would have to ignore some initial certificate info output but some 
variation on

echo "GET /version" | openssl s_client -connect test.example.com -crlf -ign_eof 
-quiet

would be as pure OpenSSL as you can get…

Andrew

From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf Of 
Ivan Medoedov
Sent: Saturday, March 30, 2019 08:03
To: openssl-users@openssl.org
Subject: [EXTERNAL] The smallest minimal example of an HTTPS GET request with 
openssl

Hello,

I've only managed to find this, but it seems to do too much for what I need:

https://wiki.openssl.org/index.php/SSL/TLS_Client

Basically I need something like Go's 
http.Get("https://mysite.com/version")
 to just get a one line of text from the server.

I can't use libcurl, I have to use pure openssl.

Thanks

Re: The smallest minimal example of an HTTPS GET request with openssl

[Viktor Dukhovni]

> On Mar 30, 2019, at 4:28 PM, Ivan Medoedov  wrote:
> 
> Thanks, Viktor.

You're welcome.  One important note about the example on the Wiki.
Since OpenSSL 1.0.2, there is internal support for certificate
name checks.  You should not roll your own.

The SSL_set1_host(3) interface is present since OpenSSL 1.1.0.
In OpenSSL 1.0.2 you can use SSL_CTX_get0_param(3) and
X509_VERIFY_PARAM_set1_host(3):

X509_VERIFY_PARAM *vpm = SSL_get0_param(ssl);
X509_VERIFY_PARAM_set1_host(vpm, "www.foo.com", 0);

Either of the above needs to happen before the handshake starts and
then the checks are made automatically as part of the handshake,
resulting in a certificate verification failure if the name checks
fail.

Alternatively, you can call  X509_check_host(3) after the handshake
completes.  This might also then need to happen after session resumption,
because the cached certificate validity would only cover the trust path,
and not the name checks.  But if you never resume sessions that failed
name checks previously, and never re-use sessions across different
host names (for the same IP e.g.) then you might be safe without, some
care is recommended.

-- 
Viktor.

Re: The smallest minimal example of an HTTPS GET request with openssl

[Ivan Medoedov]

Thanks, Viktor.

On Sat, Mar 30, 2019 at 8:59 PM Viktor Dukhovni 
wrote:

> On Sat, Mar 30, 2019 at 08:09:01PM +0100, Ivan Medoedov wrote:
>
> > You are right of course. I handle HTTP myself. A TLS connection example
> > will suffice.
> >
> > > > https://wiki.openssl.org/index.php/SSL/TLS_Client
>
> That example can be simplified, but OpenSSL does not presently
> provide a built-in high-level function to make a verified connection
> to a given host:port.
>
> With library initialization now implicit in 1.1.0 and later, and
> thread-safety no longer requiring application callbacks, this would
> now be easier to provide, but:
>
> * SSL_CTX creation is moderately expensive, and one should
>   generally create and use the same SSL_CTX for multiple SSL
>   connecitons in any applications that uses more than one SSL
>   connection.  So typically, there would still be a one-time
>   initialization step to create the SSL_CTX (application context).
>
> * Some applications will want to do post-handshake I/O via the
>   socket API, and prefer to hand OpenSSL an already connected
>   socket on which to perform the TLS handshake.  Others may
>   want OpenSSL to establish the connection and may prefer to
>   use BIO interface to interact with the peer.
>
> * While many systems have a usable default "trust store",
>   some do not, or the default "trust store" is overly inclusive.
>   Applications should generally allow the user to specify the
>   set of trusted CAs.
>
> ...
>
> Through in enough similar qualifiers, and you end up with the rather
> complex example.  It could perhaps be refactored as (hypothetical
> interface):
>
> /*
>  * Context with default trust store, protocols, ciphers, ...
>  * and peer verification enabled.
>  */
> SSL_CTX *app_ctx = SSL_default_ssl_ctx(TLS_method());
>
> /*
>  * Prepare verified (with name checks) SSL handle for the
>  * requested host:port (with SNI).
>  */
> SSL *con = SSL_default_ssl(app_ctx, host, port);
>
> Then depending on whether you want to give OpenSSL an already
> connected socket, or have OpenSSL make the connection for you:
>
> int err = SSL_connect_socket(con, fd);
>
> or (something along the lines of):
>
> BIO *bp = SSL_connect_hostport(con, host, port, );
>
> At this point a stock error handler may be required, but if all
> went well, you now need an I/O loop.  And so would need to either
> use the connected file descriptor with your own buffering, ...
> or use OpenSSL BIOs, or some other buffering I/O layer that
> supports custom read/write wrappers.
>
> On FreeBSD/NetBSD it might be nice to have stock "fittings" to
> wrap stdio around OpenSSL connections via:
>
>  #include 
>
>  FILE *
>  funopen(const void *cookie,
>  int(*readfn)  (void *, char *, int),
>  int(*writefn) (void *, const char *, int),
>  fpos_t (*seekfn)  (void *, fpos_t, int),
>  int(*closefn) (void *));
>
> Providing higher level interfaces to the core TLS library would
> make a good project for someone good at API design, who is familiar
> with OpenSSL, but more interested in the API than the internals.
>
> --
> Viktor.
>

Re: The smallest minimal example of an HTTPS GET request with openssl

[Viktor Dukhovni]

On Sat, Mar 30, 2019 at 08:09:01PM +0100, Ivan Medoedov wrote:

> You are right of course. I handle HTTP myself. A TLS connection example
> will suffice.
> 
> > > https://wiki.openssl.org/index.php/SSL/TLS_Client

That example can be simplified, but OpenSSL does not presently
provide a built-in high-level function to make a verified connection
to a given host:port.

With library initialization now implicit in 1.1.0 and later, and
thread-safety no longer requiring application callbacks, this would
now be easier to provide, but:

* SSL_CTX creation is moderately expensive, and one should
  generally create and use the same SSL_CTX for multiple SSL
  connecitons in any applications that uses more than one SSL
  connection.  So typically, there would still be a one-time
  initialization step to create the SSL_CTX (application context).

* Some applications will want to do post-handshake I/O via the
  socket API, and prefer to hand OpenSSL an already connected
  socket on which to perform the TLS handshake.  Others may
  want OpenSSL to establish the connection and may prefer to
  use BIO interface to interact with the peer.

* While many systems have a usable default "trust store",
  some do not, or the default "trust store" is overly inclusive.
  Applications should generally allow the user to specify the
  set of trusted CAs.

...

Through in enough similar qualifiers, and you end up with the rather
complex example.  It could perhaps be refactored as (hypothetical
interface):

/*
 * Context with default trust store, protocols, ciphers, ...
 * and peer verification enabled.
 */
SSL_CTX *app_ctx = SSL_default_ssl_ctx(TLS_method());

/*
 * Prepare verified (with name checks) SSL handle for the
 * requested host:port (with SNI).
 */
SSL *con = SSL_default_ssl(app_ctx, host, port);

Then depending on whether you want to give OpenSSL an already
connected socket, or have OpenSSL make the connection for you:

int err = SSL_connect_socket(con, fd);

or (something along the lines of):

BIO *bp = SSL_connect_hostport(con, host, port, );

At this point a stock error handler may be required, but if all
went well, you now need an I/O loop.  And so would need to either
use the connected file descriptor with your own buffering, ...
or use OpenSSL BIOs, or some other buffering I/O layer that
supports custom read/write wrappers.

On FreeBSD/NetBSD it might be nice to have stock "fittings" to
wrap stdio around OpenSSL connections via:

 #include 

 FILE *
 funopen(const void *cookie,
 int(*readfn)  (void *, char *, int),
 int(*writefn) (void *, const char *, int),
 fpos_t (*seekfn)  (void *, fpos_t, int),
 int(*closefn) (void *));

Providing higher level interfaces to the core TLS library would
make a good project for someone good at API design, who is familiar
with OpenSSL, but more interested in the API than the internals.

-- 
Viktor.

Re: The smallest minimal example of an HTTPS GET request with openssl

[Ivan Medoedov]

You are right of course. I handle HTTP myself. A TLS connection example
will suffice.

On Sat, Mar 30, 2019 at 7:53 PM Viktor Dukhovni 
wrote:

> On Sat, Mar 30, 2019 at 04:02:55PM +0100, Ivan Medoedov wrote:
>
> > I've only managed to find this, but it seems to do too much for what I
> need:
> >
> > https://wiki.openssl.org/index.php/SSL/TLS_Client
> >
> > Basically I need something like Go's http.Get("
> https://mysite.com/version;)
> > to just get a one line of text from the server.
> >
> > I can't use libcurl, I have to use pure openssl.
>
> OpenSSL is NOT an HTTPS (client or server) library.  OpenSSL can
> establish a TLS connection, but does not implement anything like
> "http.Get".  That's what libcurl and the like are for.
>
> If you're willing to roll your own HTTP client code, then you
> can use OpenSSL for the TLS part of the HTTPS transaction.
>
> HTTP clients handle all sorts of complications over and above TLS:
>
> * Proxies
> * Redirects
> * Client requests with bodies
> * Authentication
> * Chunked transfer encoding
> * Content encoding (e.g. gzip)
> * Cookies
> ...
>
> None of these are in scope for OpenSSL.
>
> --
> Viktor.
>

Re: The smallest minimal example of an HTTPS GET request with openssl

[Viktor Dukhovni]

On Sat, Mar 30, 2019 at 04:02:55PM +0100, Ivan Medoedov wrote:

> I've only managed to find this, but it seems to do too much for what I need:
> 
> https://wiki.openssl.org/index.php/SSL/TLS_Client
> 
> Basically I need something like Go's http.Get("https://mysite.com/version;)
> to just get a one line of text from the server.
> 
> I can't use libcurl, I have to use pure openssl.

OpenSSL is NOT an HTTPS (client or server) library.  OpenSSL can
establish a TLS connection, but does not implement anything like
"http.Get".  That's what libcurl and the like are for.

If you're willing to roll your own HTTP client code, then you
can use OpenSSL for the TLS part of the HTTPS transaction.

HTTP clients handle all sorts of complications over and above TLS:

* Proxies
* Redirects
* Client requests with bodies
* Authentication
* Chunked transfer encoding
* Content encoding (e.g. gzip)
* Cookies
...

None of these are in scope for OpenSSL.

-- 
Viktor.