Control: usertag -1 bsp-2019-04-se-gothenburg Hi there,
strace(1) shows a select(2) syscall indicating that the socket is ready for both read and write, but is later blocking on a read(2) without any write(2) taking place. select(8, [3], [3], NULL, {tv_sec=180, tv_usec=0}) = 2 (in [3], out [3], left {tv_sec=179, tv_usec=999998}) read(3, "…", 5) = 5 read(3, "…", 156) = 156 read(3, Net::SSLeay warns: If you need to select(2) on the socket, go right ahead, but be warned that OpenSSL does some internal buffering so SSL_read does not always return data even if the socket selected for reading (just keep on selecting and trying to read). "Net::SSLeay" is no different from the C language OpenSSL in this respect. And indeed LWP::Protocol::http's use of select(2) on SSL sockets *does* assume that read/write readiness won't block. (If Net::SSLeay::read() returns -1, then the loop will retry later with SSL_ERROR_WANT_READ/WRITE.) However since OpenSSL 1.1.1 the SSL_MODE_AUTO_RETRY flag is on by default, which breaks that assumption: ssl_read(3) might block, even when select(2) claimed the socket had data to be read. SSL_MODE_AUTO_RETRY During normal operations, non-application data records might need to be sent or received that the application is not aware of. If a non-application data record was processed, SSL_read_ex(3) and SSL_read(3) can return with a failure and indicate the need to retry with SSL_ERROR_WANT_READ. If such a non-application data record was processed, the flag SSL_MODE_AUTO_RETRY causes it to try to process the next record instead of returning. […] In a blocking environment, applications are not always prepared to deal with the functions returning intermediate reports such as retry requests, and setting the SSL_MODE_AUTO_RETRY flag will cause the functions to only return after successfully processing an application data record or a failure. […] All modes are off by default except for SSL_MODE_AUTO_RETRY which is on by default since 1.1.1. — https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_set_mode.html See also https://github.com/openssl/openssl/issues/6234 . I see several paths forward here: - Refactor LWP::Protocol::http's select loop to solve the assumption that's now broken with OpenSSL ≥1.1.1; or - Unset SSL_MODE_AUTO_RETRY in IO::Socket::SSL; or - Make context flags configurable in IO::Socket::SSL, and unset SSL_MODE_AUTO_RETRY from LWP. IMHO the first option is not ideal so late in the release cycle. The second option is the easiest to implement, and should™ be regression-free, but might confuse people who became used to OpenSSL ≥1.1.1's new context default flags. SSL_CTX_clear_mode(3) and SSL_CTRL_CLEAR_MODE macros are unfortunately not exposed to Net::SSLeay 1.85-2. The proper fix would be to expose these and release a new version of Net::SSLeay, of course, but for tests the macros can be taken from /usr/include/openssl/ssl.h: # define SSL_CTRL_CLEAR_MODE 78 […] # define SSL_CTX_clear_mode(ctx,op) \ SSL_CTX_ctrl((ctx),SSL_CTRL_CLEAR_MODE,(op),NULL) and used as is in IO::Socket::SSL.pm. With the following patch I'm again able to POST to HTTPS servers using TLS 1.3. --8<--------------------------------------------------------------->8-- --- a/IO/Socket/SSL.pm +++ b/IO/Socket/SSL.pm @@ -2433,6 +2433,7 @@ # cannot guarantee, that the location of the buffer stays constant Net::SSLeay::CTX_set_mode( $ctx, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER|SSL_MODE_ENABLE_PARTIAL_WRITE); + Net::SSLeay::CTX_ctrl($ctx, 78, Net::SSLeay::MODE_AUTO_RETRY(), undef); if ( my $proto_list = $arg_hash->{SSL_npn_protocols} ) { return IO::Socket::SSL->_internal_error("NPN not supported in Net::SSLeay",9) --8<--------------------------------------------------------------->8-- (Again, I'm not proposing to patch IO::Socket::SSL as above :-) With MODE_AUTO_RETRY set — the default for OpenSSL ≥1.1.1 — one gets: $ strace -e trace=read,write,select perl -MLWP::UserAgent -e 'LWP::UserAgent->new(ssl_opts => {SSL_version => "TLSv1_3"})->post("https://facebook.com", { data => "plonc" })'; […] select(8, [3], [3], NULL, {tv_sec=180, tv_usec=0}) = 2 (in [3], out [3], left {tv_sec=179, tv_usec=999998}) read(3, "…", 5) = 5 read(3, "…", 156) = 156 read(3, And now with the MODE_AUTO_RETRY flag unset: $ select(8, [3], [3], NULL, {tv_sec=180, tv_usec=0}) = 2 (in [3], out [3], left {tv_sec=179, tv_usec=999998}) read(3, "…", 5) = 5 read(3, "…", 156) = 156 write(3, "…", 217) = 217 select(8, [3], NULL, NULL, {tv_sec=180, tv_usec=0}) = 1 (in [3], left {tv_sec=179, tv_usec=870931}) read(3, "…", 5) = 5 read(3, "…", 361) = 361 Cheers, -- Guilhem.
signature.asc
Description: PGP signature