Re: Getting started with ktls
Nope. As far as I know KTLS does not need wireguard. The NFS stuff deleted was just ancient cruft left over by the old NFS code (pre-FreeBSD7ish). rick From: owner-freebsd-curr...@freebsd.org on behalf of tech-lists Sent: Thursday, March 25, 2021 10:52 AM To: freebsd-current@freebsd.org Subject: Re: Getting started with ktls On Fri, Mar 19, 2021 at 09:37:30PM +, Rick Macklem wrote: >J. wrote: >>on the (main/14) server, /etc/rpc.tlsservd was not already there; I had >>to create it. Is this correct? >> >>version is main-n245454 >I'll admit I have no idea what n245454 means, but the daemons were >committed to main on Feb 18, 2021. >Installing them from ports should be fine. Hi Rick, Does the recent reloval of wireguard have any bearing on ktls? The reason I ask is that when I happened to see stable/13 updating, it removed the if_wg stuff and also some nfs stuff. thanks, -- J. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Fri, Mar 19, 2021 at 09:37:30PM +, Rick Macklem wrote: J. wrote: on the (main/14) server, /etc/rpc.tlsservd was not already there; I had to create it. Is this correct? version is main-n245454 I'll admit I have no idea what n245454 means, but the daemons were committed to main on Feb 18, 2021. Installing them from ports should be fine. Hi Rick, Does the recent reloval of wireguard have any bearing on ktls? The reason I ask is that when I happened to see stable/13 updating, it removed the if_wg stuff and also some nfs stuff. thanks, -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
On 3/18/21 8:31 AM, tech-lists wrote: On Wed, Mar 17, 2021 at 08:39:02PM +, Rick Macklem wrote: Make sure you've done the following: ktls_ocf - is loaded these sysctls are set to 1 kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs [on stable/13] % sysctl kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs kern.ipc.tls.enable: 1 kern.ipc.mb_use_ext_pgs: 1 % kldstat | grep ktls 71 0x0135300025520 ktls_ocf.ko % % sysctl -a | fgrep kern.ipc.tls.stats kern.ipc.tls.stats.ocf.retries: 0 kern.ipc.tls.stats.ocf.separate_output: 0 kern.ipc.tls.stats.ocf.inplace: 0 kern.ipc.tls.stats.ocf.tls13_gcm_crypts: 0 kern.ipc.tls.stats.ocf.tls12_gcm_crypts: 0 kern.ipc.tls.stats.ocf.tls11_cbc_crypts: 0 kern.ipc.tls.stats.ocf.tls10_cbc_crypts: 0 kern.ipc.tls.stats.switch_failed: 0 kern.ipc.tls.stats.switch_to_sw: 0 kern.ipc.tls.stats.switch_to_ifnet: 0 kern.ipc.tls.stats.failed_crypto: 0 kern.ipc.tls.stats.corrupted_records: 0 kern.ipc.tls.stats.active: 0 kern.ipc.tls.stats.enable_calls: 535 kern.ipc.tls.stats.offload_total: 0 kern.ipc.tls.stats.sw_rx_inqueue: 0 kern.ipc.tls.stats.sw_tx_inqueue: 0 kern.ipc.tls.stats.threads: 4 % FYI, you can do this a bit more efficiently with just 'sysctl kern.ipc.tls.stats' The 'enable_calls' means that OpenSSL is trying to offload connections, but those attempts are all failing (offload_total is a count of how many of those setsockopt() calls succeed). If you are familiar with dtrace, you can use some DTrace probes to see why 'ktls_enable_tx' and 'ktls_enable_rx' are failing, or barring that printf. For example, does ktls_create_session() fail, or does ktls_try_sw() fail? It is probably easiest to debug this using a userland application using openssl than trying NFS over TLS. -- John Baldwin ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
J. wrote: >on the (main/14) server, /etc/rpc.tlsservd was not already there; I had >to create it. Is this correct? > >version is main-n245454 I'll admit I have no idea what n245454 means, but the daemons were committed to main on Feb 18, 2021. Installing them from ports should be fine. rick -- J. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Sun, Mar 14, 2021 at 11:07:23PM +, tech-lists wrote: On Sun, Mar 14, 2021 at 08:55:18PM +, Rick Macklem wrote: If you want to try NFS-over-TLS, see this: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt Please let us know if you try it, rick Hi, on the (main/14) server, /etc/rpc.tlsservd was not already there; I had to create it. Is this correct? version is main-n245454 -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
On Wed, Mar 17, 2021 at 08:39:02PM +, Rick Macklem wrote: Make sure you've done the following: ktls_ocf - is loaded these sysctls are set to 1 kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs [on stable/13] % sysctl kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs kern.ipc.tls.enable: 1 kern.ipc.mb_use_ext_pgs: 1 % kldstat | grep ktls 71 0x0135300025520 ktls_ocf.ko % % sysctl -a | fgrep kern.ipc.tls.stats kern.ipc.tls.stats.ocf.retries: 0 kern.ipc.tls.stats.ocf.separate_output: 0 kern.ipc.tls.stats.ocf.inplace: 0 kern.ipc.tls.stats.ocf.tls13_gcm_crypts: 0 kern.ipc.tls.stats.ocf.tls12_gcm_crypts: 0 kern.ipc.tls.stats.ocf.tls11_cbc_crypts: 0 kern.ipc.tls.stats.ocf.tls10_cbc_crypts: 0 kern.ipc.tls.stats.switch_failed: 0 kern.ipc.tls.stats.switch_to_sw: 0 kern.ipc.tls.stats.switch_to_ifnet: 0 kern.ipc.tls.stats.failed_crypto: 0 kern.ipc.tls.stats.corrupted_records: 0 kern.ipc.tls.stats.active: 0 kern.ipc.tls.stats.enable_calls: 535 kern.ipc.tls.stats.offload_total: 0 kern.ipc.tls.stats.sw_rx_inqueue: 0 kern.ipc.tls.stats.sw_tx_inqueue: 0 kern.ipc.tls.stats.threads: 4 % now to try setting up the nfs thing -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
J. wrote: >On Tue, Mar 16, 2021 at 11:46:27PM +, Rick Macklem wrote: >>Well, if you do "sysctl -a | fgrep kern.ipc.tls.stats" and it is working, >>you should see the count for at least one of the "crypts" ticking up. >>If they are all zero, it isn't working. That might depend on the apps >>or setup and does not necessarily indicate broken. > >OK. it's "not working" by those criteria on the stable/13 rpi4. >This one has mutt (imaps) and lynx (https) installed. mutt appears to >use tlsv1.3 to connect with my email provider. I know that the receive direction only works for TLS1.2. Not sure about the xmit direction? Make sure you've done the following: ktls_ocf - is loaded these sysctls are set to 1 kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs Beyond that, it will take someone more knowledgible to figure out if it can work for these apps? (To be honest, for userspace applications I'm not sure there is any advantage to using KTLS unless you have specialized hardware. rick >Trying the nfs-over-tls should definitely test it. When it works, the >data on the wire after the first couple of Null RPCs is encrypted. >Also, if you start the daemons with "-v", This is what i'll try once buildworld etc completes on the main/14 rpi4. -- J. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Tue, Mar 16, 2021 at 11:46:27PM +, Rick Macklem wrote: Well, if you do "sysctl -a | fgrep kern.ipc.tls.stats" and it is working, you should see the count for at least one of the "crypts" ticking up. If they are all zero, it isn't working. That might depend on the apps or setup and does not necessarily indicate broken. OK. it's "not working" by those criteria on the stable/13 rpi4. This one has mutt (imaps) and lynx (https) installed. mutt appears to use tlsv1.3 to connect with my email provider. Trying the nfs-over-tls should definitely test it. When it works, the data on the wire after the first couple of Null RPCs is encrypted. Also, if you start the daemons with "-v", This is what i'll try once buildworld etc completes on the main/14 rpi4. -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
On Sun, Mar 14, 2021 at 08:55:18PM +, Rick Macklem wrote: If you want to try NFS-over-TLS, see this: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt Please let us know if you try it, rick Hi, I'm going to try this with 2x rpi4 machines, client on stable/13 and server on main/14. both have zfs. -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
J. wrote: >On Sun, Mar 14, 2021 at 08:55:18PM +, Rick Macklem wrote: >>Alan explains how to set it up, below. >>However, I thought I'd note that maybe one person has tested KTLS >>on arm64, so you should consider doing this for test purposes only. >>If you do do some testing, please post with your results, >>success or failure. >> >>>It's present in current kernels for both 13 and 14, amd64 and aarch64. >>>However, it's not present in 13's openssl. To use it, you must either >>>rebuild world with WITH_OPENSSL_KTLS=YES in /etc/src.conf, > >>Doing it this way means that everything linked to OpenSSL will use >>it. Probably a better testsituation, but expect at least the apache >>server to break. (Most breakage was fixed by a recent patch to the >>serf library, but I think the apache server is still broken. > >OK, it's been built and all ports recompiled and reinstalled. Things >that use openssl on this machine are mutt (imaps) lynx (https) and >nginx (https) and py-certbot. They all seem to work. How would I test? Well, if you do "sysctl -a | fgrep kern.ipc.tls.stats" and it is working, you should see the count for at least one of the "crypts" ticking up. If they are all zero, it isn't working. That might depend on the apps or setup and does not necessarily indicate broken. Trying the nfs-over-tls should definitely test it. When it works, the data on the wire after the first couple of Null RPCs is encrypted. Also, if you start the daemons with "-v", then it will log how the handshake etc. goes in /var/log/daemon.log. rick thanks, -- J. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Sun, Mar 14, 2021 at 08:55:18PM +, Rick Macklem wrote: Alan explains how to set it up, below. However, I thought I'd note that maybe one person has tested KTLS on arm64, so you should consider doing this for test purposes only. If you do do some testing, please post with your results, success or failure. It's present in current kernels for both 13 and 14, amd64 and aarch64. However, it's not present in 13's openssl. To use it, you must either rebuild world with WITH_OPENSSL_KTLS=YES in /etc/src.conf, Doing it this way means that everything linked to OpenSSL will use it. Probably a better testsituation, but expect at least the apache server to break. (Most breakage was fixed by a recent patch to the serf library, but I think the apache server is still broken. OK, it's been built and all ports recompiled and reinstalled. Things that use openssl on this machine are mutt (imaps) lynx (https) and nginx (https) and py-certbot. They all seem to work. How would I test? thanks, -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
On Sun, Mar 14, 2021 at 11:07:23PM +, tech-lists wrote: On Sun, Mar 14, 2021 at 08:55:18PM +, Rick Macklem wrote: Alan explains how to set it up, below. However, I thought I'd note that maybe one person has tested KTLS on arm64, so you should consider doing this for test purposes only. If you do do some testing, please post with your results, success or failure. will do. The end-point I want is to have the arm64 machine accessible externally for wireguard-based vpn. ok, upgraded to stable/13-n244890-4e535a1ba72 with ktls. now, need to rebuild the poudriere instance and rebuild all installed ports. -- J. -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
On Sun, Mar 14, 2021 at 08:55:18PM +, Rick Macklem wrote: Alan explains how to set it up, below. However, I thought I'd note that maybe one person has tested KTLS on arm64, so you should consider doing this for test purposes only. If you do do some testing, please post with your results, success or failure. will do. The end-point I want is to have the arm64 machine accessible externally for wireguard-based vpn. Doing it this way means that everything linked to OpenSSL will use it. Probably a better testsituation, but expect at least the apache server to break. (Most breakage was fixed by a recent patch to the serf library, but I think the apache server is still broken. I have latest serf installed with nginx. The machine, a rpi4/8GB, builds its ports with poudriere. I have WITH_OPENSSL_KTLS= set in /etc/src.conf and it's building a new world now. If you want to try NFS-over-TLS, see this: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt Please let us know if you try it, rick yep, I'll try that too and post results here. thanks :D -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
[stuff snipped] > J. wrote: >> >> I'd like to have it (ktls) available on the ARM64 >> stable/13-n244876-0b45290603b. Is it just a matter of adding the option, >> and then the sysctls become available? Is it "better" with openssl[-devel] >> in ports or openssl in base? >> >> thanks, >> -- >> J.\ Alan explains how to set it up, below. However, I thought I'd note that maybe one person has tested KTLS on arm64, so you should consider doing this for test purposes only. If you do do some testing, please post with your results, success or failure. >It's present in current kernels for both 13 and 14, amd64 and aarch64. >However, it's not present in 13's openssl. To use it, you must either >rebuild world with WITH_OPENSSL_KTLS=YES in /etc/src.conf, Doing it this way means that everything linked to OpenSSL will use it. Probably a better testsituation, but expect at least the apache server to break. (Most breakage was fixed by a recent patch to the serf library, but I think the apache server is still broken. >(or) install >security/openssl-devel from pkg, or built security/openssl from ports with >the KTLS option enabled. I don't know if any version of openssl is >"better" than another. The sysctls should be available in any case. Only applications built using includes from /usr/local/include and linked to libraries in /usr/local/lib will use it for these cases. If you want to try NFS-over-TLS, see this: https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt Please let us know if you try it, rick -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Sun, Mar 14, 2021 at 09:54:33AM -0600, Alan Somers wrote: > On Sun, Mar 14, 2021 at 8:57 AM tech-lists wrote: > > > I'd like to have it (ktls) available on the ARM64 > > stable/13-n244876-0b45290603b. Is it just a matter of adding the option, > > and then the sysctls become available? Is it "better" with openssl[-devel] > > in ports or openssl in base? > > > > thanks, > > -- > > J.\ > > > It's present in current kernels for both 13 and 14, amd64 and aarch64. > However, it's not present in 13's openssl. To use it, you must either > rebuild world with WITH_OPENSSL_KTLS=YES in /etc/src.conf, install > security/openssl-devel from pkg, or built security/openssl from ports with > the KTLS option enabled. I don't know if any version of openssl is > "better" than another. The sysctls should be available in any case. security/openssl-devel is alpha-quality software; I don't recommend running it in production. -Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Sun, Mar 14, 2021 at 8:57 AM tech-lists wrote: > On Thu, Mar 11, 2021 at 03:42:55PM +, Rick Macklem wrote: > >I'm going to cheat and top post (the discussion looks > >pretty convoluted). > > > >- The kernel must be built with "options KERN_TLS" > >- OpenSSL must be built with KTLS enabled > >- These two sysctls need to be set to 1 > > kern.ipc.tls.enable > > kern.ipc.mb_use_ext_pgs > > Hello, > > I'd like to try ktls but have found the following: > > On AMD64 (stable/13) this option is present in the GENERIC kernel > of world built about a month ago: stable/13-n244496-618dee60231 > and openssl version is 1.1.1i-freebsd > > On ARM64 (stable/13) it's *not* present in a world built earlier > today from stable/13-n244876-0b45290603b. Here, the openssl version > is 1.1.1j-freebsd > > On another ARM64 (main/14) it *is* present in main-n245445-07564e17620 > built with sources from the 11th March. openssl is 1.1.1j-freebsd here > as well. > > I'd like to have it (ktls) available on the ARM64 > stable/13-n244876-0b45290603b. Is it just a matter of adding the option, > and then the sysctls become available? Is it "better" with openssl[-devel] > in ports or openssl in base? > > thanks, > -- > J.\ It's present in current kernels for both 13 and 14, amd64 and aarch64. However, it's not present in 13's openssl. To use it, you must either rebuild world with WITH_OPENSSL_KTLS=YES in /etc/src.conf, install security/openssl-devel from pkg, or built security/openssl from ports with the KTLS option enabled. I don't know if any version of openssl is "better" than another. The sysctls should be available in any case. -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Thu, Mar 11, 2021 at 03:42:55PM +, Rick Macklem wrote: I'm going to cheat and top post (the discussion looks pretty convoluted). - The kernel must be built with "options KERN_TLS" - OpenSSL must be built with KTLS enabled - These two sysctls need to be set to 1 kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs Hello, I'd like to try ktls but have found the following: On AMD64 (stable/13) this option is present in the GENERIC kernel of world built about a month ago: stable/13-n244496-618dee60231 and openssl version is 1.1.1i-freebsd On ARM64 (stable/13) it's *not* present in a world built earlier today from stable/13-n244876-0b45290603b. Here, the openssl version is 1.1.1j-freebsd On another ARM64 (main/14) it *is* present in main-n245445-07564e17620 built with sources from the 11th March. openssl is 1.1.1j-freebsd here as well. I'd like to have it (ktls) available on the ARM64 stable/13-n244876-0b45290603b. Is it just a matter of adding the option, and then the sysctls become available? Is it "better" with openssl[-devel] in ports or openssl in base? thanks, -- J. signature.asc Description: PGP signature
Re: Getting started with ktls
On Thu, Mar 11, 2021 at 11:49 AM John Baldwin wrote: > On 3/10/21 4:18 PM, Alan Somers wrote: > > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > > the CPU utilization of applications like zrepl. But I have a few > questions: > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > > Libraries" section says "Applications using a supported library should > > generally work with ktls without any changes". These sentences seem to > be > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > necessary, but OpenSSL sets it automatically? > > Yes, you can do it by hand if you want but you'd have to do all the key > exchange by hand as well. > > > * When using OpenSSL, the library will automatically call setsockopt(_, > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > application to tell if ktls is enabled on a particular socket or OpenSSL > > session? > > BIO_get_ktls_send() and BIO_get_ktls_recv() on the write and read BIO's of > the connection, respectively. > > > * From experiment, I can see that OpenSSL attempts to set > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > > From reading ktls_start and ossl_statem_server_post_work, it looks like > > maybe a single socket cannot have ktls enabled for both sending and > > receiving at the same time. Is that true? > > Neither FreeBSD nor OpenSSL yet support RX offload on TLS 1.3. If you use > TLS 1.2 you will get KTLS in both directions (or if you use TLS 1.1 with > TOE offload on a Chelsio T6). > > -- > John Baldwin > Switching to TLS 1.2 turned out to be key. Once I did that, ... it just worked. I was expecting to need minor changes throughout the kernel and libzfs. However, that wasn't necessary. Here is my proof-of-concept program. So far only the recv path is implemented. I'll probably implement the send path too, but I'm not currently planning to integrate this into any open-source application. Thanks for all the help! https://github.com/asomers/freebsd-src/tree/ktls-zfs/tools/zfs-ktls -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On 3/10/21 4:18 PM, Alan Somers wrote: I'm trying to make ktls work with "zfs send/recv" to substantially reduce the CPU utilization of applications like zrepl. But I have a few questions: * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported Libraries" section says "Applications using a supported library should generally work with ktls without any changes". These sentences seem to be contradictory. I think it means that the TCP_TXTLS_ENABLE option is necessary, but OpenSSL sets it automatically? Yes, you can do it by hand if you want but you'd have to do all the key exchange by hand as well. * When using OpenSSL, the library will automatically call setsockopt(_, TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an application to tell if ktls is enabled on a particular socket or OpenSSL session? BIO_get_ktls_send() and BIO_get_ktls_recv() on the write and read BIO's of the connection, respectively. * From experiment, I can see that OpenSSL attempts to set TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? From reading ktls_start and ossl_statem_server_post_work, it looks like maybe a single socket cannot have ktls enabled for both sending and receiving at the same time. Is that true? Neither FreeBSD nor OpenSSL yet support RX offload on TLS 1.3. If you use TLS 1.2 you will get KTLS in both directions (or if you use TLS 1.1 with TOE offload on a Chelsio T6). -- John Baldwin ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
I'm going to cheat and top post (the discussion looks pretty convoluted). - The kernel must be built with "options KERN_TLS" - OpenSSL must be built with KTLS enabled - These two sysctls need to be set to 1 kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs then it all happens "behind the curtain" in the OpenSSL libraries. To find out if it is enabled, do the following after the handshake. (SSL_connect() or SSL_accept() or ??) ret = BIO_get_ktls_send(SSL_get_wbio(ssl)); if (ret != 0) ret = BIO_get_ktls_recv(SSL_get_rbio(ssl)); if (ret != 0) /* KTLS enabled for both send and recv. */ The calls return non-zero if it is enabled for that direction. You'll need it to use TLS 1.2 if you want both directions to work at this time. (The code is in usr.sbin/rpc.tlsclntd and usr.sbin/rpc.tlsservd. Much easier to look at than man pages imho.) --> Now you can sosend()/soreceive() on the socket in the kernel. If your data is already in mbufs, then they must be M_EXTPG mbufs. There is a function that copies an mbuf chain into M_EXTPG mbufs, but I can't remember what I called it. rick From: owner-freebsd-curr...@freebsd.org on behalf of Alan Somers Sent: Wednesday, March 10, 2021 10:55 PM To: Benjamin Kaduk Cc: FreeBSD CURRENT Subject: Re: Getting started with ktls CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to ith...@uoguelph.ca On Wed, Mar 10, 2021 at 8:15 PM Benjamin Kaduk wrote: > On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote: > > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > > > > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > > > I'm trying to make ktls work with "zfs send/recv" to substantially > reduce > > > > the CPU utilization of applications like zrepl. But I have a few > > > questions: > > > > > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by > a > > > > successful set of the TCP_TXTLS_ENABLE socket option", but the > "Supported > > > > Libraries" section says "Applications using a supported library > should > > > > generally work with ktls without any changes". These sentences seem > to > > > be > > > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > > > necessary, but OpenSSL sets it automatically? > > > > > > Yes, OpenSSL sets it automatically for the builtin socket and > connection > > > BIO classes. Applications using other BIO classes will need to do > things > > > manually (or implement the appropriate _ctrl() parameters for their BIO > > > class). > > > > > > > * When using OpenSSL, the library will automatically call > setsockopt(_, > > > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > > > application to tell if ktls is enabled on a particular socket or > OpenSSL > > > > session? > > > > > > IIRC the lack of answer for this is part of why upstream OpenSSL > doesn't > > > have specific KTLS tests enabled in the automated test suite. > > > > > > > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason > > why it's not implemented? That might be the easiest way to check for the > > ktls status of an individual socket. > > I think that's probably more of a question for jhb than me. I don't know > of a reason why not, but I do know that there is some desire to keep the > functionality that openssl exposes roughly compatible between linux and > FreeBSD KTLS. I don't know whether Linux has something similar. > > > > > > > > > > * From experiment, I can see that OpenSSL attempts to set > > > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why > not? > > > > From reading ktls_start and ossl_statem_server_post_work, it looks > like > > > > maybe a single socket cannot have ktls enabled for both sending and > > > > receiving at the same time. Is that true? > > > > > > No. They just get enabled separately, since change_cipher_state() is > > > called separately for read and write transitions. > > > > > > > Apologies if I'm too ignorant, but what is a transition in SSL-speak? > This > > is my first attempt at any kind of SSL programming. What I know from > > ktrace is tha
Re: Getting started with ktls
On Wed, Mar 10, 2021 at 8:15 PM Benjamin Kaduk wrote: > On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote: > > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > > > > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > > > I'm trying to make ktls work with "zfs send/recv" to substantially > reduce > > > > the CPU utilization of applications like zrepl. But I have a few > > > questions: > > > > > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by > a > > > > successful set of the TCP_TXTLS_ENABLE socket option", but the > "Supported > > > > Libraries" section says "Applications using a supported library > should > > > > generally work with ktls without any changes". These sentences seem > to > > > be > > > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > > > necessary, but OpenSSL sets it automatically? > > > > > > Yes, OpenSSL sets it automatically for the builtin socket and > connection > > > BIO classes. Applications using other BIO classes will need to do > things > > > manually (or implement the appropriate _ctrl() parameters for their BIO > > > class). > > > > > > > * When using OpenSSL, the library will automatically call > setsockopt(_, > > > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > > > application to tell if ktls is enabled on a particular socket or > OpenSSL > > > > session? > > > > > > IIRC the lack of answer for this is part of why upstream OpenSSL > doesn't > > > have specific KTLS tests enabled in the automated test suite. > > > > > > > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason > > why it's not implemented? That might be the easiest way to check for the > > ktls status of an individual socket. > > I think that's probably more of a question for jhb than me. I don't know > of a reason why not, but I do know that there is some desire to keep the > functionality that openssl exposes roughly compatible between linux and > FreeBSD KTLS. I don't know whether Linux has something similar. > > > > > > > > > > * From experiment, I can see that OpenSSL attempts to set > > > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why > not? > > > > From reading ktls_start and ossl_statem_server_post_work, it looks > like > > > > maybe a single socket cannot have ktls enabled for both sending and > > > > receiving at the same time. Is that true? > > > > > > No. They just get enabled separately, since change_cipher_state() is > > > called separately for read and write transitions. > > > > > > > Apologies if I'm too ignorant, but what is a transition in SSL-speak? > This > > is my first attempt at any kind of SSL programming. What I know from > > ktrace is that TCP_RXTLS_ENABLE never gets set. > > Sorry! I'm pretty conversant with this stuff (I'm the security area > director that is responsible for the IETF TLS working group) and don't > always target the right level. Basically, for a decent encrypting protocol > you want different encrytion keys for the read and write direction > (whichever peer you are), and the TLS (1.3) handshake has a multi-stage key > hierarchy to try to encrypt as much of it as possible. So, for example, > the client will need to update it's encryption key for reading once it > reads the ServerHello (and before reading the Encrypted Extensions) > message, even though the keys the client uses for writing don't change at > that time. Internally, OpenSSL implements this "transition" of key > material with a change_cipher_state() abstraction, that takes a flags > argument (`which`). The flags indicate which set of keys to update, and > which direction (read or write). So, by my read of the code, what's > *supposed* to happen is that we call: > > if (BIO_set_ktls(bio, _info, which & SSL3_CC_WRITE)) > > And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with > an `is_txt` value that evaluates to true; otherwise, `is_txt` is false, > which corresponds to the RX case that you're failing to see happen. > > Just to get the boring stuff out of the way: what version of openssl are > you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not > defined when ktls_start() is being compiled (so that the setsockopt(fd, > IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)? > > Thanks, > > Ben > I'm using the OpenSSL that's in base in 14.0-CURRENT: 1.1.1j-freebsd . I haven't recompiled the code to check whether OPENSSL_NO_KTLS_RX is defined, but it sure looks like it shouldn't be, based on my reading of the source. -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote: > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > > > the CPU utilization of applications like zrepl. But I have a few > > questions: > > > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > > > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > > > Libraries" section says "Applications using a supported library should > > > generally work with ktls without any changes". These sentences seem to > > be > > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > > necessary, but OpenSSL sets it automatically? > > > > Yes, OpenSSL sets it automatically for the builtin socket and connection > > BIO classes. Applications using other BIO classes will need to do things > > manually (or implement the appropriate _ctrl() parameters for their BIO > > class). > > > > > * When using OpenSSL, the library will automatically call setsockopt(_, > > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > > application to tell if ktls is enabled on a particular socket or OpenSSL > > > session? > > > > IIRC the lack of answer for this is part of why upstream OpenSSL doesn't > > have specific KTLS tests enabled in the automated test suite. > > > > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason > why it's not implemented? That might be the easiest way to check for the > ktls status of an individual socket. I think that's probably more of a question for jhb than me. I don't know of a reason why not, but I do know that there is some desire to keep the functionality that openssl exposes roughly compatible between linux and FreeBSD KTLS. I don't know whether Linux has something similar. > > > > > > * From experiment, I can see that OpenSSL attempts to set > > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > > > From reading ktls_start and ossl_statem_server_post_work, it looks like > > > maybe a single socket cannot have ktls enabled for both sending and > > > receiving at the same time. Is that true? > > > > No. They just get enabled separately, since change_cipher_state() is > > called separately for read and write transitions. > > > > Apologies if I'm too ignorant, but what is a transition in SSL-speak? This > is my first attempt at any kind of SSL programming. What I know from > ktrace is that TCP_RXTLS_ENABLE never gets set. Sorry! I'm pretty conversant with this stuff (I'm the security area director that is responsible for the IETF TLS working group) and don't always target the right level. Basically, for a decent encrypting protocol you want different encrytion keys for the read and write direction (whichever peer you are), and the TLS (1.3) handshake has a multi-stage key hierarchy to try to encrypt as much of it as possible. So, for example, the client will need to update it's encryption key for reading once it reads the ServerHello (and before reading the Encrypted Extensions) message, even though the keys the client uses for writing don't change at that time. Internally, OpenSSL implements this "transition" of key material with a change_cipher_state() abstraction, that takes a flags argument (`which`). The flags indicate which set of keys to update, and which direction (read or write). So, by my read of the code, what's *supposed* to happen is that we call: if (BIO_set_ktls(bio, _info, which & SSL3_CC_WRITE)) And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with an `is_txt` value that evaluates to true; otherwise, `is_txt` is false, which corresponds to the RX case that you're failing to see happen. Just to get the boring stuff out of the way: what version of openssl are you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not defined when ktls_start() is being compiled (so that the setsockopt(fd, IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)? Thanks, Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > > the CPU utilization of applications like zrepl. But I have a few > questions: > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > > Libraries" section says "Applications using a supported library should > > generally work with ktls without any changes". These sentences seem to > be > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > necessary, but OpenSSL sets it automatically? > > Yes, OpenSSL sets it automatically for the builtin socket and connection > BIO classes. Applications using other BIO classes will need to do things > manually (or implement the appropriate _ctrl() parameters for their BIO > class). > > > * When using OpenSSL, the library will automatically call setsockopt(_, > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > application to tell if ktls is enabled on a particular socket or OpenSSL > > session? > > IIRC the lack of answer for this is part of why upstream OpenSSL doesn't > have specific KTLS tests enabled in the automated test suite. > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason why it's not implemented? That might be the easiest way to check for the ktls status of an individual socket. > > > * From experiment, I can see that OpenSSL attempts to set > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > > From reading ktls_start and ossl_statem_server_post_work, it looks like > > maybe a single socket cannot have ktls enabled for both sending and > > receiving at the same time. Is that true? > > No. They just get enabled separately, since change_cipher_state() is > called separately for read and write transitions. > Apologies if I'm too ignorant, but what is a transition in SSL-speak? This is my first attempt at any kind of SSL programming. What I know from ktrace is that TCP_RXTLS_ENABLE never gets set. > > -Ben > > > Based on the man page and rmacklem's previous mailing list posts, I think > > this should be workable with minor modifications to the kernel and > libzfs. > > I just need to figure out how to use ktls first. > > > > -Alan > > ___ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > the CPU utilization of applications like zrepl. But I have a few questions: > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > Libraries" section says "Applications using a supported library should > generally work with ktls without any changes". These sentences seem to be > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > necessary, but OpenSSL sets it automatically? Yes, OpenSSL sets it automatically for the builtin socket and connection BIO classes. Applications using other BIO classes will need to do things manually (or implement the appropriate _ctrl() parameters for their BIO class). > * When using OpenSSL, the library will automatically call setsockopt(_, > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > application to tell if ktls is enabled on a particular socket or OpenSSL > session? IIRC the lack of answer for this is part of why upstream OpenSSL doesn't have specific KTLS tests enabled in the automated test suite. > * From experiment, I can see that OpenSSL attempts to set > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > From reading ktls_start and ossl_statem_server_post_work, it looks like > maybe a single socket cannot have ktls enabled for both sending and > receiving at the same time. Is that true? No. They just get enabled separately, since change_cipher_state() is called separately for read and write transitions. -Ben > Based on the man page and rmacklem's previous mailing list posts, I think > this should be workable with minor modifications to the kernel and libzfs. > I just need to figure out how to use ktls first. > > -Alan > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Getting started with ktls
I'm trying to make ktls work with "zfs send/recv" to substantially reduce the CPU utilization of applications like zrepl. But I have a few questions: * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported Libraries" section says "Applications using a supported library should generally work with ktls without any changes". These sentences seem to be contradictory. I think it means that the TCP_TXTLS_ENABLE option is necessary, but OpenSSL sets it automatically? * When using OpenSSL, the library will automatically call setsockopt(_, TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an application to tell if ktls is enabled on a particular socket or OpenSSL session? * From experiment, I can see that OpenSSL attempts to set TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? >From reading ktls_start and ossl_statem_server_post_work, it looks like maybe a single socket cannot have ktls enabled for both sending and receiving at the same time. Is that true? Based on the man page and rmacklem's previous mailing list posts, I think this should be workable with minor modifications to the kernel and libzfs. I just need to figure out how to use ktls first. -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
