Re: Getting started with ktls
On Wed, Mar 10, 2021 at 8:15 PM Benjamin Kaduk wrote: > On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote: > > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > > > > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > > > I'm trying to make ktls work with "zfs send/recv" to substantially > reduce > > > > the CPU utilization of applications like zrepl. But I have a few > > > questions: > > > > > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by > a > > > > successful set of the TCP_TXTLS_ENABLE socket option", but the > "Supported > > > > Libraries" section says "Applications using a supported library > should > > > > generally work with ktls without any changes". These sentences seem > to > > > be > > > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > > > necessary, but OpenSSL sets it automatically? > > > > > > Yes, OpenSSL sets it automatically for the builtin socket and > connection > > > BIO classes. Applications using other BIO classes will need to do > things > > > manually (or implement the appropriate _ctrl() parameters for their BIO > > > class). > > > > > > > * When using OpenSSL, the library will automatically call > setsockopt(_, > > > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > > > application to tell if ktls is enabled on a particular socket or > OpenSSL > > > > session? > > > > > > IIRC the lack of answer for this is part of why upstream OpenSSL > doesn't > > > have specific KTLS tests enabled in the automated test suite. > > > > > > > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason > > why it's not implemented? That might be the easiest way to check for the > > ktls status of an individual socket. > > I think that's probably more of a question for jhb than me. I don't know > of a reason why not, but I do know that there is some desire to keep the > functionality that openssl exposes roughly compatible between linux and > FreeBSD KTLS. I don't know whether Linux has something similar. > > > > > > > > > > * From experiment, I can see that OpenSSL attempts to set > > > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why > not? > > > > From reading ktls_start and ossl_statem_server_post_work, it looks > like > > > > maybe a single socket cannot have ktls enabled for both sending and > > > > receiving at the same time. Is that true? > > > > > > No. They just get enabled separately, since change_cipher_state() is > > > called separately for read and write transitions. > > > > > > > Apologies if I'm too ignorant, but what is a transition in SSL-speak? > This > > is my first attempt at any kind of SSL programming. What I know from > > ktrace is that TCP_RXTLS_ENABLE never gets set. > > Sorry! I'm pretty conversant with this stuff (I'm the security area > director that is responsible for the IETF TLS working group) and don't > always target the right level. Basically, for a decent encrypting protocol > you want different encrytion keys for the read and write direction > (whichever peer you are), and the TLS (1.3) handshake has a multi-stage key > hierarchy to try to encrypt as much of it as possible. So, for example, > the client will need to update it's encryption key for reading once it > reads the ServerHello (and before reading the Encrypted Extensions) > message, even though the keys the client uses for writing don't change at > that time. Internally, OpenSSL implements this "transition" of key > material with a change_cipher_state() abstraction, that takes a flags > argument (`which`). The flags indicate which set of keys to update, and > which direction (read or write). So, by my read of the code, what's > *supposed* to happen is that we call: > > if (BIO_set_ktls(bio, _info, which & SSL3_CC_WRITE)) > > And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with > an `is_txt` value that evaluates to true; otherwise, `is_txt` is false, > which corresponds to the RX case that you're failing to see happen. > > Just to get the boring stuff out of the way: what version of openssl are > you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not > defined when ktls_start() is being compiled (so that the setsockopt(fd, > IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)? > > Thanks, > > Ben > I'm using the OpenSSL that's in base in 14.0-CURRENT: 1.1.1j-freebsd . I haven't recompiled the code to check whether OPENSSL_NO_KTLS_RX is defined, but it sure looks like it shouldn't be, based on my reading of the source. -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote: > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > > > the CPU utilization of applications like zrepl. But I have a few > > questions: > > > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > > > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > > > Libraries" section says "Applications using a supported library should > > > generally work with ktls without any changes". These sentences seem to > > be > > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > > necessary, but OpenSSL sets it automatically? > > > > Yes, OpenSSL sets it automatically for the builtin socket and connection > > BIO classes. Applications using other BIO classes will need to do things > > manually (or implement the appropriate _ctrl() parameters for their BIO > > class). > > > > > * When using OpenSSL, the library will automatically call setsockopt(_, > > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > > application to tell if ktls is enabled on a particular socket or OpenSSL > > > session? > > > > IIRC the lack of answer for this is part of why upstream OpenSSL doesn't > > have specific KTLS tests enabled in the automated test suite. > > > > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason > why it's not implemented? That might be the easiest way to check for the > ktls status of an individual socket. I think that's probably more of a question for jhb than me. I don't know of a reason why not, but I do know that there is some desire to keep the functionality that openssl exposes roughly compatible between linux and FreeBSD KTLS. I don't know whether Linux has something similar. > > > > > > * From experiment, I can see that OpenSSL attempts to set > > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > > > From reading ktls_start and ossl_statem_server_post_work, it looks like > > > maybe a single socket cannot have ktls enabled for both sending and > > > receiving at the same time. Is that true? > > > > No. They just get enabled separately, since change_cipher_state() is > > called separately for read and write transitions. > > > > Apologies if I'm too ignorant, but what is a transition in SSL-speak? This > is my first attempt at any kind of SSL programming. What I know from > ktrace is that TCP_RXTLS_ENABLE never gets set. Sorry! I'm pretty conversant with this stuff (I'm the security area director that is responsible for the IETF TLS working group) and don't always target the right level. Basically, for a decent encrypting protocol you want different encrytion keys for the read and write direction (whichever peer you are), and the TLS (1.3) handshake has a multi-stage key hierarchy to try to encrypt as much of it as possible. So, for example, the client will need to update it's encryption key for reading once it reads the ServerHello (and before reading the Encrypted Extensions) message, even though the keys the client uses for writing don't change at that time. Internally, OpenSSL implements this "transition" of key material with a change_cipher_state() abstraction, that takes a flags argument (`which`). The flags indicate which set of keys to update, and which direction (read or write). So, by my read of the code, what's *supposed* to happen is that we call: if (BIO_set_ktls(bio, _info, which & SSL3_CC_WRITE)) And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with an `is_txt` value that evaluates to true; otherwise, `is_txt` is false, which corresponds to the RX case that you're failing to see happen. Just to get the boring stuff out of the way: what version of openssl are you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not defined when ktls_start() is being compiled (so that the setsockopt(fd, IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)? Thanks, Ben ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: main (bad9fa56620e based): some unexpected SIGSEGV's using poudriere-devel to build armv7 ports on aarch64 (cortex-a72) system
On 2021-Mar-10, at 10:09, Olivier Houchard wrote: > On Tue, Mar 09, 2021 at 03:39:42PM -0800, Mark Millard via freebsd-arm wrote: >> After using poudriere to build ports for native cortex-a72 >> on the MACCHIATObin Double Shot (and similarly for >> cortex-a57 on the OverDrive 1000) I attempted to do my >> usual bulk build targeting cortex-a7 via poudriere-devel: >> >> # poudriere jail -i -jFBSDFSSDjailArmV7 >> Jail name: FBSDFSSDjailArmV7 >> Jail version: 14.0-CURRENT >> Jail arch: arm.armv7 >> Jail method: null >> Jail mount:/usr/obj/DESTDIRs/clang-armv7-installworld-poud >> Jail fs: >> Jail updated: 2021-01-27 14:47:10 >> Jail pkgbase: disabled >> >> But I got some SIGSEGV failures that I've never before >> had analogous failures. I'll show the 6 backtraces. >> They all have a similar type-of-context but in various >> programs, summarized as (from the lldb bt outputs): >> > > FREEBSD_COMPAT32 was indeed broken on arm64, and the process would crash > when receiving a signal. I believe I fixed it in -CURRENT with commit > c328f64d81079bad5064c8a387883df50ab5aaed > I built and updated FreeBSD based on that vintage and the port builds are part way through. The ports that I had observed problems for have built just fine and no others have failed so far. If it all builds, it will be tomorrow sometime before the bulk builds finish. But I figured I'd indicate that the fix looks to have fully worked for my context that had the problem. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar) ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > > the CPU utilization of applications like zrepl. But I have a few > questions: > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > > Libraries" section says "Applications using a supported library should > > generally work with ktls without any changes". These sentences seem to > be > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > necessary, but OpenSSL sets it automatically? > > Yes, OpenSSL sets it automatically for the builtin socket and connection > BIO classes. Applications using other BIO classes will need to do things > manually (or implement the appropriate _ctrl() parameters for their BIO > class). > > > * When using OpenSSL, the library will automatically call setsockopt(_, > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > application to tell if ktls is enabled on a particular socket or OpenSSL > > session? > > IIRC the lack of answer for this is part of why upstream OpenSSL doesn't > have specific KTLS tests enabled in the automated test suite. > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason why it's not implemented? That might be the easiest way to check for the ktls status of an individual socket. > > > * From experiment, I can see that OpenSSL attempts to set > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > > From reading ktls_start and ossl_statem_server_post_work, it looks like > > maybe a single socket cannot have ktls enabled for both sending and > > receiving at the same time. Is that true? > > No. They just get enabled separately, since change_cipher_state() is > called separately for read and write transitions. > Apologies if I'm too ignorant, but what is a transition in SSL-speak? This is my first attempt at any kind of SSL programming. What I know from ktrace is that TCP_RXTLS_ENABLE never gets set. > > -Ben > > > Based on the man page and rmacklem's previous mailing list posts, I think > > this should be workable with minor modifications to the kernel and > libzfs. > > I just need to figure out how to use ktls first. > > > > -Alan > > ___ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Getting started with ktls
On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > the CPU utilization of applications like zrepl. But I have a few questions: > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > Libraries" section says "Applications using a supported library should > generally work with ktls without any changes". These sentences seem to be > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > necessary, but OpenSSL sets it automatically? Yes, OpenSSL sets it automatically for the builtin socket and connection BIO classes. Applications using other BIO classes will need to do things manually (or implement the appropriate _ctrl() parameters for their BIO class). > * When using OpenSSL, the library will automatically call setsockopt(_, > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > application to tell if ktls is enabled on a particular socket or OpenSSL > session? IIRC the lack of answer for this is part of why upstream OpenSSL doesn't have specific KTLS tests enabled in the automated test suite. > * From experiment, I can see that OpenSSL attempts to set > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > From reading ktls_start and ossl_statem_server_post_work, it looks like > maybe a single socket cannot have ktls enabled for both sending and > receiving at the same time. Is that true? No. They just get enabled separately, since change_cipher_state() is called separately for read and write transitions. -Ben > Based on the man page and rmacklem's previous mailing list posts, I think > this should be workable with minor modifications to the kernel and libzfs. > I just need to figure out how to use ktls first. > > -Alan > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Getting started with ktls
I'm trying to make ktls work with "zfs send/recv" to substantially reduce the CPU utilization of applications like zrepl. But I have a few questions: * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported Libraries" section says "Applications using a supported library should generally work with ktls without any changes". These sentences seem to be contradictory. I think it means that the TCP_TXTLS_ENABLE option is necessary, but OpenSSL sets it automatically? * When using OpenSSL, the library will automatically call setsockopt(_, TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an application to tell if ktls is enabled on a particular socket or OpenSSL session? * From experiment, I can see that OpenSSL attempts to set TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? >From reading ktls_start and ossl_statem_server_post_work, it looks like maybe a single socket cannot have ktls enabled for both sending and receiving at the same time. Is that true? Based on the man page and rmacklem's previous mailing list posts, I think this should be workable with minor modifications to the kernel and libzfs. I just need to figure out how to use ktls first. -Alan ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: main (bad9fa56620e based): some unexpected SIGSEGV's using poudriere-devel to build armv7 ports on aarch64 (cortex-a72) system
Hi Mark, On Tue, Mar 09, 2021 at 03:39:42PM -0800, Mark Millard via freebsd-arm wrote: > After using poudriere to build ports for native cortex-a72 > on the MACCHIATObin Double Shot (and similarly for > cortex-a57 on the OverDrive 1000) I attempted to do my > usual bulk build targeting cortex-a7 via poudriere-devel: > > # poudriere jail -i -jFBSDFSSDjailArmV7 > Jail name: FBSDFSSDjailArmV7 > Jail version: 14.0-CURRENT > Jail arch: arm.armv7 > Jail method: null > Jail mount:/usr/obj/DESTDIRs/clang-armv7-installworld-poud > Jail fs: > Jail updated: 2021-01-27 14:47:10 > Jail pkgbase: disabled > > But I got some SIGSEGV failures that I've never before > had analogous failures. I'll show the 6 backtraces. > They all have a similar type-of-context but in various > programs, summarized as (from the lldb bt outputs): > FREEBSD_COMPAT32 was indeed broken on arm64, and the process would crash when receiving a signal. I believe I fixed it in -CURRENT with commit c328f64d81079bad5064c8a387883df50ab5aaed Regards, Olivier ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: "panic: malloc(M_WAITOK) with sleeping prohibited" at main-n245363-b3dac3913dc9
On Wed, Mar 10, 2021 at 08:25:00AM -0700, Warner Losh wrote:
> ...
> > Also (as yesterday), neither laptop exhibited a problem after the
> > corresponding update.
> >
>
> Yes it "works" without invariants. I'm working on a better fix that isn't
> such a huge game of whack-a-mole that defers the async messages to an
> taskqueue where they can sleep for memory, if need be.
>
Fair enough -- we see that ("works" without INVARIANTS option) often
enough at work. :-}
I'm happy to test
Peace,
david
--
David H. Wolfskill da...@catwhisker.org
"So much money is being raised and completely wasted by people that do not
have the GOP's best interests in mind." - D. Trump, who would know firsthand.
See https://www.catwhisker.org/~david/publickey.gpg for my public key.
signature.asc
Description: PGP signature
Re: "panic: malloc(M_WAITOK) with sleeping prohibited" at main-n245363-b3dac3913dc9
On Wed, Mar 10, 2021 at 5:37 AM David Wolfskill wrote: > On Tue, Mar 09, 2021 at 04:04:56PM -0700, Warner Losh wrote: > > On Tue, Mar 9, 2021 at 2:46 PM David Wolfskill > wrote: > > ... > > > uma_zalloc_debug: zone "malloc-1024"umass0 on uhub2 > > > with the following non-sleepable locks held: > > > umass0: on usbus0 > > > exclusive sleep mutex CAM device lockumass0: SCSI over Bulk-Only; > quirks > > > = 0x4000 > > > (CAM device lock) r = 0 (0xf800122c9cd0) locked @ > > > /usr/src/sys/cam/cam_xpt.c:2333 > > > umass0:6:0: Attached to scbus6 > > > stack backtrace: > > > (probe0:umass-sim0:0:0:0): Down reving Protocol Version from 2 to 0? > > > #0 0x80c7cce1 at witnesuhub3: 6 ports with 6 removable, self > > > powered > > > s_debugger+0x71 > > > pass6 at umass-sim0 bus 0 scbus6 target 0 lun 0 > > > #1 0x80pass6: uhub4: 8 ports with 8 removable, self powered > > > c7ddfd at witness_warn+0x40d > > > #2 Removable Direct Access SCSI device > > > 0x80f42fe6 at uma_zallpass6: Serial Number 2010081884130 > > > oc_arg+0x46 > > > #3 0x80be34pass6: 40.000MB/s transfers > > > panic: malloc(M_WAITOK) with sleeping prohibited > > > cpuid = 1 > > > time = 22 > > > KDB: stack backtrace: > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > > 0xfe00e157a2d0 > > > vpanic() at vpanic+0x181/frame 0xfe00e157a320 > > > panic() at panic+0x43/frame 0xfe00e157a380 > > > malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e157a3a0 > > > malloc() at malloc+0x34/frame 0xfe00e157a400 > > > g_post_event_x() at g_post_event_x+0x5a/frame 0xfe00e157a450 > > > g_post_event() at g_post_event+0x48/frame 0xfe00e157a4b0 > > > disk_create() at disk_create+0x16f/frame 0xfe00e157a600 > > > daregister() at daregister+0x70a/frame 0xfe00e157a880 > > > cam_periph_alloc() at cam_periph_alloc+0x57b/frame 0xfe00e157a950 > > > daasync() at daasync+0x2c2/frame 0xfe00e157a9c0 > > > xpt_async_process_dev() at xpt_async_process_dev+0x152/frame > > > 0xfe00e157aa10 > > > xpt_async_process() at xpt_async_process+0x334/frame 0xfe00e157ab20 > > > xpt_done_process() at xpt_done_process+0x3a3/frame 0xfe00e157ab60 > > > xpt_done_td() at xpt_done_td+0xf5/frame 0xfe00e157abb0 > > > fork_exit() at fork_exit+0x80/frame 0xfe00e157abf0 > > > fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e157abf0 > > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > > KDB: enter: panic > > > [ thread pid 17 tid 100095 ] > > > Stopped at kdb_enter+0x37: movq$0,0x128b8ce(%rip) > > > db> > > > > > I'm willing to "poke at it" a bit, given a hint or two... > > > > > > > OK. I know what's happening here. disk_create() posts an event and also > > expects to sleep to get memory. I can fix that too, but then that's one > > more 'no memory' hole. > > > > I'm unsure if we should keep playing whack-a-mole, or just defer this > > creation to the taskqueue we already have hanging around... > > > > Warner > > > > Same (build) machine had a similar panic after updating source to > main-n245372-d1cbe7908986 & rebuilding. FWIW. > > Also (as yesterday), neither laptop exhibited a problem after the > corresponding update. > Yes it "works" without invariants. I'm working on a better fix that isn't such a huge game of whack-a-mole that defers the async messages to an taskqueue where they can sleep for memory, if need be. Warner > Peace, > david > -- > David H. Wolfskill da...@catwhisker.org > It is supremely disingenuous to claim a lack of jurisdiction, then > proceed to participate in a decision on the same matter. > > See https://www.catwhisker.org/~david/publickey.gpg for my public key. > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: panic: malloc(M_WAITOK) with sleeping prohibited
On Wed, Mar 10, 2021 at 01:33:34PM +0100, Hans Petter Selasky wrote: > On 3/10/21 12:41 PM, Peter Holm wrote: > > On Wed, Mar 10, 2021 at 10:52:53AM +0100, Hans Petter Selasky wrote: > >> On 3/10/21 10:15 AM, Peter Holm wrote: > >>> I just got this panic: > >>> > >>> igb0: port 0xd020-0xd03f > >>> mem 0xfb32-0xfb33,0xfb344000-0xfb347fff irq 16 at device 0.0 on > >>> pci8 > >>> igb0: Using 1024 TX descriptors and 1024 RX descriptors > >>> igb0: queue equality override not set, capping rx_queues at 6 and > >>> tx_queues at 6 > >>> igb0: Using 6 RX queues 6 TX queues > >>> igb0: Using MSI-X interrupts with 7 vectors > >>> igb0: > >>> db> > >>> db> show panic > >>> panic: malloc(M_WAITOK) with sleeping prohibited > >>> db> bt > >>> Tracing pid 12 tid 100172 td 0xfe010dce2100 > >>> kdb_enter() at kdb_enter+0x37/frame 0xfe00e4f72980 > >>> vpanic() at vpanic+0x1b2/frame 0xfe00e4f729d0 > >>> panic() at panic+0x43/frame 0xfe00e4f72a30 > >>> malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4f72a50 > >>> malloc() at malloc+0x34/frame 0xfe00e4f72ab0 > >>> linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4f72b00 > >>> linux_irq_handler() at linux_irq_handler+0x3a/frame 0xfe00e4f72b20 > >>> ithread_loop() at ithread_loop+0x279/frame 0xfe00e4f72bb0 > >>> fork_exit() at fork_exit+0x80/frame 0xfe00e4f72bf0 > >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4f72bf0 > >>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > >>> db> x/s version > >>> version:FreeBSD 14.0-CURRENT #0 main-n245371-ce53f92e6c81: Wed > >>> Mar 10 10:00:29 CET 2021\012 > >>> p...@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 > >>> db> > >> > >> This should fix it: > >> https://cgit.freebsd.org/src/commit/?id=d1cbe79089868226625c12ef49f51214d79aa427 > >> > >> --HPS > > > > Yes, thank you. Now I see this: > > > > ugen0.3: at usbus0 > > ukbd0 on uhub3 > > ukbd0: on > > usbus0 > > kbd2 at ukbd0 > > panic: malloc(M_WAITOK) with sleeping prohibited > > cpuid = 0 > > time = 1615375651 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > 0xfe00e4974890 > > vpanic() at vpanic+0x181/frame 0xfe00e49748e0 > > panic() at panic+0x43/frame 0xfe00e4974940 > > malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4974960 > > malloc() at malloc+0x34/frame 0xfe00e49749c0 > > linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4974a10 > > linux_timer_callback_wrapper() at linux_timer_callback_wrapper+0x37/frame > > 0xfe00e4974a30 > > softclock_call_cc() at softclock_call_cc+0x15d/frame 0xfe00e4974b00 > > softclock() at softclock+0x66/frame 0xfe00e4974b20 > > ithread_loop() at ithread_loop+0x279/frame 0xfe00e4974bb0 > > fork_exit() at fork_exit+0x80/frame 0xfe00e4974bf0 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4974bf0 > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > KDB: enter: panic > > [ thread pid 12 tid 100088 ] > > Try this: > https://cgit.freebsd.org/src/commit/?id=dfb33cb0ef48084da84072244e8ca486dfcf3a96 > Works for me. Thank you! - Peter > There will be a more comprehensive fix coming: > https://reviews.freebsd.org/D29183 > > --HPS ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: "panic: malloc(M_WAITOK) with sleeping prohibited" at main-n245363-b3dac3913dc9
On Tue, Mar 09, 2021 at 04:04:56PM -0700, Warner Losh wrote: > On Tue, Mar 9, 2021 at 2:46 PM David Wolfskill wrote: > ... > > uma_zalloc_debug: zone "malloc-1024"umass0 on uhub2 > > with the following non-sleepable locks held: > > umass0: on usbus0 > > exclusive sleep mutex CAM device lockumass0: SCSI over Bulk-Only; quirks > > = 0x4000 > > (CAM device lock) r = 0 (0xf800122c9cd0) locked @ > > /usr/src/sys/cam/cam_xpt.c:2333 > > umass0:6:0: Attached to scbus6 > > stack backtrace: > > (probe0:umass-sim0:0:0:0): Down reving Protocol Version from 2 to 0? > > #0 0x80c7cce1 at witnesuhub3: 6 ports with 6 removable, self > > powered > > s_debugger+0x71 > > pass6 at umass-sim0 bus 0 scbus6 target 0 lun 0 > > #1 0x80pass6: uhub4: 8 ports with 8 removable, self powered > > c7ddfd at witness_warn+0x40d > > #2 Removable Direct Access SCSI device > > 0x80f42fe6 at uma_zallpass6: Serial Number 2010081884130 > > oc_arg+0x46 > > #3 0x80be34pass6: 40.000MB/s transfers > > panic: malloc(M_WAITOK) with sleeping prohibited > > cpuid = 1 > > time = 22 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > 0xfe00e157a2d0 > > vpanic() at vpanic+0x181/frame 0xfe00e157a320 > > panic() at panic+0x43/frame 0xfe00e157a380 > > malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e157a3a0 > > malloc() at malloc+0x34/frame 0xfe00e157a400 > > g_post_event_x() at g_post_event_x+0x5a/frame 0xfe00e157a450 > > g_post_event() at g_post_event+0x48/frame 0xfe00e157a4b0 > > disk_create() at disk_create+0x16f/frame 0xfe00e157a600 > > daregister() at daregister+0x70a/frame 0xfe00e157a880 > > cam_periph_alloc() at cam_periph_alloc+0x57b/frame 0xfe00e157a950 > > daasync() at daasync+0x2c2/frame 0xfe00e157a9c0 > > xpt_async_process_dev() at xpt_async_process_dev+0x152/frame > > 0xfe00e157aa10 > > xpt_async_process() at xpt_async_process+0x334/frame 0xfe00e157ab20 > > xpt_done_process() at xpt_done_process+0x3a3/frame 0xfe00e157ab60 > > xpt_done_td() at xpt_done_td+0xf5/frame 0xfe00e157abb0 > > fork_exit() at fork_exit+0x80/frame 0xfe00e157abf0 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e157abf0 > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > KDB: enter: panic > > [ thread pid 17 tid 100095 ] > > Stopped at kdb_enter+0x37: movq$0,0x128b8ce(%rip) > > db> > > > I'm willing to "poke at it" a bit, given a hint or two... > > > > OK. I know what's happening here. disk_create() posts an event and also > expects to sleep to get memory. I can fix that too, but then that's one > more 'no memory' hole. > > I'm unsure if we should keep playing whack-a-mole, or just defer this > creation to the taskqueue we already have hanging around... > > Warner > Same (build) machine had a similar panic after updating source to main-n245372-d1cbe7908986 & rebuilding. FWIW. Also (as yesterday), neither laptop exhibited a problem after the corresponding update. Peace, david -- David H. Wolfskill da...@catwhisker.org It is supremely disingenuous to claim a lack of jurisdiction, then proceed to participate in a decision on the same matter. See https://www.catwhisker.org/~david/publickey.gpg for my public key. signature.asc Description: PGP signature
Re: panic: malloc(M_WAITOK) with sleeping prohibited
On 3/10/21 12:41 PM, Peter Holm wrote: On Wed, Mar 10, 2021 at 10:52:53AM +0100, Hans Petter Selasky wrote: On 3/10/21 10:15 AM, Peter Holm wrote: I just got this panic: igb0: port 0xd020-0xd03f mem 0xfb32-0xfb33,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: db> db> show panic panic: malloc(M_WAITOK) with sleeping prohibited db> bt Tracing pid 12 tid 100172 td 0xfe010dce2100 kdb_enter() at kdb_enter+0x37/frame 0xfe00e4f72980 vpanic() at vpanic+0x1b2/frame 0xfe00e4f729d0 panic() at panic+0x43/frame 0xfe00e4f72a30 malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4f72a50 malloc() at malloc+0x34/frame 0xfe00e4f72ab0 linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4f72b00 linux_irq_handler() at linux_irq_handler+0x3a/frame 0xfe00e4f72b20 ithread_loop() at ithread_loop+0x279/frame 0xfe00e4f72bb0 fork_exit() at fork_exit+0x80/frame 0xfe00e4f72bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4f72bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> x/s version version:FreeBSD 14.0-CURRENT #0 main-n245371-ce53f92e6c81: Wed Mar 10 10:00:29 CET 2021\012 p...@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db> This should fix it: https://cgit.freebsd.org/src/commit/?id=d1cbe79089868226625c12ef49f51214d79aa427 --HPS Yes, thank you. Now I see this: ugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 panic: malloc(M_WAITOK) with sleeping prohibited cpuid = 0 time = 1615375651 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00e4974890 vpanic() at vpanic+0x181/frame 0xfe00e49748e0 panic() at panic+0x43/frame 0xfe00e4974940 malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4974960 malloc() at malloc+0x34/frame 0xfe00e49749c0 linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4974a10 linux_timer_callback_wrapper() at linux_timer_callback_wrapper+0x37/frame 0xfe00e4974a30 softclock_call_cc() at softclock_call_cc+0x15d/frame 0xfe00e4974b00 softclock() at softclock+0x66/frame 0xfe00e4974b20 ithread_loop() at ithread_loop+0x279/frame 0xfe00e4974bb0 fork_exit() at fork_exit+0x80/frame 0xfe00e4974bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4974bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic [ thread pid 12 tid 100088 ] Try this: https://cgit.freebsd.org/src/commit/?id=dfb33cb0ef48084da84072244e8ca486dfcf3a96 There will be a more comprehensive fix coming: https://reviews.freebsd.org/D29183 --HPS ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: panic: malloc(M_WAITOK) with sleeping prohibited
On Wed, Mar 10, 2021 at 10:52:53AM +0100, Hans Petter Selasky wrote: > On 3/10/21 10:15 AM, Peter Holm wrote: > > I just got this panic: > > > > igb0: port 0xd020-0xd03f mem > > 0xfb32-0xfb33,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 > > igb0: Using 1024 TX descriptors and 1024 RX descriptors > > igb0: queue equality override not set, capping rx_queues at 6 and tx_queues > > at 6 > > igb0: Using 6 RX queues 6 TX queues > > igb0: Using MSI-X interrupts with 7 vectors > > igb0: > > db> > > db> show panic > > panic: malloc(M_WAITOK) with sleeping prohibited > > db> bt > > Tracing pid 12 tid 100172 td 0xfe010dce2100 > > kdb_enter() at kdb_enter+0x37/frame 0xfe00e4f72980 > > vpanic() at vpanic+0x1b2/frame 0xfe00e4f729d0 > > panic() at panic+0x43/frame 0xfe00e4f72a30 > > malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4f72a50 > > malloc() at malloc+0x34/frame 0xfe00e4f72ab0 > > linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4f72b00 > > linux_irq_handler() at linux_irq_handler+0x3a/frame 0xfe00e4f72b20 > > ithread_loop() at ithread_loop+0x279/frame 0xfe00e4f72bb0 > > fork_exit() at fork_exit+0x80/frame 0xfe00e4f72bf0 > > fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4f72bf0 > > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > > db> x/s version > > version:FreeBSD 14.0-CURRENT #0 main-n245371-ce53f92e6c81: Wed Mar > > 10 10:00:29 CET 2021\012 > > p...@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 > > db> > > This should fix it: > https://cgit.freebsd.org/src/commit/?id=d1cbe79089868226625c12ef49f51214d79aa427 > > --HPS Yes, thank you. Now I see this: ugen0.3: at usbus0 ukbd0 on uhub3 ukbd0: on usbus0 kbd2 at ukbd0 panic: malloc(M_WAITOK) with sleeping prohibited cpuid = 0 time = 1615375651 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe00e4974890 vpanic() at vpanic+0x181/frame 0xfe00e49748e0 panic() at panic+0x43/frame 0xfe00e4974940 malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4974960 malloc() at malloc+0x34/frame 0xfe00e49749c0 linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4974a10 linux_timer_callback_wrapper() at linux_timer_callback_wrapper+0x37/frame 0xfe00e4974a30 softclock_call_cc() at softclock_call_cc+0x15d/frame 0xfe00e4974b00 softclock() at softclock+0x66/frame 0xfe00e4974b20 ithread_loop() at ithread_loop+0x279/frame 0xfe00e4974bb0 fork_exit() at fork_exit+0x80/frame 0xfe00e4974bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4974bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic [ thread pid 12 tid 100088 ] Stopped at kdb_enter+0x37: movq$0,0x12891fe(%rip) db> x/s version version:FreeBSD 14.0-CURRENT #0 main-n245372-d1cbe7908986: Wed Mar 10 12:25:03 CET 2021\012 p...@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db> - Peter ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: panic: malloc(M_WAITOK) with sleeping prohibited
On 3/10/21 10:15 AM, Peter Holm wrote: I just got this panic: igb0: port 0xd020-0xd03f mem 0xfb32-0xfb33,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: db> db> show panic panic: malloc(M_WAITOK) with sleeping prohibited db> bt Tracing pid 12 tid 100172 td 0xfe010dce2100 kdb_enter() at kdb_enter+0x37/frame 0xfe00e4f72980 vpanic() at vpanic+0x1b2/frame 0xfe00e4f729d0 panic() at panic+0x43/frame 0xfe00e4f72a30 malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4f72a50 malloc() at malloc+0x34/frame 0xfe00e4f72ab0 linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4f72b00 linux_irq_handler() at linux_irq_handler+0x3a/frame 0xfe00e4f72b20 ithread_loop() at ithread_loop+0x279/frame 0xfe00e4f72bb0 fork_exit() at fork_exit+0x80/frame 0xfe00e4f72bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4f72bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> x/s version version:FreeBSD 14.0-CURRENT #0 main-n245371-ce53f92e6c81: Wed Mar 10 10:00:29 CET 2021\012 p...@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db> This should fix it: https://cgit.freebsd.org/src/commit/?id=d1cbe79089868226625c12ef49f51214d79aa427 --HPS ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
panic: malloc(M_WAITOK) with sleeping prohibited
I just got this panic: igb0: port 0xd020-0xd03f mem 0xfb32-0xfb33,0xfb344000-0xfb347fff irq 16 at device 0.0 on pci8 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6 igb0: Using 6 RX queues 6 TX queues igb0: Using MSI-X interrupts with 7 vectors igb0: db> db> show panic panic: malloc(M_WAITOK) with sleeping prohibited db> bt Tracing pid 12 tid 100172 td 0xfe010dce2100 kdb_enter() at kdb_enter+0x37/frame 0xfe00e4f72980 vpanic() at vpanic+0x1b2/frame 0xfe00e4f729d0 panic() at panic+0x43/frame 0xfe00e4f72a30 malloc_dbg() at malloc_dbg+0xd4/frame 0xfe00e4f72a50 malloc() at malloc+0x34/frame 0xfe00e4f72ab0 linux_alloc_current() at linux_alloc_current+0x3d/frame 0xfe00e4f72b00 linux_irq_handler() at linux_irq_handler+0x3a/frame 0xfe00e4f72b20 ithread_loop() at ithread_loop+0x279/frame 0xfe00e4f72bb0 fork_exit() at fork_exit+0x80/frame 0xfe00e4f72bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe00e4f72bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db> x/s version version:FreeBSD 14.0-CURRENT #0 main-n245371-ce53f92e6c81: Wed Mar 10 10:00:29 CET 2021\012 p...@mercat1.netperf.freebsd.org:/usr/src/sys/amd64/compile/PHO\012 db> - Peter ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
