[mdb-discuss] ::kmastat output different in meaning between x86 32-bit and 64-bit?
Jonathan, thanks! I understood the difference now. Raymond Jonathan Adams wrote: (oops; I forgot to CC mdb-discuss on this earlier) On 8/2/07, *Jonathan Adams* jwadams at gmail.com mailto:jwadams at gmail.com wrote: On 8/2/07, * Raymond LI* Raymond.Li at sun.com mailto:Raymond.Li at sun.com wrote: Guys, I met a puzzle when I work with my amd64 box. When I observe a stream slab of 2048 sizes, the buf in use, buf total and memory in use seems to mean different thing between 32/64 bit kernels. To answer your questions, I need to know more about your caches; what is the output of ::kmem_cache ! grep streams_dblk_19.. in MDB on your 32-bit and 64-bit systems? That will tell me what the cache flags for your system are. (replace .. with 36 or 84 for 64-bit and 32-bit, respectively) While you're at it, take the cache pointer (the first field output by the above command), and do: pointer::print kmem_cache_t cache_chunksize cache_bufsize cache_slabsize He responded with the following data: --- cut here --- On 32bit kernel, with driver unloaded at startup ::kmem_cache ! grep streams_dblk_1984 cac2e2b0 streams_dblk_1984 020f 00 2048 9 cac2e2b0::print kmem_cache_t cache_chunksize cache_bufsize cache_slabsize cache_chunksize = 0x840 cache_bufsize = 0x800 cache_slabsize = 0x5000 After add_drv, ::kmem_cache ! grep streams_dblk_1984 cac2e2b0 streams_dblk_1984 020f 00 2048 612 cac2e2b0::print kmem_cache_t cache_chunksize cache_bufsize cache_slabsize And then rem_drv, ::kmem_cache ! grep streams_dblk_1984 cac2e2b0 streams_dblk_1984 020f 00 2048 612 cac2e2b0::print kmem_cache_t cache_chunksize cache_bufsize cache_slabsize cache_chunksize = 0x840 cache_bufsize = 0x800 cache_slabsize = 0x5000 -- On 64bit kernel, with driver unloaded ::kmem_cache ! grep streams_dblk_1936 ec0033c08 streams_dblk_1936 0269 00 2048 2 with driver loaded, ::kmem_cache ! grep streams_dblk_1936 ec0033c08 streams_dblk_1936 0269 00 2048 602 ec0033c08::print kmem_cache_t cache_chunksize cache_bufsize cache_slabsize cache_chunksize = 0x800 cache_bufsize = 0x800 cache_slabsize = 0x1000 --- cut here --- Looking at the ::kmem_cache output: 32-bit: ::kmem_cache ! grep streams_dblk_1984 cac2e2b0 streams_dblk_1984 020f 00 2048 9 64-bit ::kmem_cache ! grep streams_dblk_1936 ec0033c08 streams_dblk_1936 0269 00 2048 602 The third field is the flags field; this contains the full reason for all the differences you noticed. 32-bit: KMF_HASH | KMF_CONTENTS | KMF_AUDIT | KMF_DEADBEEF | KMF_REDZONE 64-bit: KMF_HASH | KMF_CONTENTS | KMF_AUDIT | KMF_FIREWALL | KMF_NOMAGAZINE The flags they both have are KMF_HASH (buffers require a hash table to track control structures), KMF_CONTENTS (record contents of buffer upon free), KMF_AUDIT (record information about each allocation and free). The 64-bit cache is a firewall cache; this means the buffer size is rounded up to a multiple of PAGESIZE, and all buffers are allocated so that the end of the buffer is at the end of a page. The allocation is then done in such a way that there is an unmapped VA hole *after* that page, and so that allocation addresses are not re-used recently. The magazine (that is, caching) layer of the cache is disabled, which means that the objects are freed and unmapped immediately upon kmem_cache_free(). The 32-bit cache is a standard debugging cache; the slabs are five pages long, which is 9 buffers / slab (0x5000 / 0x540); the extra 40 bytes is a REDZONE, used to detect buffer overruns, etc. The magazine layer is *enabled*, which means that freed buffers are cached, until the system notices that we're running low on space. The difference only exists on DEBUG kernels, and is because firewalling is not done on 32-bit platforms, since the allocation patterns used waste and fragment VA space. On a non-debug system, the setup will be pretty much the same between 32-bit and 64-bit systems. I allocated mbuf of 1600 bytes, In 64-bit mode, the cache name of 2048 should have name of streams_dblk_1936, output like below: cache buf buf buf memory alloc alloc name size in use total in use succeed fail - -- -- -- - - - streams_dblk_1936 2048 602 602 2465792 1382 0 While with 32-bit mode, with the name of streams_dblk_1984, output like below: cache buf buf buf memory alloc alloc name size in use total in use succeed fail
[mdb-discuss] ::kmastat output different in meaning between x86 32-bit and 64-bit?
Guys, I met a puzzle when I work with my amd64 box. When I observe a stream slab of 2048 sizes, the buf in use, buf total and memory in use seems to mean different thing between 32/64 bit kernels. I allocated mbuf of 1600 bytes, In 64-bit mode, the cache name of 2048 should have name of streams_dblk_1936, output like below: cache buf buf buf memory alloc alloc name size in use total in use succeed fail - -- -- -- - - - streams_dblk_1936 2048 602 602 2465792 1382 0 While with 32-bit mode, with the name of streams_dblk_1984, output like below: cache buf buf buf memory alloc alloc name size in use total in use succeed fail - -- -- -- - - - streams_dblk_1984 2048 600 603 1372160 608 0 My first question is: why in 64-bit kernel, the memory in use = (buf in use + buf total) * buf size? And we see in 32-bit kernel, it is buf total * buf size. Another thing I don't understand is my driver allocated 600 mblk of 1600 Bytes during attach. After rem_drv, I found in 64-bit kernel, kmastat reports below: cache buf buf buf memory alloc alloc name size in use total in use succeed fail - -- -- -- - - - streams_dblk_1936 2048 2 2 8192 1493 0 buf total also reduced by 600 after freemsg. And it seems next time allocate, stream module will allocate new area of streams_dblk_1936. While on 32-bit kernel, the streams_dblk_1984 seems to be still there: cache buf buf buf memory alloc alloc name size in use total in use succeed fail - -- -- -- - - - streams_dblk_1984 2048 0 603 1372160 608 0 Why buf total still 603? Thanks, Raymond
[mdb-discuss] help, crashdump analysis, x86
rivanwang wrote: I have some questions as follows. Would you be so kind as to give me some suggestions? d2c84de0::findstack -v will give some clue. ::cpuinfo -v ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 0 fec20ae4 1b80 104 nono t-740847 d2c84de0 sched ||| RUNNING --+|+-- PIL THREAD READY | 5 d2c84de0 EXISTS | 3 d2ca0de0 ENABLE | - d2c28de0 (idle) | +-- PRI THREAD PROC 99 d2c9ade0 sched 99 d2c97de0 sched 60 d3264a00 fsflush 60 d2e1ade0 sched 60 d2e37de0 sched 60 d4644de0 sched 60 d96dcde0 sched 59 d38e7400 Xsun d2c84de0::thread ADDRSTATE FLG PFLG SFLG PRI EPRI PIL INTR DISPTIME BOUND PR d2c84de0 onproc80903 104 0 5 d2ca0de00-1 2 d2ca0de0::thread ADDRSTATE FLG PFLG SFLG PRI EPRI PIL INTR DISPTIME BOUND PR d2ca0de0 onproc 903 102 0 3 d2c28de046a51-1 1 d2ca0de0::findstack -v stack pointer for thread d2ca0de0: d2ca0c2c d2ca0de0 0xd94c62bc() After I pressed F1+A?the kernel created the thread d2c84de0 to give responses to keyboard interruption(PIL = 5, PRI= 104). but another thread d2ca0de0,at same time, is still running on CPU. ( PIL = 3 , PRI = 102 ). I guess one event may causes the kernel to create the thread d2ca0de0 , but then the kernel hangs, until I have pressed F1+A , the kernel creates another thead d2c84de0 , and finally crashed down. I have no idea what causes the kernel to create thread d2ca0de0 (PRI=102,PIL=3)? [[ Q3 ]] ::cpuinfo ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 0 fec20ae4 1b80 104 nono t-740847 d2c84de0 sched ::cycinfo -v CPU CYC_CPU STATE NELEMS ROOTFIRE HANDLER 0 d9aabe00 online 4 d9aabd80 96b6b848e80 clock 2 | +--+--+ 0 1 | | +-++ +-+-+ 3 | +++ ADDR NDX HEAP LEVL PENDFIRE USECINT HANDLER d9aabd80 01 high 0 96b6b848e80 1 cbe_hres_tick d9aabda0 12 low 74125396b6b848e80 1 apic_redistribute_compute d9aabdc0 20 lock 406 96b6b848e80 1 clock d9aabde0 33 high 0 96b6d4e5200 100 deadman --- The value of SWITCH of thread d2c84de0 is 740847 ; The value of PEND of apic_redistribute_compute is 741253 ; The value of PEND of clock is 406 . (741253 - 406) == 740847 What does it mean ? Could you please account for it ? [[ Q4 ]] ::ipcs Message queues: failed to read 'msq_svc'; module not present Shared memory: ADDR REFID KEY MODE PRJID ZONEID OWNER GROUP CREAT CGRP d4915f50 1 3 103 0666 3 0 1002 102 1002 102 d3f0b090 1 2 101 0666 3 0 0 0 0 0 d3f0b2c0 1 1 102 0666 3 0 1002 102 1002 102 d3f0bbf0 1 0 100 0666 3 0 1002 102 1002 102 Semaphores: ADDR REFID KEY MODE PRJID ZONEID OWNER GROUP CREAT CGRP d4915ee0 3 3 103 0666 3 0 1002 102 1002 102 d3f0b1e0 3 2 101 0666 3 0 0 0 0 0 d3f0b250 4 1 102 0666 3 0 1002 102 1002 102 d3f0bb80 7 0 100 0666 3 0 1002 102 1002 102 --- I dont know what threads are accessing to the semaphore d3f0b1e0 ? How can I find these unkown threads? ::showrev Hostname: cetc.a28.com Release: 5.10 Kernel architecture: i86pc Application architecture: i386 Kernel version: SunOS 5.10 i86pc Generic Platform: i86pc ::msgbuf MESSAGE /pci at 0,0/pci103c,3013 at 1d,2 (uhci2): failed to attach pcplusmp: pciclass,0c0300 (uhci) instance 3 vector 0x16 ioapic 0x1 intin 0x16 is bound to cpu 0 /pci at 0,0/pci103c,3013 at 1d,3 (uhci3): failed to attach cpu0: x86 (GenuineIntel family 15 model 4 step 10 clock 3000 MHz)
[mdb-discuss] [Fwd: About debug hang issues with deadman]
Typically this is deadloop in kernel/module code. This case is bge's long loop in ISR, which prevent lbolt from being cleared by clock. Oliver Yang wrote: Hi Guys, I have a question about using mdb debug hang issue, does anybody can give me the answers in below mail? Original Message Subject: About debug hang issues with deadman Date: Mon, 27 Nov 2006 16:40:00 +0800 From: Oliver Yang Oliver.Yang at Sun.COM To: opensolaris-code at opensolaris.org Hi All, Recently, I ran into a hang issue, and I enabled deadman timer by setting set snooping =1 in /etc/system file. Finally, we got a crashdump file. Does anybody can give me a hint on how to debug hang issue with enabling deadman? Here are steps what I have tried on one of crashdump files: ::msgbuf panic[cpu0]/thread=a13f7de0: deadman: timed out after 50 seconds of clock inactivity a13f4f24 genunix:deadman+159 (0) a13f4f5c genunix:cyclic_expire+280 (a14fe000, 3, aa2183) a13f4fb8 genunix:cyclic_fire+17d (fec21d30) a13f4fd4 unix:cbe_fire+4a (0, 0) a13f75b8 unix:_interrupt+2a1 (1b0, abb1, ab36) a13f7638 unix:bcopy+39 (abb11780) a13f765c genunix:copymsg+2e (aa228340) a13f7688 genunix:copymsgchain+23 (aa228340) a13f76c0 dls:i_dls_link_rx_func+9d (a3041d78, 0, a13f77) a13f7704 dls:i_dls_link_rx_common_promisc+3b (a3041d78, 0, a13f77) a13f77a8 dls:i_dls_link_rx_common+11c (a3041d78, 0, aa2283) a13f77c4 dls:i_dls_link_txloop+18 (a3041d78, aa228340) a13f77f0 mac:mac_txloop+92 (a3042ee8, aa228340) a13f780c dls:dls_tx+16 (a3026f80, abb08e80) a13f7828 dld:dld_tx_single+1e (a2bfdea8, abb08e80) a13f7840 dld:str_mdata_fastpath_put+60 (a2bfdea8, abb08e80) a13f78b0 ip:tcp_send_data+85c (aafda500, aafdfc68,) a13f791c ip:tcp_send+6f1 (aafdfc68, aafda500,) a13f79bc ip:tcp_wput_data+663 (aafda500, 0, 0) a13f7aa0 ip:tcp_rput_data+29b2 (aafda100, a70a13c0,) a13f7af8 ip:squeue_drain+1c3 (a1df0d00, 4, 4bb30e) a13f7b48 ip:squeue_enter_chain+5be (a1df0d00, abb16f40,) a13f7bdc ip:ip_input+731 (a2581754, a2afc020,) a13f7c78 dls:i_dls_link_rx_common+26e (a3041d78, a2afc020,) a13f7c90 dls:i_dls_link_rx_promisc+19 (a3041d78, a2afc020,) a13f7ccc mac:mac_rx+53 (a3042ee8, a2afc020,) a13f7d60 bge:bge_receive+598 (a2c78000, a2f26800) a13f7dac bge:bge_intr+30f (a2c78000, 0) syncing file systems... done dumping to /dev/dsk/c2d0s1, offset 1719074816, content: kernel Then I used mdb checking the crash dump file, and we found most of CPUs are IDLE while system paniced: ::cpuinfo -v ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 0 fec254f8 1b5 10 105 nono t-5676 a13f7de0 sched ||| RUNNING --+|+-- PIL THREAD READY | 10 a13fade0 EXISTS | 6 a13f7de0 ENABLE | - a11ccde0 (idle) | +-- PRI THREAD PROC 109 a13fade0 sched 60 a1b7ede0 sched 60 a1587de0 sched 60 a27a1de0 sched 59 a727f000 snoop ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 1 a1d6e200 1f10 -1 nono t-0a22a9de0 (idle) || RUNNING --++-- PRI THREAD PROC READY60 a24a3de0 sched QUIESCED EXISTS ENABLE ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 2 a1d6d180 1f00 -1 nono t-0a23f6de0 (idle) | RUNNING --+ READY QUIESCED EXISTS ENABLE ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 3 a1d6c100 1f00 -1 nono t-0a242fde0 (idle) | RUNNING --+ READY QUIESCED EXISTS ENABLE The CPU 1,2,3 status is QUIESCED, I think it must be disabled by deadman via a cross-call. On CPU0, a network interrupt thread a13f7de0 was interrupted by a clock(cyclic) interrupt: a13f7de0::findstack -v stack pointer for thread a13f7de0: a13f75a8 a13f75b8 _interrupt+0xe7() a13f7638 bcopy+0x39(abb11780) a13f765c copymsg+0x2e(aa228340) a13f7688 copymsgchain+0x23(aa228340) a13f76c0 i_dls_link_rx_func+0x9d(a3041d78, 0, a13f773c, aa228340, 1, 0) a13f7704 i_dls_link_rx_common_promisc+0x3b(a3041d78, 0, a13f773c, aa228340, 0, e669fcdc) a13f77a8 i_dls_link_rx_common+0x11c(a3041d78, 0, aa228340, e669fcdc) a13f77c4 i_dls_link_txloop+0x18(a3041d78, aa228340) a13f77f0 mac_txloop+0x92(a3042ee8, aa228340) a13f780c dls_tx+0x16(a3026f80, abb08e80) a13f7828 dld_tx_single+0x1e(a2bfdea8, abb08e80) a13f7840 str_mdata_fastpath_put+0x60(a2bfdea8, abb08e80) a13f78b0 tcp_send_data+0x85c(aafda500, aafdfc68, abb08e80) a13f791c tcp_send+0x6f1(aafdfc68, aafda500, 5a8, 34, 20, 0) a13f79bc tcp_wput_data+0x663(aafda500, 0, 0) a13f7aa0 tcp_rput_data+0x29b2(aafda100, a70a13c0, a1df0d00) a13f7af8 squeue_drain+0x1c3(a1df0d00, 4, 4bb30e5e, bd) a13f7b48
