On Wed, Apr 15, 2009 at 4:54 PM, Cathy Zhou <[email protected]> wrote:
>
>> I'm running snv_108 I see something that looks a lot like 6545725,
>> which was supposedly fixed in snv_105. When I search the history via
>> opengrok and mercurial I see no mention of 6545725. The hang may be
>> similar to 6799144 as well.
>>
> It looks like some pending DLPI/ioctl operation is stuck somewhere. You will
> have to go though the thread list to see anything suspicious. Please try to
> start with running "::stacks" in mdb and see any stack suspicious. Pay
> attention to the functions start with "mac_", "dls_", "dld_" or "vsw_".
::stacks doesn't work for me, but ::threadlist -v seems to do the trick...
I see several of these, which I assume are just waiting for work...
000002a100467ca0 185ee40 0 0 99 3002edbafa8
PC: cv_wait+0x3c THREAD: mac_soft_ring_worker()
stack pointer for thread 2a100467ca0: 2a100467111
[ 000002a100467111 cv_wait+0x3c() ]
mac_soft_ring_worker+0x210()
thread_start+4()
000002a101cb1ca0 185ee40 0 0 99 60017ae32e8
PC: cv_wait+0x3c THREAD: mac_srs_worker()
stack pointer for thread 2a101cb1ca0: 2a101cb1111
[ 000002a101cb1111 cv_wait+0x3c() ]
mac_srs_worker+0x204()
thread_start+4()
000002a100c0bca0 185ee40 0 0 60 6001848b7f8
PC: cv_wait+0x3c TASKQ: vsw_vsw_taskq0
stack pointer for thread 2a100c0bca0: 2a100c0b111
[ 000002a100c0b111 cv_wait+0x3c() ]
taskq_thread+0x10c()
thread_start+4()
000002a100517ca0 185ee40 0 0 60 60021aafca0
PC: cv_wait+0x3c THREAD: i_mac_notify_thread()
stack pointer for thread 2a100517ca0: 2a100517111
[ 000002a100517111 cv_wait+0x3c() ]
i_mac_notify_thread+0x90()
thread_start+4()
000002a100de9ca0 185ee40 0 0 99 6002829bb20
PC: cv_wait+0x3c THREAD: vsw_ldc_tx_worker()
stack pointer for thread 2a100de9ca0: 2a100de9111
[ 000002a100de9111 cv_wait+0x3c() ]
vsw_ldc_tx_worker+0xa0()
thread_start+4()
> ::threadlist -v ! egrep -e 'cv_wait\+0x3c.*(mac|dls|dld|vsw)_' | sort |uniq -c
| sort -n
1 PC: cv_wait+0x3c TASKQ: vsw_vsw_taskq0
1 PC: cv_wait+0x3c TASKQ: vsw_vsw_taskq1
1 PC: cv_wait+0x3c THREAD: dld_taskq_dispatch()
2 PC: cv_wait+0x3c THREAD: vsw_ldc_rx_worker()
2 PC: cv_wait+0x3c THREAD: vsw_ldc_tx_worker()
10 PC: cv_wait+0x3c THREAD: i_mac_notify_thread()
18 PC: cv_wait+0x3c THREAD: mac_srs_worker()
216 PC: cv_wait+0x3c THREAD: mac_soft_ring_worker()
Then there is this...
000002a1001c7ca0 185ee40 0 0 60 6002ba55940
PC: cv_wait+0x3c TASKQ: system_taskq
stack pointer for thread 2a1001c7ca0: 2a1001c6801
[ 000002a1001c6801 cv_wait+0x3c() ]
mac_flow_wait+0x50()
mac_rx_srs_quiesce+0xb8()
mac_rx_classify_flow_quiesce+0x2c()
mac_rx_client_quiesce+0x20()
mac_rx_set+0x10()
vsw_set_if_hw_addr+0x124()
vsw_mac_client_init+0x64()
vsw_m_start+0x4c()
mac_start+0x20()
mac_promisc_add+0x14()
dls_promisc+0x48()
proto_promiscon_req+0x100()
dld_wput_nondata_task+0x70()
taskq_d_thread+0x9c()
thread_start+4()
000002a100157ca0 185ee40 0 0 60 60013262a54
PC: cv_wait+0x3c TASKQ: system_taskq
stack pointer for thread 2a100157ca0: 2a100156e41
[ 000002a100156e41 cv_wait+0x3c() ]
i_mac_perim_enter+0x5c()
mac_perim_enter_by_mh+4()
ioc_fast+0x128()
dld_wput_nondata_task+0x80()
taskq_thread+0x19c()
thread_start+4()
Here's the thread that I mentioned initially...
0000030008303720 60027e438f8 60026718fa8 3 1 60017ace550
PC: cv_wait+0x3c CMD: snoop -d vsw0
stack pointer for thread 30008303720: 2a100506bf1
[ 000002a100506bf1 cv_wait+0x3c() ]
dld_close+0x2c()
qdetach+0x8c()
strclose+0x36c()
device_close+0x84()
spec_close+0x13c()
fop_close+0x48()
closef+0x50()
closeandsetf+0x348()
close+8()
syscall_trap32+0xcc()
And the mutex seems to be held by....
> 000002a100506bf1::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
000002a100506bf1 adapt 2a1005068d900 - - no
And a stuck ifconfig -a...
0000030007fcd240 600235b44a8 600274f1370 3 1 60013262a54
PC: cv_wait+0x3c CMD: /usr/sbin/ifconfig -a
stack pointer for thread 30007fcd240: 2a1024da361
[ 000002a1024da361 cv_wait+0x3c() ]
i_mac_perim_enter+0x5c()
mac_perim_enter_by_mh+4()
mac_perim_enter_by_macname+0x2c()
dls_devnet_open+0x68()
devnet_create_rvp+0xc()
devnet_lookup+0x130()
fop_lookup+0x104()
lookuppnvp+0x370()
lookuppnat+0x10c()
lookupnameat+0x5c()
vn_openat+0x104()
copen+0x3a0()
syscall_trap32+0xcc()
> 000002a1024da361::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
000002a1024da361 adapt 2a1024dab7800 - - no
My mdb-foo is weak and doesn't allow me to get much beyond this.
Thanks for the quick reply.
--
Mike Gerdts
http://mgerdts.blogspot.com/
_______________________________________________
networking-discuss mailing list
[email protected]
