Vladislav Bolkhovitin wrote:
Cameron Harr wrote:
Vladislav Bolkhovitin wrote:
** Sometimes the benchmark "zombied" (process doing no work, but
process can't be killed) after running a certain amount of time.
However, it wasn't repeatable in a reliable way, so I mark that
this particular run has zombied before.
That means that there is a bug somewhere. Usually such bugs are
found in few hours of code auditing (srpt driver is pretty simple)
or by using kernel debug facilities (example diff to .config
attached). I personally always prefer put my effort on fixing real
things, not inventing various workarounds, like srpt_thread in this
case.
So I would:
1. Completely remove srpt thread and all related code. It doesn't do
anything, which can't be done in SIRQ context (tasklet)
2. Audit the code to check if it does any action, which it
shouldn't do on SIRQ and fix it. This step isn't required, but
usually it saves a lot of time of puzzled debugging in the future.
3. Change in srpt_handle_rdma_comp() and srpt_handle_new_iu()
SCST_CONTEXT_THREAD to SCST_CONTEXT_DIRECT_ATOMIC.
I'm assuming you didn't want me to implement this change this time, correct?
I also changed it in srpt_handle_err_comp()
Then I would run the problematic tests (heavy tpc-h workload, e.g.)
on debug kernel and fix found problems.
Anyway, Cameron, can you get the latest code from SCST trunk and try
with it? It was recently updated. Also please add the case with
changes from (3) above.
This is all with version 1.0.1 of SCST (v532).
In my fio test, I do runs with srpt thread=1 and then =0. When it was
set to zero during the test, I got many errors printed out by FIO,
and the target eventually crashed. This is the first part of a long
call trace.
NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in: ib_srpt(U) scst_vdisk(U) scst(U) fio_driver(PU)
fio_port(PU) autofs4 hidp rfcomm l2cap bluetooth sunrpc ib_ipoib
mlx4_ib ib_cm ib_sa ib_mad ib_core ipv6 xfrm_nalgo crypto_api
nls_utf8 hfsplus dm_mirror dm_multipath dm_mod video sbs backlight
i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp
parport i2c_i801 shpchp i2c_core e1000e mlx4_core i5000_edac edac_mc
pcspkr ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd
ehci_hcd
Pid: 25732, comm: scsi_tgt0 Tainted: P 2.6.18-92.1.13.el5 #1
RIP: 0010:[<ffffffff80064bcb>] [<ffffffff80064bcb>]
.text.lock.spinlock+0x29/0x30
RSP: 0018:ffffffff80418a88 EFLAGS: 00000086
RAX: ffff810785307fd8 RBX: ffffffff884e68a0 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff884e68a0
RBP: ffffffff884e62a0 R08: ffff810790926900 R09: ffff8107909268e8
R10: 0000000000000018 R11: ffffffff884fcab3 R12: 0000000000000001
R13: 0000000000000001 R14: 0000000000000000 R15: ffff8107f0f374c0
FS: 0000000000000000(0000) GS:ffffffff803a0000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000037bc0986d0 CR3: 0000000000201000 CR4: 00000000000006e0
Process scsi_tgt0 (pid: 25732, threadinfo ffff810785306000, task
ffff810810852100)
Stack: 0000000000000000 ffffffff884c509d ffff8107909268e8
ffff810790926900
00000002071dd688 0000020000000220 0000000000000200 00000000da984c08
0000000000000000 ffff8107909267f0 ffff810806ceee20 0000000000000001
Call Trace:
<IRQ> [<ffffffff884c509d>] :scst:sgv_pool_alloc+0x10c/0x5d3
[<ffffffff884c1f85>] :scst:scst_alloc_space+0x5b/0x106
[<ffffffff884bdc90>] :scst:scst_process_active_cmd+0x4fc/0x131c
[<ffffffff884bee46>] :scst:scst_cmd_init_done+0x17f/0x3ef
[<ffffffff884fb1ff>] :ib_srpt:srpt_handle_new_iu+0x281/0x4e7
[<ffffffff8835ec3d>] :mlx4_ib:mlx4_ib_free_srq_wqe+0x27/0x4f
[<ffffffff883591da>] :mlx4_ib:get_sw_cqe+0x12/0x30
[<ffffffff88359c97>] :mlx4_ib:mlx4_ib_poll_cq+0x432/0x48f
[<ffffffff884fcc43>] :ib_srpt:srpt_completion+0x190/0x250
[<ffffffff8811aa5b>] :mlx4_core:mlx4_eq_int+0x3b/0x26f
[<ffffffff8811ac9e>] :mlx4_core:mlx4_msi_x_interrupt+0xf/0x17
According to this trace, Vu was incorrect when he wrote that
srpt_handle_new_iu called on tasklet context. It at least sometimes
called from IRQ context. Try with the attached patch. It's against the
latest trunk.
I tried with the latest scst and srpt as of this morning. Previously, I
had used srpt-1.0.0. The following results are with BLOCKIO, and I'll
have a NULLIO in a bit. You can see from here that I don't hang any
more, but the srpt thread=0 are a little lower.
As before this run was done with ioengine=libaio and iodepth=16. I
pretty much always get significantly better performance with libaio than
with sync or other engines. Also, the iodepth setting tended to give me
better results.
----------------------------------------------
type=randwrite bs=512 drives=1 scst_threads=1 srptthread=1 iops=67073.48
type=randwrite bs=4k drives=1 scst_threads=1 srptthread=1 iops=54876.82
type=randwrite bs=512 drives=2 scst_threads=1 srptthread=1 iops=74858.00
type=randwrite bs=4k drives=2 scst_threads=1 srptthread=1 iops=75357.15
type=randwrite bs=512 drives=3 scst_threads=1 srptthread=1 iops=83257.72
type=randwrite bs=4k drives=3 scst_threads=1 srptthread=1 iops=82186.79
type=randwrite bs=512 drives=1 scst_threads=2 srptthread=1 iops=59908.06
type=randwrite bs=4k drives=1 scst_threads=2 srptthread=1 iops=50982.91
type=randwrite bs=512 drives=2 scst_threads=2 srptthread=1 iops=99243.07
type=randwrite bs=4k drives=2 scst_threads=2 srptthread=1 iops=79670.62
type=randwrite bs=512 drives=3 scst_threads=2 srptthread=1 iops=102898.37
type=randwrite bs=4k drives=3 scst_threads=2 srptthread=1 iops=92248.25
type=randwrite bs=512 drives=1 scst_threads=3 srptthread=1 iops=63086.77
type=randwrite bs=4k drives=1 scst_threads=3 srptthread=1 iops=53020.41
type=randwrite bs=512 drives=2 scst_threads=3 srptthread=1 iops=95990.06
type=randwrite bs=4k drives=2 scst_threads=3 srptthread=1 iops=77487.26
type=randwrite bs=512 drives=3 scst_threads=3 srptthread=1 iops=105945.85
type=randwrite bs=4k drives=3 scst_threads=3 srptthread=1 iops=95389.01
type=randwrite bs=512 drives=1 scst_threads=1 srptthread=0 iops=50299.36
type=randwrite bs=4k drives=1 scst_threads=1 srptthread=0 iops=48070.11
type=randwrite bs=512 drives=2 scst_threads=1 srptthread=0 iops=54017.21
type=randwrite bs=4k drives=2 scst_threads=1 srptthread=0 iops=50407.20
type=randwrite bs=512 drives=3 scst_threads=1 srptthread=0 iops=55822.11
type=randwrite bs=4k drives=3 scst_threads=1 srptthread=0 iops=50447.82
type=randwrite bs=512 drives=1 scst_threads=2 srptthread=0 iops=60672.48
type=randwrite bs=4k drives=1 scst_threads=2 srptthread=0 iops=48811.93
type=randwrite bs=512 drives=2 scst_threads=2 srptthread=0 iops=81919.87
type=randwrite bs=4k drives=2 scst_threads=2 srptthread=0 iops=72912.99
type=randwrite bs=512 drives=3 scst_threads=2 srptthread=0 iops=91036.45
type=randwrite bs=4k drives=3 scst_threads=2 srptthread=0 iops=88994.63
type=randwrite bs=512 drives=1 scst_threads=3 srptthread=0 iops=58929.21
type=randwrite bs=4k drives=1 scst_threads=3 srptthread=0 iops=48698.90
type=randwrite bs=512 drives=2 scst_threads=3 srptthread=0 iops=83967.58
type=randwrite bs=4k drives=2 scst_threads=3 srptthread=0 iops=73932.36
type=randwrite bs=512 drives=3 scst_threads=3 srptthread=0 iops=96686.46
type=randwrite bs=4k drives=3 scst_threads=3 srptthread=0 iops=88689.27
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
