Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread w...@vandenberge.us
Here is the output for "iostat -xn" for the smaller of the two servers/pools (80TB). The three c8 drives with the 0 across the board are the hot spares. Nothing jumps out at me. The c5 drives are boot (mirror), ZIL (mirror) and L2ARC (stripe) r/sw/s kr/s kw/s wait actv wsvc_t asvc_t

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread Willem R. van den Berge
Here is the output for "iostat -xn" for the smaller of the two servers/pools (80TB). The three c8 drives with the 0 across the board are the hot spares. Nothing jumps out at me. The c5 drives are boot (mirror), ZIL (mirror) and L2ARC (stripe) r/sw/s kr/s kw/s wait actv wsvc_t asvc_t

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread Saso Kiselkov
On 05/07/2013 21:00, w...@vandenberge.us wrote: > Latencytop reports the following continuously for the pool process and it > doesn’t change significantly under load, which looks ok to me: > > genunix`cv_wait genunix`taskq_threa144919.6 msec6.2 sec > 99.3 % > > Wait for availab

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread w...@vandenberge.us
Latencytop reports the following continuously for the pool process and it doesn’t change significantly under load, which looks ok to me: genunix`cv_wait genunix`taskq_threa144919.6 msec6.2 sec 99.3 % Wait for available CPU 14618 44.9 usec9.8 msec 0.

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread w...@vandenberge.us
"iostat -Exn" reports"Soft Errors: 0 Hard Errors: 0 Transport Errors: 0" across the board. Going to try latencytop next. We're not deduping or compressing. Wim > On July 5, 2013 at 2:28 PM Lucas Van Tol wrote: > > > Have you tried looking with 'latencytop'? > pkg install diagnostic/latencytop

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread Saso Kiselkov
On 05/07/2013 19:09, Irek Szczesniak wrote: > On Fri, Jul 5, 2013 at 8:00 PM, Saso Kiselkov wrote: >> On 05/07/2013 17:08, w...@vandenberge.us wrote: >>> Good morning, >>> >>> I have a weird problem with two of the 15+ OpenSolaris storage servers in >>> our >>> environment. All the Nearline serve

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread Lucas Van Tol
Have you tried looking with 'latencytop'? pkg install diagnostic/latencytop latencytop and arrow over to the zpool process for your data pool? That may help you track down what is specifically slowing you down; for example if you see something about space map loading, you need echo 'metaslab_debu

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread Irek Szczesniak
On Fri, Jul 5, 2013 at 8:00 PM, Saso Kiselkov wrote: > On 05/07/2013 17:08, w...@vandenberge.us wrote: >> Good morning, >> >> I have a weird problem with two of the 15+ OpenSolaris storage servers in our >> environment. All the Nearline servers are essentially the same. Supermicro >> X9DR3-F based

Re: [OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread Saso Kiselkov
On 05/07/2013 17:08, w...@vandenberge.us wrote: > Good morning, > > I have a weird problem with two of the 15+ OpenSolaris storage servers in our > environment. All the Nearline servers are essentially the same. Supermicro > X9DR3-F based server, Dual E5-2609's, 64GB memory, Dual 10Gb SFP+ NICs, L

[OpenIndiana-discuss] Sudden ZFS performance issue

2013-07-05 Thread w...@vandenberge.us
Good morning, I have a weird problem with two of the 15+ OpenSolaris storage servers in our environment. All the Nearline servers are essentially the same. Supermicro X9DR3-F based server, Dual E5-2609's, 64GB memory, Dual 10Gb SFP+ NICs, LSI 9200-8e HBA, Supermicro CSE-826E26-R1200LPB storage arr