hdparm -W /dev/xxx should show you On Mon, 19 Nov 2018 at 12:28 AM, Alex Litvak <alexander.v.lit...@gmail.com> wrote:
> All machines state the same. > > /opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -DskCache -Lall -a0 > > Adapter 0-VD 0(target id: 0): Disk Write Cache : Disk's Default > Adapter 0-VD 1(target id: 1): Disk Write Cache : Disk's Default > > I assume they are all on which is actually bad based on common sense. > > > https://notesbytom.wordpress.com/2016/10/21/dell-perc-megaraid-disk-cache-policy/ > > An I couldn't find how to confirm it if it is true but vendor wouldn't > ship drives with cache disabled. > > I am getting logs in the controller log which are not shown on other > servers > > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03407a0, localAddr e03407a0 > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03409e0, localAddr e03409e0 > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e0340c20, localAddr e0340c20 > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e0340e60, localAddr e0340e60 > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03410a0, localAddr e03410a0 > 11/18/18 8:21:55: C0:SysDma: localAddrPlb 50e03401a0, localAddr e03401a0 > > Not sure if it has any relation to the issue of latency but search > returned nothing substantial. > > On 11/18/2018 8:52 AM, Serkan Çoban wrote: > > I am not saying controller cache, you should check ssd disk caches. > > On Sun, Nov 18, 2018 at 11:40 AM Alex Litvak > > <alexander.v.lit...@gmail.com> wrote: > >> > >> All 3 nodes have this status for SSD mirror. Controller cache is on > for all 3. > >> > >> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache > if Bad BBU > >> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache > if Bad BBU > >> > >> > >> On 11/18/2018 12:45 AM, Serkan Çoban wrote: > >>> Does write cache on SSDs enabled on three servers? Can you check them? > >>> On Sun, Nov 18, 2018 at 9:05 AM Alex Litvak > >>> <alexander.v.lit...@gmail.com> wrote: > >>>> > >>>> Raid card for journal disks is Perc H730 (Megaraid), RAID 1, battery > back cache is on > >>>> > >>>> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache > if Bad BBU > >>>> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache > if Bad BBU > >>>> > >>>> I have 2 other nodes with older Perc H710 and similar SSDs with > slightly higher wear (6.3% vs 5.18%) but from observation they hardly hit > 1.5 ms on rear occasion > >>>> Cache, RAID, and battery situation is the same. > >>>> > >>>> On 11/17/2018 11:38 PM, Serkan Çoban wrote: > >>>>>> 10ms w_await for SSD is too much. How that SSD is connected to the > system? Any raid card installed on this system? What is the raid mode? > >>>>> On Sun, Nov 18, 2018 at 8:25 AM Alex Litvak > >>>>> <alexander.v.lit...@gmail.com> wrote: > >>>>>> > >>>>>> Here is another snapshot. I wonder if this write io wait is too big > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 23.00 0.00 336.00 > 29.22 0.34 14.74 0.00 14.74 2.87 6.60 > >>>>>> dm-15 0.00 0.00 0.00 16.00 0.00 200.00 > 25.00 0.01 0.75 0.00 0.75 0.75 1.20 > >>>>>> dm-16 0.00 0.00 0.00 17.00 0.00 276.00 > 32.47 0.25 14.94 0.00 14.94 3.35 5.70 > >>>>>> dm-17 0.00 0.00 0.00 17.00 0.00 252.00 > 29.65 0.32 18.65 0.00 18.65 4.00 6.80 > >>>>>> dm-18 0.00 0.00 0.00 15.00 0.00 152.00 > 20.27 0.25 16.80 0.00 16.80 4.07 6.10 > >>>>>> dm-19 0.00 0.00 0.00 13.00 0.00 152.00 > 23.38 0.21 15.92 0.00 15.92 4.85 6.30 > >>>>>> dm-20 0.00 0.00 0.00 20.00 0.00 248.00 > 24.80 0.27 13.60 0.00 13.60 3.25 6.50 > >>>>>> dm-21 0.00 0.00 0.00 17.00 0.00 188.00 > 22.12 0.27 16.00 0.00 16.00 3.59 6.10 > >>>>>> dm-22 0.00 0.00 0.00 20.00 0.00 156.00 > 15.60 0.11 5.55 0.00 5.55 2.95 5.90 > >>>>>> dm-24 0.00 0.00 0.00 8.00 0.00 56.00 > 14.00 0.12 14.62 0.00 14.62 4.75 3.80 > >>>>>> dm-25 0.00 0.00 0.00 19.00 0.00 200.00 > 21.05 0.21 10.89 0.00 10.89 2.74 5.20 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 11.00 0.00 136.00 > 24.73 0.11 9.73 0.00 9.73 1.82 2.00 > >>>>>> dm-15 0.00 0.00 0.00 12.00 0.00 136.00 > 22.67 0.04 3.75 0.00 3.75 1.08 1.30 > >>>>>> dm-16 0.00 0.00 0.00 9.00 0.00 104.00 > 23.11 0.09 10.44 0.00 10.44 2.44 2.20 > >>>>>> dm-17 0.00 0.00 0.00 5.00 0.00 160.00 > 64.00 0.02 4.00 0.00 4.00 4.00 2.00 > >>>>>> dm-18 0.00 0.00 0.00 5.00 0.00 52.00 > 20.80 0.03 5.80 0.00 5.80 3.60 1.80 > >>>>>> dm-19 0.00 0.00 0.00 10.00 0.00 104.00 > 20.80 0.08 7.90 0.00 7.90 2.10 2.10 > >>>>>> dm-20 0.00 0.00 0.00 9.00 0.00 132.00 > 29.33 0.10 11.22 0.00 11.22 2.56 2.30 > >>>>>> dm-21 0.00 0.00 0.00 6.00 0.00 68.00 > 22.67 0.07 12.33 0.00 12.33 3.83 2.30 > >>>>>> dm-22 0.00 0.00 0.00 3.00 0.00 20.00 > 13.33 0.01 3.67 0.00 3.67 3.67 1.10 > >>>>>> dm-24 0.00 0.00 0.00 4.00 0.00 24.00 > 12.00 0.07 18.00 0.00 18.00 5.25 2.10 > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 64.00 > 21.33 0.06 10.33 0.00 10.33 3.67 2.20 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 5.00 0.00 140.00 > 56.00 0.08 15.20 0.00 15.20 5.40 2.70 > >>>>>> dm-15 0.00 0.00 0.00 6.00 0.00 236.00 > 78.67 0.18 30.67 0.00 30.67 6.83 4.10 > >>>>>> dm-16 0.00 0.00 0.00 8.00 0.00 84.00 > 21.00 0.06 7.25 0.00 7.25 1.62 1.30 > >>>>>> dm-17 0.00 0.00 0.00 3.00 0.00 84.00 > 56.00 0.00 0.33 0.00 0.33 0.33 0.10 > >>>>>> dm-18 0.00 0.00 0.00 2.00 0.00 20.00 > 20.00 0.02 12.00 0.00 12.00 12.00 2.40 > >>>>>> dm-19 0.00 0.00 0.00 12.00 0.00 80.00 > 13.33 0.05 4.00 0.00 4.00 2.33 2.80 > >>>>>> dm-20 0.00 0.00 0.00 16.00 0.00 256.00 > 32.00 0.00 0.06 0.00 0.06 0.06 0.10 > >>>>>> dm-21 0.00 0.00 0.00 8.00 0.00 500.00 > 125.00 0.00 0.12 0.00 0.12 0.12 0.10 > >>>>>> dm-22 0.00 0.00 0.00 1.00 0.00 8.00 > 16.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-24 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-25 0.00 0.00 0.00 2.00 0.00 32.00 > 32.00 0.08 40.00 0.00 40.00 20.50 4.10 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 10.00 0.00 108.00 > 21.60 0.11 10.80 0.00 10.80 1.90 1.90 > >>>>>> dm-15 0.00 0.00 0.00 5.00 0.00 60.00 > 24.00 0.03 6.20 0.00 6.20 3.40 1.70 > >>>>>> dm-16 0.00 0.00 0.00 6.00 0.00 68.00 > 22.67 0.00 0.17 0.00 0.17 0.17 0.10 > >>>>>> dm-17 0.00 0.00 0.00 7.00 0.00 100.00 > 28.57 0.09 12.29 0.00 12.29 2.86 2.00 > >>>>>> dm-18 0.00 0.00 0.00 5.00 0.00 44.00 > 17.60 0.07 14.00 0.00 14.00 4.00 2.00 > >>>>>> dm-19 0.00 0.00 0.00 2.00 0.00 24.00 > 24.00 0.03 14.00 0.00 14.00 8.50 1.70 > >>>>>> dm-20 0.00 0.00 0.00 7.00 0.00 96.00 > 27.43 0.07 9.57 0.00 9.57 2.00 1.40 > >>>>>> dm-21 0.00 0.00 0.00 7.00 0.00 84.00 > 24.00 0.12 16.57 0.00 16.57 4.86 3.40 > >>>>>> dm-22 0.00 0.00 0.00 5.00 0.00 60.00 > 24.00 0.05 10.20 0.00 10.20 3.40 1.70 > >>>>>> dm-24 0.00 0.00 0.00 2.00 0.00 16.00 > 16.00 0.03 17.00 0.00 17.00 9.00 1.80 > >>>>>> dm-25 0.00 0.00 0.00 4.00 0.00 76.00 > 38.00 0.05 11.75 0.00 11.75 4.75 1.90 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 9.00 0.00 104.00 > 23.11 0.14 15.33 0.00 15.33 4.33 3.90 > >>>>>> dm-15 0.00 0.00 0.00 11.00 0.00 128.00 > 23.27 0.01 0.82 0.00 0.82 0.82 0.90 > >>>>>> dm-16 0.00 0.00 0.00 10.00 0.00 120.00 > 24.00 0.09 9.30 0.00 9.30 3.30 3.30 > >>>>>> dm-17 0.00 0.00 0.00 5.00 0.00 72.00 > 28.80 0.04 8.20 0.00 8.20 8.20 4.10 > >>>>>> dm-18 0.00 0.00 0.00 7.00 0.00 68.00 > 19.43 0.08 11.57 0.00 11.57 6.29 4.40 > >>>>>> dm-19 0.00 0.00 0.00 7.00 0.00 76.00 > 21.71 0.10 13.86 0.00 13.86 6.29 4.40 > >>>>>> dm-20 0.00 0.00 0.00 15.00 0.00 176.00 > 23.47 0.12 7.87 0.00 7.87 1.60 2.40 > >>>>>> dm-21 0.00 0.00 0.00 9.00 0.00 112.00 > 24.89 0.17 18.56 0.00 18.56 6.56 5.90 > >>>>>> dm-22 0.00 0.00 0.00 4.00 0.00 56.00 > 28.00 0.04 10.75 0.00 10.75 7.50 3.00 > >>>>>> dm-24 0.00 0.00 0.00 5.00 0.00 532.00 > 212.80 0.02 4.00 0.00 4.00 3.60 1.80 > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 60.00 > 20.00 0.11 19.00 0.00 19.00 7.17 4.30 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 9.00 0.00 112.00 > 24.89 0.30 33.22 0.00 33.22 4.33 3.90 > >>>>>> dm-15 0.00 0.00 0.00 8.00 0.00 96.00 > 24.00 0.08 9.88 0.00 9.88 2.62 2.10 > >>>>>> dm-16 0.00 0.00 0.00 5.00 0.00 92.00 > 36.80 0.01 1.80 0.00 1.80 1.80 0.90 > >>>>>> dm-17 0.00 0.00 0.00 4.00 0.00 104.00 > 52.00 0.04 10.00 0.00 10.00 10.00 4.00 > >>>>>> dm-18 0.00 0.00 0.00 9.00 0.00 92.00 > 20.44 0.11 12.00 0.00 12.00 4.44 4.00 > >>>>>> dm-19 0.00 0.00 0.00 7.00 0.00 64.00 > 18.29 0.10 14.43 0.00 14.43 4.00 2.80 > >>>>>> dm-20 0.00 0.00 0.00 7.00 0.00 84.00 > 24.00 0.10 14.00 0.00 14.00 3.71 2.60 > >>>>>> dm-21 0.00 0.00 0.00 2.00 0.00 16.00 > 16.00 0.08 39.50 0.00 39.50 20.50 4.10 > >>>>>> dm-22 0.00 0.00 0.00 5.00 0.00 36.00 > 14.40 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-24 0.00 0.00 0.00 5.00 0.00 52.00 > 20.80 0.09 17.20 0.00 17.20 5.40 2.70 > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 48.00 > 16.00 0.10 16.00 0.00 16.00 5.50 3.30 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 5.00 0.00 48.00 > 19.20 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-15 0.00 0.00 0.00 5.00 0.00 44.00 > 17.60 0.01 3.00 0.00 3.00 3.00 1.50 > >>>>>> dm-16 0.00 0.00 0.00 3.00 0.00 32.00 > 21.33 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-17 0.00 0.00 0.00 4.00 0.00 40.00 > 20.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-18 0.00 0.00 0.00 3.00 0.00 32.00 > 21.33 0.04 13.00 0.00 13.00 6.67 2.00 > >>>>>> dm-19 0.00 0.00 0.00 3.00 0.00 28.00 > 18.67 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-20 0.00 0.00 0.00 6.00 0.00 64.00 > 21.33 0.11 17.67 0.00 17.67 3.17 1.90 > >>>>>> dm-21 0.00 0.00 0.00 1.00 0.00 12.00 > 24.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-22 0.00 0.00 0.00 1.00 0.00 12.00 > 24.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> dm-24 0.00 0.00 0.00 3.00 0.00 20.00 > 13.33 0.00 0.33 0.00 0.33 0.33 0.10 > >>>>>> dm-25 0.00 0.00 0.00 6.00 0.00 52.00 > 17.33 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 18.00 0.00 236.00 > 26.22 0.42 23.28 0.00 23.28 3.72 6.70 > >>>>>> dm-15 0.00 0.00 0.00 15.00 0.00 240.00 > 32.00 0.12 8.27 0.00 8.27 2.00 3.00 > >>>>>> dm-16 0.00 0.00 0.00 10.00 0.00 120.00 > 24.00 0.23 22.80 0.00 22.80 6.50 6.50 > >>>>>> dm-17 0.00 0.00 0.00 6.00 0.00 68.00 > 22.67 0.06 9.83 0.00 9.83 5.00 3.00 > >>>>>> dm-18 0.00 0.00 0.00 12.00 0.00 152.00 > 25.33 0.10 8.17 0.00 8.17 3.17 3.80 > >>>>>> dm-19 0.00 0.00 0.00 12.00 0.00 136.00 > 22.67 0.14 11.58 0.00 11.58 3.42 4.10 > >>>>>> dm-20 0.00 0.00 0.00 19.00 0.00 296.00 > 31.16 0.20 10.63 0.00 10.63 1.84 3.50 > >>>>>> dm-21 0.00 0.00 0.00 14.00 0.00 132.00 > 18.86 0.17 12.14 0.00 12.14 3.71 5.20 > >>>>>> dm-22 0.00 0.00 0.00 10.00 0.00 84.00 > 16.80 0.12 12.00 0.00 12.00 6.60 6.60 > >>>>>> dm-24 0.00 0.00 0.00 5.00 0.00 40.00 > 16.00 0.10 19.40 0.00 19.40 9.80 4.90 > >>>>>> dm-25 0.00 0.00 0.00 9.00 0.00 152.00 > 33.78 0.08 9.33 0.00 9.33 3.44 3.10 > >>>>>> > >>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>> dm-14 0.00 0.00 0.00 8.00 0.00 112.00 > 28.00 0.08 9.75 0.00 9.75 3.88 3.10 > >>>>>> dm-15 0.00 0.00 0.00 14.00 0.00 156.00 > 22.29 0.13 9.43 0.00 9.43 2.71 3.80 > >>>>>> dm-16 0.00 0.00 0.00 12.00 0.00 132.00 > 22.00 0.11 8.83 0.00 8.83 2.17 2.60 > >>>>>> dm-17 0.00 0.00 0.00 8.00 0.00 116.00 > 29.00 0.08 10.12 0.00 10.12 4.25 3.40 > >>>>>> dm-18 0.00 0.00 0.00 8.00 0.00 100.00 > 25.00 0.08 9.62 0.00 9.62 5.00 4.00 > >>>>>> dm-19 0.00 0.00 0.00 20.00 0.00 212.00 > 21.20 0.25 12.75 0.00 12.75 2.60 5.20 > >>>>>> dm-20 0.00 0.00 0.00 16.00 0.00 168.00 > 21.00 0.04 2.69 0.00 2.69 0.81 1.30 > >>>>>> dm-21 0.00 0.00 0.00 5.00 0.00 48.00 > 19.20 0.05 10.80 0.00 10.80 7.80 3.90 > >>>>>> dm-22 0.00 0.00 0.00 5.00 0.00 52.00 > 20.80 0.03 6.80 0.00 6.80 3.40 1.70 > >>>>>> dm-24 0.00 0.00 0.00 8.00 0.00 60.00 > 15.00 0.09 11.75 0.00 11.75 2.25 1.80 > >>>>>> dm-25 0.00 0.00 0.00 12.00 0.00 160.00 > 26.67 0.06 5.08 0.00 5.08 1.25 1.50 > >>>>>> > >>>>>> On 11/17/2018 10:19 PM, Alex Litvak wrote: > >>>>>>> I stand corrected, I looked at the device iostat, but it was > partitioned. Here is a more correct picture of what is going on now. > >>>>>>> > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>>> dm-14 0.00 0.00 0.00 19.00 0.00 4116.00 > 433.26 0.01 0.68 0.00 0.68 0.05 0.10 > >>>>>>> dm-15 0.00 0.00 0.00 35.00 0.00 8224.00 > 469.94 0.03 0.86 0.00 0.86 0.06 0.20 > >>>>>>> dm-16 0.00 0.00 0.00 53.00 0.00 12428.00 > 468.98 0.11 2.04 0.00 2.04 0.17 0.90 > >>>>>>> dm-17 0.00 0.00 0.00 43.00 0.00 8344.00 > 388.09 0.09 2.14 0.00 2.14 0.42 1.80 > >>>>>>> dm-18 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-19 0.00 0.00 0.00 75.00 0.00 16824.00 > 448.64 0.08 1.11 0.00 1.11 0.08 0.60 > >>>>>>> dm-20 0.00 0.00 0.00 70.00 0.00 16452.00 > 470.06 0.06 0.90 0.00 0.90 0.09 0.60 > >>>>>>> dm-21 0.00 0.00 0.00 18.00 0.00 4112.00 > 456.89 0.02 1.00 0.00 1.00 0.11 0.20 > >>>>>>> dm-22 0.00 0.00 0.00 53.00 0.00 12324.00 > 465.06 0.06 0.70 0.00 0.70 0.08 0.40 > >>>>>>> dm-24 0.00 0.00 0.00 18.00 0.00 4272.00 > 474.67 0.02 1.06 0.00 1.06 0.17 0.30 > >>>>>>> dm-25 0.00 0.00 0.00 74.00 0.00 16916.00 > 457.19 0.09 1.26 0.00 1.26 0.18 1.30 > >>>>>>> > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>>> dm-14 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-15 0.00 0.00 0.00 17.00 0.00 4108.00 > 483.29 0.02 1.00 0.00 1.00 0.06 0.10 > >>>>>>> dm-16 0.00 0.00 0.00 34.00 0.00 8208.00 > 482.82 0.03 1.00 0.00 1.00 0.06 0.20 > >>>>>>> dm-17 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-18 0.00 0.00 0.00 36.00 0.00 8220.00 > 456.67 0.05 1.33 0.00 1.33 0.08 0.30 > >>>>>>> dm-19 0.00 0.00 0.00 1.00 0.00 8.00 > 16.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-20 0.00 0.00 0.00 36.00 0.00 8288.00 > 460.44 0.05 1.42 0.00 1.42 0.08 0.30 > >>>>>>> dm-21 0.00 0.00 0.00 34.00 0.00 8208.00 > 482.82 0.03 1.00 0.00 1.00 0.06 0.20 > >>>>>>> dm-22 0.00 0.00 0.00 18.00 0.00 4128.00 > 458.67 0.04 3.22 0.00 3.22 0.17 0.30 > >>>>>>> dm-24 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-25 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>>> dm-14 0.00 0.00 0.00 20.00 0.00 4032.00 > 403.20 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-15 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-16 0.00 0.00 0.00 1.00 0.00 20.00 > 40.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-17 0.00 0.00 0.00 4.00 0.00 28.00 > 14.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-18 0.00 0.00 0.00 3.00 0.00 36.00 > 24.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-19 0.00 0.00 0.00 2.00 0.00 20.00 > 20.00 0.01 2.50 0.00 2.50 2.50 0.50 > >>>>>>> dm-20 0.00 0.00 0.00 6.00 0.00 96.00 > 32.00 0.02 3.33 0.00 3.33 2.00 1.20 > >>>>>>> dm-21 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-22 0.00 0.00 0.00 2.00 0.00 32.00 > 32.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-24 0.00 0.00 0.00 22.00 0.00 4184.00 > 380.36 0.10 4.59 0.00 4.59 0.95 2.10 > >>>>>>> dm-25 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> > >>>>>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s > avgrq-sz avgqu-sz await r_await w_await svctm %util > >>>>>>> dm-14 0.00 0.00 0.00 8.00 0.00 1928.00 > 482.00 0.01 1.00 0.00 1.00 0.12 0.10 > >>>>>>> dm-15 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-16 0.00 0.00 0.00 3.00 0.00 312.00 > 208.00 0.00 0.33 0.00 0.33 0.33 0.10 > >>>>>>> dm-17 0.00 0.00 0.00 18.00 0.00 4264.00 > 473.78 0.03 1.67 0.00 1.67 0.11 0.20 > >>>>>>> dm-18 0.00 0.00 0.00 17.00 0.00 4104.00 > 482.82 0.03 1.82 0.00 1.82 0.12 0.20 > >>>>>>> dm-19 0.00 0.00 0.00 18.00 0.00 4112.00 > 456.89 0.02 1.06 0.00 1.06 0.11 0.20 > >>>>>>> dm-20 0.00 0.00 0.00 32.00 0.00 4308.00 > 269.25 0.03 0.81 0.00 0.81 0.34 1.10 > >>>>>>> dm-21 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-22 0.00 0.00 0.00 8.00 0.00 540.00 > 135.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> dm-24 0.00 0.00 0.00 35.00 0.00 8228.00 > 470.17 0.03 0.97 0.00 0.97 0.06 0.20 > >>>>>>> dm-25 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 11/17/2018 5:27 PM, John Petrini wrote: > >>>>>>>> The iostat isn't very helpful because there are not many writes. > I'd recommend disabling cstates entirely, not sure it's your problem but > it's good practice and if your cluster goes as idle as your > >>>>>>>> iostat suggests it could be the culprit. > >>>>>>>> > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> ceph-users mailing list > >>>>>>>> ceph-users@lists.ceph.com > >>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> ceph-users mailing list > >>>>>> ceph-users@lists.ceph.com > >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>>> > >>>> > >>>> _______________________________________________ > >>>> ceph-users mailing list > >>>> ceph-users@lists.ceph.com > >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> _______________________________________________ > >>> ceph-users mailing list > >>> ceph-users@lists.ceph.com > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
