Hi Charles, Looks to me like all of the drives are performing roughly the same – there’s certainly not 1 that sticks out (also note this is happening on all 23 nodes in the cluster).
Thanks! Kelly [root@r1k1.kafka.log10.blackberry sys] # cat /proc/mdstat Personalities : [raid10] md127 : active raid10 sdc[2] sdh[7] sdb[1] sdf[5] sde[13] sdg[12] sdj[9] sdk[10] sda[0] sdl[11] sdd[3] sdi[8] 23441323008 blocks super 1.2 512K chunks 2 near-copies [12/12] [UUUUUUUUUUUU] [>....................] check = 0.0% (618944/23441323008) finish=108288.4min speed=3607K/sec unused devices: <none> [root@r1k1.kafka.log10.blackberry sys] # iostat -xdmc 1 10 Linux 3.10.0-327.18.2.el7.x86_64 (r1k1.kafka.log10.blackberry) 05/26/16 _x86_64_ (32 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 12.76 0.07 2.48 0.16 0.00 84.53 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 0.01 0.56 0.26 26.44 0.06 11.30 871.39 9.67 362.22 5.14 365.71 6.68 17.83 sdk 0.01 0.56 0.26 26.56 0.06 11.30 867.70 9.53 355.45 5.05 358.84 6.58 17.65 sdc 0.01 0.46 0.26 26.34 0.06 11.29 874.67 9.73 365.89 4.86 369.38 6.81 18.11 sdd 0.01 0.46 0.20 26.34 0.07 11.29 876.98 10.40 391.99 5.33 394.93 7.17 19.02 sda 0.01 0.49 0.26 26.53 0.06 11.29 868.24 9.48 353.91 4.96 357.36 6.57 17.61 sdj 0.01 0.56 0.20 26.44 0.07 11.30 873.73 10.04 376.87 5.48 379.68 6.91 18.40 sdl 0.01 0.56 0.20 26.56 0.07 11.30 869.99 9.77 365.16 5.92 367.92 6.72 17.99 sdh 0.01 0.57 0.21 26.79 0.07 11.30 862.30 9.65 357.60 5.27 360.31 6.63 17.90 sde 0.01 0.47 0.26 26.13 0.06 11.29 881.38 10.60 401.47 6.62 405.41 7.35 19.41 sdf 0.01 0.47 0.20 26.13 0.07 11.29 883.71 9.53 361.85 5.24 364.64 6.73 17.73 sdg 0.01 0.57 0.26 26.79 0.06 11.30 859.99 10.15 375.20 5.26 378.82 6.86 18.57 sdb 0.01 0.49 0.20 26.53 0.07 11.29 870.69 9.85 368.48 5.35 371.23 6.79 18.15 md127 0.00 0.00 2.51 156.82 0.77 67.77 881.06 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 25.51 0.03 4.37 1.05 0.00 69.04 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 0.00 1.00 8.00 30.00 0.50 14.18 791.16 1.06 28.03 0.50 35.37 6.97 26.50 sdk 0.00 0.00 8.00 30.00 0.50 14.52 809.47 0.93 24.32 0.00 30.80 7.87 29.90 sdc 0.00 1.00 9.00 32.00 0.56 15.21 787.90 1.13 27.54 0.67 35.09 6.90 28.30 sdd 0.00 1.00 10.00 32.00 0.62 15.21 772.19 1.29 30.69 0.70 40.06 6.76 28.40 sda 0.00 0.00 8.00 38.00 0.50 15.54 714.09 1.40 30.35 0.38 36.66 7.91 36.40 sdj 0.00 1.00 8.00 30.00 0.50 14.18 791.16 1.05 27.68 0.50 34.93 7.00 26.60 sdl 0.00 0.00 8.00 30.00 0.50 14.52 809.47 0.90 23.61 0.25 29.83 7.66 29.10 sdh 0.00 1.00 13.00 34.00 0.81 14.11 650.04 1.17 24.98 0.31 34.41 6.60 31.00 sde 0.00 0.00 16.00 31.00 1.00 14.54 676.94 1.20 25.45 0.31 38.42 7.13 33.50 sdf 0.00 0.00 16.00 31.00 1.00 14.54 676.94 1.19 25.38 0.31 38.32 5.57 26.20 sdg 0.00 1.00 13.00 34.00 0.81 14.11 650.04 1.22 25.98 0.31 35.79 6.70 31.50 sdb 0.00 0.00 8.00 38.00 0.50 15.54 714.09 1.31 28.41 0.25 34.34 8.02 36.90 md127 0.00 0.00 0.00 198.00 0.00 86.59 895.60 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 21.31 0.00 2.99 0.00 0.00 75.69 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 0.00 0.00 8.00 7.00 0.50 3.50 546.13 0.13 8.47 6.25 11.00 8.47 12.70 sdk 0.00 0.00 8.00 8.00 0.50 3.98 574.00 0.16 9.88 1.00 18.75 10.06 16.10 sdc 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.12 7.25 0.62 13.88 7.25 11.60 sdd 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.12 7.44 0.50 14.38 7.44 11.90 sda 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.13 8.00 0.50 15.50 8.00 12.80 sdj 0.00 0.00 8.00 7.00 0.50 3.50 546.13 0.18 12.20 9.25 15.57 12.20 18.30 sdl 0.00 0.00 8.00 9.00 0.50 4.48 600.47 0.11 6.94 1.00 12.22 6.59 11.20 sdh 0.00 0.00 8.00 9.00 0.50 3.51 482.82 0.10 6.12 0.50 11.11 6.12 10.40 sde 0.00 0.00 8.00 9.00 0.50 4.00 542.59 0.16 9.65 0.25 18.00 9.65 16.40 sdf 0.00 0.00 8.00 9.00 0.50 4.00 542.59 0.13 7.65 0.25 14.22 7.65 13.00 sdg 0.00 0.00 8.00 9.00 0.50 3.51 482.82 0.13 7.59 0.50 13.89 7.59 12.90 sdb 0.00 0.00 8.00 8.00 0.50 4.00 576.00 0.11 6.62 2.12 11.12 6.62 10.60 md127 0.00 0.00 0.00 49.00 0.00 23.00 961.14 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 18.70 4.21 4.24 0.00 0.00 72.85 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 0.00 0.00 8.00 15.00 0.50 6.09 586.78 0.22 9.17 0.25 13.93 6.26 14.40 sdk 0.00 0.00 8.00 14.00 0.50 5.55 563.27 0.25 11.68 2.38 17.00 7.59 16.70 sdc 0.00 0.00 8.00 13.00 0.50 6.50 682.67 0.15 7.00 0.25 11.15 6.00 12.60 sdd 0.00 0.00 8.00 13.00 0.50 6.50 682.67 0.17 7.95 0.25 12.69 6.86 14.40 sda 0.00 0.00 8.00 14.00 0.50 6.50 652.00 0.26 11.77 0.62 18.14 7.86 17.30 sdj 0.00 0.00 8.00 15.00 0.50 6.09 586.78 0.34 14.35 2.00 20.93 9.87 22.70 sdl 0.00 0.00 8.00 13.00 0.50 5.05 541.33 0.25 11.86 0.50 18.85 7.57 15.90 sdh 0.00 0.00 10.00 17.00 0.62 7.14 589.04 0.33 12.19 0.60 19.00 7.41 20.00 sde 0.00 0.00 8.00 18.00 0.50 6.68 565.85 0.31 11.77 0.25 16.89 7.00 18.20 sdf 0.00 0.00 8.00 18.00 0.50 6.68 565.85 0.42 16.12 2.25 22.28 9.96 25.90 sdg 0.00 0.00 10.00 17.00 0.62 7.14 589.04 0.33 12.30 0.60 19.18 6.59 17.80 sdb 0.00 0.00 8.00 14.00 0.50 6.50 652.00 0.27 12.14 2.25 17.79 8.00 17.60 md127 0.00 0.00 0.00 91.00 0.00 38.47 865.76 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 16.69 0.03 3.08 0.03 0.00 80.16 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 0.00 0.00 18.00 14.00 1.12 7.00 520.00 0.15 4.84 0.50 10.43 4.62 14.80 sdk 0.00 0.00 16.00 14.00 1.00 7.00 546.13 0.14 4.77 0.38 9.79 4.77 14.30 sdc 0.00 0.00 16.00 13.00 1.00 6.50 529.66 0.14 5.00 0.38 10.69 5.00 14.50 sdd 0.00 0.00 16.00 13.00 1.00 6.50 529.66 0.15 5.10 0.38 10.92 5.10 14.80 sda 0.00 0.00 16.00 18.00 1.00 7.54 514.59 0.21 6.12 1.31 10.39 6.26 21.30 sdj 0.00 0.00 18.00 14.00 1.12 7.00 520.00 0.13 4.25 0.50 9.07 4.03 12.90 sdl 0.00 0.00 16.00 14.00 1.00 7.00 546.13 0.13 4.47 0.31 9.21 4.47 13.40 sdh 7.00 0.00 10.00 13.00 1.06 6.50 673.39 0.10 4.57 0.50 7.69 4.57 10.50 sde 6.00 0.00 10.00 13.00 1.00 6.50 667.83 0.15 6.35 0.60 10.77 6.35 14.60 sdf 6.00 0.00 10.00 13.00 1.00 6.50 667.83 0.14 6.22 0.60 10.54 6.22 14.30 sdg 7.00 0.00 10.00 13.00 1.06 6.50 673.39 0.10 4.39 0.50 7.38 4.39 10.10 sdb 0.00 0.00 16.00 19.00 1.00 7.57 501.71 0.13 3.77 0.31 6.68 3.77 13.20 md127 0.00 0.00 0.00 85.00 0.00 40.57 977.60 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 22.73 0.00 5.91 0.06 0.00 71.30 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 242.00 0.00 1048.00 1.00 80.62 0.01 157.43 4.38 4.18 4.17 8.00 0.37 38.50 sdk 334.00 0.00 954.00 0.00 80.50 0.00 172.81 7.27 7.62 7.62 0.00 0.45 43.20 sdc 294.00 0.00 994.00 0.00 80.50 0.00 165.86 5.56 5.59 5.59 0.00 0.40 39.80 sdd 249.00 0.00 1039.00 0.00 80.50 0.00 158.68 4.11 3.95 3.95 0.00 0.37 38.00 sda 268.00 0.00 1020.00 11.00 80.50 0.18 160.26 5.47 5.31 5.14 21.36 0.58 60.20 sdj 253.00 0.00 1037.00 1.00 80.62 0.01 159.10 4.42 4.26 4.26 3.00 0.37 38.80 sdl 257.00 0.00 1031.00 0.00 80.50 0.00 159.91 5.13 4.98 4.98 0.00 0.37 38.30 sdh 224.00 0.00 1064.00 1.00 80.50 0.00 154.81 3.80 3.57 3.57 10.00 0.36 38.30 sde 247.00 0.00 1041.00 0.00 80.50 0.00 158.37 4.96 4.77 4.77 0.00 0.37 38.60 sdf 220.00 0.00 1068.00 0.00 80.50 0.00 154.37 3.70 3.47 3.47 0.00 0.33 35.40 sdg 242.00 0.00 1046.00 1.00 80.50 0.00 157.47 5.05 4.82 4.81 13.00 0.39 40.80 sdb 239.00 0.00 1049.00 10.00 80.50 0.15 155.97 4.77 4.51 4.43 12.10 0.45 47.90 md127 0.00 0.00 0.00 13.00 0.00 0.17 26.46 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 28.14 0.03 6.00 0.00 0.00 65.83 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 0.00 5.00 12.00 37.00 0.75 0.29 43.27 1.13 23.04 0.67 30.30 4.98 24.40 sdk 0.00 17.00 16.00 34.00 1.00 0.34 54.88 2.95 59.02 0.56 86.53 3.98 19.90 sdc 0.00 4.00 8.00 33.00 0.50 0.25 37.66 1.14 27.88 0.25 34.58 5.66 23.20 sdd 0.00 4.00 8.00 33.00 0.50 0.25 37.66 0.52 12.83 0.25 15.88 4.02 16.50 sda 0.00 3.00 16.00 21.00 1.00 0.17 64.86 0.26 7.14 1.06 11.76 3.92 14.50 sdj 0.00 5.00 12.00 37.00 0.75 0.29 43.27 0.84 17.24 0.42 22.70 4.47 21.90 sdl 0.00 17.00 16.00 34.00 1.00 0.34 54.88 2.98 59.56 0.56 87.32 3.92 19.60 sdh 0.00 4.00 8.00 26.00 0.50 0.20 41.88 0.67 19.71 1.75 25.23 4.50 15.30 sde 0.00 4.00 8.00 22.00 0.50 0.19 47.20 0.39 12.83 2.38 16.64 3.93 11.80 sdf 0.00 4.00 8.00 22.00 0.50 0.19 47.20 0.35 11.60 2.25 15.00 3.67 11.00 sdg 0.00 4.00 8.00 26.00 0.50 0.20 41.88 0.67 19.62 1.50 25.19 4.12 14.00 sdb 0.00 3.00 16.00 21.00 1.00 0.17 64.86 0.42 11.27 1.00 19.10 6.73 24.90 md127 0.00 0.00 0.00 210.00 0.00 1.44 14.06 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 35.57 0.00 10.34 0.00 0.00 54.08 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 0.00 0.00 9.00 11.00 0.56 0.04 62.00 0.14 7.00 1.78 11.27 6.50 13.00 sdk 0.00 0.00 8.00 11.00 0.50 0.05 58.95 0.10 5.26 0.88 8.45 5.26 10.00 sdc 0.00 0.00 16.00 16.00 1.00 0.07 68.75 0.17 5.44 0.44 10.44 5.44 17.40 sdd 0.00 0.00 16.00 16.00 1.00 0.07 68.75 0.17 5.38 1.00 9.75 5.38 17.20 sda 0.00 0.00 8.00 16.00 0.50 0.06 48.08 0.20 8.54 0.75 12.44 5.42 13.00 sdj 0.00 0.00 9.00 11.00 0.56 0.04 62.00 0.13 6.65 0.44 11.73 5.90 11.80 sdl 0.00 0.00 8.00 11.00 0.50 0.05 58.95 0.12 6.16 1.62 9.45 6.16 11.70 sdh 0.00 0.00 16.00 30.00 1.00 0.11 49.63 0.32 6.85 0.44 10.27 4.39 20.20 sde 0.00 0.00 16.00 6.00 1.00 0.02 95.27 0.10 4.41 0.44 15.00 4.41 9.70 sdf 0.00 0.00 16.00 6.00 1.00 0.02 95.27 0.14 6.59 4.06 13.33 6.55 14.40 sdg 0.00 0.00 16.00 29.00 1.00 0.11 50.56 0.31 6.80 0.44 10.31 4.82 21.70 sdb 0.00 0.00 8.00 16.00 0.50 0.06 48.08 0.24 10.17 0.62 14.94 6.75 16.20 md127 0.00 0.00 0.00 89.00 0.00 0.36 8.24 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 35.50 0.03 7.43 0.00 0.00 57.04 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 74.00 0.00 21.00 7.00 5.94 1.53 546.00 0.47 16.71 17.57 14.14 4.89 13.70 sdk 70.00 0.00 26.00 10.00 6.00 1.41 421.56 0.41 10.94 11.73 8.90 4.33 15.60 sdc 77.00 0.00 11.00 9.00 5.50 1.57 723.60 0.64 32.00 42.64 19.00 10.65 21.30 sdd 77.00 0.00 11.00 9.00 5.50 1.57 723.60 1.19 59.60 96.36 14.67 12.10 24.20 sda 71.00 1.00 24.00 11.00 5.94 1.53 437.09 0.51 14.46 14.38 14.64 5.09 17.80 sdj 74.00 0.00 21.00 7.00 5.94 1.53 546.00 0.58 20.79 20.57 21.43 7.04 19.70 sdl 70.00 0.00 26.00 11.00 6.00 1.91 437.84 0.39 10.54 11.04 9.36 4.32 16.00 sdh 77.00 0.00 11.00 7.00 5.50 1.52 798.67 0.43 24.17 33.82 9.00 6.61 11.90 sde 77.00 0.00 11.00 6.00 5.50 1.52 845.18 0.58 34.24 36.91 29.33 13.71 23.30 sdf 77.00 0.00 11.00 6.00 5.50 1.52 845.18 0.60 35.35 45.36 17.00 10.06 17.10 sdg 77.00 0.00 11.00 8.00 5.50 1.52 757.05 0.43 22.95 32.00 10.50 6.89 13.10 sdb 71.00 1.00 24.00 11.00 5.94 1.53 437.09 0.60 17.14 13.67 24.73 9.03 31.60 md127 0.00 0.00 0.00 52.00 0.00 9.57 376.96 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 27.06 0.03 6.00 0.00 0.00 66.91 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdi 14.00 0.00 10.00 9.00 1.50 4.06 599.58 0.13 6.84 2.60 11.56 6.63 12.60 sdk 14.00 0.00 10.00 10.00 1.50 5.00 665.60 0.13 7.05 2.50 11.60 6.35 12.70 sdc 14.00 0.00 11.00 10.00 1.56 4.01 543.24 0.15 7.33 1.00 14.30 7.24 15.20 sdd 14.00 0.00 11.00 10.00 1.56 4.01 543.24 0.15 7.14 1.00 13.90 7.05 14.80 sda 14.00 0.00 11.00 10.00 1.56 4.20 561.52 0.12 5.38 0.91 10.30 5.43 11.40 sdj 14.00 0.00 10.00 9.00 1.50 4.06 599.58 0.26 13.68 3.60 24.89 13.47 25.60 sdl 14.00 0.00 10.00 9.00 1.50 4.50 646.74 0.13 6.63 1.30 12.56 6.47 12.30 sdh 13.00 0.00 11.00 9.00 1.50 4.00 563.60 0.11 5.70 1.18 11.22 5.55 11.10 sde 14.00 0.00 10.00 8.00 1.50 4.00 625.78 0.09 4.78 1.10 9.38 4.67 8.40 sdf 14.00 0.00 10.00 8.00 1.50 4.00 625.78 0.14 8.06 4.00 13.12 7.17 12.90 sdg 13.00 0.00 11.00 9.00 1.50 4.00 563.60 0.14 7.00 1.91 13.22 6.80 13.60 sdb 14.00 0.00 11.00 10.00 1.56 4.20 561.52 0.17 7.67 1.73 14.20 7.71 16.20 md127 0.00 0.00 0.00 56.00 0.00 25.27 924.14 0.00 0.00 0.00 0.00 0.00 0.00 On 2016-05-25, 5:43 PM, "centos-boun...@centos.org on behalf of cpol...@surewest.net" <centos-boun...@centos.org on behalf of cpol...@surewest.net> wrote: >On 2016-05-25 19:13, Kelly Lesperance wrote: >> Hdparm didn’t get far: >> >> [root@r1k1 ~] # hdparm -tT /dev/sda >> >> /dev/sda: >> Timing cached reads: Alarm clock >> [root@r1k1 ~] # > >Hi Kelly, > >Try running 'iostat -xdmc 1'. Look for a single drive that has >substantially greater await than ~10msec. If all the drives >except one are taking 6-8msec, but one is very much more, you've >got a drive that drags down the whole array's performance. > >Ignore the very first output from the command - it's an >average of the disk subsystem since boot. > >Post a representative output along with the contents /proc/mdstat. > >Good luck, >-- >Charles Polisher > >_______________________________________________ >CentOS mailing list >CentOS@centos.org >https://lists.centos.org/mailman/listinfo/centos _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos