Hi, I've got a 6 node C* cluster (all nodes are equal both in OS and HW setup, they are DL380 Gen9 with Smart Array RAID 50,3 on SAS 15K HDDs) which has been recently updated from 2.2.5 to 3.5. As part of the update I've done the upgradesstables.
On 4 nodes the average request size issued to the block dev was never higher than 8 (that maps to 4K reads) while on remaining 2 nodes it was basically always maxed 512 (256K reads). Nodes doing 4K reads were pumping max 2K read IOPs while the 2 nodes never went up above 30 IOPs. We are quite pedantic about OS settings. All nodes got same settings and C* configuration. On all nodes block dev got noop scheduler set and read ahead aligned with strip size. During heavy read workloads we've also noticed that those 4 nodes can swing up to 10K IOPs to get data from storage, the 2 are much below. What can cause such difference? -Jacek
