[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
How bizarre, I haven’t dealt with this specific SKU before. Some Dell / LSI HBAs call this passthrough mode, some “personality”, some “jbod mode”, dunno why they can’t be consistent. > We are testing an experimental Ceph cluster with server and controller at > subject. > > The controller have not an HBA mode, but only a 'NonRAID' mode, come sort of > 'auto RAID0' configuration. Dell’s CLI guide describes setting individual drives in Non-RAID, which *smells* like passthrough, not the more-complex RAID0 workaround we had to do before passthrough. https://www.dell.com/support/manuals/en-nz/perc-h750-sas/perc_cli_rg/set-drive-state-commands?guid=guid-d4750845-1f57-434c-b4a9-935876ee1a8e&lang=en-us > > We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well, > and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad > (particulary, very low IOPS). Spinners are slow, this is news? That said, how slow is slow? Testing commands and results or it didn’t happen. Also, firmware matters. Run Dell’s DSU. > There's some hint for disk/controller configuration/optimization? Give us details, perccli /c0 show, test results etc. Use a different HBA if you have to use an HBA, one that doesn’t suffer an RoC. Better yet, take an expansive look at TCO and don’t write off NVMe as infeasible. If your cluster is experimental hopefully you aren’t stuck with a lot of these. Add up the cost of an RoC HBA, optionally with cache RAM and BBU/supercap, add in the cost delta for SAS HDDs over SATA. Add in the operational hassle of managing WAL+DB on those boot SSDs. Add in the extra HDDs you’ll need to provision because of IOPS. > > > Thanks. > > -- > Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso... > mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli) > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
spinners are slow anyway, but on top of that SAS disks often default to writecache=off. In use as a single disk with no risk of raid write-holes, you can turn on writecache. On SAS, I would assume the firmware does not lie about writes reaching stable storage (flushes). # turn on temporarily: sdparm --set=WCE /dev/sdX # turn on persistently: sdparm --set=WCE --save /dev/sdX To check current state: sdparm --get=WCE /dev/sdf /dev/sdf: SEAGATE ST2000NM0045 DS03 WCE 0 [cha: y, def: 0, sav: 0] "WCE 0" means: off "sav: 0" means: off next time the disk is powered on Matthias On Thu, Apr 06, 2023 at 09:26:27AM -0400, Anthony D'Atri wrote: > How bizarre, I haven’t dealt with this specific SKU before. Some Dell / LSI > HBAs call this passthrough mode, some “personality”, some “jbod mode”, dunno > why they can’t be consistent. > > > > We are testing an experimental Ceph cluster with server and controller at > > subject. > > > > The controller have not an HBA mode, but only a 'NonRAID' mode, come sort of > > 'auto RAID0' configuration. > > Dell’s CLI guide describes setting individual drives in Non-RAID, which > *smells* like passthrough, not the more-complex RAID0 workaround we had to do > before passthrough. > > https://www.dell.com/support/manuals/en-nz/perc-h750-sas/perc_cli_rg/set-drive-state-commands?guid=guid-d4750845-1f57-434c-b4a9-935876ee1a8e&lang=en-us > > > > We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well, > > and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad > > (particulary, very low IOPS). > > Spinners are slow, this is news? > > That said, how slow is slow? Testing commands and results or it didn’t > happen. > > Also, firmware matters. Run Dell’s DSU. > > > There's some hint for disk/controller configuration/optimization? > > Give us details, perccli /c0 show, test results etc. > > Use a different HBA if you have to use an HBA, one that doesn’t suffer an > RoC. Better yet, take an expansive look at TCO and don’t write off NVMe as > infeasible. If your cluster is experimental hopefully you aren’t stuck with > a lot of these. Add up the cost of an RoC HBA, optionally with cache RAM and > BBU/supercap, add in the cost delta for SAS HDDs over SATA. Add in the > operational hassle of managing WAL+DB on those boot SSDs. Add in the extra > HDDs you’ll need to provision because of IOPS. > > > > > > > Thanks. > > > > -- > > Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso... > > mi va bene fino a quando non riguarda me :)(Emanuele Pucciarelli) > > > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Mandi! Matthias Ferdinand In chel di` si favelave... > To check current state: > sdparm --get=WCE /dev/sdf > /dev/sdf: SEAGATE ST2000NM0045 DS03 > WCE 0 [cha: y, def: 0, sav: 0] > "WCE 0" means: off > "sav: 0" means: off next time the disk is powered on Checking current state lead to: root@pppve1:~# sdparm --get=WCE /dev/sdd /dev/sdd: ATA HGST HUS726T4TAL PV07 WCE 0 [cha: y] So seems off, right?! -- Voi non ci crederete la mia ragazza sogna (R. Vecchioni) ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Mandi! Anthony D'Atri In chel di` si favelave... > Dell???s CLI guide describes setting individual drives in Non-RAID, which > *smells* like passthrough, not the more-complex RAID0 workaround we had to do > before passthrough. > https://www.dell.com/support/manuals/en-nz/perc-h750-sas/perc_cli_rg/set-drive-state-commands?guid=guid-d4750845-1f57-434c-b4a9-935876ee1a8e&lang=en-us??? Exactly. NonRAID 'smells' also to us more like passthrough. The firs strange things came from the fact that SATA/SSD disk are passed in a way that seems 'fully transparent' (eg, linux see even the disk S/N), while SAS disk are passed more like an RAID0 disk, with a different S/N. > Spinners are slow, this is news? ;-) > That said, how slow is slow? Testing commands and results or it didn???t > happen. A test done some month ago: root@pppve1:~# fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=128k --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=hdd-rw-128 hdd-rw-128: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256 ... fio-3.12 Starting 4 processes Jobs: 4 (f=4): [m(4)][0.0%][eta 08d:17h:11m:29s] hdd-rw-128: (groupid=0, jobs=4): err= 0: pid=26198: Wed May 18 19:11:04 2022 read: IOPS=84, BW=10.5MiB/s (11.0MB/s)(1279MiB/121557msec) slat (usec): min=4, max=303887, avg=23029.19, stdev=61832.29 clat (msec): min=1329, max=6673, avg=4737.71, stdev=415.84 lat (msec): min=1543, max=6673, avg=4760.74, stdev=420.10 clat percentiles (msec): | 1.00th=[ 2802], 5.00th=[ 4329], 10.00th=[ 4463], 20.00th=[ 4530], | 30.00th=[ 4597], 40.00th=[ 4665], 50.00th=[ 4732], 60.00th=[ 4799], | 70.00th=[ 4866], 80.00th=[ 4933], 90.00th=[ 5134], 95.00th=[ 5336], | 99.00th=[ 5805], 99.50th=[ 6007], 99.90th=[ 6342], 99.95th=[ 6409], | 99.99th=[ 6611] bw ( KiB/s): min= 256, max= 5120, per=25.18%, avg=2713.08, stdev=780.45, samples=929 iops: min=2, max= 40, avg=21.13, stdev= 6.10, samples=929 write: IOPS=87, BW=10.9MiB/s (11.5MB/s)(1328MiB/121557msec); 0 zone resets slat (usec): min=9, max=309914, avg=23025.13, stdev=61676.77 clat (msec): min=1444, max=13086, avg=6943.12, stdev=2068.26 lat (msec): min=1543, max=13086, avg=6966.15, stdev=2069.28 clat percentiles (msec): | 1.00th=[ 2769], 5.00th=[ 4597], 10.00th=[ 4799], 20.00th=[ 5067], | 30.00th=[ 5403], 40.00th=[ 5873], 50.00th=[ 6409], 60.00th=[ 7148], | 70.00th=[ 8020], 80.00th=[ 9060], 90.00th=[10134], 95.00th=[10671], | 99.00th=[11610], 99.50th=[11879], 99.90th=[12550], 99.95th=[12550], | 99.99th=[12684] bw ( KiB/s): min= 256, max= 5376, per=24.68%, avg=2762.20, stdev=841.30, samples=926 iops: min=2, max= 42, avg=21.52, stdev= 6.56, samples=926 cpu : usr=0.05%, sys=0.09%, ctx=2847, majf=0, minf=49 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8% submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=10233,10627,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-11.0MB/s), io=1279MiB (1341MB), run=121557-121557msec WRITE: bw=10.9MiB/s (11.5MB/s), 10.9MiB/s-10.9MiB/s (11.5MB/s-11.5MB/s), io=1328MiB (1393MB), run=121557-121557msec Disk stats (read/write): sdc: ios=10282/10601, merge=0/0, ticks=3041312/27373721, in_queue=30373472, util=99.99% > Also, firmware matters. Run Dell???s DSU. Controller does not have the latest-latest firmware, but a decent new one; i've looked at chaneglogs and found nothing that seems relevant to performance trouble. Indeed, i'll do an upgrade ASAP. > Give us details, perccli /c0 show, test results etc. root@pppve1:~# perccli /c0 show Generating detailed summary of the adapter, it may take a while to complete. CLI Version = 007.1910.. Oct 08, 2021 Operating system = Linux 5.4.203-1-pve Controller = 0 Status = Success Description = None Product Name = PERC H750 Adapter Serial Number = 23L01Y6 SAS Address = 5f4ee0802ba3a400 PCI Address = 00:b3:00:00 System Time = 04/11/2023 12:11:50 Mfg. Date = 03/25/22 Controller Time = 04/11/2023 10:11:47 FW Package Build = 52.16.1-4405 BIOS Version = 7.16.00.0_0x07100501 FW Version = 5.160.02-3552 Driver Name = megaraid_sas Driver Version = 07.713.01.00-rc1 Current Personality = RAID-Mode Vendor Id = 0x1000 Device Id = 0x10E2 SubVendor Id = 0x1028 SubDevice Id = 0x2176 Host Interface = PCI-E Device Interface = SAS-12G Bus Number = 179 Device Number = 0 Function Number = 0 Domain ID = 0 Security Protocol = None JBOD Drives = 6 JBOD LIST : =
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
> iops: min=2, max= 40, avg=21.13, stdev= 6.10, samples=929 > iops: min=2, max= 42, avg=21.52, stdev= 6.56, samples=926 That looks horrible. We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s read or write. Originally, I was also a bit afraid that these disks would drag performance down, but they are on par with the NL-SAS drives. For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 Mini (Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is equivalent to JBOD mode or pass-through. These controllers have no cache options, if your do, disable all of them. Mode should be write-through. For your disk type I saw "volatile write cache available = yes" on "the internet". This looks a bit odd, but maybe these HDDs do have some volatile cache. Try to disable it with smartctl and do the benchmark again. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Hi, do you want to hear the truth from real experience? Or the myth? The truth is that: - hdd are too slow for ceph, the first time you need to do a rebalance or similar you will discover... - if you want to use hdds do a raid with your controller and use the controller BBU cache (do not consider controllers with hdd cache), and present the raid as one ceph disk. - enabling single hdd write cache (that is not battery protected) is far worse than enabling controller cache (which I assume is always protected by BBU) - anyway the best thing for ceph is to use nvme disks. Mario Il giorno gio 6 apr 2023 alle ore 13:40 Marco Gaiarin < g...@lilliput.linux.it> ha scritto: > > We are testing an experimental Ceph cluster with server and controller at > subject. > > The controller have not an HBA mode, but only a 'NonRAID' mode, come sort > of > 'auto RAID0' configuration. > > We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well, > and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad > (particulary, very low IOPS). > > > There's some hint for disk/controller configuration/optimization? > > > Thanks. > > -- > Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso... > mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli) > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
> > The truth is that: > - hdd are too slow for ceph, the first time you need to do a rebalance or > similar you will discover... Depends on the needs. For cold storage, or sequential use-cases that aren't performance-sensitive ... Can't say "too slow" without context. In Marco's case, I wonder how the results might differ with numjobs=1 -- with a value of 4 as reported, seems to me like the drive will be seeking an awful lot. Mind you many Ceph multi-client workloads exhibit the "IO Blender" effect where they present to the drives as random, but this FIO job may not be entirely indicative. If you have to expand just to get more IOPs, that's a different story. > - if you want to use hdds do a raid with your controller and use the > controller BBU cache (do not consider controllers with hdd cache), and > present the raid as one ceph disk. Take care regarding OSD and PG counts with that strategy. Plus, Ceph does replication, so replication under the OSD layer can be ... gratuitous. > - enabling single hdd write cache (that is not battery protected) is far > worse than enabling controller cache (which I assume is always protected by > BBU) There are plenty of RoC HBAs out there without cache RAM or BBU/supercap, and also ones with cache RAM but without BBU/supercap. These often default to writethrough caching and arguably don't have much or any net benefit. > - anyway the best thing for ceph is to use nvme disks. I wouldn't disagree, but it's not entirely cut and dried. Notably the cost and hassle of an RoC HBA, cache, BBU/supercap, additional monitoring, replacement ... See my post a few years back about reasons I don't like RoC HBAs. Go with a plain, non-RoC HBA and the savings can help justify going with SATA SSDs at a minimum. > > Mario > > Il giorno gio 6 apr 2023 alle ore 13:40 Marco Gaiarin < > g...@lilliput.linux.it> ha scritto: > >> >> We are testing an experimental Ceph cluster with server and controller at >> subject. >> >> The controller have not an HBA mode, but only a 'NonRAID' mode, come sort >> of >> 'auto RAID0' configuration. >> >> We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well, >> and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad >> (particulary, very low IOPS). >> >> >> There's some hint for disk/controller configuration/optimization? >> >> >> Thanks. >> >> -- >> Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso... >> mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli) >> >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Mandi! Frank Schilder In chel di` si favelave... >> iops: min=2, max= 40, avg=21.13, stdev= 6.10, samples=929 >> iops: min=2, max= 42, avg=21.52, stdev= 6.56, samples=926 > That looks horrible. Exactly, horrible. The strange thing is that we came from an homegrown Ceph cluster built using old hardware (HP G6 servers) and spare disks, that perform 'better', or at least perform 'more uniformly' that this 'new'. With these embarassing IOPS, sooner or later we reach the point where performances go to the ground, and sometimes it suffices to launch some 'find' on the filesystem involved... > We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s > read or write. Originally, I was also a bit afraid that these disks would > drag performance down, but they are on par with the NL-SAS drives. > For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 > Mini (Embedded)) and it works perfectly. All ceph-disks are configured > non-raid, which is equivalent to JBOD mode or pass-through. These controllers > have no cache options, if your do, disable all of them. Mode should be > write-through. > For your disk type I saw "volatile write cache available = yes" on "the > internet". This looks a bit odd, but maybe these HDDs do have some volatile > cache. Try to disable it with smartctl and do the benchmark again. Sorry, i'm a bit puzzled here. Matthias suggest to enable write cache, you suggest to disble it... or i'm cache-confused?! ;-) My actually controller configuration is: root@pppve1:~# perccli /c0 show Generating detailed summary of the adapter, it may take a while to complete. CLI Version = 007.1910.. Oct 08, 2021 Operating system = Linux 5.4.203-1-pve Controller = 0 Status = Success Description = None Product Name = PERC H750 Adapter Serial Number = 23L01Y6 SAS Address = 5f4ee0802ba3a400 PCI Address = 00:b3:00:00 System Time = 04/14/2023 18:03:24 Mfg. Date = 03/25/22 Controller Time = 04/14/2023 16:03:22 FW Package Build = 52.16.1-4405 BIOS Version = 7.16.00.0_0x07100501 FW Version = 5.160.02-3552 Driver Name = megaraid_sas Driver Version = 07.713.01.00-rc1 Current Personality = RAID-Mode Vendor Id = 0x1000 Device Id = 0x10E2 SubVendor Id = 0x1028 SubDevice Id = 0x2176 Host Interface = PCI-E Device Interface = SAS-12G Bus Number = 179 Device Number = 0 Function Number = 0 Domain ID = 0 Security Protocol = None JBOD Drives = 6 JBOD LIST : = - ID EID:Slt DID State Intf Med Size SeSz ModelVendor Port - 0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDTATA x1 1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDTATA x1 3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1 5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 - ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down Physical Drives = 6 PD LIST : === -- EID:Slt DID State DG Size Intf Med SED PI SeSz ModelSp Type -- 64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDTU JBOD 64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDTU JBOD 64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD 64:5 9 Onln -3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD 64:6 10 Onln -3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD 64:7 11 Onln -3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD -- EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded UBUnsp=UBad Unsupported|Rbld=Rebuild Enclosures = 1 Enclosure LIST : ==
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Hi, Current controller mode is RAID. You can switch to HBA mode and disable cache in controller settings at the BIOS k Sent from my iPhone > On 15 Apr 2023, at 12:11, Marco Gaiarin wrote: > > Mandi! Frank Schilder > In chel di` si favelave... > >>> iops: min=2, max= 40, avg=21.13, stdev= 6.10, samples=929 >>> iops: min=2, max= 42, avg=21.52, stdev= 6.56, samples=926 >> That looks horrible. > > Exactly, horrible. > > The strange thing is that we came from an homegrown Ceph cluster built using > old hardware (HP G6 servers) and spare disks, that perform 'better', or at > least perform 'more uniformly' that this 'new'. > > With these embarassing IOPS, sooner or later we reach the point where > performances go to the ground, and sometimes it suffices to launch some > 'find' on the filesystem involved... > > >> We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s >> read or write. Originally, I was also a bit afraid that these disks would >> drag performance down, but they are on par with the NL-SAS drives. >> For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 >> Mini (Embedded)) and it works perfectly. All ceph-disks are configured >> non-raid, which is equivalent to JBOD mode or pass-through. These >> controllers have no cache options, if your do, disable all of them. Mode >> should be write-through. >> For your disk type I saw "volatile write cache available = yes" on "the >> internet". This looks a bit odd, but maybe these HDDs do have some volatile >> cache. Try to disable it with smartctl and do the benchmark again. > > Sorry, i'm a bit puzzled here. > > Matthias suggest to enable write cache, you suggest to disble it... or i'm > cache-confused?! ;-) > > > My actually controller configuration is: > > root@pppve1:~# perccli /c0 show > Generating detailed summary of the adapter, it may take a while to complete. > > CLI Version = 007.1910.. Oct 08, 2021 > Operating system = Linux 5.4.203-1-pve > Controller = 0 > Status = Success > Description = None > > Product Name = PERC H750 Adapter > Serial Number = 23L01Y6 > SAS Address = 5f4ee0802ba3a400 > PCI Address = 00:b3:00:00 > System Time = 04/14/2023 18:03:24 > Mfg. Date = 03/25/22 > Controller Time = 04/14/2023 16:03:22 > FW Package Build = 52.16.1-4405 > BIOS Version = 7.16.00.0_0x07100501 > FW Version = 5.160.02-3552 > Driver Name = megaraid_sas > Driver Version = 07.713.01.00-rc1 > Current Personality = RAID-Mode > Vendor Id = 0x1000 > Device Id = 0x10E2 > SubVendor Id = 0x1028 > SubDevice Id = 0x2176 > Host Interface = PCI-E > Device Interface = SAS-12G > Bus Number = 179 > Device Number = 0 > Function Number = 0 > Domain ID = 0 > Security Protocol = None > JBOD Drives = 6 > > JBOD LIST : > = > > - > ID EID:Slt DID State Intf Med Size SeSz ModelVendor > Port > - > 0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDTATA x1 > > 1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDTATA x1 > > 3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1 > > 5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 > > 6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 > > 7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1 > > - > > ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device > ID|Onln=Online > Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size > SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down > > Physical Drives = 6 > > PD LIST : > === > > -- > EID:Slt DID State DG Size Intf Med SED PI SeSz ModelSp > Type > -- > 64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDTU > JBOD > 64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDTU > JBOD > 64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U > JBOD > 64:5 9 Onln -3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U > JBOD > 64:6 10 Onln -3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U > JBOD > 64:7 11 Onln -3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U > JBOD > -- > > EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup > DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare > UBad=Unconfigured Bad|Sntze=Sanitize|Onl
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
With the LSI HBAs I’ve used, HBA cache seemed to only be used for VDs, not for passthrough drives. And then with various nasty bugs. Be careful not to conflate HBA cache with cache on the HDD itself. > On Apr 15, 2023, at 11:51 AM, Konstantin Shalygin wrote: > > Hi, > > Current controller mode is RAID. You can switch to HBA mode and disable cache > in controller settings at the BIOS > > > k > Sent from my iPhone > >> On 15 Apr 2023, at 12:11, Marco Gaiarin wrote: >> >> Mandi! Frank Schilder >> In chel di` si favelave... >> iops: min=2, max= 40, avg=21.13, stdev= 6.10, samples=929 iops: min=2, max= 42, avg=21.52, stdev= 6.56, samples=926 >>> That looks horrible. >> >> Exactly, horrible. >> >> The strange thing is that we came from an homegrown Ceph cluster built using >> old hardware (HP G6 servers) and spare disks, that perform 'better', or at >> least perform 'more uniformly' that this 'new'. >> >> With these embarassing IOPS, sooner or later we reach the point where >> performances go to the ground, and sometimes it suffices to launch some >> 'find' on the filesystem involved... >> >> >>> We also have a few SATA HDDs in Dell servers and they do about 100-150 >>> IOP/s read or write. Originally, I was also a bit afraid that these disks >>> would drag performance down, but they are on par with the NL-SAS drives. >>> For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 >>> Mini (Embedded)) and it works perfectly. All ceph-disks are configured >>> non-raid, which is equivalent to JBOD mode or pass-through. These >>> controllers have no cache options, if your do, disable all of them. Mode >>> should be write-through. >>> For your disk type I saw "volatile write cache available = yes" on "the >>> internet". This looks a bit odd, but maybe these HDDs do have some volatile >>> cache. Try to disable it with smartctl and do the benchmark again. >> >> Sorry, i'm a bit puzzled here. >> >> Matthias suggest to enable write cache, you suggest to disble it... or i'm >> cache-confused?! ;-) >> >> >> My actually controller configuration is: >> >> root@pppve1:~# perccli /c0 show >> Generating detailed summary of the adapter, it may take a while to complete. >> >> CLI Version = 007.1910.. Oct 08, 2021 >> Operating system = Linux 5.4.203-1-pve >> Controller = 0 >> Status = Success >> Description = None >> >> Product Name = PERC H750 Adapter >> Serial Number = 23L01Y6 >> SAS Address = 5f4ee0802ba3a400 >> PCI Address = 00:b3:00:00 >> System Time = 04/14/2023 18:03:24 >> Mfg. Date = 03/25/22 >> Controller Time = 04/14/2023 16:03:22 >> FW Package Build = 52.16.1-4405 >> BIOS Version = 7.16.00.0_0x07100501 >> FW Version = 5.160.02-3552 >> Driver Name = megaraid_sas >> Driver Version = 07.713.01.00-rc1 >> Current Personality = RAID-Mode >> Vendor Id = 0x1000 >> Device Id = 0x10E2 >> SubVendor Id = 0x1028 >> SubDevice Id = 0x2176 >> Host Interface = PCI-E >> Device Interface = SAS-12G >> Bus Number = 179 >> Device Number = 0 >> Function Number = 0 >> Domain ID = 0 >> Security Protocol = None >> JBOD Drives = 6 >> >> JBOD LIST : >> = >> >> - >> ID EID:Slt DID State Intf Med Size SeSz ModelVendor >> Port >> - >> 0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDTATA >> x1 >> 1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDTATA >> x1 >> 3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA >> x1 >> 5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA >> x1 >> 6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA >> x1 >> 7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA >> x1 >> - >> >> ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device >> ID|Onln=Online >> Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size >> SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down >> >> Physical Drives = 6 >> >> PD LIST : >> === >> >> -- >> EID:Slt DID State DG Size Intf Med SED PI SeSz ModelSp >> Type >> -- >> 64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDTU >> JBOD >> 64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDTU >> JBOD >> 64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U >> JBOD >> 64:5 9 Onln -3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U >> JBOD >> 64:6 10 Onln -3.638 TB SATA
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Hi Marco. >> For your disk type I saw "volatile write cache available = yes" on "the >> internet". This looks a bit odd, but maybe these HDDs do have some volatile >> cache. Try to disable it with smartctl and do the benchmark again. > > Sorry, i'm a bit puzzled here. > > Matthias suggest to enable write cache, you suggest to disble it... or i'm > cache-confused?! ;-) You need to disable *volatile* write cache. In order to figure out what is volatile or not, the best is to start disabling everything. This is usually the best way to run ceph anyways. Some expensive controllers claim to have non-volatile write cache. This is usually activated by disabling cache (yes, I know how this sounds), because this option typically refers to volatile and not the non-volatile cache. I would start by getting the disks into non-raid mode and disable all caches I can find. Then benchmark again. Then, maybe, enable caches again one by one and see if anything improves. If not or if it gets worse, disable it again. Best regards, Frank = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Konstantin Shalygin Sent: Saturday, April 15, 2023 11:49 AM To: Marco Gaiarin Cc: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller... Hi, Current controller mode is RAID. You can switch to HBA mode and disable cache in controller settings at the BIOS k Sent from my iPhone > On 15 Apr 2023, at 12:11, Marco Gaiarin wrote: > > Mandi! Frank Schilder > In chel di` si favelave... > >>> iops: min=2, max= 40, avg=21.13, stdev= 6.10, samples=929 >>> iops: min=2, max= 42, avg=21.52, stdev= 6.56, samples=926 >> That looks horrible. > > Exactly, horrible. > > The strange thing is that we came from an homegrown Ceph cluster built using > old hardware (HP G6 servers) and spare disks, that perform 'better', or at > least perform 'more uniformly' that this 'new'. > > With these embarassing IOPS, sooner or later we reach the point where > performances go to the ground, and sometimes it suffices to launch some > 'find' on the filesystem involved... > > >> We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s >> read or write. Originally, I was also a bit afraid that these disks would >> drag performance down, but they are on par with the NL-SAS drives. >> For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 >> Mini (Embedded)) and it works perfectly. All ceph-disks are configured >> non-raid, which is equivalent to JBOD mode or pass-through. These >> controllers have no cache options, if your do, disable all of them. Mode >> should be write-through. >> For your disk type I saw "volatile write cache available = yes" on "the >> internet". This looks a bit odd, but maybe these HDDs do have some volatile >> cache. Try to disable it with smartctl and do the benchmark again. > > Sorry, i'm a bit puzzled here. > > Matthias suggest to enable write cache, you suggest to disble it... or i'm > cache-confused?! ;-) > > > My actually controller configuration is: > > root@pppve1:~# perccli /c0 show > Generating detailed summary of the adapter, it may take a while to complete. > > CLI Version = 007.1910.. Oct 08, 2021 > Operating system = Linux 5.4.203-1-pve > Controller = 0 > Status = Success > Description = None > > Product Name = PERC H750 Adapter > Serial Number = 23L01Y6 > SAS Address = 5f4ee0802ba3a400 > PCI Address = 00:b3:00:00 > System Time = 04/14/2023 18:03:24 > Mfg. Date = 03/25/22 > Controller Time = 04/14/2023 16:03:22 > FW Package Build = 52.16.1-4405 > BIOS Version = 7.16.00.0_0x07100501 > FW Version = 5.160.02-3552 > Driver Name = megaraid_sas > Driver Version = 07.713.01.00-rc1 > Current Personality = RAID-Mode > Vendor Id = 0x1000 > Device Id = 0x10E2 > SubVendor Id = 0x1028 > SubDevice Id = 0x2176 > Host Interface = PCI-E > Device Interface = SAS-12G > Bus Number = 179 > Device Number = 0 > Function Number = 0 > Domain ID = 0 > Security Protocol = None > JBOD Drives = 6 > > JBOD LIST : > = > > - > ID EID:Slt DID State Intf Med Size SeSz ModelVendor > Port > - > 0 64:0 6 Onln SATA SSD 447.130 GB 512B MT
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Hi, > > Matthias suggest to enable write cache, you suggest to disble it... or i'm > > cache-confused?! ;-) there were some discussions about write cache settings last year, e.g. https://www.spinics.net/lists/ceph-users/msg73263.html https://www.spinics.net/lists/ceph-users/msg69489.html The entire threads are worth reading, but it is easy to get throroghly confused. In the first linked mail, Dan van der Ster points to this page: https://docs.ceph.com/en/latest/start/hardware-recommendations/#write-caches Essentially as Frank Schilder said: fio-test the drives with different settings, as some perform better when write-back cache is disabled, and some perform better with write-back cache enabled. And in order to not lose all of your data at once in case of a power outage: make sure your drives are not el-cheapo consumer ones which might lie to you about write completion (on either flushes or write-through writes). If a drive is not advertised for DC (data center) use, most probably it is such a dangerous consumer-grade drive and you better stay away from it. In case of power loss, drives with "power loss prevention" (large capacitor) do have a little more time to persist in-cache data to stable storage. Such drives may safely signal back write completion _before_ data has actually reached flash chips (or platters), their capacitor allows them to still persist the cache data on power loss. This gives them more opportunity for internal optimizations without risking durability, and could increase IOPS very much. Of course, individual testing is still necessary to find optimal cache settings for each drive. Matthias On Mon, Apr 17, 2023 at 07:32:42AM +, Frank Schilder wrote: > Hi Marco. > > >> For your disk type I saw "volatile write cache available = yes" on "the > >> internet". This looks a bit odd, but maybe these HDDs do have some > >> volatile cache. Try to disable it with smartctl and do the benchmark again. > > > > Sorry, i'm a bit puzzled here. > > > > Matthias suggest to enable write cache, you suggest to disble it... or i'm > > cache-confused?! ;-) > > You need to disable *volatile* write cache. In order to figure out what is > volatile or not, the best is to start disabling everything. This is usually > the best way to run ceph anyways. Some expensive controllers claim to have > non-volatile write cache. This is usually activated by disabling cache (yes, > I know how this sounds), because this option typically refers to volatile and > not the non-volatile cache. > > I would start by getting the disks into non-raid mode and disable all caches > I can find. Then benchmark again. Then, maybe, enable caches again one by one > and see if anything improves. If not or if it gets worse, disable it again. > > Best regards, > Frank > > = > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ____________________ > From: Konstantin Shalygin > Sent: Saturday, April 15, 2023 11:49 AM > To: Marco Gaiarin > Cc: Frank Schilder; ceph-users@ceph.io > Subject: Re: [ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 > Controller... > > Hi, > > Current controller mode is RAID. You can switch to HBA mode and disable cache > in controller settings at the BIOS > > > k > Sent from my iPhone > > > On 15 Apr 2023, at 12:11, Marco Gaiarin wrote: > > > > Mandi! Frank Schilder > > In chel di` si favelave... > > > >>> iops: min=2, max= 40, avg=21.13, stdev= 6.10, samples=929 > >>> iops: min=2, max= 42, avg=21.52, stdev= 6.56, samples=926 > >> That looks horrible. > > > > Exactly, horrible. > > > > The strange thing is that we came from an homegrown Ceph cluster built using > > old hardware (HP G6 servers) and spare disks, that perform 'better', or at > > least perform 'more uniformly' that this 'new'. > > > > With these embarassing IOPS, sooner or later we reach the point where > > performances go to the ground, and sometimes it suffices to launch some > > 'find' on the filesystem involved... > > > > > >> We also have a few SATA HDDs in Dell servers and they do about 100-150 > >> IOP/s read or write. Originally, I was also a bit afraid that these disks > >> would drag performance down, but they are on par with the NL-SAS drives. > >> For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 > >> Mini (Embedded)) and it works perfectly. All ceph-disks are configured > >> non-raid, which is equivalent to JBOD mode or pass-throu
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Il giorno sab 15 apr 2023 alle ore 11:10 Marco Gaiarin < g...@lilliput.linux.it> ha scritto: > > Sorry, i'm a bit puzzled here. > > Matthias suggest to enable write cache, you suggest to disble it... or i'm > cache-confused?! ;-) > > > > There is a cache in each disk, and a cache in the controller. The disk cache is: - volatile, if power goes away content is lost (some nvme disks have PLP that is a capacitor so cache in this case is not volatile) - ecc only in sas disks - needed for ssd (most ssd wear leveling algorithms have problems without cache) The controller cache: - is not volatile, if the controller has BBU - usually has ecc ram Usually when you have a BBU raid controller you disable single disks cache and keep only controller cache which is protected by battery. Ceph experts tell you to disable controller cache and raid but no one was able to tell me what to do with disk cache: - if you disable it disk performance becomes very slow and you risk problems in ssd - if you enable it disk will lose content if power goes away. Now ceph experts will tell you to buy only disks with PLP to solve the problem... but hdds do not have plp (if you find an hdd with plp tell me I will buy it) ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Mandi! Matthias Ferdinand In chel di` si favelave... > In the first linked mail, Dan van der Ster points to this page: > https://docs.ceph.com/en/latest/start/hardware-recommendations/#write-caches root@pppve1:~# for d in a b c d e f; do smartctl -g wcache /dev/sd$d | grep ^Write; done Write cache is: Enabled Write cache is: Enabled Write cache is: Enabled Write cache is: Disabled Write cache is: Disabled Write cache is: Disabled configuration seems just correct: sda, sdb and sdc, SDD disks, have cache enabled; sdd, sde and sdf that are HDD are disabled. -- Se non trovi nessuno vuol dire che siamo scappati alle sei-shell (bash, tcsh,csh...) (Possi) ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Mandi! Konstantin Shalygin In chel di` si favelave... > Current controller mode is RAID. You can switch to HBA mode and disable cache > in controller settings at the BIOS No, is a bit complex then that. Controller does not have an 'HBA-mode', but a 'AutoRAID0' mode. -- Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing. (from texinfo documentation) Dick Brandon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Mandi! Mario Giammarco In chel di` si favelave... > The disk cache is: If the controller does not lie, disk cache is disabled, see my previous messages. > The controller cache: Manual say that for Non-RAID/Automatic RAID0 disks, «The only supported cache policy for non???RAID disks is Write-Through»; so , AFAIK, disabled. https://dl.dell.com/content/manual61357984-dell-poweredge-raid-controller-11-user-s-guide-perc-h755-adapter-h755-front-sas-h755n-front-nvme-h755-mx-adapter-h750-adapter-sas-h355-adapter-sas-h355-front-sas-h350-adapter-sas-h350-mini-monolithic-sas.pdf?language=en-us reading through the controller docs, seems to me that there's very little to tackle... -- Il mondo del calcio sta andando a puttane! Perché, hanno finito le letterine?(Marco Citi) ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Actually there was a firmware bug around that a while back. The HBA and storcli claimed to not touch drive cache, but actually were enabling it and lying. > On Apr 19, 2023, at 1:41 PM, Marco Gaiarin wrote: > > Mandi! Mario Giammarco > In chel di` si favelave... > >> The disk cache is: > > If the controller does not lie, disk cache is disabled, see my previous > messages. > > >> The controller cache: > > Manual say that for Non-RAID/Automatic RAID0 disks, «The only supported > cache policy for non???RAID disks is Write-Through»; so , AFAIK, disabled. > > > https://dl.dell.com/content/manual61357984-dell-poweredge-raid-controller-11-user-s-guide-perc-h755-adapter-h755-front-sas-h755n-front-nvme-h755-mx-adapter-h750-adapter-sas-h355-adapter-sas-h355-front-sas-h350-adapter-sas-h350-mini-monolithic-sas.pdf?language=en-us > > > reading through the controller docs, seems to me that there's very little to > tackle... > > -- > Il mondo del calcio sta andando a puttane! > Perché, hanno finito le letterine?(Marco Citi) > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Mandi! Anthony D'Atri In chel di` si favelave... > Actually there was a firmware bug around that a while back. The HBA and > storcli claimed to not touch drive cache, but actually were enabling it and > lying. Some pointer to the issue? I doubt hit me but... Thanks. -- ma l'impresa eccezionale, dammi retta e` essere normale (L. Dalla) ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
