[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
> > First, the simple cases, called Seagate ;-) > I have checked 3 different types of Seagate HDDs one labeled Enterprise, > one labeled IronWolf, and one labeled Exos. Non of this drives seams to > support Advanced power management (APM). > I tested the auto spin down beaver of the HDD labeled Enterprise and it > works as expected. In my last E-Mail I was a little bit confused because > all the Seagate drives often report the state “unknown”. However, now I’m > pretty sure that hdparm reports “unknown” when the drive is doing some IO, > if the drive is running but doing no IO the state “active/idle” is > reported and if the spindle has stoped the state is “standby”. > I was not able to check the auto spin down for the IronWolf and Exos until > now. > > I have found 4 HDD types that supports APM for testing. Two HGST HDDs, one > with 4 TB and one with 6 TB. A 4 TB WD drive labeled as Red Pro and a 12 > TB WD drive labeled as Gold. > I did set the APM to 127 for all 4 drives. The two HGST drives and the 4TB > WD drive did reach the state “standby”. While the 12 TB WD drive always > reported “active/idle” and never reached “standby”. > The spin down behaviour of the two HGST drives did not change when I set > the APM to 128 on all drives. Also the 12 TB WD still reports > “active/idle” permanently. Only the behaviour of the 4 TB WD drive has > changed since it seams that it can not reach the state “standby” any more. > So from 4 drives only 1 behave as the man page of hdparm suggests. > > > For now my HDD type sample size is too small to draw a final conclusion. > But I think it is possible to say that the APM setting is not so > straightforward as outlined by the man page of hdparm and you should test > this for every drive type. Maybe nice to collect the power usage during these tests with influx/prometheus? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
> Hmm, I see on the man page > > -B > Get/set Advanced Power Management feature, if the drive > supports it. A low value means aggressive power management > and a high value means better performance. Possible > settings range from values 1 through 127 (which permit > spin-down), and values 128 through 254 (which do not > permit spin-down). The highest degree of power management > is attained with a setting of 1, and the highest I/O > performance with a setting of 254. A value of 255 tells > hdparm to disable Advanced Power Management altogether on > the drive (not all drives support disabling it, but most > do). > > which seems to say that your value of 128 actually disables spindown. > Thank you for point that out. I did investigate that a little bit further now. First, the simple cases, called Seagate ;-) I have checked 3 different types of Seagate HDDs one labeled Enterprise, one labeled IronWolf, and one labeled Exos. Non of this drives seams to support Advanced power management (APM). I tested the auto spin down beaver of the HDD labeled Enterprise and it works as expected. In my last E-Mail I was a little bit confused because all the Seagate drives often report the state “unknown”. However, now I’m pretty sure that hdparm reports “unknown” when the drive is doing some IO, if the drive is running but doing no IO the state “active/idle” is reported and if the spindle has stoped the state is “standby”. I was not able to check the auto spin down for the IronWolf and Exos until now. I have found 4 HDD types that supports APM for testing. Two HGST HDDs, one with 4 TB and one with 6 TB. A 4 TB WD drive labeled as Red Pro and a 12 TB WD drive labeled as Gold. I did set the APM to 127 for all 4 drives. The two HGST drives and the 4TB WD drive did reach the state “standby”. While the 12 TB WD drive always reported “active/idle” and never reached “standby”. The spin down behaviour of the two HGST drives did not change when I set the APM to 128 on all drives. Also the 12 TB WD still reports “active/idle” permanently. Only the behaviour of the 4 TB WD drive has changed since it seams that it can not reach the state “standby” any more. So from 4 drives only 1 behave as the man page of hdparm suggests. For now my HDD type sample size is too small to draw a final conclusion. But I think it is possible to say that the APM setting is not so straightforward as outlined by the man page of hdparm and you should test this for every drive type. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
> The OSD daemon would crash I would assume. Since I don't understand why the OSDs should crash just because a disk goes into standby, I just tried it now. The result is very unspectacular und fits perfectly to Gregorys great explanation. The drive goes into standby for around 2 or 3 seconds and spines up again. The OSD daemon doesn't seem to care at all about the spin down. @Gregorys: Thank you for you explanation. Best regards, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
> When having software raid solutions, I was also thinking about spinning them > down and researching how to do this. I can't exactly remember, but a simple > hdparm/sdparm command was not sufficient. Now I am bit curious if you solved > this problem with mdadm / software raid? > On the first two boxes I used HW RAID controllers from Areca. At the web interface of this controllers you can select the timeout for a spin down easily. However, I was forced to manually increase the response timeout for the disks, since the RAID Controllers did mark the drives as failed if the did not spin up quickly enough, but with a time out of 20 seconds or so I did not face that problem anymore. Now I use BTRFS on the backup servers and the automatic spin down was straight forward, at least for WD and HGST drives. I think that it also works for the Seagate drives, but they sometimes report “unknown” instead of “standby” as drive state (hdparm -C /dev/sda). What I did: $ hdparm -S 241 /dev/sda # = spindown after 30 minutes And only for the WD & HGST drives: $ hdparm -B 128 /dev/sda ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
I would not recommend this on Ceph. There was a project where somebody tried to make RADOS amenable to spinning down drives, but I don't think it ever amounted to anything. The issue is just that the OSDs need to do disk writes whenever they get new OSDMaps, there's a lot of random stuff that updates them, and nothing tries to constrain it to restrict writes in mostly-idle clusters. So they wake up constantly to do internal maintenance and heartbeats even if the cluster is idle. If you *really* don't use the data often, the best approach is probably just to turn it all off. You'll need to make sure it turns on fast enough, but if you do a clean shutdown of everything with the right settings applied (you may or may not need things like nodown/noup when changing states, to prevent a lot of map churn) you should be able to make it work. -Greg On Fri, Jan 21, 2022 at 7:40 AM Sebastian Mazza wrote: > > > > > On 21.01.2022, at 14:36, Marc wrote: > > > >> > >> I wonder if it is possible to let the HDDs sleep or if the OSD daemons > >> prevent a hold of the spindle motors. Or can it even create some problems > >> for the OSD deamon if the HDD spines down? > >> However, it should be easy to check on a cluster without any load and > >> optimally on a Custer that is not in production, by something like: > >> > > > > From what I can remember was always the test result of spinning down/up > > drives that it causes more wear/damage then just leaving them spinning. > > > > If you do a spin down / up every 20 minutes or so the wear/damage of the > motors is probably a problem. But Christoph stated that the cluster is not > used for several days and I don't think one spin up/down per day generates > enough spin ups of the spindle motor to be concerned about that. > I have backup storage servers (no ceph) that are running for many years now. > The HDDs in this server are spinning only for one or two hours per day and > compared to HHDs in productive server that reading and writing 24 / 7, they > hardly ever fail. So I wouldn't worry about wear and tear of the motors from > spin ups on an archive system that are only used once in a view days. > However, it could be that it heavily depends on the drives and I was only > extraordinary lucky with all the WD, HGST and Seagate drives in our backup > machines. > > Best regards, > Sebastian > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
> enough spin ups of the spindle motor to be concerned about that. > I have backup storage servers (no ceph) that are running for many years > now. The HDDs in this server are spinning only for one or two hours per > day and compared to HHDs in productive server that reading and writing 24 > / 7, they hardly ever fail. Interesting. When having software raid solutions, I was also thinking about spinning them down and researching how to do this. I can't exactly remember, but a simple hdparm/sdparm command was not sufficient. Now I am bit curious if you solved this problem with mdadm / software raid? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
> On 21.01.2022, at 14:36, Marc wrote: > >> >> I wonder if it is possible to let the HDDs sleep or if the OSD daemons >> prevent a hold of the spindle motors. Or can it even create some problems >> for the OSD deamon if the HDD spines down? >> However, it should be easy to check on a cluster without any load and >> optimally on a Custer that is not in production, by something like: >> > > From what I can remember was always the test result of spinning down/up > drives that it causes more wear/damage then just leaving them spinning. > If you do a spin down / up every 20 minutes or so the wear/damage of the motors is probably a problem. But Christoph stated that the cluster is not used for several days and I don't think one spin up/down per day generates enough spin ups of the spindle motor to be concerned about that. I have backup storage servers (no ceph) that are running for many years now. The HDDs in this server are spinning only for one or two hours per day and compared to HHDs in productive server that reading and writing 24 / 7, they hardly ever fail. So I wouldn't worry about wear and tear of the motors from spin ups on an archive system that are only used once in a view days. However, it could be that it heavily depends on the drives and I was only extraordinary lucky with all the WD, HGST and Seagate drives in our backup machines. Best regards, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
On 21.01.22 14:23, Sebastian Mazza wrote: Or can it even create some problems for the OSD deamon if the HDD spins down? The OSD daemon would crash I would assume. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
> > I wonder if it is possible to let the HDDs sleep or if the OSD daemons > prevent a hold of the spindle motors. Or can it even create some problems > for the OSD deamon if the HDD spines down? > However, it should be easy to check on a cluster without any load and > optimally on a Custer that is not in production, by something like: > >From what I can remember was always the test result of spinning down/up drives >that it causes more wear/damage then just leaving them spinning. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
Hi Christoph, I do not have any answer for you, but I find the question very interesting. I wonder if it is possible to let the HDDs sleep or if the OSD daemons prevent a hold of the spindle motors. Or can it even create some problems for the OSD deamon if the HDD spines down? However, it should be easy to check on a cluster without any load and optimally on a Custer that is not in production, by something like: $ hdparm -y /dev/sda # spin down the hdd sda Wait a minute or so and check if the HDD is sill sleeping by $ hdparm -C /dev/sda If the "drive state is: standby” at least for several minutes, you could try to set a timeout for an automatic spinn down of the drive. e.g.: $ hdparm -S 243 /dev/sda # = spindown after (243-240)*30 minutes = 1.5 hours. You may need to change the Advanced power management (APM) level before the drive does an automatic spinn down. The current APM level can be checkt by $ hdparm -I /dev/sda | grep level > Advanced power management level: 128 You can set it by $ hdparm -B 128 /dev/sda The man page of hdparam states ..“values 128 through 254 (which do not permit spin-down)”… Best regards, Sebastian > On 12.01.2022, at 10:22, Christoph Adomeit > wrote: > > Hi, > > a customer has a ceph cluster which is used for archiving large amounts of > video data. > > The cluster sometimes is not used for several days but if data is needed the > cluster > should be available within a few minutes. > > The cluster consists of 5 Servers and 180 physical seagate harddisks and > wastes a lot of power > for drives and cooling. > > Any ideas what can be done to reduce the power usage an heat output in this > scenario ? > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ideas for Powersaving on archive Cluster ?
If performance isn't as big a concern, most servers have firmware settings that enable more aggressive power saving, at the cost of added latency/reduced cpu power/etc. HPE would be accessible/configurable via HP's ILO, Dells with DRAC, etc. They'd want to test and see how much of an impact it made on performance vs. power consumption at idle, to see if it's worth the tradeoffs. On Wed, Jan 12, 2022 at 3:22 AM Christoph Adomeit < christoph.adom...@gatworks.de> wrote: > Hi, > > a customer has a ceph cluster which is used for archiving large amounts of > video data. > > The cluster sometimes is not used for several days but if data is needed > the cluster > should be available within a few minutes. > > The cluster consists of 5 Servers and 180 physical seagate harddisks and > wastes a lot of power > for drives and cooling. > > Any ideas what can be done to reduce the power usage an heat output in > this scenario ? > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
