Re: [ceph-users] Ceph for "home lab" / hobbyist use?
On Fri, Sep 6, 2019 at 6:37 PM Peter Woodman wrote: > > 2GB ram is gonna be really tight, probably. Bummer. So it won't be enough for any Ceph component to run reliably? Drat. The nice thing about the HC2 is the fact that it can power the attached SATA disk and itself through one barrel connector and includes a stackable case to mount the disk and act as a heat sink. Would have been so easy to stack those up and use fewer cables/power adapters to do it. I wish I could convince the Hardkernel guys to drop a new model with this form factor that uses their 64-bit 4GB chipset. > However, I do something similar at home with a bunch of rock64 4gb boards, > and it works well. Nice! Would you mind describing your setup in more detail? It's encouraging to hear this works on arm (I wasn't sure if that was still supported). I'm also curious how you manage all the power supplies, cables and disks. Assuming you're using external 3.5" USB 3 disks that require their own power, you're already at 8 power adapters for just 4 nodes. > There are sometimes issues with the released ARM packages (frequently crc32 > doesn;'t work, which isn't great), so you may have to build your own on the > board you're targeting or on something like scaleway, YMMV. Thanks, that's good to know. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
2GB ram is gonna be really tight, probably. However, I do something similar at home with a bunch of rock64 4gb boards, and it works well. There are sometimes issues with the released ARM packages (frequently crc32 doesn;'t work, which isn't great), so you may have to build your own on the board you're targeting or on something like scaleway, YMMV. On Fri, Sep 6, 2019 at 6:16 PM Cranage, Steve wrote: > I use those HC2 nodes for my home Ceph cluster, but my setup only has to > support the librados API, my software does HSM between regular XFS file > systems and the RADOS api so I don’t need the other MDS and the rest so I > can’t tell you if you’ll be happy in your configuration. > > > > Steve Cranage > > Principal Architect, Co-Founder > > DeepSpace Storage > > 719-930-6960 > > > -- > *From:* ceph-users on behalf of > William Ferrell > *Sent:* Friday, September 6, 2019 3:16:30 PM > *To:* ceph-users@lists.ceph.com > *Subject:* [ceph-users] Ceph for "home lab" / hobbyist use? > > Hello everyone! > > After years of running several ZFS pools on a home server and several > disk failures along the way, I've decided that my current home storage > setup stinks. So far there hasn't been any data loss, but > recovering/"resilvering" a ZFS pool after a disk failure is a > nail-biting experience. I also think the way things are set up now > isn't making the best use of all the disks attached to the server; > they were acquired over time instead of all at once, so I've got 4 > 4-disk raidz1 pools, each in their own enclosures. If any enclosure > dies, all that pool's data is lost. Despite having a total of 16 disks > in use for storage, the entire system can only "safely" lose one disk > before there's a risk of a second failure taking a bunch of data with > it. > > I'd like to ask the list's opinions on running a Ceph cluster in a > home environment as a filer using cheap, low-power systems. I don't > have any expectations for high performance (this will be built on a > gigabit network, and just used for backups and streaming videos, > music, etc. for two people); the main concern is resiliency if one or > two disks fail, and the secondary concern is having a decent usable > storage capacity. Being able to slowly add capacity to the cluster one > disk at a time is a very appealing bonus. > > I'm interested in using these things as OSDs (and hopefully monitors > and metadata servers): > https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/ > > They're about $50 each, can boot from MicroSD or eMMC flash (basically > an SSD with a custom connector), and have one SATA port. They have > 8-core 32-bit CPUs, 2GB of RAM and a gigabit ethernet port. Four of > them (including disks) can run off a single 12V/8A power adapter > (basically 100 watts per set of 4). The obvious appeal is price, plus > they're stackable so they'd be easy to hide away in a closet. > > Is it feasible for these to work as OSDs at all? The Ceph hardware > recommendations page suggests OSDs need 1GB per TB of space, so does > this mean these wouldn't be suitable with, say, a 4TB or 8TB disk? Or > would they work, but just more slowly? > > Pushing my luck further (assuming the HC2 can handle OSD duties at > all), is that enough muscle to run the monitor and/or metadata > servers? Should monitors and MDS's be run separately, or can/should > they piggyback on hosts running OSDs? > > I'd be perfectly happy with a setup like this even if it could only > achieve speeds in the 20-30MB/sec range. > > Is this a dumb idea, or could it actually work? Are there any other > recommendations among Ceph users for low-end hardware to cobble > together a working cluster? > > Any feedback is sincerely appreciated. > > Thanks! > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph for "home lab" / hobbyist use?
I use those HC2 nodes for my home Ceph cluster, but my setup only has to support the librados API, my software does HSM between regular XFS file systems and the RADOS api so I don’t need the other MDS and the rest so I can’t tell you if you’ll be happy in your configuration. Steve Cranage Principal Architect, Co-Founder DeepSpace Storage 719-930-6960 [cid:image001.png@01D3FCBC.58FDB6F0] From: ceph-users on behalf of William Ferrell Sent: Friday, September 6, 2019 3:16:30 PM To: ceph-users@lists.ceph.com Subject: [ceph-users] Ceph for "home lab" / hobbyist use? Hello everyone! After years of running several ZFS pools on a home server and several disk failures along the way, I've decided that my current home storage setup stinks. So far there hasn't been any data loss, but recovering/"resilvering" a ZFS pool after a disk failure is a nail-biting experience. I also think the way things are set up now isn't making the best use of all the disks attached to the server; they were acquired over time instead of all at once, so I've got 4 4-disk raidz1 pools, each in their own enclosures. If any enclosure dies, all that pool's data is lost. Despite having a total of 16 disks in use for storage, the entire system can only "safely" lose one disk before there's a risk of a second failure taking a bunch of data with it. I'd like to ask the list's opinions on running a Ceph cluster in a home environment as a filer using cheap, low-power systems. I don't have any expectations for high performance (this will be built on a gigabit network, and just used for backups and streaming videos, music, etc. for two people); the main concern is resiliency if one or two disks fail, and the secondary concern is having a decent usable storage capacity. Being able to slowly add capacity to the cluster one disk at a time is a very appealing bonus. I'm interested in using these things as OSDs (and hopefully monitors and metadata servers): https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/ They're about $50 each, can boot from MicroSD or eMMC flash (basically an SSD with a custom connector), and have one SATA port. They have 8-core 32-bit CPUs, 2GB of RAM and a gigabit ethernet port. Four of them (including disks) can run off a single 12V/8A power adapter (basically 100 watts per set of 4). The obvious appeal is price, plus they're stackable so they'd be easy to hide away in a closet. Is it feasible for these to work as OSDs at all? The Ceph hardware recommendations page suggests OSDs need 1GB per TB of space, so does this mean these wouldn't be suitable with, say, a 4TB or 8TB disk? Or would they work, but just more slowly? Pushing my luck further (assuming the HC2 can handle OSD duties at all), is that enough muscle to run the monitor and/or metadata servers? Should monitors and MDS's be run separately, or can/should they piggyback on hosts running OSDs? I'd be perfectly happy with a setup like this even if it could only achieve speeds in the 20-30MB/sec range. Is this a dumb idea, or could it actually work? Are there any other recommendations among Ceph users for low-end hardware to cobble together a working cluster? Any feedback is sincerely appreciated. Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph for "home lab" / hobbyist use?
Hello everyone! After years of running several ZFS pools on a home server and several disk failures along the way, I've decided that my current home storage setup stinks. So far there hasn't been any data loss, but recovering/"resilvering" a ZFS pool after a disk failure is a nail-biting experience. I also think the way things are set up now isn't making the best use of all the disks attached to the server; they were acquired over time instead of all at once, so I've got 4 4-disk raidz1 pools, each in their own enclosures. If any enclosure dies, all that pool's data is lost. Despite having a total of 16 disks in use for storage, the entire system can only "safely" lose one disk before there's a risk of a second failure taking a bunch of data with it. I'd like to ask the list's opinions on running a Ceph cluster in a home environment as a filer using cheap, low-power systems. I don't have any expectations for high performance (this will be built on a gigabit network, and just used for backups and streaming videos, music, etc. for two people); the main concern is resiliency if one or two disks fail, and the secondary concern is having a decent usable storage capacity. Being able to slowly add capacity to the cluster one disk at a time is a very appealing bonus. I'm interested in using these things as OSDs (and hopefully monitors and metadata servers): https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/ They're about $50 each, can boot from MicroSD or eMMC flash (basically an SSD with a custom connector), and have one SATA port. They have 8-core 32-bit CPUs, 2GB of RAM and a gigabit ethernet port. Four of them (including disks) can run off a single 12V/8A power adapter (basically 100 watts per set of 4). The obvious appeal is price, plus they're stackable so they'd be easy to hide away in a closet. Is it feasible for these to work as OSDs at all? The Ceph hardware recommendations page suggests OSDs need 1GB per TB of space, so does this mean these wouldn't be suitable with, say, a 4TB or 8TB disk? Or would they work, but just more slowly? Pushing my luck further (assuming the HC2 can handle OSD duties at all), is that enough muscle to run the monitor and/or metadata servers? Should monitors and MDS's be run separately, or can/should they piggyback on hosts running OSDs? I'd be perfectly happy with a setup like this even if it could only achieve speeds in the 20-30MB/sec range. Is this a dumb idea, or could it actually work? Are there any other recommendations among Ceph users for low-end hardware to cobble together a working cluster? Any feedback is sincerely appreciated. Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] regurlary 'no space left on device' when deleting on cephfs
Yeah, no ENOSPC error code on deletion is a little bit unintuitive, but what it means is: the purge queue is full. You've already told the MDS to purge faster. Not sure how to tell it to increase the maximum backlog for deletes/purges, though, but you should be able to find something with the search term "purge queue". :) Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Sep 6, 2019 at 4:00 PM Stefan Kooman wrote: > > Quoting Kenneth Waegeman (kenneth.waege...@ugent.be): > > The cluster is healthy at this moment, and we have certainly enough space > > (see also osd df below) > > It's not well balanced though ... do you use ceph balancer (with > balancer in upmap mode)? > > Gr. Stefan > > > -- > | BIT BV https://www.bit.nl/Kamer van Koophandel 09090351 > | GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] regurlary 'no space left on device' when deleting on cephfs
Quoting Kenneth Waegeman (kenneth.waege...@ugent.be): > The cluster is healthy at this moment, and we have certainly enough space > (see also osd df below) It's not well balanced though ... do you use ceph balancer (with balancer in upmap mode)? Gr. Stefan -- | BIT BV https://www.bit.nl/Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] regurlary 'no space left on device' when deleting on cephfs
Hi all,
We are using cephfs to make a copy of another fs via rsync, and also use
snapshots.
I'm seeing this issue now and then when I try to delete files on cephFS:
|[root@osd001 ~]# rm -f /mnt/ceph/backups/osd00*||
||rm: cannot remove
‘/mnt/ceph/backups/osd001.gigalith.os-3eea7740.1542483’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd001.gigalith.os-b2f21740.1557247’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd001.gigalith.os-ca3be740.1549780’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd002.gigalith.os-1b437740.1173950’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd002.gigalith.os-92186740.1169503’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd002.gigalith.os-e9260740.1178280’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd003.gigalith.os-2dec5740.2025571’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd003.gigalith.os-e5f94740.2029993’: No space left
on device||
||rm: cannot remove
‘/mnt/ceph/backups/osd004.gigalith.os-f4a9740.364609’: No space left on
device||
|
The cluster is healthy at this moment, and we have certainly enough
space (see also osd df below)
[root@osd001 ~]# ceph -s
cluster:
id: 92bfcf0a-1d39-43b3-b60f-44f01b630e47
health: HEALTH_OK
services:
mon: 3 daemons, quorum mds01,mds02,mds03
mgr: mds02(active), standbys: mds03, mds01
mds: ceph_fs-2/2/2 up {0=mds01=up:active,1=mds02=up:active}, 1
up:standby
osd: 536 osds: 536 up, 536 in
data:
pools: 3 pools, 3328 pgs
objects: 451.5 M objects, 762 TiB
usage: 1.1 PiB used, 2.0 PiB / 3.2 PiB avail
pgs: 1280 active+clean
1138 active+clean+snaptrim_wait
899 active+clean+snaptrim
9 active+clean+scrubbing+deep
2 active+clean+scrubbing
io:
client: 0 B/s wr, 0 op/s rd, 0 op/s wr
There is also nothing in the mds log.
We tuned the mds already before:
|[mds]||
||mds_cache_memory_limit=21474836480||
||mds_log_max_expiring=200||
||mds_log_max_segments=200||
||mds_max_purge_files=2560||
||mds_max_purge_ops=327600||
||mds_max_purge_ops_per_pg=20|
I tried restarting the mds, flushing the mds journals, and do a remount
on the client, but that does not help - after some time it just works
again..
What can I do to debug / tune this further ?
Thanks!!
Kenneth
ceph osd df :
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
546 ssd 0.14699 1.0 151 GiB 67 GiB 84 GiB 44.30 1.23 512
547 ssd 0.14699 1.0 151 GiB 68 GiB 83 GiB 45.27 1.26 512
0 ssd 0.14699 1.0 151 GiB 66 GiB 85 GiB 43.88 1.22 515
1 ssd 0.14699 1.0 151 GiB 66 GiB 85 GiB 43.65 1.21 509
2 ssd 0.14699 1.0 151 GiB 67 GiB 84 GiB 44.53 1.24 511
3 ssd 0.14699 1.0 151 GiB 67 GiB 84 GiB 44.39 1.23 513
540 fast 0.14799 1.0 151 GiB 3.1 GiB 148 GiB 2.05 0.06 0
541 fast 0.14799 1.0 151 GiB 3.1 GiB 148 GiB 2.05 0.06 0
528 hdd 3.63899 1.0 3.6 TiB 1.4 TiB 2.2 TiB 38.50 1.07 31
529 hdd 3.63899 1.0 3.6 TiB 2.1 TiB 1.6 TiB 56.96 1.58 42
530 hdd 3.63899 1.0 3.6 TiB 1.5 TiB 2.2 TiB 39.85 1.11 31
531 hdd 3.63899 1.0 3.6 TiB 1.5 TiB 2.2 TiB 39.91 1.11 30
532 hdd 3.63899 1.0 3.6 TiB 1.6 TiB 2.1 TiB 42.75 1.19 34
533 hdd 3.63899 1.0 3.6 TiB 1.3 TiB 2.3 TiB 35.60 0.99 31
534 hdd 3.63899 1.0 3.6 TiB 2.2 TiB 1.4 TiB 61.22 1.70 46
535 hdd 3.63899 1.0 3.6 TiB 1.3 TiB 2.3 TiB 37.02 1.03 33
536 hdd 3.63899 1.0 3.6 TiB 1.5 TiB 2.2 TiB 39.89 1.11 30
537 hdd 3.63899 1.0 3.6 TiB 1.1 TiB 2.5 TiB 31.35 0.87 24
538 hdd 3.63899 1.0 3.6 TiB 2.3 TiB 1.4 TiB 62.68 1.74 53
539 hdd 3.63899 1.0 3.6 TiB 1.9 TiB 1.8 TiB 51.27 1.43 40
542 hdd 3.63899 1.0 3.6 TiB 2.2 TiB 1.5 TiB 59.81 1.66 46
543 hdd 3.63899 1.0 3.6 TiB 1.5 TiB 2.1 TiB 41.27 1.15 35
544 hdd 3.63899 1.0 3.6 TiB 1.3 TiB 2.3 TiB 35.58 0.99 28
545 hdd 3.63899 1.0 3.6 TiB 1.2 TiB 2.4 TiB 32.73 0.91 28
520 fast 0.14799 1.0 151 GiB 3.8 GiB 147 GiB 2.53 0.07 0
522 fast 0.14799 1.0 151 GiB 3.8 GiB 147 GiB 2.53 0.07 0
496 hdd 3.63899 1.0 3.6 TiB 1.8 TiB 1.9 TiB 48.41 1.35 35
498 hdd 3.63899 1.0 3.6 TiB 1.1 TiB 2.6 TiB 29.88 0.83 27
500 hdd 3.63899 1.0 3.6 TiB 2.0 TiB 1.6 TiB 55.49 1.54 43
502 hdd 3.63899 1.0 3.6 TiB 1.4 TiB 2.2 TiB 38.48 1.07 31
504 hdd 3.63899 1.0 3.6 TiB 1.1 TiB 2.6 TiB 29.90 0.83 24
510 hdd 3.63899 1.0 3.6 TiB 1.2 TiB 2.4 TiB 34.16 0.95 28
512 hdd 3.63899 1.0 3.6 TiB 955 GiB 2.7 TiB 25.64 0.71 22
514 hdd 3.63899 1.0 3.6 TiB 1.3 TiB 2.3 TiB 37.03 1.03 31
516 hdd 3.63899 1.0 3.6 TiB 1.6 TiB 2.1 TiB 42.73 1.19 37
518 hdd 3.63899 1.0 3.6 TiB 1.8 TiB 1.9 TiB 48.39 1.35 38
524 hdd 3.63899 1.0
[ceph-users] Automatic balancing vs supervised optimization
Hi I have question regarding supervised/automatic balancing using upmap. I created a plan in supervised mode, but its score was not expected to improve the data distribution. But the automatic balancer triggered a considerable rebalance. Is this normal ? I thought that automatic balancing basically means running from time to time the balancer in supervised mode, More details on what I tried: I changed the balancer mode of my luminous cluster from crush-compact to upmap: ceph balancer off ceph osd set-require-min-compat-client luminous ceph osd crush weight-set rm-compat This triggered some rebalance. When done (i.e. when the status was HEALTH_OK) I issued: ceph balancer mode upmap I then tried to run the balancer in supervised mode: # ceph balancer optimize myplan # ceph balancer eval myplan But the returned value was greater than the value returned by "ceph balancer eval" I then tried with the automatic balancer: ceph balancer on which started moving objects improving A LOT the data distribution. Thanks, Massimo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 14.2.2 -> 14.2.3 upgrade [WRN] failed to encode map e905 with expected crc
Hi, While upgrading the monitors on a Nautilus test cluster warning messages apear: [WRN] failed to encode map e905 with expected crc Is this expected? I have only seen this in the past when mixing different releases (major versions), not when upgrading within a release. What is the impact of this? Gr. Stefan -- | BIT BV https://www.bit.nl/Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
