[ceph-users] Re: quincy v17.2.1 QE Validation status
On Sun, Jun 19, 2022 at 9:43 PM Yuri Weinstein wrote: > > rados, rgw, rbd and fs suits ran on the latest sha1 > (https://shaman.ceph.com/builds/ceph/quincy-release/eb0eac1a195f1d8e9e3c472c7b1ca1e9add581c2/) > > pls see the summary: > https://tracker.ceph.com/issues/55974#note-1 > > seeking final approvals. fs approved. > > Thx > YuriW > > On Fri, Jun 17, 2022 at 1:43 PM Yuri Weinstein wrote: >> >> Neha and all, >> >> https://github.com/ceph/ceph/pull/46706 has been tested and merged >> >> I am building a new branch and will run rados on it and email and update. >> >> Thx >> >> >> On Wed, Jun 15, 2022 at 6:59 PM Neha Ojha wrote: >>> >>> On Wed, Jun 15, 2022 at 7:23 AM Venky Shankar wrote: >>> > >>> > On Tue, Jun 14, 2022 at 10:51 PM Yuri Weinstein >>> > wrote: >>> > > >>> > > Details of this release are summarized here: >>> > > >>> > > https://tracker.ceph.com/issues/55974 >>> > > Release Notes - https://github.com/ceph/ceph/pull/46576 >>> > > >>> > > Seeking approvals for: >>> > > >>> > > rados - Neha, Travis, Ernesto, Adam >>> >>> All the rados tests have passed, so this looks good! However, some >>> users are waiting for a remediation for >>> https://tracker.ceph.com/issues/53729, and we want to make the offline >>> tool available in 17.2.1 by means of >>> https://github.com/ceph/ceph/pull/46706. Given that this PR is >>> restricted to a new option being added to the ceph-objectstore-tool, >>> the testing requirement is minimal - just running the rados suite on >>> the PR should be enough and we can merge it by the end of this week. >>> >>> If anybody has any objections, please let me know. >>> >>> Thanks, >>> Neha >>> >>> > >>> > > rgw - Casey >>> > > fs - Venky, Gerg >>> > >>> > fs approved. >>> > >>> > > orch - Adam >>> > > rbd - Ilya, Deepika >>> > > krbd Ilya, Deepika >>> > > upgrade/octopus-x - Casey >>> > > >>> > > Please reply to this email with approval and/or trackers of known >>> > > issues/PRs to address them. >>> > > >>> > > Josh, David - it's ready for LRC upgrade if you'd like. >>> > > >>> > > Thx >>> > > YuriW >>> > > ___ >>> > > Dev mailing list -- d...@ceph.io >>> > > To unsubscribe send an email to dev-le...@ceph.io >>> > >>> > >>> > >>> > -- >>> > Cheers, >>> > Venky >>> > >>> > ___ >>> > Dev mailing list -- d...@ceph.io >>> > To unsubscribe send an email to dev-le...@ceph.io >>> > >>> > ___ > Dev mailing list -- d...@ceph.io > To unsubscribe send an email to dev-le...@ceph.io -- Cheers, Venky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] multi-site replication not syncing metadata
Hi all, Running into some trouble. I just setup ceph multi-site replication. Good news is that it is syncing the data. But the metadata is NOT syncing. I was trying to follow the instructions from here: https://docs.ceph.com/en/quincy/radosgw/multisite/#create-a-secondary-zone I see there is open issue on syncing, not sure if this is related: https://github.com/ceph/ceph/pull/46148 I I'm using: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) Any suggestions? BTW, docs seem a bit out of date. I opened an issue: https://tracker.ceph.com/issues/56131 $ radosgw-admin sync status (from slave) realm bbd51090-42ff-4795-adea-4b9dbaaf573e () zonegroup d66ae4f6-c090-40c6-b05f-eeaa9c279e45 (-1) zone 7bb893c7-23a4-4c86-99f6-71aeec8209d5 (slave-1) metadata sync preparing for full sync full sync: 64/64 shards full sync: 0 entries to sync incremental sync: 0/64 shards metadata is behind on 64 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63] data sync source: 661233b5-b3da-4c3f-a401-d6874c11cdb8 (master-1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 9 shards behind shards: [23,33,36,39,47,83,84,96,123] oldest incremental change not applied: 2022-06-20T22:51:43.724881+ [23] $ radosgw-admin sync status (from master) realm bbd51090-42ff-4795-adea-4b9dbaaf573e () zonegroup d66ae4f6-c090-40c6-b05f-eeaa9c279e45 (-1) zone 661233b5-b3da-4c3f-a401-d6874c11cdb8 (master-1) metadata sync no sync (zone is master) 2022-06-20T23:03:14.408+ 7f1e8e309840 0 ERROR: failed to fetch datalog info data sync source: 7bb893c7-23a4-4c86-99f6-71aeec8209d5 (slave-1) failed to retrieve sync info: (13) Permission denied ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Suggestion to build ceph storage
I’ve come close more than once to removing that misleading 4% guidance. The OP plans to use a single M.2 NVMe device - I’m a bit suspcious that the M.2 connector may only be SATA, and 12 OSDs sharing one SATA device for WAL+DB, plus potential CephFS metadata and RGW index pools seems like a sound strategy for disappointment. Sometimes people assume that M.2 connectors are SATA, or are NVMe, and may have a rude awakening. Similarly, be very careful to not provision a client / desktop class NVMe for this duty; many drives in this form factor are not enterprise class. If there are PCI-e slots for future AIC NVMe devices, and/or a rear cage option, that would allow rather more flexibility. > On Jun 20, 2022, at 7:59 AM, Stefan Kooman wrote: > > On 6/20/22 16:47, Jake Grimmett wrote: >> Hi Stefan >> We use cephfs for our 7200CPU/224GPU HPC cluster, for our use-case >> (large-ish image files) it works well. >> We have 36 ceph nodes, each with 12 x 12TB HDD, 2 x 1.92TB NVMe, plus a >> 240GB System disk. Four dedicated nodes have NVMe for metadata pool, and >> provide mon,mgr and MDS service. >> I'm not sure you need 4% of OSD for wal/db, search this mailing list archive >> for a definitive answer, but my personal notes are as follows: >> "If you expect lots of small files: go for a DB that's > ~300 GB >> For mostly large files you are probably fine with a 60 GB DB. >> 266 GB is the same as 60 GB, due to the way the cache multiplies at each >> level, spills over during compaction." > > There is (experimental ...) support for dynamic sizing in Pacific [1]. Not > sure if it's stable yet in Quincy. > > Gr. Stefan > > [1]: > https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#sizing > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Suggestion to build ceph storage
Thanks Christophe, On Mon, Jun 20, 2022 at 11:45 AM Christophe BAILLON wrote: > Hi > > We have 20 ceph node, each with 12 x 18Tb, 2 x nvme 1Tb > > I try this method to create osd > > ceph orch apply -i osd_spec.yaml > > with this conf > > osd_spec.yaml > service_type: osd > service_id: osd_spec_default > placement: > host_pattern: '*' > data_devices: > rotational: 1 > db_devices: > paths: > - /dev/nvme0n1 > - /dev/nvme1n1 > > this created 6 osd with wal/db on /dev/nvme0n1 and 6 on /dev/nvme1n1 per > node > > Does cephadm automatically create partitions for Wal/DB or is it something I have to define in config? ( sorry i am new to cephadm because we are using ceph-ansible and i heard cephadm will replace ceph-ansible soon, is that correct?) > but when I do a lvs, I see only 6 x 80Go partitions on each nvme... > > I think this is dynamic sizing, but I'm not sure, I don't know how to > check it... > > Our cluster will only host couple of files, a small one and a big one ~2GB > for cephfs only use, and with only 8 users accessing datas > How many MDS nodes do you have for your cluster size? Do you have dedicated or shared MDS with OSDs? > I don't know if this is optimum, we are in testing process... > > - Mail original - > > De: "Stefan Kooman" > > À: "Jake Grimmett" , "Christian Wuerdig" < > christian.wuer...@gmail.com>, "Satish Patel" > > > > Cc: "ceph-users" > > Envoyé: Lundi 20 Juin 2022 16:59:58 > > Objet: [ceph-users] Re: Suggestion to build ceph storage > > > On 6/20/22 16:47, Jake Grimmett wrote: > >> Hi Stefan > >> > >> We use cephfs for our 7200CPU/224GPU HPC cluster, for our use-case > >> (large-ish image files) it works well. > >> > >> We have 36 ceph nodes, each with 12 x 12TB HDD, 2 x 1.92TB NVMe, plus a > >> 240GB System disk. Four dedicated nodes have NVMe for metadata pool, and > >> provide mon,mgr and MDS service. > >> > >> I'm not sure you need 4% of OSD for wal/db, search this mailing list > >> archive for a definitive answer, but my personal notes are as follows: > >> > >> "If you expect lots of small files: go for a DB that's > ~300 GB > >> For mostly large files you are probably fine with a 60 GB DB. > >> 266 GB is the same as 60 GB, due to the way the cache multiplies at each > >> level, spills over during compaction." > > > > There is (experimental ...) support for dynamic sizing in Pacific [1]. > > Not sure if it's stable yet in Quincy. > > > > Gr. Stefan > > > > [1]: > > > https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#sizing > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > -- > Christophe BAILLON > Mobile :: +336 16 400 522 > Work :: https://eyona.com > Twitter :: https://twitter.com/ctof > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Suggestion to build ceph storage
Thank Jake On Mon, Jun 20, 2022 at 10:47 AM Jake Grimmett wrote: > Hi Stefan > > We use cephfs for our 7200CPU/224GPU HPC cluster, for our use-case > (large-ish image files) it works well. > > We have 36 ceph nodes, each with 12 x 12TB HDD, 2 x 1.92TB NVMe, plus a > 240GB System disk. Four dedicated nodes have NVMe for metadata pool, and > provide mon,mgr and MDS service. > This is great info, Assuming we don't need redundancy for NvMe because if it fails then it will impact only on 6 OSDs and that is acceptable. At present, because of limited HW supply i am planning to host MDS on the same OSDs nodes (not dedicated HW for MDS) agreed that not a best practice but again currently i am dealing with all unknown and i don't want to throw money on something which we don't know. I may have more data as we start using and then I can adjust requirements accordingly. > > I'm not sure you need 4% of OSD for wal/db, search this mailing list > archive for a definitive answer, but my personal notes are as follows: > > "If you expect lots of small files: go for a DB that's > ~300 GB > For mostly large files you are probably fine with a 60 GB DB. > 266 GB is the same as 60 GB, due to the way the cache multiplies at each > level, spills over during compaction." > We don't know what kind of workload we are going to run because currently all they ask for is large storage with many many drives. In future if they ask for more IOPs then we may replace some box with NvME or SSD and adjust requirements. > > We use a single enterprise quality 1.9TB NVMe for each 6 OSDs to good > effect, you probably need 1DWPD to be safe. I suspect you might be able > to increase the ratio of HDD per NVMe with PCIe gen4 NVMe drives. > > Can you share what company NvME drives are you using? > best regards, > > Jake > > On 20/06/2022 08:22, Stefan Kooman wrote: > > On 6/19/22 23:23, Christian Wuerdig wrote: > >> On Sun, 19 Jun 2022 at 02:29, Satish Patel > wrote: > >> > >>> Greeting folks, > >>> > >>> We are planning to build Ceph storage for mostly cephFS for HPC > workload > >>> and in future we are planning to expand to S3 style but that is yet > >>> to be > >>> decided. Because we need mass storage, we bought the following HW. > >>> > >>> 15 Total servers and each server has a 12x18TB HDD (spinning disk) . We > >>> understand SSD/NvME would be best fit but it's way out of budget. > >>> > >>> I hope you have extra HW on hand for Monitor and MDS servers > > > > ^^ this. It also depends on the uptime guarantees you have to provide > > (if any). Are the HPC users going to write large files? Or loads of > > small files? The more metadata operations the busier the MDSes will be, > > but if it's mainly large files the load on them will be much lower. > >> > >>> Ceph recommends using a faster disk for wal/db if the data disk is > >>> slow and > >>> in my case I do have a slower disk for data. > >>> > >>> Question: > >>> 1. Let's say if i want to put a NvME disk for wal/db then what size i > >>> should buy. > >>> > >> > >> The official recommendation is to budget 4% of OSD size for WAL/DB - > >> so in > >> your case that would be 720GB per OSD. Especially if you want to go to > S3 > >> later you should stick closer to that limit since RGW is a heavy meta > >> data > >> user. > > > > CephFS can be metadata heavy also, depending on work load. You can > > co-locate the S3 service on this cluster later on, but from an > > operational perspective this might not be preferred: you can tune the > > hardware / configuration for each use case. Easier to troubleshoot, > > independent upgrade cycle, etc. > > > > Gr. Stefan > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > For help, read https://www.mrc-lmb.cam.ac.uk/scicomp/ > then contact unixad...@mrc-lmb.cam.ac.uk > -- > Dr Jake Grimmett > Head Of Scientific Computing > MRC Laboratory of Molecular Biology > Francis Crick Avenue, > Cambridge CB2 0QH, UK. > Phone 01223 267019 > Mobile 0776 9886539 > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: quincy v17.2.1 QE Validation status
On Sun, Jun 19, 2022 at 6:13 PM Yuri Weinstein wrote: > > rados, rgw, rbd and fs suits ran on the latest sha1 > (https://shaman.ceph.com/builds/ceph/quincy-release/eb0eac1a195f1d8e9e3c472c7b1ca1e9add581c2/) > > pls see the summary: > https://tracker.ceph.com/issues/55974#note-1 > > seeking final approvals. rbd approved. Thanks, Ilya ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Suggestion to build ceph storage
Hi We have 20 ceph node, each with 12 x 18Tb, 2 x nvme 1Tb I try this method to create osd ceph orch apply -i osd_spec.yaml with this conf osd_spec.yaml service_type: osd service_id: osd_spec_default placement: host_pattern: '*' data_devices: rotational: 1 db_devices: paths: - /dev/nvme0n1 - /dev/nvme1n1 this created 6 osd with wal/db on /dev/nvme0n1 and 6 on /dev/nvme1n1 per node but when I do a lvs, I see only 6 x 80Go partitions on each nvme... I think this is dynamic sizing, but I'm not sure, I don't know how to check it... Our cluster will only host couple of files, a small one and a big one ~2GB for cephfs only use, and with only 8 users accessing datas I don't know if this is optimum, we are in testing process... - Mail original - > De: "Stefan Kooman" > À: "Jake Grimmett" , "Christian Wuerdig" > , "Satish Patel" > > Cc: "ceph-users" > Envoyé: Lundi 20 Juin 2022 16:59:58 > Objet: [ceph-users] Re: Suggestion to build ceph storage > On 6/20/22 16:47, Jake Grimmett wrote: >> Hi Stefan >> >> We use cephfs for our 7200CPU/224GPU HPC cluster, for our use-case >> (large-ish image files) it works well. >> >> We have 36 ceph nodes, each with 12 x 12TB HDD, 2 x 1.92TB NVMe, plus a >> 240GB System disk. Four dedicated nodes have NVMe for metadata pool, and >> provide mon,mgr and MDS service. >> >> I'm not sure you need 4% of OSD for wal/db, search this mailing list >> archive for a definitive answer, but my personal notes are as follows: >> >> "If you expect lots of small files: go for a DB that's > ~300 GB >> For mostly large files you are probably fine with a 60 GB DB. >> 266 GB is the same as 60 GB, due to the way the cache multiplies at each >> level, spills over during compaction." > > There is (experimental ...) support for dynamic sizing in Pacific [1]. > Not sure if it's stable yet in Quincy. > > Gr. Stefan > > [1]: > https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#sizing > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io -- Christophe BAILLON Mobile :: +336 16 400 522 Work :: https://eyona.com Twitter :: https://twitter.com/ctof ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Suggestion to build ceph storage
Hi Stefan We use cephfs for our 7200CPU/224GPU HPC cluster, for our use-case (large-ish image files) it works well. We have 36 ceph nodes, each with 12 x 12TB HDD, 2 x 1.92TB NVMe, plus a 240GB System disk. Four dedicated nodes have NVMe for metadata pool, and provide mon,mgr and MDS service. I'm not sure you need 4% of OSD for wal/db, search this mailing list archive for a definitive answer, but my personal notes are as follows: "If you expect lots of small files: go for a DB that's > ~300 GB For mostly large files you are probably fine with a 60 GB DB. 266 GB is the same as 60 GB, due to the way the cache multiplies at each level, spills over during compaction." We use a single enterprise quality 1.9TB NVMe for each 6 OSDs to good effect, you probably need 1DWPD to be safe. I suspect you might be able to increase the ratio of HDD per NVMe with PCIe gen4 NVMe drives. best regards, Jake On 20/06/2022 08:22, Stefan Kooman wrote: On 6/19/22 23:23, Christian Wuerdig wrote: On Sun, 19 Jun 2022 at 02:29, Satish Patel wrote: Greeting folks, We are planning to build Ceph storage for mostly cephFS for HPC workload and in future we are planning to expand to S3 style but that is yet to be decided. Because we need mass storage, we bought the following HW. 15 Total servers and each server has a 12x18TB HDD (spinning disk) . We understand SSD/NvME would be best fit but it's way out of budget. I hope you have extra HW on hand for Monitor and MDS servers ^^ this. It also depends on the uptime guarantees you have to provide (if any). Are the HPC users going to write large files? Or loads of small files? The more metadata operations the busier the MDSes will be, but if it's mainly large files the load on them will be much lower. Ceph recommends using a faster disk for wal/db if the data disk is slow and in my case I do have a slower disk for data. Question: 1. Let's say if i want to put a NvME disk for wal/db then what size i should buy. The official recommendation is to budget 4% of OSD size for WAL/DB - so in your case that would be 720GB per OSD. Especially if you want to go to S3 later you should stick closer to that limit since RGW is a heavy meta data user. CephFS can be metadata heavy also, depending on work load. You can co-locate the S3 service on this cluster later on, but from an operational perspective this might not be preferred: you can tune the hardware / configuration for each use case. Easier to troubleshoot, independent upgrade cycle, etc. Gr. Stefan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io For help, read https://www.mrc-lmb.cam.ac.uk/scicomp/ then contact unixad...@mrc-lmb.cam.ac.uk -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-container: docker restart, mon's unable to join
This was related to an older post back from 2020: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/GBUBEGTZTAMNEQNKUGS47M6W6B4AEVVS/ Von: Kilian Ries Gesendet: Freitag, 17. Juni 2022 17:23:03 An: ceph-users@ceph.io; ste...@bit.nl Betreff: [ceph-users] ceph-container: docker restart, mon's unable to join @Stefan Did you find any solution to your problem? I just got the same error ... I have a running pacific cluster with 4x monitor servers and wanted to join a fifth monitor via docker container. My container start command is: docker run --name ceph-mon-arbiter01 --net=host -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -e MON_IP=192.168.2.5 -e CEPH_PUBLIC_NETWORK=192.168.2.0/24 quay.io/ceph/daemon:latest-pacific mon This works for the first time - then if i stop the container and start it again i just geht the following output: 2022-06-17 14:47:41 /opt/ceph-container/bin/entrypoint.sh: Existing mon, trying to rejoin cluster... After that, the container exits. If i manually delete /var/lib/ceph/mon/ceph-arbiter01/ then the container is starting again. Regards, Kilian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: What is the max size of cephfs (filesystem)
On 6/20/22 3:45 PM, Arnaud M wrote: Hello to everyone I have looked on the internet but couldn't find an answer. Do you know the maximum size of a ceph filesystem ? Not the max size of a single file but the limit of the whole filesystem ? For example a quick search on zfs on google output : A ZFS file system can store up to *256 quadrillion zettabytes* (ZB). I would like to have the same answer with cephfs. And if there is a limit, where is this limit coded ? Is it hard-coded or is it configurable ? Let's say someone wants to have a cephfs up to ExaByte, would it be completely foolish or would the system, given enough mds and servers and everything needed, be usable ? Is there any other limit to a ceph filesystem ? From the cephfs layer I didn't see there is any limitation about this. -- Xiubo All the best Arnaud ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: What is the max size of cephfs (filesystem)
> Currently the biggest HDD is 20TB. According to news articles, HDDs up to 26TB are sampling. Mind you they’re SMR. And for many applications having that much capacity behind a tired SATA interface is a serious bottleneck; I’ve seen deployments cap HDD size at 8TB because of this. But I digress... 30TB SSDs (eg. Intel / Solidigm P5316) have been shipping for a while now. > 1 exabyte means 50.000 OSD > cluster(without replication or EC) > AFAIK Cern did some tests using 5000 OSDs. Bigbang III used 10800! The CERN bigbang findings have been priceless contributions to Ceph scalability. > I don't know any larger clusters than Cern’s. The bigbangs were all transient I think. I would expect that there are clusters even larger than CERN’s production deployment in certain organizations that don’t talk about them. > So I am not saying it is impossible but it is very unlikely to grow a single > Ceph cluster to that size. In high school I couldn’t imagine using all 48KB on an Apple ][ 640KB ought to be enough for anyone (apocryphal) When I ran Dumpling 450x 3TB OSDs were among the larger clusters according to Inktank at the time. Basically, never say never. > Maybe you should search for alternatives, like hdfs which I > know/worked with more than 50.000 HDDs without problems. HDFS is a different beast FWIW. > > On Mon, Jun 20, 2022 at 10:46 AM Arnaud M wrote: >> >> Hello to everyone >> >> I have looked on the internet but couldn't find an answer. >> Do you know the maximum size of a ceph filesystem ? Not the max size of a >> single file but the limit of the whole filesystem ? >> >> For example a quick search on zfs on google output : >> A ZFS file system can store up to *256 quadrillion zettabytes* (ZB). >> >> I would like to have the same answer with cephfs. >> >> And if there is a limit, where is this limit coded ? Is it hard-coded or is >> it configurable ? >> >> Let's say someone wants to have a cephfs up to ExaByte, would it be >> completely foolish or would the system, given enough mds and servers and >> everything needed, be usable ? >> >> Is there any other limit to a ceph filesystem ? >> >> All the best >> >> Arnaud >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: What is the max size of cephfs (filesystem)
Am 20.06.22 um 10:34 schrieb Arnaud M: Yes i know for zfs such an amount of data is impossible I am not trying to store that much data My question is really for pure curiosity What is the téorical max size of a ceph filesystem There is no theoretical limit on the size of a Ceph cluster as far as I know. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: What is the max size of cephfs (filesystem)
Yes i know for zfs such an amount of data is impossible I am not trying to store that much data My question is really for pure curiosity What is the téorical max size of a ceph filesystem For example is it theoretically possible to store 1 exabyte ? 1 zettabyte ? More ? in cephfs ? Let's suppose I have all the servers/osd I want (again just théorically) would I be able to store 1 zettabyte ? More ? Or is there an hardcoded limit to the maximum size of a cephfs cluster Just for curiosity All the best Arnaud Le lun. 20 juin 2022 à 10:26, Robert Sander a écrit : > Am 20.06.22 um 09:45 schrieb Arnaud M: > > > A ZFS file system can store up to *256 quadrillion zettabytes* (ZB). > > How would a storage system look like in reality that could hold such an > amount of data? > > Regards > -- > Robert Sander > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > http://www.heinlein-support.de > > Tel: 030 / 405051-43 > Fax: 030 / 405051-19 > > Zwangsangaben lt. §35a GmbHG: > HRB 220009 B / Amtsgericht Berlin-Charlottenburg, > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: What is the max size of cephfs (filesystem)
Am 20.06.22 um 09:45 schrieb Arnaud M: A ZFS file system can store up to *256 quadrillion zettabytes* (ZB). How would a storage system look like in reality that could hold such an amount of data? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: What is the max size of cephfs (filesystem)
Hello thanks for the answer But is there any hard coded limit ? Like in zfs ? Maybe a limit to the maximum files a cephfs can have ? All the best Arnaud Le lun. 20 juin 2022 à 10:18, Serkan Çoban a écrit : > Currently the biggest HDD is 20TB. 1 exabyte means 50.000 OSD > cluster(without replication or EC) > AFAIK Cern did some tests using 5000 OSDs. I don't know any larger > clusters than Cern's. > So I am not saying it is impossible but it is very unlikely to grow a > single Ceph cluster to that size. > Maybe you should search for alternatives, like hdfs which I > know/worked with more than 50.000 HDDs without problems. > > On Mon, Jun 20, 2022 at 10:46 AM Arnaud M > wrote: > > > > Hello to everyone > > > > I have looked on the internet but couldn't find an answer. > > Do you know the maximum size of a ceph filesystem ? Not the max size of a > > single file but the limit of the whole filesystem ? > > > > For example a quick search on zfs on google output : > > A ZFS file system can store up to *256 quadrillion zettabytes* (ZB). > > > > I would like to have the same answer with cephfs. > > > > And if there is a limit, where is this limit coded ? Is it hard-coded or > is > > it configurable ? > > > > Let's say someone wants to have a cephfs up to ExaByte, would it be > > completely foolish or would the system, given enough mds and servers and > > everything needed, be usable ? > > > > Is there any other limit to a ceph filesystem ? > > > > All the best > > > > Arnaud > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: What is the max size of cephfs (filesystem)
Currently the biggest HDD is 20TB. 1 exabyte means 50.000 OSD cluster(without replication or EC) AFAIK Cern did some tests using 5000 OSDs. I don't know any larger clusters than Cern's. So I am not saying it is impossible but it is very unlikely to grow a single Ceph cluster to that size. Maybe you should search for alternatives, like hdfs which I know/worked with more than 50.000 HDDs without problems. On Mon, Jun 20, 2022 at 10:46 AM Arnaud M wrote: > > Hello to everyone > > I have looked on the internet but couldn't find an answer. > Do you know the maximum size of a ceph filesystem ? Not the max size of a > single file but the limit of the whole filesystem ? > > For example a quick search on zfs on google output : > A ZFS file system can store up to *256 quadrillion zettabytes* (ZB). > > I would like to have the same answer with cephfs. > > And if there is a limit, where is this limit coded ? Is it hard-coded or is > it configurable ? > > Let's say someone wants to have a cephfs up to ExaByte, would it be > completely foolish or would the system, given enough mds and servers and > everything needed, be usable ? > > Is there any other limit to a ceph filesystem ? > > All the best > > Arnaud > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] What is the max size of cephfs (filesystem)
Hello to everyone I have looked on the internet but couldn't find an answer. Do you know the maximum size of a ceph filesystem ? Not the max size of a single file but the limit of the whole filesystem ? For example a quick search on zfs on google output : A ZFS file system can store up to *256 quadrillion zettabytes* (ZB). I would like to have the same answer with cephfs. And if there is a limit, where is this limit coded ? Is it hard-coded or is it configurable ? Let's say someone wants to have a cephfs up to ExaByte, would it be completely foolish or would the system, given enough mds and servers and everything needed, be usable ? Is there any other limit to a ceph filesystem ? All the best Arnaud ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
