Thank you for the explanation Frank. I also agree with you, Ceph is not designed for this kind of use case but I tried to continue what I know. My idea was exactly what you described, I was trying to automate cleaning or recreating on any failure.
As you can see below, rep1 pool is very fast: - Create: time for i in {00001..99999}; do head -c 1K </dev/urandom >randfile$i; done replication 2 : 31m59.917s replication 1 : 7m6.046s -------------------------------- - Delete: time rm -rf testdir/ replication 2 : 11m56.994s replication 1 : 0m40.756s ------------------------------------- I started learning DRBD, I will also check BeeGFS thanks for the advice. Regards. Frank Schilder <fr...@dtu.dk>, 1 May 2023 Pzt, 10:27 tarihinde şunu yazdı: > > I think you misunderstood Janne's reply. The main statement is at the end, > ceph is not designed for an "I don't care about data" use case. If you need > speed for temporary data where you can sustain data loss, go for something > simpler. For example, we use beegfs with great success for a burst buffer for > an HPC cluster. It is very lightweight and will pull out all performance your > drives can offer. In case of disaster it is easily possible to clean up. > Beegfs does not care about lost data, such data will simply become > inaccessible while everything else just moves on. It will not try to > self-heal either. It doesn't even scrub data, so no competition of users with > admin IO. > > Its pretty much your use case. We clean it up every 6-8 weeks and if > something breaks we just redeploy the whole thing from scratch. Performance > is great and its a very simple and economic system to administrate. No need > for the whole ceph daemon engine with large RAM requirements and extra admin > daemons. > > Use ceph for data you want to survive a nuclear blast. Don't use it for > things its not made for and then complain. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: mhnx <morphinwith...@gmail.com> > Sent: Saturday, April 29, 2023 5:48 AM > To: Janne Johansson > Cc: Ceph Users > Subject: [ceph-users] Re: How can I use not-replicated pool (replication 1 or > raid-0) > > Hello Janne, thank you for your response. > > I understand your advice and be sure that I've designed too many EC > pools and I know the mess. This is not an option because I need SPEED. > > Please let me tell you, my hardware first to meet the same vision. > Server: R620 > Cpu: 2 x Xeon E5-2630 v2 @ 2.60GHz > Ram: 128GB - DDR3 > Disk1: 20x Samsung SSD 860 2TB > Disk2: 10x Samsung SSD 870 2TB > > My ssds does not have PLP. Because of that, every ceph write also > waits for TRIM. I want to know how much latency we are talking about > because I'm thinking of adding PLP NVME for wal+db cache to gain some > speed. > As you can see, I even try to gain from every TRIM command. > Currently I'm testing replication 2 pool and even this speed is not > enough for my use case. > Now I'm trying to boost the deletion speed because I'm writing and > deleting files all the time and this never ends. > I write this mail because replication 1 will decrease the deletion > speed but still I'm trying to tune some MDS+ODS parameters to increase > delete speed. > > Any help and idea will be great for me. Thanks. > Regards. > > > > Janne Johansson <icepic...@gmail.com>, 12 Nis 2023 Çar, 10:10 > tarihinde şunu yazdı: > > > > Den mån 10 apr. 2023 kl 22:31 skrev mhnx <morphinwith...@gmail.com>: > > > Hello. > > > I have a 10 node cluster. I want to create a non-replicated pool > > > (replication 1) and I want to ask some questions about it: > > > > > > Let me tell you my use case: > > > - I don't care about losing data, > > > - All of my data is JUNK and these junk files are usually between 1KB to > > > 32MB. > > > - These files will be deleted in 5 days. > > > - Writable space and I/O speed is more important. > > > - I have high Write/Read/Delete operations, minimum 200GB a day. > > > > That is "only" 18MB/s which should easily be doable even with > > repl=2,3,4. or EC. This of course depends on speed of drives, network, > > cpus and all that, but in itself it doesn't seem too hard to achieve > > in terms of average speeds. We have EC8+3 rgw backed by some 12-14 OSD > > hosts with hdd and nvme (for wal+db) that can ingest over 1GB/s if you > > parallelize the rgw streams, so 18MB/s seems totally doable with 10 > > decent machines. Even with replication. > > > > > I'm afraid that, in any failure, I won't be able to access the whole > > > cluster. Losing data is okay but I have to ignore missing files, > > > > Even with repl=1, in case of a failure, the cluster will still aim at > > fixing itself rather than ignoring currently lost data and moving on, > > so any solution that involves "forgetting" about lost data would need > > a ceph operator telling the cluster to ignore all the missing parts > > and to recreate the broken PGs. This would not be automatic. > > > > > > -- > > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io