Hi Darren, >> I'm going to deploy a new cluster soon based on 6.4TB NVME PCI-E Cards, I >> will have only 1 NVME card per node and 38 nodes. >> >> The use case is to offer cephfs volumes for a k8s platform, I plan to use an >> EC-POOL 8+3 for the cephfs_data pool. >> >> Do you have recommendations for the setup or mistakes to avoid? I use >> ceph-ansible to deploy all myclusters.
As describe in another mail, the context is: 3 Intel Server 1U for MONs/MDSs/MGRs services + K8s daemons CPU : 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (24c/48t) Memory : 64GB Disk OS : 2x Intel SSD DC S3520 240GB Network : SFP+ 10Gb/s 38 Dell C4140 1U for OSD node : CPU : 2 x Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz (28c/56t) Memory : 384GB GPU : 4 Nvidia V100 32GB NVLink Disk OS : M.2 240G NVME : Dell 6.4TB NVME PCI-E Drive (Samsung PM1725b), only 1 slot available Network : 2x SFP+ 10Gb/s (1 NIC for public network and 1 NIC for private Network) Each server is used in a k8s cluster to give access to GPUs and CPUs for X-learning labs. Ceph have to share the CPU and memory with the K8s cluster. I plan to have only 1 filesystem > So I would not be putting 1 x 6.4TB device in but multiple smaller devices in > each node. Because I don't have place for more > What CPU are you thinking of using? > How many CPU's? If you have 1 PCIe card then it will only be connected to 1 > CPU so will you be able to use all of the performance of multiple CPU's. We plan to use two Xeon Gold 6132. > What network are you thinking of? Wouldn't be doing less than 100G or > multiple 25G connections. Unfortunately, we are not ready for 25Gb/s neither 100Gb/s yet, "only" 10Gbits/s fat tree network. > Where is you CephFS metadata being stored? What are you doing about CephFS > metadata servers? The CephFS metadata will be on separate servers as describe above. > What about some faster storage for the WAL? Dell 6.4TB NVME PCI-E Drives (Samsung PM1725b) look fast for me, don't they? > What is your IO Profile? Read/Write split? IO Profile will be mixed cases, this cephfs will be used as scratch storage to compute processes in our K8s clusters. > It may be the case that EC is not the best fit for the workload you are > trying to do. As usual, we have to choose between performance and usable space... Thanks, Best regards, -- Yoann Moulin EPFL IC-IT _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
