Oh wow, a lot to read piled up in one night :)
First things first: I want to thank you all for your insights and for the
really valuable knowledge I pulled from this mailthread.
Regarding flash only:
We use flash only clusters for our RBD clusters. This is very nice and most
of the maintanance is
We have a 8 hosts cluster with 4TB NVMe drive per host for now. The pool
size is 2 and it's hosting RBD images for VMs.
Each host has 128GB RAM installed.
This week one of the hosts went down.
Right when the recovery started, everything went crazy. OSDs from other
hosts went down being killed by O
Hi Josh,
There is 1 OSD per host.
There are 3 pools of 256, 128 and 32 PGs (total = 416 PGs across 8 OSDs).
ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus
(stable)
I still have 1 OSD where docker reports 61GB RAM being consumed by the
container (we have containerized dep
Hi Sage,
I uploaded a lot of debug logs from the OSDs and Mons:
ceph-post-file: 4ebc2eeb-7bb1-48c4-bbfa-ed581faca74f
At 13:24:25 I stopped OSD 122 and one Minute later I started it again.
In both cases I got slow ops.
Currently I running the upstream Version (without crude patches)
ceph versio
Hi Marius,
Your mempools show quite high usage in osd_pglog and buffer_anon,
which reminds me of this issue:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7IMIWCKIHXNULEBHVUIXQQGYUDJAO2SF/#Y2225AVEZYCBIJXXATJIJAXUWKNP4H3I
You can configure the pglog size to reduce the memory us
Also similar: https://tracker.ceph.com/issues/51609
On Fri, Nov 12, 2021 at 5:02 PM Dan van der Ster wrote:
>
> Hi Marius,
>
> Your mempools show quite high usage in osd_pglog and buffer_anon,
> which reminds me of this issue:
>
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7I
Folks,
New here - I tried searching for this topic in the archive, couldn't find
any since 2018 or so. So starting a new thread. I am looking at the impact
of node failures. I found this doc:
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/operations_guide/handling-a-n
Hi,
Another question is why “ ceph orch ls osd” reports in the RUNNING
column the value x/24, why 24?
can you share your 'ceph osd tree' and maybe also 'ceph -s'? I would
assume that you have a few dead or down OSDs, but it's hard to tell.
1/ see which disk are in each OSD service_id?
Hello all,
I have an issue that developed after restarting my cluster. I shut down a
healthy working cluster. Before shutting down the nodes, I issued the
following:
sudo ceph osd set noout
sudo ceph osd set nobackfill
sudo ceph osd set norecover
When I restarted all the nodes, several of the
Hi Stephen,
it would be nice to see failing OSD startup log...
Thanks,
Igor
On 11/12/2021 11:37 PM, Stephen J. Thompson wrote:
Before shutting down
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 8
Hello Igor,
The OSD logs a empty.
Enclosed is the end of the ceph-volume.log
Regards,
Stephen
[2021-11-12 21:23:54,191][ceph_volume.main][INFO ] Running command: ceph-
volume inventory --format=json --filter-for-batch
[2021-11-12 21:23:54,194][ceph_volume.process][INFO ] Running command: /
> - IIUC, if a root SSD fails, there is pretty much >no way to rebuild a
> new>node with the same OSDs and avoid data >shuffling - is this correct?You
> can still rebuild the node and add old OSDs and avoid shuffling. Might need
> to enable NOOUT flag while you work on configuration of new node.
Thanks for the answers - some clarifications:
- RE WAL drive loss: Can I assume the WAL drive is used ONLY if there is
write traffic? IOW, can I protect against a SPOF like that by keeping
clusters cold after the initial data load and serving only reads?
- I do see the docs around 1-4% in terms of
I am running Pacific 16.2.4 cluster and recently noticed that rm -rf
visibly hangs on the old directories. Cluster is healthy, has a
light load, and any newly created directories deleted immediately (well rm
returns command prompt immediately). The directories in question have 10 -
20 small text
The metadata pool is on the same type of drives as other pools; every node
uses SATA SSDs. They are all read / write mix DC types. Intel and Seagate.
On Fri, Nov 12, 2021 at 8:02 PM Anthony D'Atri
wrote:
> MDS RAM cache vs going to the metadata pool? What type of drives is your
> metadata poo
15 matches
Mail list logo
