thanks. I tried and it improved the situation by double the speed to 10MB/s
or something. Good catch for the fix!
It would be good to be at 50MB/s of recovery as far as cluster
infrastructure could support in my case. there may be other constraint on
resource utilization for recovery that I am
Just to be clear, you should remove the osd by stopping the daemon and
marking it out before you repair the PG. The pg may not be able to be
repaired until you remove the bad disk.
1 - identify the bad disk (via scrubs or SMART/dmesg inspection)
2 - stop daemon and mark it out
3 - wait for PG to
If I recall correctly When the acting or up_set of an PG changes the scrub
information is lost. This was likely lost when you stopped osd.238 and
changed the sets.
I do not believe based on your initial post you need to be using the
objectstore tool currently. Inconsistent PGs are a common
I've ran additional tests with Pacific releases and with "ceph-volume
inventory" things went wrong with the first v16.11 release
(v16.2.11-20230125)
=== Ceph v16.2.10-20220920 ===
Device Path Size rotates available Model name
/dev/sdc
This afternoon I have a look at the python file but do not manage how it
works with containers as I am only a Fortran HPC programmer... but I
found that "cephadm gather-facts" shows all the HDD in Pacific.
Some quick tests show:
== Nautilus ==
[root@mostha1 ~]#
Thank you, Frank. This confirms that monitors indeed do this, and
Our boot drives in 3 systems are smaller 1 DWPD drives (RAID1 to protect
against a random single drive failure), and over 3 years mons have eaten
through 60% of their endurance. Other systems have larger boot drives and
2% of their
Here are the notes from this week's CLT call. The call focused heavily on
release process, specifically around figuring out which patches are
required for a release.
- 17.2.7 status
- A few more FS PRs and one core PR then we can start release process
- Trying to finalize list of
That's really strange. Just out of curiosity, have you tried Quincy
(and/or Reef) as well? I don't recall what inventory does in the
background exactly, I believe Adam King mentioned that in some thread,
maybe that can help here. I'll search for that thread tomorrow.
Zitat von Patrick
Oh wow! I never bothered looking, because on our hardware the wear is so low:
# iotop -ao -bn 2 -d 300
Total DISK READ : 0.00 B/s | Total DISK WRITE : 6.46 M/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 6.47 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN
Eugon,
Thank you, however I'm am still lost. I can create a subvolume group, that I
understand. The issue is '/volume/' isn't a real directory on the host,
it's a 'virtual directory' in cephfs. '/mnt/tank/database' is a real folder
structure on the host. I can't `mv /mnt/tank/database
(transferred from https://github.com/ceph/ceph-csi/discussions/4181)
> What's the best practices of accessing ceph over flaky network connection?
> For example, can I setup a local dm-cache binding ceph with a local SSD to
> buffer the I/O? Thanks.
A flaky network will usually be quite
Hello Wes,
Thank you for your response.
brc1admin:~ # rados list-inconsistent-obj 15.f4f
No scrub information available for pg 15.f4f
brc1admin:~ # ceph osd ok-to-stop osd.238
OSD(s) 238 are ok to stop without reducing availability or risking data,
provided there are no other concurrent
Hello All,
Greetings. We've a Ceph Cluster with the version
*ceph version 14.2.16-402-g7d47dbaf4d
(7d47dbaf4d0960a2e910628360ae36def84ed913) nautilus (stable)
===
Issues: 1 pg in inconsistent state and does not recover.
# ceph -s
cluster:
id:
Eugen,
Thanks for your response. May I ask what numbers you're referring to?
I am not referring to monitor store.db sizes. I am specifically referring
to writes monitors do to their store.db file by frequently rotating and
replacing them with new versions during compactions. The size of the
Hi Eugen,
[root@mostha1 ~]# rpm -q cephadm
cephadm-16.2.14-0.el8.noarch
Log associated to the
2023-10-11 16:16:02,167 7f820515fb80 DEBUG
cephadm ['gather-facts']
2023-10-11 16:16:02,208 7f820515fb80 DEBUG
Can you check which cephadm version is installed on the host? And then
please add (only the relevant) output from the cephadm.log when you
run the inventory (without the --image ). Sometimes the
version mismatch on the host and the one the orchestrator uses can
cause some disruptions. You
Hi Eugen,
first many thanks for the time spent on this problem.
"ceph osd purge 2 --force --yes-i-really-mean-it" works and clean all
the bas status.
*[root@mostha1 ~]# cephadm shell
*Inferring fsid 250f9864-0142-11ee-8e5f-00266cf8869c
Using recent ceph image
That all looks normal to me, to be honest. Can you show some details
how you calculate the "hundreds of GB per day"? I see similar stats as
Frank on different clusters with different client IO.
Zitat von Zakhar Kirpichenko :
Sure, nothing unusual there:
---
cluster:
id:
Your response is a bit confusing since it seems to be mixed up with
the previous answer. So you still need to remove the OSD properly, so
purge it from the crush tree:
ceph osd purge 2 --force --yes-i-really-mean-it (only in a test cluster!)
If everything is clean (OSD has been removed,
Hi Eugen,
sorry for posting twice, my zimbra server returns an error at the first
attempt.
My initial problem is that ceph cannot detect these HDD since Pacific.
So I have deployed Octopus, where "ceph orch apply osd
--all-available-devices" works fine and then upgraded to Pacific.
But
Hi Eugen,
- the OS is Alma Linux 8 with latests updates.
- this morning I've worked with ceph-volume but it ends with a strange
final state. I was connected on host mostha1 where /dev/sdc was not
reconized. These are the steps followed based on the ceph-volume
documentation I've read:
Don't use ceph-volume manually to deploy OSDs if your cluster is
managed by cephadm. I just wanted to point out that you hadn't wiped
the disk properly to be able to re-use it. Let the orchestrator handle
the OSD creation and activation. I recommend to remove the OSD again,
wipe it
Hi Eugen,
- the OS is Alma Linux 8 with latests updates.
- this morning I've worked with ceph-volume but it ends with a strange
final state. I was connected on host mostha1 where /dev/sdc was not
reconized. These are the steps followed based on the ceph-volume
documentation I've read:
Sure, nothing unusual there:
---
cluster:
id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86
health: HEALTH_OK
services:
mon: 5 daemons, quorum ceph01,ceph03,ceph04,ceph05,ceph02 (age 2w)
mgr: ceph01.vankui(active, since 12d), standbys: ceph02.shsinf
osd: 96 osds: 96 up
Can you add some more details as requested by Frank? Which mgr modules
are enabled? What's the current 'ceph -s' output?
Is autoscaler running and doing stuff?
Is balancer running and doing stuff?
Is backfill going on?
Is recovery going on?
Is your ceph version affected by the "excessive
We don't use CephFS at all and don't have RBD snapshots apart from some
cloning for Openstack images.
The size of mon stores isn't an issue, it's < 600 MB. But it gets
overwritten often causing lots of disk writes, and that is an issue for us.
/Z
On Wed, 11 Oct 2023 at 14:37, Eugen Block
Do you use many snapshots (rbd or cephfs)? That can cause a heavy
monitor usage, we've seen large mon stores on customer clusters with
rbd mirroring on snapshot basis. In a healthy cluster they have mon
stores of around 2GB in size.
@Eugen: Was there not an option to limit logging to the
Thank you, Frank.
The cluster is healthy, operating normally, nothing unusual is going on. We
observe lots of writes by mon processes into mon rocksdb stores,
specifically:
/var/lib/ceph/mon/ceph-cephXX/store.db:
65M 3675511.sst
65M 3675512.sst
65M 3675513.sst
65M 3675514.sst
65M
I need to ask here: where exactly do you observe the hundreds of GB written per
day? Are the mon logs huge? Is it the mon store? Is your cluster unhealthy?
We have an octopus cluster with 1282 OSDs, 1650 ceph fs clients and about 800
librbd clients. Per week our mon logs are about 70M, the
Thanks Eugen. Operation complete:
root@hcn03:/imagework# ceph osd pool delete infra-pool infra-pool
--yes-i-really-really-mean-it
pool 'infra-pool' removed
Everything clean and tidy again.
Thanks for your help and support.
--- Original Message ---
On Wednesday, October 11th, 2023
Hello Kamil,
There is stackhpc.cephadm Ansible collection
(https://galaxy.ansible.com/ui/repo/published/stackhpc/cephadm/) that would
probably fit most of your needs - other option is to ceph orch ls —export and
then store it in git for import purposes - but it won’t cover everything.
Best
Thank you, Eugen.
I'm interested specifically to find out whether the huge amount of data
written by monitors is expected. It is eating through the endurance of our
system drives, which were not specced for high DWPD/TBW, as this is not a
documented requirement, and monitors produce hundreds of
On 10/11/23 11:42, Kamil Madac wrote:
Is it possible to do it with cephadm? Is it possible to have some config
files in git and then apply same cluster configuration on multiple
clusters? Or is this approach not aligned with cephadm and we should do it
different way?
You can export the
On 9/5/23 16:53, Max Carrara wrote:
> Hello there,
>
> could you perhaps provide some more information on how (or where) this
> got fixed? It doesn't seem to be fixed yet on the latest Ceph Quincy
> and Reef versions, but maybe I'm mistaken. I've provided some more
> context regarding this below,
Hello ceph community,
Currently we have deployed ceph clusters with ceph-ansible and whole
configuration (number od daemons, osd configurations, rgw configurations,
crush configuration, ...) of each cluster is stored in git and ansible
variables and we can recreate clusters with ceph-ansible in
Hi,
what you report is the expected behaviour, at least I see the same on
all clusters. I can't answer why the compaction is required that
often, but you can control the log level of the rocksdb output:
ceph config set mon debug_rocksdb 1/5 (default is 4/5)
This reduces the log entries
Any input from anyone, please?
On Tue, 10 Oct 2023 at 09:44, Zakhar Kirpichenko wrote:
> Any input from anyone, please?
>
> It's another thing that seems to be rather poorly documented: it's unclear
> what to expect, what 'normal' behavior should be, and what can be done
> about the huge amount
Hi,
just wondering if 'ceph-volume lvm zap --destroy /dev/sdc' would help
here. From your previous output you didn't specify the --destroy flag.
Which cephadm version is installed on the host? Did you also upgrade
the OS when moving to Pacific? (Sorry if I missed that.
Zitat von Patrick
Hi Casey,
thank you for an update. Now it's clear why it happened and we will adopt
our code for multipart.
Cheers,
Arvydas
On Tue, Oct 10, 2023, 18:36 Casey Bodley wrote:
> hi Arvydas,
>
> it looks like this change corresponds to
> https://tracker.ceph.com/issues/48322 and
>
Le 02/10/2023 à 18:22, Patrick Bégou a écrit :
Hi all,
still stuck with this problem.
I've deployed octopus and all my HDD have been setup as osd. Fine.
I've upgraded to pacific and 2 osd have failed. They have been
automatically removed and upgrade finishes. Cluster Health is finaly
OK, no
Hi,
check out this thread [1] as well, where Anh Phan pointed out:
Not really sure what you want, but for simplicity, just move folder
to following structure:
/volumes/[Sub Volume Group Name]/[Sub Volume Name]
ceph will recognize it (no extend attr needed), if you use
subvolumegroup name
Hi,
then I misinterpreted your message and thought you were actually
surprised about the trash image. Yeah I don't think messing with
hexedit really helped here, but I'm not sure either. Anyway, let us
know how it went.
Zitat von Rhys Goodwin :
Thanks again Eugen. Looking at my command
42 matches
Mail list logo
