Good morning,
Cephadm Reef 18.2.1. We recently added 4 hosts and changed a failure
domain from host to datacenter which is the reason for the large
misplaced percentage.
We were seeing some pretty crazy spikes in "OSD Read Latencies" and "OSD
Write Latencies" on the dashboard. Most of the ti
Should we bring to life this again?
On Tue, Mar 19, 2024, 8:14 PM Mark Nelson wrote:
> A long time ago Wido used to have a bot logging IRC afaik, but I think
> that's been gone for some time.
>
>
> Mark
>
>
> On 3/19/24 19:36, Alvaro Soto wrote:
> > Hi Community!!!
> > Are we logging IRC channel
Hi Michael,
i think yesterday i found the culprit in my case.
After inspecting "ceph pg dump" and especially the column
"last_scrub_duration". I found, that any PG without proper scrubbing
was located on one of three OSDs (and all these OSDs share the same
SSD for their DB). I put them on "out" a
Hi Torkil,
highly likely you're facing a well known issue with RocksDB performance
drop after bulk data removal. The latter might occur at source OSDs
after PG migration completion.
You might want to use DB compaction (preferably offline one using
ceph-kvstore-tool) to get OSD out of this "d
On 22-03-2024 08:38, Igor Fedotov wrote:
Hi Torkil,
Hi Igor
highly likely you're facing a well known issue with RocksDB performance
drop after bulk data removal. The latter might occur at source OSDs
after PG migration completion.
Aha, thanks.
You might want to use DB compaction (prefera
Hi,
what does your node-exporter spec look like?
ceph orch ls node-exporter --export
If other node-exporter daemons are running in the cluster, what's the
difference between them? Do they all have the same container image?
ceph config get mgr mgr/cephadm/container_image_node_exporter
and c
Hi,
As I said in my initial message, I'd in mind to do exactly the same as I
identified in my initial analysis that all the PGs with this problem
where sharing one OSD (but only 20 PGs had the problem over ~200 hosted
by the OSD). But as I don't feel I'm in an urgent situation, I was
wonderin
Hello Torkil,
The easiest way (in my opinion) to perform offline compaction is a bit
different than what Igor suggested. We had a prior off-list
conversation indicating that the results would be equivalent.
1. ceph config set osd osd_compact_on_start true
2. Restart the OSD that you want to compa
Hello Michel,
It might be worth mentioning that the next releases of Reef and Quincy
should increase the default value of osd_max_scrubs from 1 to 3. See the
Reef pull request: https://github.com/ceph/ceph/pull/55173
You could try increasing this configuration setting if you haven't already,
but n
Pierre,
Yes, as mentioned in my initial email, I checked the OSD state and found
nothing wrong either in the OSD logs or in the system logs (SMART errors).
Thanks for the advice of increasing osd_max_scrubs, I may try it, but I
doubt it is a contention problem because it really only affects a
Hi all,
I have setup a test cluster with 3 servers,
Everything has default values with a replication
of 3.
I have created one volume called gds-common
and the data pool has been configured with compression lz4
and compression_mode aggressive.
I have copied 71TB data to this volume but I can
no
Hello Michel,
Pierre also suggested checking the performance of this OSD's device(s) which
can be done by running a ceph tell osd.x bench.
One think I can think of is how the scrubbing speed of this very OSD could be
influenced by mclock sheduling, would the max iops capacity calculated by thi
Hello!
After upgrading "5.15.0-84-generic" to "5.15.0-100-generic" (Ubuntu 22.04.2
LTS) , commit latency started acting weird with "CT4000MX500SSD" drives.
osd commit_latency(ms) apply_latency(ms)
36 867867
373045 3045
38
Hi Frédéric,
I think you raise the right point, sorry if I misunderstood Pierre's
suggestion to look at OSD performances. Just before reading your email,
I was implementing Pierre's suggestion for max_osd_scrubs and I saw the
osd_mclock_max_capacity_iops_hdd for a few OSDs (I guess those with
https://askubuntu.com/questions/1454997/how-to-stop-sys-from-changing-usb-ssd-provisioning-mode-from-unmap-to-full-in-ub
How to stop sys from changing USB SSD provisioning_mode from unmap to full in
Ubuntu 22.04?
askubuntu.com
?
> On Mar 22, 2024, at 09:36, Özkan Göksu wrote:
>
> Hello!
>
>
Hello Anthony, thank you for the answer.
While researching I also found out this type of issues but the thing I did
not understand is in the same server the OS drives "SAMSUNG MZ7WD480" is
all good.
root@sd-01:~# lsblk -D
NAME DISC-ALN DISC-GRAN DISC-MAX
Hello again.
In ceph recommendations I found this:
https://docs.ceph.com/en/quincy/start/hardware-recommendations/
WRITE CACHES
Enterprise SSDs and HDDs normally include power loss protection features
which ensure data durability when power is lost while operating, and use
multi-level caches to
Maybe because the Crucial units are detected as client drives? But also look
at the device paths and the output of whatever "disklist" is. Your boot drives
are SATA and the others are SAS which seems even more likely to be a factor.
> On Mar 22, 2024, at 10:42, Özkan Göksu wrote:
>
> Hello A
Michel,
Glad to know that was it.
I was wondering when would per OSD osd_mclock_max_capacity_iops_hdd value be
set in cluster's config database since I don't have any set in my lab.
Turns out the per OSD osd_mclock_max_capacity_iops_hdd is only set when the
calculated value is belo
Personally, I don't think the compaction is actually required. Reef
has compact-on-iteration enabled, which should take care of this
automatically. We see this sort of delay pretty often during PG
cleaning, at the end of a PG being cleaned, when the PG has a high
count of objects, whether or not OS
After I set these 2 udev rules:
root@sd-02:~# cat /etc/udev/rules.d/98-ceph-provisioning-mode.rules
ACTION=="add", SUBSYSTEM=="scsi_disk", ATTR{provisioning_mode}:="unmap"
root@sd-02:~# cat /etc/udev/rules.d/99-ceph-write-through.rules
ACTION=="add", SUBSYSTEM=="scsi_disk", ATTR{cache_type}:="wri
Hi,
The attempt to rerun the bench was not really a success. I got the
following messages:
-
Mar 22 14:48:36 idr-osd2 ceph-osd[326854]: osd.29 83873
maybe_override_max_osd_capacity_for_qos osd bench result - bandwidth
(MiB/sec): 10.910 iops: 2792.876 elapsed_sec: 1.074
Mar 22 14:48:36 i
Hey Eugen,
Thank you for the quick reply.
The 5 missing disks on the one host were completely installed after I fully
cleaned them up as I described.
So, seems a smaller number of disks can make it.
Regarding the other host with 40 disks:
Failing the MGR didn't have any effect.
There are nor er
Frédéric,
We arrived at the same conclusions! I agree that an insane low value
would be a good addition: the idea would be that the benchmark emits a
warning about the value but the it will not put a value lower than the
minimum defined. I don't have a precise idea of the possible bad side
ef
Perhaps emitting an extremely low value could have value for identifying a
compromised drive?
> On Mar 22, 2024, at 12:49, Michel Jouvin
> wrote:
>
> Frédéric,
>
> We arrived at the same conclusions! I agree that an insane low value would be
> a good addition: the idea would be that the benc
Follow-up, changing the title to the real topic...
I did more tests on my OSDs using "ceph tell osd.x bench..." as advised
by
https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#benchmarking-test-steps-using-osd-bench
(exact impact of "cache drop" is not clear/visible based
On Fri, Mar 22, 2024 at 04:29:21PM +0100, Frédéric Nass wrote:
A/ these incredibly low values were calculated a while back with an unmature
version of the code or under some specific hardware conditions and you can hope
this won't happen again
The OSD run bench and update osd_mclock_max_capac
I'm afraid the parameter mgr mgr/cephadm/default_cephadm_command_timeout is
buggy.
Once not on default anymore, MGR is preparing the parameter a bit (e.g.
substracting 5 secs)
And there making it float, but cephadm is not having it (not even if I try the
default 900 myself):
[WRN] CEPHADM_APPLY
Michel,
Log says that osd.29 is providing 2792 '4k' iops at 10.910 MiB/s. These figures
suggest that a controller write-back cache is in use along the IO path. Is that
right?
Since 2792 is above 500, osd_mclock_max_capacity_iops_hdd falls back to 315 and
OSD is suggesting runni
> The OSD run bench and update osd_mclock_max_capacity_iops_{hdd,ssd} every
> time the OSD is started.
> If you check the OSD log you'll see it does the bench.
Are you sure about the update on every start? Does the update happen only if
the benchmark result is < 500 iops?
Loo
On Fri, Mar 22, 2024 at 06:51:44PM +0100, Frédéric Nass wrote:
The OSD run bench and update osd_mclock_max_capacity_iops_{hdd,ssd} every time
the OSD is started.
If you check the OSD log you'll see it does the bench.
Are you sure about the update on every start? Does the update happen only
Hi John,
> A few major features we have planned include:
> * Standalone servers (internally defined users/groups)
No concerns here
> * Active Directory Domain Member Servers
In the second case, what is the plan regarding UID mapping? Is NFS
coexistence planned, or a concurrent mount of the same
Hi,
Very basic question : 2 days ago I reboot all the cluster. Everything work
fine. But I'm guessing during the shutdown 4 osd was mark as crash
[WRN] RECENT_CRASH: 4 daemons have recently crashed
osd.381 crashed on host cthulhu5 at 2024-03-20T18:33:12.017102Z
osd.379 crashed on host ct
Hi,
With our small cluster (11 nodes) I notice ceph log a lot
Beside to keep that somewhere «just in case», is they are anything to check
regularly in the log (in prevention of more serious problem) ? Or can we
trust «ceph health» and use the log only for debug.
Regards
--
Albert SHIH 🦫 🐸
Fr
You can use the `ceph crash` interface to view/archive recent crashes. [1]
To list recent crashes: ceph crash ls-new
To get information about a particular crash: ceph crash info
To silence a crash: ceph crash archive
To silence all active crashes: ceph crash archive-all
[1]
https://docs.ceph.c
If the documentation is to be believed, it's just install the zabbix
sender, then;
ceph mgr module enable zabbix
ceph zabbix config-set zabbix_host my-zabbix-server
(Optional) Set the identifier to the fsid.
And poof. I should now have a discovered entity on my zabbix server to add
templates to
Sure! I think Wido just did it all unofficially, but afaik we've lost
all of those records now. I don't know if Wido still reads the mailing
list but he might be able to chime in. There was a ton of knowledge in
the irc channel back in the day. With slack, it feels like a lot of
discussions
I put it on the list for the next CLT. :) (though I imagine it will move to
the infrastructure meeting from there.)
On Fri, Mar 22, 2024 at 4:42 PM Mark Nelson wrote:
> Sure! I think Wido just did it all unofficially, but afaik we've lost
> all of those records now. I don't know if Wido still
Hi All,
I'm looking for some help/advice to solve the issue outlined in the heading.
I'm running CepfFS (name: cephfs) on a Ceph Reef (v18.2.2 - latest
update) cluster, connecting from a laptop running Rocky Linux v9.3
(latest update) with KDE v5 (latest update).
I've set up the laptop to co
Hi All,
I'm trying to mount a Ceph Reef (v18.2.2 - latest version) RBD Image as
a 2nd HDD on a Rocky Linux v9.3 (latest version) host.
The EC pool has been created and initialised and the image has been
created.
The ceph-common package has been installed on the host.
The correct keyring ha
40 matches
Mail list logo
