We have a reef 18.2.2 cluster with 6 radosgw servers on Rocky 8.9. The radosgw
servers are not fronted by anything like HAProxy as the clients connect
directly to a DNS name via a round-robin DNS. Each of the radosgw servers have
a certificate using SAN entries for all 6 radosgw servers as
I have both a small test cluster and a larger production cluster. They are
(were, for the test cluster) running Rocky Linux 8.9. They are both updated
originally from Pacific, currently at reef 18.2.2.These are all rpm installs.
It has come time to consider upgrades to Rocky 9.3. As there is
wishes,
Pierre Riteau
On Wed, 8 May 2024 at 19:51, Christopher Durham wrote:
> Hello,
> I am uisng 18.2.2 on Rocky 8 Linux.
>
> I am getting http error 500 whe trying to hit the ceph dashboard on reef
> 18.2.2 when trying to look at any of the radosgw pages.
> I tracked this
Hello,
I am uisng 18.2.2 on Rocky 8 Linux.
I am getting http error 500 whe trying to hit the ceph dashboard on reef 18.2.2
when trying to look at any of the radosgw pages.
I tracked this down to /usr/share/ceph/mgr/dashboard/controllers/rgw.py
It appears to parse the metadata for a given radosgw
Hi,
I have a reef cluster 18.2.2 on Rocky 8.9. This cluster has been upgraded from
pacific->quincy->reef over the past few years. It is a multi site with one
other cluster that works fine with s3/radosgw on both sides, with proper
bidirectional data replication.
On one of the master cluster's
Hi,
I am upgrading my test cluster from 17.2.6 (quincy) to 18.2.2 (reef).
As it was an rpm install, i am following the directions here:
Reef — Ceph Documentation
|
|
| |
Reef — Ceph Documentation
|
|
|
The upgrade worked, but I have some observations and questions before I move to
Hi,
The following article:
https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/
suggests that dsabling C-states on your CPUs (on the OSD nodes) as one method
to improve performance. The article seems to indicate that the scenariobeing
addressed in the article was with NVMEs as OSDs.
to go to reef, but we are not quite there yet.
-Chris
On Thursday, September 21, 2023 at 11:17:51 AM MDT, Casey Bodley
wrote:
On Thu, Sep 21, 2023 at 12:21 PM Christopher Durham wrote:
>
>
> Hi Casey,
>
> This is indeed a multisite setup. The other side shows that fo
narrow
things down as soon as the issue starts, and maybe garner more datato track
this down further. We'll eee.
-chris
On Thursday, September 21, 2023 at 11:17:51 AM MDT, Casey Bodley
wrote:
On Thu, Sep 21, 2023 at 12:21 PM Christopher Durham wrote:
>
>
> Hi Casey,
>
>
tion log entries
for multisite. can you confirm that this is a multisite setup? is the
'bucket sync status' mostly caught up on each zone? in a healthy
multisite configuration, these log entries would eventually get
trimmed automatically
On Wed, Sep 20, 2023 at 7:08 PM Christopher Durham wrote:
>
>
I am using ceph 17.2.6 on Rocky 8.
I have a system that started giving me large omap object warnings.
I tracked this down to a specific index shard for a single s3 bucket.
rados -p listomapkeys .dir..bucketid.nn.shardid
shows over 3 million keys for that shard. There are only about 2
million
-verify disabled. But I didn't check the tracker
for any pending tickets, so someone might be working on it.
Regards,
Eugen
[1] https://github.com/ceph/ceph/pull/47207/files
Zitat von Christopher Durham :
> Hi,
> I am using 17.2.6 on Rocky Linux 8
> The ceph mgr dashboard, in my situation,
Hi,
I am using 17.2.6 on Rocky Linux 8
The ceph mgr dashboard, in my situation, (bare metal install, upgraded from
15->16-> 17.2.6), can no longer hit the ObjectStore->(Daemons,Users,Buckets)
pages.
When I try to hit those pages, it gives an error:
RGW REST API failed request with status code
^.1001_/echo -n -e x801001_/'; echo ' > mykey &&
rados rmomapkey -p default.rgw.buckets.index
.dir.zone.bucketid.xx.indexshardnumber --omap-key-file mykey';
done < uglykeys.txt
On Tue, Jul 18, 2023 at 9:27 AM Christopher Durham wrote:
Hi,
I am using ceph 17.2.6 on rocky
do_remove.sh | sh -x"
while read f;
do
echo -n $f | sed 's/^.1001_/echo -n -e x801001_/'; echo ' > mykey &&
rados rmomapkey -p default.rgw.buckets.index
.dir.zone.bucketid.xx.indexshardnumber --omap-key-file mykey';
done < uglykeys.txt
On Tue, Jul 18, 2023 at 9:27
1001_/'; echo ' > mykey &&
rados rmomapkey -p default.rgw.buckets.index
.dir.zone.bucketid.xx.indexshardnumber --omap-key-file mykey';
done < uglykeys.txt
On Tue, Jul 18, 2023 at 9:27 AM Christopher Durham wrote:
Hi,
I am using ceph 17.2.6 on rocky linux 8.
I got a large omap object war
Hi,
I am using ceph 17.2.6 on rocky linux 8.
I got a large omap object warning today.
Ok, So I tracked it down to a shard for a bucket in the index pool of an s3
pool.
However, when lisitng the omapkeys with:
# rados -p pool.index listomapkeys .dir.zone.bucketid.xx.indexshardnumber
it is clear
Hi,
I am using 17.2.6 on rocky linux for both the master and the slave site
I noticed that:
radosgw-admin sync status
often shows that the metadata sync is behind a minute or two on the slave. This
didn't make sense, as the metadata isn't changing as far as I know.
radosgw-admin mdlog list
(on
Hi,
After doing a 'radosgw-admin metadata sync init' on a secondary site in a
2-cluster multisite configuration, (See my previous post on doing a quincy
upgade with sts roles manually synced between primary and secondary), and
letting it sync all metadata from a master zone, I get the following
-Original Message-
From: Casey Bodley
To: Christopher Durham
Cc: ceph-users@ceph.io
Sent: Tue, Apr 11, 2023 1:59 pm
Subject: [ceph-users] Re: ceph 17.2.6 and iam roles (pr#48030)
On Tue, Apr 11, 2023 at 3:53 PM Casey Bodley wrote:
>
> On Tue, Apr 11, 2023 at 3:19 PM Christopher
Hi,
I see that this PR: https://github.com/ceph/ceph/pull/48030
made it into ceph 17.2.6, as per the change log at:
https://docs.ceph.com/en/latest/releases/quincy/ That's great.
But my scenario is as follows:
I have two clusters set up as multisite. Because of the lack of replication
for
Question:
What does the future hold with regard to cephadm vs rpm/deb packages? If it is
now suggested to use cephadm and thus containers to deploy new clusters, what
does the future hold? Is there an intent, at sometime in the future, to no
longer support rpm/deb packages for Linux systems,
-Original Message-
From: Stephen Smith6
To: Christopher Durham ; ceph-users@ceph.io
Sent: Wed, Jan 11, 2023 2:00 pm
Subject: Re: [ceph-users] pg mapping verification
#yiv3876159325 filtered {}#yiv3876159325 filtered {}#yiv3876159325
p.yiv3876159325MsoNormal, #yiv3876159325
:
crushtool -i crushmap.bin --test --rule 5 --show-mappings --num-rep 6 | head
CRUSH rule 5 x 0 [19,7,13,22,16,28]
CRUSH rule 5 x 1 [21,3,15,31,19,7]
[...]
Regards,
Eugen
Zitat von Christopher Durham :
> Hi,
> For a given crush rule and pool that uses it, how can I verify hat
> the
Hi,
For a given crush rule and pool that uses it, how can I verify hat the pgs in
that pool folllow the rule? I have a requirement to 'prove' that the pgs are
mapping correctly.
I see: https://pypi.org/project/crush/
This allows me to read in a crushmap file that I could then use to verify a
Hi,
Is there any information on this issue? Max number of OSDs per pool, or maxpool
size (data) as opposed to cluster size? Thanks!
-Chris
-Original Message-
From: Christopher Durham
To: ceph-users@ceph.io
Sent: Thu, Dec 15, 2022 5:36 pm
Subject: max pool size (amount of data
Hi,
There are various articles, case studies, etc about large ceph clusters,
storing 10s of PiB,with CERN being the largest cluster as far as I know.
Is there a largest pool capacity limit? In other words, while you may have a
30PiB cluster,is there a limit or recommendation as to max pool
Hello,
I have an ec pool set as 6+2. I have noticed that when rebooting servers during
system upgrades, I get pgs set to inactive while the osds are down. I then
discovered that my min_size for the pool is set to 7, which makes sense that if
I reboot two servers that host a pg with OSDs on
Hello,
In what releases of ceph is s3 select in rgw available? From what I can glean,
we have:
pacific: csvquincy: csv, parquet
Am I correct?
Thanks!
-Chris
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to
Hi,
I've seen Dan's talk:
https://www.youtube.com/watch?v=0i7ew3XXb7Q
and other similar ones that talk about CLUSTER size.
But, I see nothing (perhaps I have not looked hard enough), on any
recommendations regarding max POOL size.
So, are there any limitations on a given pool that has all OSDs of
know what happens, although it could take a bit of time both
inscheduling it and then execution.
Thank you for your help, it is appreciated.
-Chris
-Original Message-
From: Dan van der Ster
To: Christopher Durham
Cc: ceph-users@ceph.io
Sent: Fri, Oct 14, 2022 2:25 am
Subject: [ceph
int.
Any other thoughts would be appreciated.
-Chris
-Original Message-
From: Dan van der Ster
To: Christopher Durham
Cc: Ceph Users
Sent: Tue, Oct 11, 2022 11:39 am
Subject: [ceph-users] Re: crush hierarchy backwards and upmaps ...
Hi Chris,
Just curious, does this rule make se
indep 4 type rack
step chooseleaf indep 2 type host
step emit }
-Chris
-Original Message-
From: Dan van der Ster
To: Christopher Durham
Cc: Ceph Users
Sent: Mon, Oct 10, 2022 12:22 pm
Subject: [ceph-users] Re: crush hierarchy backwards and upmaps ...
Hi,
Here's a similar bug
Hello,
I am using pacific 16.2.10 on Rocky 8.6 Linux.
After setting upmap_max_deviation to 1 on the ceph balancer in ceph-mgr, I
achieved a near perfect balance of PGs and space on my OSDs. This is great.
However, I started getting the following errors on my ceph-mon logs, every
three minutes,
Hi,
There is a potential that my Ceph RGW multi site soluton may be down for an
extended time (2 weeks?) for a physical relocation. Some questions,
particularly in regard to RGW
1. Is there any limit on downtime after which I might have to restart an entire
sync? I want to still be able to
Hi,
I am uisng 15.2.7 on CentOS 8.1. I have a number of old buckets that are listed
with
# radosgw-admin metadata list bucket.instance
but are not listed with:
# radosgw-admin bucket list
Lets say that one of them is:
'old-bucket' and its instance is 'c100feda-5e16-48a4-b908-7be61aa877ef.123.1'
36 matches
Mail list logo
