Hi,
Do slow ops impact data integrity => No
Can I generally ignore it => No :)
This means that some client transactions are blocked for 120 sec (that's a
lot).
This could be a lock on the client side (CephFS, essentially), an incident
on the infrastructure side (a disk about to fall, network
Hi Daniel,
Changing pg_num when some OSD is almost full is not a good strategy (or
even dangerous).
What is causing this backfilling? loss of an OSD? balancer? other ?
What is the least busy OSD level (sort -nrk17)
Is the balancer activated? (upmap?)
Once the situation stabilizes, it becomes
> mon weight myself, do you know how that happened?
>
> Zitat von "David C." :
>
> Ok, got it :
>>
>> [root@pprod-admin:/var/lib/ceph/]# ceph mon dump -f json-pretty
>> |egrep "name|weigh"
>> dumped monmap epoch 14
>>
;name": "pprod-mon3",
"weight": 10,
"name": "pprod-osd2",
"weight": 0,
"name": "pprod-osd1",
"weight": 0,
"name": "pprod-osd
I took the wrong ligne =>
https://github.com/ceph/ceph/blob/v17.2.6/src/mon/MonClient.cc#L822
Le jeu. 7 mars 2024 à 18:21, David C. a écrit :
>
> Hello everybody,
>
> I'm encountering strange behavior on an infrastructure (it's
> pre-production but it's very ugly). After a
Hello everybody,
I'm encountering strange behavior on an infrastructure (it's pre-production
but it's very ugly). After a "drain" on monitor (and a manager). MGRs all
crash on startup:
Mar 07 17:06:47 pprod-mon1 ceph-mgr[564045]: mgr ms_dispatch2 standby
mgrmap(e 1310) v1
Mar 07 17:06:47
sent to devices and the discard latency, as well (via
> node_exporter, for example).
>
> Matt
>
>
> On 2024-03-02 06:18, David C. wrote:
> > I came across an enterprise NVMe used for BlueFS DB whose performance
> > dropped sharply after a few months of de
I came across an enterprise NVMe used for BlueFS DB whose performance
dropped sharply after a few months of delivery (I won't mention the brand
here but it was not among these 3: Intel, Samsung, Micron).
It is clear that enabling bdev_enable_discard impacted performance, but
this option also saved
Hello,
Each rack works on different trees or is everything parallelized ?
The meta pools would be distributed over racks 1,2,4,5 ?
If it is distributed, even if the addressed MDS is on the same switch as
the client, you will always have this MDS which will consult/write (nvme)
OSDs on the other
if rebalancing tasks have been launched it's not a big deal, but I don't
think it's the priority.
The priority being to get the MDS back on its feet.
I haven't seen an answer to this question: can you stop/unmount cephfs
clients or not ?
There are other solutions but as you are not comfortable I
Do you have the possibility to stop/unmount cephfs clients ?
If so, do that and restart the MDS.
It should restart.
Have the clients restart one by one and check that the MDS does not crash
(by monitoring the logs)
Cordialement,
*David
look at ALL cephfs kernel clients (no effect on RGW)
Le ven. 23 févr. 2024 à 16:38, a écrit :
> And we dont have parameter folder
>
> cd /sys/module/ceph/
> [root@cephgw01 ceph]# ls
> coresize holders initsize initstate notes refcnt rhelversion
> sections srcversion taint uevent
>
>
Hi,
The problem seems to come from the clients (reconnect).
Test by disabling metrics on all clients:
echo Y > /sys/module/ceph/parameters/disable_send_metrics
Cordialement,
*David CASIER*
Hello Albert,
this should return you the sockets used on the network cluster :
ceph report | jq '.osdmap.osds[] | .cluster_addrs.addrvec[] | .addr'
Cordialement,
*David CASIER*
mple shows that it's impossible to get any metrics in an IPv6 only
> network (Discovery is impossible) and it's visible at install so there's no
> test for IPv6 only environnement before release ?
>
> Now I'm seriously asking myself to put a crappy IPv4 subnet only for my
rue
> then this should definitely be in the docs as a warning for EC pools
> in cephfs!
>
> Zitat von "David C." :
>
> > In case the root is EC, it is likely that is not possible to apply the
> > disaster recovery procedure,
# ceph fs new cephfs cephfs_metadata cephfs_data --force
> new fs with metadata pool 6 and data pool 8
>
> CC'ing Zac here to hopefully clear that up.
>
> Zitat von "David C." :
>
> > Albert,
> > Never used EC for (root) data pool.
> >
> > Le jeu. 25 ja
Albert,
Never used EC for (root) data pool.
Le jeu. 25 janv. 2024 à 12:08, Albert Shih a écrit :
> Le 25/01/2024 à 08:42:19+, Eugen Block a écrit
> > Hi,
> >
> > it's really as easy as it sounds (fresh test cluster on 18.2.1 without
> any
> > pools yet):
> >
> > ceph:~ # ceph fs volume
Hi,
The client calculates the location (PG) of an object from its name and the
crushmap.
This is what makes it possible to parallelize the flows directly from the
client.
The client also has the map of the PGs which are relocated to other OSDs
(upmap, temp, etc.)
Hi Albert,
In this scenario, it is more consistent to work with subvolumes.
Regarding security, you can use namespaces to isolate access at the OSD
level.
What Robert emphasizes is that creating pools dynamically is not without
effect on the number of PGs and (therefore) on the architecture (PG
n daemon type
> node-exporter
>
> Tried to remove & recreate service : it's the same ... how to stop the
> rotation now :-/
>
>
>
> Le mar. 23 janv. 2024 à 17:18, David C. a écrit :
>
>> Is the cep
ion to create this certificate inside the Key
> store but how ... that's the point :-(
>
> Regards.
>
>
>
> Le mar. 23 janv. 2024 à 15:52, David C. a écrit :
>
>> Hello Nicolas,
>>
>> I don't know if it's an update issue.
>>
>> If this is
Hello Nicolas,
I don't know if it's an update issue.
If this is not a problem for you, you can consider redeploying
grafana/prometheus.
It is also possible to inject your own certificates :
https://docs.ceph.com/en/latest/cephadm/services/monitoring/#example
Hi Sake,
I would start by decrementing max_mds by 1:
ceph fs set atlassian-prod max_mds 2
The mds.1 no longer restarts?
logs?
Le jeu. 21 déc. 2023 à 08:11, Sake Ceph a écrit :
> Starting a new thread, forgot subject in the previous.
> So our FS down. Got the following error, what can I
Hi Mohamed,
Changing weights is no longer a good practice.
The balancer is supposed to do the job.
The number of pg per osd is really tight on your infrastructure.
Can you display the ceph osd tree command?
Cordialement,
*David CASIER*
the point of view of
> global storage capacity.
>
> Patrick
>
> Le 05/12/2023 à 12:19, David C. a écrit :
>
> Hi,
>
> To return to my comparison with SANs, on a SAN you have spare disks to
> repair a failed disk.
>
> On Ceph, you therefore need at least on
Hi,
To return to my comparison with SANs, on a SAN you have spare disks to
repair a failed disk.
On Ceph, you therefore need at least one more host (k+m+1).
If we take into consideration the formalities/delivery times of a new
server, k+m+2 is not luxury (Depending on the growth of your
Hi Matthew,
To make a simplistic comparison, it is generally not recommended to raid 5
with large disks (>1 TB) due to the probability (low but not zero) of
losing another disk during the rebuild.
So imagine losing a host full of disks.
Additionally, min_size=1 means you can no longer maintain
scard": "false",
> "bdev_enable_discard": "false",
>
>
>
> Istvan Szabo
> Staff Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> -----
*
Le lun. 4 déc. 2023 à 06:01, Szabo, Istvan (Agoda)
a écrit :
> With the nodes that has some free space on that namespace, we don't have
> issue, only with this which is weird.
> --
> *From:* Anthony D'Atri
> *Sent:* Friday, December 1, 2023 10:53 PM
Hi,
It looks like a trim/discard problem.
I would try my luck by activating the discard on a disk, to validate.
I have no feedback on the reliability of the bdev_*_discard parameters.
Maybe dig a little deeper into the subject or if anyone has any feedback...
Hi Guiseppe,
Wouldn't you have clients who heavily load the MDS with concurrent access
on the same trees ?
Perhaps, also, look at the stability of all your clients (even if there are
many) [dmesg -T, ...]
How are your 4 active MDS configured (pinning?) ?
Probably nothing to do but normal for 2
Hello Albert,
5 vs 3 MON => you won't notice any difference
5 vs 3 MGR => by default, only 1 will be active
Le sam. 18 nov. 2023 à 09:28, Albert Shih a écrit :
> Le 17/11/2023 à 11:23:49+0100, David C. a écrit
>
> Hi,
>
> >
> > 5 instead of 3 mon will allow
figure out how to enable cephadm's access to the
> machines.
>
> Anyway, thanks for your reply.
>
> Luis Domingues
> Proton AG
>
>
> On Friday, 17 November 2023 at 13:55, David C.
> wrote:
>
>
> > Hi,
> >
> > You can use the cephadm account (instea
Hi,
You can use the cephadm account (instead of root) to control machines with
the orchestrator.
Le ven. 17 nov. 2023 à 13:30, Luis Domingues a
écrit :
> Hi,
>
> I noticed when installing the cephadm rpm package, to bootstrap a cluster
> for example, that a user cephadm was created. But I do
ven. 17 nov. 2023 à 11:22, Jean-Marc FONTANA
a écrit :
> Hello, everyone,
>
> There's nothing cephadm.log in /var/log/ceph.
>
> To get something else, we tried what David C. proposed (thanks to him !!)
> and found:
>
> nov. 17 10:53:54 svtcephmonv3 ceph-mgr[727]: [balancer
Hi Albert ,
5 instead of 3 mon will allow you to limit the impact if you break a mon
(for example, with the file system full)
5 instead of 3 MDS, this makes sense if the workload can be distributed
over several trees in your file system. Sometimes it can also make sense to
have several FSs in
each rbd?
>
> Respectfully,
>
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>
>
> On Wed, Nov 15, 2023 at 1:14 PM David C. wrote:
>
>> rbd create testpool/test3 --size=100M
>> rbd snap limit set testp
rbd create testpool/test3 --size=100M
rbd snap limit set testpool/test3 --limit 3
Le mer. 15 nov. 2023 à 17:58, Wesley Dillingham a
écrit :
> looking into how to limit snapshots at the ceph level for RBD snapshots.
> Ideally ceph would enforce an arbitrary number of snapshots allowable per
>
Hi Jean Marc,
maybe look at this parameter "rgw_enable_apis", if the values you have
correspond to the default (need rgw restart) :
https://docs.ceph.com/en/quincy/radosgw/config-ref/#confval-rgw_enable_apis
ceph config get client.rgw rgw_enable_apis
Hi Daniel,
it's perfectly normal for a PG to freeze when the primary osd is not stable.
It can sometimes happen that the disk fails but doesn't immediately send
back I/O errors (which crash the osd).
When the OSD is stopped, there's a 5-minute delay before it goes down in
the crushmap.
Le
on this architecture.
Cordialement,
*David CASIER*
Le jeu. 9 nov. 2023 à 08:48, Albert Shih a écrit :
> Le 08/11/2023 à 19:29:19+0100, David C. a écrit
> Hi David.
>
>
Hi Albert,
What would be the number of replicas (in total and on each row) and their
distribution on the tree ?
Le mer. 8 nov. 2023 à 18:45, Albert Shih a écrit :
> Hi everyone,
>
> I'm totally newbie with ceph, so sorry if I'm asking some stupid question.
>
> I'm trying to understand how the
Without (raid/jbod) controller ?
Le mer. 8 nov. 2023 à 18:36, Peter a écrit :
> Hi All,
>
> I note that HDD cluster commit delay improves after i turn off HDD cache.
> However, i also note that not all HDDs are able to turn off the cache.
> special I found that two HDD with same model number,
so the next step is to place the pools on the right rule :
ceph osd pool set db-pool crush_rule fc-r02-ssd
Le mer. 8 nov. 2023 à 12:04, Denny Fuchs a écrit :
> hi,
>
> I've forget to write the command, I've used:
>
> =
> ceph osd crush move fc-r02-ceph-osd-01 root=default
> ceph osd
, David C. a écrit :
> Hi,
>
> It seems to me that before removing buckets from the crushmap, it is
> necessary to do the migration first.
> I think you should restore the initial crushmap by adding the default root
> next to it and only then do the migration.
> There should be so
Hi,
It seems to me that before removing buckets from the crushmap, it is
necessary to do the migration first.
I think you should restore the initial crushmap by adding the default root
next to it and only then do the migration.
There should be some backfill (probably a lot).
Hi Dominique,
The consistency of the data should not be at risk with such a problem.
But on the other hand, it's better to solve the network problem.
Perhaps look at the state of bond0 :
cat /proc/net/bonding/bond0
As well as the usual network checks
Hi,
I've just checked with the team and the situation is much more serious than
it seems: the lost disks contained the MONs AND OSDs databases (5 servers
down out of 8, replica 3).
It seems that the team fell victim to a bad batch of Samsung 980 Pros (I'm
not a big fan of this "Pro" range, but
Hi Mohamed,
I understand there's one operational monitor, isn't there?
If so, you need to reprovision the other monitors on an empty base so that
they synchronize with the only remaining monitor.
Cordialement,
*David CASIER*
Hi Hubert,
It's an error "125" (ECANCELED) (and there may be many reasons for it).
I see a high latency (144sec), is the object big ?
No network problems ?
Cordialement,
*David CASIER*
. 2023 à 18:11, David C. a écrit :
> Hi Michel,
>
> (I'm just discovering the existence of this module, so it's possible I'm
> making mistakes)
>
> The rgw module is new and only seems to be there to configure multisite.
>
> It is present on the v17.2.6 branch but I don't
Hi Michel,
(I'm just discovering the existence of this module, so it's possible I'm
making mistakes)
The rgw module is new and only seems to be there to configure multisite.
It is present on the v17.2.6 branch but I don't see it in the container for
this version.
In any case, if you're not
. 20 oct. 2023 à 13:08, David C. a écrit :
> Someone correct me if I'm saying something stupid but from what I see in
> the code, there is a check each time to make sure rctime doesn't go back.
> Which seems logical to me because otherwise you would have to go through
> all
Someone correct me if I'm saying something stupid but from what I see in
the code, there is a check each time to make sure rctime doesn't go back.
Which seems logical to me because otherwise you would have to go through
all the children to determine the correct ctime.
I don't have the impression
Hi,
Michel,
the pool already appears to be in automatic autoscale ("autoscale_mode on").
If you're worried (if, for example, the platform is having trouble handling
a large data shift) then you can set the parameter to warn (like the
rjenkis pool).
If not, as Hervé says, the transition to 2048
>
> I don't think this is necessary. It _is_ necessary to convert all
> leveldb to rocksdb before upgrading to Pacific, on both mons and any
> filestore OSDs.
Thanks, Josh, I guess that explains why some people had issues with
Filestore OSDs post Pacific upgrade
On Tue, Dec 6, 2022 at 4:07 PM
Rocky has a
> similar path I think.
>
> - you will need to love those filestore OSD’s to Bluestore before hitting
> Pacific, might even be part of the Nautilus upgrade. This takes some time
> if I remember correctly.
>
> - You may need to upgrade monitors to RocksDB too.
>
&
Hi All
I'm planning to upgrade a Luminous 12.2.10 cluster to Pacific 16.2.10,
cluster is primarily used for CephFS, mix of Filestore and Bluestore
OSDs, mons/osds collocated, running on CentOS 7 nodes
My proposed upgrade path is: Upgrade to Nautilus 14.2.22 -> Upgrade to
EL8 on the nodes
My understanding is BeeGFS doesn't offer data redundancy by default,
you have to configure mirroring. You've not said how your Ceph cluster
is configured but my guess is you have the recommended 3x replication
- I wouldn't be surprised if BeeGFS was much faster than Ceph in this
case. I'd be
Success!
I remembered I had a server I'd taken out of the cluster to
investigate some issues, that had some good quality 800GB Intel DC
SSDs, dedicated an entire drive to swap, tuned up min_free_kbytes,
added an MDS to that server and let it run. Took 3 - 4 hours but
eventually came back online.
On Thu, Oct 22, 2020 at 6:09 PM Dan van der Ster wrote:
>
>
>
> On Thu, 22 Oct 2020, 19:03 David C, wrote:
>>
>> Thanks, guys
>>
>> I can't add more RAM right now or have access to a server that does,
>> I'd fear it wouldn't be enough anyway. I'll gi
Dan van der Ster
> Sent: 22 October 2020 18:11:57
> To: David C
> Cc: ceph-devel; ceph-users
> Subject: [ceph-users] Re: Urgent help needed please - MDS offline
>
> I assume you aren't able to quickly double the RAM on this MDS ? or
> failover to a new MDS with more ram?
>
&g
cause here https://tracker.ceph.com/issues/45090 )
> I don't know enough about the changelog diffs to suggest upgrading
> right now in the middle of this outage.
>
>
> -- dan
>
> On Thu, Oct 22, 2020 at 4:14 PM David C wrote:
> >
> > I've not touched the journal
dan
>
>
>
>
>
>
>
> -- dan
>
> On Thu, Oct 22, 2020 at 3:35 PM David C wrote:
> >
> > Dan, many thanks for the response.
> >
> > I was going down the route of looking at mds_beacon_grace but I now
> > realise when I start my MDS, it's swall
> Is it using lots of CPU/RAM? If you increase debug_mds do you see some
> progress?
>
> -- dan
>
>
> On Thu, Oct 22, 2020 at 2:01 PM David C wrote:
> >
> > Hi All
> >
> > My main CephFS data pool on a Luminous 12.2.10 cluster hit capac
Hi All
My main CephFS data pool on a Luminous 12.2.10 cluster hit capacity
overnight, metadata is on a separate pool which didn't hit capacity but the
filesystem stopped working which I'd expect. I increased the osd full-ratio
to give me some breathing room to get some data deleted once the
67 matches
Mail list logo
