[ceph-users] Re: Erasure coded pool chunk count k

[Anthony D'Atri]

The larger the value of K relative to M, the more efficient the raw :: usable 
ratio ends up.

There are tradeoffs and caveats.  Here are some of my thoughts; if I’m off-base 
here, I welcome enlightenment. 



When possible, it’s ideal to have at least K+M failure domains — often racks, 
sometimes hosts, chassis, etc.  Thus smaller clusters, say with 5-6 nodes, 
aren’t good fits for larger sums of K+M if your data is valuable.

Larger sums of K+M also mean that more drives will be touched by each read or 
write, especially during recovery.  This could be a factor if one is 
IOPS-limited.  Same with scrubs.

When using a pool for, eg. RGW buckets, larger sums of K+M may result in 
greater overhead when storing small objects, since Ceph / RGW only AIUI writes 
full stripes.  So say you have an EC pool of 17,3 on drives with the default 
4kB bluestone_min_alloc size.  A 1kB S3 object would thus allocate 17+3=20 x 
4kB == 80kB of storage, which is 7900% overhead.  This is an extreme example to 
illustrate the point.

Larger sums of K+M may present more IOPs to each storage drive, dependent on 
workload and the EC plugin selected.

With larger objects (including RBD) the modulo factor is dramatically smaller.  
One’s use-case and dataset per-pool may thus inform the EC profiles that make 
sense; workloads that are predominately smaller objects might opt for 
replication instead.

There was a post ….. a year ago? suggesting that values with small prime 
factors are advantageous, but I never saw a discussion of why that might be.

In some cases where one might be pressured to use replication with only 2 
copies of data, a 2,2 EC profile might achieve the same efficiency with greater 
safety.

Geo / stretch clusters or ones in challenging environments are a special case; 
they might choose values of M equal to or even larger than K.

That said, I think 4,2 is a reasonable place to *start*, adjusted by one’s 
specific needs.  You get a raw :: usable ratio of 1.5 without getting too 
complicated.

ymmv






> 
> Hi,
> 
> It depends of hardware, failure domain, use case, overhead.
> 
> I don’t see an easy way to chose k and m values.
> 
> -
> Etienne Menguy
> etienne.men...@croit.io
> 
> 
>> On 4 Oct 2021, at 16:57, Golasowski Martin  wrote:
>> 
>> Hello guys,
>> how does one estimate number of chunks for erasure coded pool ( k = ? ) ? I 
>> see that number of m chunks determines the pool’s resiliency, however I did 
>> not find clear guideline how to determine k.
>> 
>> Red Hat states that they support only the following combinations:
>> 
>> k=8, m=3
>> k=8, m=4
>> k=4, m=2
>> 
>> without any rationale behind them.
>> The table is taken from 
>> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/storage_strategies_guide/erasure_code_pools.
>> 
>> Thanks!
>> 
>> Regards,
>> Martin
>> 
>> 
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Daemon Version Mismatch (But Not Really?) After Deleting/Recreating OSDs

[Gregory Farnum]

On Mon, Oct 4, 2021 at 12:05 PM Edward R Huyer  wrote:
>
> Apparently the default value for container_image in the cluster configuration 
> is "docker.io/ceph/daemon-base:latest-pacific-devel".  I don't know where 
> that came from.  I didn't set it anywhere.  I'm not allowed to edit it, 
> either (from the dashboard, anyway).
>
> The container_image_base for the cephadm module is "docker.io/ceph/ceph".
>
> Also, 16.2.6 is already out, so I'm not sure why I'd be getting 16.2.5 
> development releases.
>
> Is this possibly related to the issues with docker.io and move to quay.io?

A good guess, but like I said this whole area is way outside my
wheelhouse. I just know how to decode Ceph's URL and git version
conventions. ;)

>
> -Original Message-
> From: Gregory Farnum 
> Sent: Monday, October 4, 2021 2:33 PM
> To: Edward R Huyer 
> Cc: ceph-users@ceph.io
> Subject: Re: [ceph-users] Daemon Version Mismatch (But Not Really?) After 
> Deleting/Recreating OSDs
>
> On Mon, Oct 4, 2021 at 7:57 AM Edward R Huyer  wrote:
> >
> > Over the summer, I upgraded my cluster from Nautilus to Pacific, and 
> > converted to use cephadm after doing so.  Over the past couple weeks, I've 
> > been converting my OSDs to use NVMe drives for db+wal storage.  Schedule a 
> > node's worth of OSDs to be removed, wait for that to happen, delete the PVs 
> > and zap the drives, let the orchestrator do its thing.
> >
> > Over this past weekend, the cluster threw up a HEALTH_WARN due to 
> > mismatched daemon versions.  Apparently the recreated OSDs are reporting 
> > different version information from the old daemons.
> >
> > New OSDs:
> >
> > -  Container Image Name:  
> > docker.io/ceph/daemon-base:latest-pacific-devel
> >
> > -  Container Image ID: d253896d959e
> >
> > -  Version: 16.2.5-226-g7c9eb137
>
> I haven't done any work with cephadm, but this container name and the version 
> tag look like you've installed the in-development next version of Pacific, 
> not the released 16.2.5. Did you perhaps manage to put a phrase similar to 
> "pacific-dev" somewhere instead of "pacific"?
>
> >
> > Old OSDs and other daemons:
> >
> > -  Container Image Name: docker.io/ceph/ceph:v16
> >
> > -  Container Image ID: 6933c2a0b7dd
> >
> > -  Version: 16.2.5
> >
> > I'm assuming this is not actually a problem and will go away when I next 
> > upgrade the cluster, but I figured I'd throw it out here in case someone 
> > with more knowledge than I thinks otherwise.  If it's not a problem, is 
> > there a way to silence it until I next run an upgrade?  Is there an 
> > explanation for why it happened?
> >
> > -
> > Edward Huyer
> > Golisano College of Computing and Information Sciences Rochester
> > Institute of Technology Golisano 70-2373
> > 152 Lomb Memorial Drive
> > Rochester, NY 14623
> > 585-475-6651
> > erh...@rit.edu
> >
> > Obligatory Legalese:
> > The information transmitted, including attachments, is intended only for 
> > the person(s) or entity to which it is addressed and may contain 
> > confidential and/or privileged material. Any review, retransmission, 
> > dissemination or other use of, or taking of any action in reliance upon 
> > this information by persons or entities other than the intended recipient 
> > is prohibited. If you received this in error, please contact the sender and 
> > destroy any copies of this information.
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> > email to ceph-users-le...@ceph.io
> >
>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input

[Stefan Kooman]

On 10/4/21 14:19, von Hoesslin, Volker wrote:



   -7598> 2021-10-04T11:27:17.438+0200 7f529998c700 -1 mds.0.openfiles
_load_finish: corrupted header/values: void
Anchor::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past
end  of struct encoding: Malformed input

^^ openfiles object(s) corrupted.



<<
sounds bad! what does it mean? can we fixed it?


If you still have any clients that have (or try to) mount CephFS, umount 
them for now.


AFAIK you cannot fix this, but you can remove it (it's corrupt anyway):

- Stop all the MDSes.
- rados rm -p cephfs_metadata mds0_openfiles.0. This is more or less 
harmless [1]


Not sure if that is the only issue, but you will find out after you 
start an MDS and try to mount it from a client.


Gr. Stefan

[1]: 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028981.html

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [External Email] Re: ceph-objectstore-tool core dump

[Dave Hall]

I also had a delay on the start of the repair scrub when I was dealing with
this issue.  I ultimately increased the number of simultaneous scrubs, but
I think you could also temporarily disable scrubs and then re-issue the 'pg
repair'.  (But I'm not one of the experts on this.)

My perception is that between EC pools, large HDDs, and the overall OSD
count, there might need to be some tuning to assure that scrubs can get
scheduled:  A large HDD contains pieces of more PGs.  Each PG in an EC pool
is spread across more disks than a replication pool.  Thus, especially if
the number of OSDs is not large, there is an increased chance that more
than one scrub will want to read the same OSD.   Scheduling nightmare if
the number of simultaneous scrubs is low and client traffic is given
priority.

-Dave

-Dave

--
Dave Hall
Binghamton University
kdh...@binghamton.edu
607-760-2328 (Cell)
607-777-4641 (Office)


On Sun, Oct 3, 2021 at 11:51 PM 胡 玮文  wrote:

> > 在 2021年10月4日，04:18，Michael Thomas  写道：
> >
> > On 10/3/21 12:08, 胡 玮文 wrote:
>  在 2021年10月4日，00:53，Michael Thomas  写道：
> >>>
> >>> I recently started getting inconsistent PGs in my Octopus (15.2.14)
> ceph cluster.  I was able to determine that they are all coming from the
> same OSD: osd.143.  This host recently suffered from an unplanned power
> loss, so I'm not surprised that there may be some corruption.  This PG is
> part of a EC 8+2 pool.
> >>>
> >>> The OSD logs from the PG's primary OSD show this and similar errors
> from the PG's most recent deep scrub:
> >>>
> >>> 2021-10-03T03:25:25.969-0500 7f6e6801f700 -1 log_channel(cluster) log
> [ERR] : 23.1fa shard 143(1) soid 23:5f8c3d4e:::1179969.0168:head :
> candidate had a read error
> >>>
> >>> In attempting to fix it, I first ran 'ceph pg repair 23.1fa' on the
> PG. This accomplished nothing.  Next I ran a shallow fsck on the OSD:
> >> I expect this ‘ceph pg repair’ command could handle this kind of
> errors. After issuing this command, the pg should enter a state like
> “active+clean+scrubbing+deep+inconsistent+repair”, then you wait for the
> repair to finish (this can take hours), and you should be able to recover
> from the inconsistent state. What do you mean by “This accomplished
> nothing”?
> >
> > The PG never entered the 'repair' state, nor did anything appear in the
> primary OSD logs about a request for repair.  After more than 24 hours, the
> PG remained listed as 'inconsistent'.
> >
> > --Mike
>
> I have encountered a similar situation. My case is the pg being repaired
> cannot get all the scrub reservations to enter the scrubbing state. Could
> you try “ceph tell osd. dump_scrubs”, and see whether
> 23.1fa is listed and has forced == true? If so, this may also be your case.
> I think you may wait even longer, raise “osd_max_scrubs” config, or try set
> then unset noscrub to interrupt the running scrubs.
>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Adopting "unmanaged" OSDs into OSD service specification

[David Orman]

We have an older cluster which has been iterated on many times. It's
always been cephadm deployed, but I am certain the OSD specification
used has changed over time. I believe at some point, it may have been
'rm'd.

So here's our current state:

root@ceph02:/# ceph orch ls osd --export
service_type: osd
service_id: osd_spec_foo
service_name: osd.osd_spec_foo
placement:
  label: osd
spec:
  data_devices:
rotational: 1
  db_devices:
rotational: 0
  db_slots: 12
  filter_logic: AND
  objectstore: bluestore
---
service_type: osd
service_id: unmanaged
service_name: osd.unmanaged
placement: {}
unmanaged: true
spec:
  filter_logic: AND
  objectstore: bluestore

root@ceph02:/# ceph orch ls
NAMEPORTS  RUNNING  REFRESHED  AGE  PLACEMENT
crash  7/7  10m ago14M  *
mgr5/5  10m ago7M   label:mgr
mon5/5  10m ago14M  label:mon
osd.osd_spec_foo 0/7  -  24m  label:osd
osd.unmanaged  167/167  10m ago-

The osd_spec_foo would match these devices normally, so we're curious
how we can get these 'managed' under this service specification.
What's the appropriate way in order to effectively 'adopt' these
pre-existing OSDs into the service specification that we want them to
be managed under?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Daemon Version Mismatch (But Not Really?) After Deleting/Recreating OSDs

[Edward R Huyer]

Apparently the default value for container_image in the cluster configuration 
is "docker.io/ceph/daemon-base:latest-pacific-devel".  I don't know where that 
came from.  I didn't set it anywhere.  I'm not allowed to edit it, either (from 
the dashboard, anyway).

The container_image_base for the cephadm module is "docker.io/ceph/ceph".

Also, 16.2.6 is already out, so I'm not sure why I'd be getting 16.2.5 
development releases.

Is this possibly related to the issues with docker.io and move to quay.io?

-Original Message-
From: Gregory Farnum  
Sent: Monday, October 4, 2021 2:33 PM
To: Edward R Huyer 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Daemon Version Mismatch (But Not Really?) After 
Deleting/Recreating OSDs

On Mon, Oct 4, 2021 at 7:57 AM Edward R Huyer  wrote:
>
> Over the summer, I upgraded my cluster from Nautilus to Pacific, and 
> converted to use cephadm after doing so.  Over the past couple weeks, I've 
> been converting my OSDs to use NVMe drives for db+wal storage.  Schedule a 
> node's worth of OSDs to be removed, wait for that to happen, delete the PVs 
> and zap the drives, let the orchestrator do its thing.
>
> Over this past weekend, the cluster threw up a HEALTH_WARN due to mismatched 
> daemon versions.  Apparently the recreated OSDs are reporting different 
> version information from the old daemons.
>
> New OSDs:
>
> -  Container Image Name:  
> docker.io/ceph/daemon-base:latest-pacific-devel
>
> -  Container Image ID: d253896d959e
>
> -  Version: 16.2.5-226-g7c9eb137

I haven't done any work with cephadm, but this container name and the version 
tag look like you've installed the in-development next version of Pacific, not 
the released 16.2.5. Did you perhaps manage to put a phrase similar to 
"pacific-dev" somewhere instead of "pacific"?

>
> Old OSDs and other daemons:
>
> -  Container Image Name: docker.io/ceph/ceph:v16
>
> -  Container Image ID: 6933c2a0b7dd
>
> -  Version: 16.2.5
>
> I'm assuming this is not actually a problem and will go away when I next 
> upgrade the cluster, but I figured I'd throw it out here in case someone with 
> more knowledge than I thinks otherwise.  If it's not a problem, is there a 
> way to silence it until I next run an upgrade?  Is there an explanation for 
> why it happened?
>
> -
> Edward Huyer
> Golisano College of Computing and Information Sciences Rochester 
> Institute of Technology Golisano 70-2373
> 152 Lomb Memorial Drive
> Rochester, NY 14623
> 585-475-6651
> erh...@rit.edu
>
> Obligatory Legalese:
> The information transmitted, including attachments, is intended only for the 
> person(s) or entity to which it is addressed and may contain confidential 
> and/or privileged material. Any review, retransmission, dissemination or 
> other use of, or taking of any action in reliance upon this information by 
> persons or entities other than the intended recipient is prohibited. If you 
> received this in error, please contact the sender and destroy any copies of 
> this information.
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
> email to ceph-users-le...@ceph.io
>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Stefan Kooman]

On 10/4/21 15:58, Konstantin Shalygin wrote:



On 4 Oct 2021, at 16:38, Stefan Kooman > wrote:


What procedure are you following to add the mon?


# ceph mon dump
epoch 10
fsid 677f4be1-cd98-496d-8b50-1f99df0df670
last_changed 2021-09-11 10:04:23.890922
created 2018-05-18 20:43:43.260897
min_mon_release 14 (nautilus)
0: [v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] mon.ceph-01
1: [v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0] mon.ceph-03
2: [v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0] mon.ceph-06
dumped monmap epoch 10


sudo -u ceph ceph mon getmap -o /tmp/monmap.map
got monmap epoch 10
sudo -u ceph ceph-mon -i mon2 --mkfs --monmap /tmp/monmap.map
chown -R ceph:ceph /var/lib/ceph/mon/ceph-mon2
systemctl start ceph-mon@mon2


I'm missing the part where keyring is downloaded and used:

ceph auth get mon. -o /tmp/keyring
ceph mon getmap -o /tmp/monmap
chown -R ceph:ceph /var/lib/ceph/mon/ceph-mon2
ceph-mon -i mon2 --mkfs --monmap /tmp/monmap --keyring /tmp/keyring 
--setuser ceph --setgroup ceph


Gr. Stefan

P.s. I have noticed, that sometimes you first need to reboot the node 
before you are able to start the monitor with systemd.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Daemon Version Mismatch (But Not Really?) After Deleting/Recreating OSDs

[Gregory Farnum]

On Mon, Oct 4, 2021 at 7:57 AM Edward R Huyer  wrote:
>
> Over the summer, I upgraded my cluster from Nautilus to Pacific, and 
> converted to use cephadm after doing so.  Over the past couple weeks, I've 
> been converting my OSDs to use NVMe drives for db+wal storage.  Schedule a 
> node's worth of OSDs to be removed, wait for that to happen, delete the PVs 
> and zap the drives, let the orchestrator do its thing.
>
> Over this past weekend, the cluster threw up a HEALTH_WARN due to mismatched 
> daemon versions.  Apparently the recreated OSDs are reporting different 
> version information from the old daemons.
>
> New OSDs:
>
> -  Container Image Name:  
> docker.io/ceph/daemon-base:latest-pacific-devel
>
> -  Container Image ID: d253896d959e
>
> -  Version: 16.2.5-226-g7c9eb137

I haven't done any work with cephadm, but this container name and the
version tag look like you've installed the in-development next version
of Pacific, not the released 16.2.5. Did you perhaps manage to put a
phrase similar to "pacific-dev" somewhere instead of "pacific"?

>
> Old OSDs and other daemons:
>
> -  Container Image Name: docker.io/ceph/ceph:v16
>
> -  Container Image ID: 6933c2a0b7dd
>
> -  Version: 16.2.5
>
> I'm assuming this is not actually a problem and will go away when I next 
> upgrade the cluster, but I figured I'd throw it out here in case someone with 
> more knowledge than I thinks otherwise.  If it's not a problem, is there a 
> way to silence it until I next run an upgrade?  Is there an explanation for 
> why it happened?
>
> -
> Edward Huyer
> Golisano College of Computing and Information Sciences
> Rochester Institute of Technology
> Golisano 70-2373
> 152 Lomb Memorial Drive
> Rochester, NY 14623
> 585-475-6651
> erh...@rit.edu
>
> Obligatory Legalese:
> The information transmitted, including attachments, is intended only for the 
> person(s) or entity to which it is addressed and may contain confidential 
> and/or privileged material. Any review, retransmission, dissemination or 
> other use of, or taking of any action in reliance upon this information by 
> persons or entities other than the intended recipient is prohibited. If you 
> received this in error, please contact the sender and destroy any copies of 
> this information.
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Stefan Kooman]

On 10/4/21 15:27, Konstantin Shalygin wrote:

Hi,

I was make a mkfs for new mon, but mon stuck on probing. On debug I see: fault 
on lossy channel, failing. This is a bad (lossy) network (crc mismatch)?


What procedure are you following to add the mon?

Is this physical hardware? Or a (cloned) VM?

Have you checked that you can connect with the other monitors on port 
6789 / 3300 (and vice versa)?


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: osd marked down

[Abdelillah Asraoui]

it is on rook ceph tool container:

rook-ceph-tools-5b5bfc786-nptecv /]# ls -l /var/lib/ceph/osd/ceph-3/keyring

-rwxrwxrwx. 1 ceph ceph 171 Oct  4 16:56 /var/lib/ceph/osd/ceph-3/keyring


thanks!

On Mon, Oct 4, 2021 at 12:16 PM Eugen Block  wrote:

> Did you put that file in the container?
>
>
> Zitat von Abdelillah Asraoui :
>
> > I have create the keyring file: andvar/lib/ceph/osd/ceph-3/keyring and
> > chown to ceph but still getting these error on the osd pod log:
> >
> > k -n rook-ceph logs rook-ceph-osd-3-6497bdc65b-5cvx3
> >
> > debug 2021-10-04T16:06:38.287+ 7f8633cc1f00 -1 auth: unable to find a
> > keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied
> >
> > debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 auth: unable to find a
> > keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied
> >
> > debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 auth: unable to find a
> > keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied
> >
> > debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 monclient: keyring not
> > found
> >
> > failed to fetch mon config (--no-mon-config to skip)
> >
> >
> > thanks!
> >
> >
> > On Fri, Oct 1, 2021 at 2:02 AM Eugen Block  wrote:
> >
> >> I'm not sure if anything else could break, but since the OSD isn't
> >> starting anyway... I guess you could delete osd.3 from ceph auth:
> >>
> >> ceph auth del osd.3
> >>
> >> And then recreate it with:
> >>
> >> ceph auth get-or-create osd.3 mon 'allow profile osd' osd 'allow *'
> >> mgr 'allow profile osd'
> >> [osd.3]
> >>  key = 
> >>
> >> Then create a keyring file /var/lib/ceph/osd/ceph-3/keyring with the
> >> respective content:
> >>
> >> [osd.3]
> >>  key = 
> >>  caps mgr = "allow profile osd"
> >>  caps mon = "allow profile osd"
> >>  caps osd = "allow *"
> >>
> >>
> >> Make sure the file owner is ceph and try to restart the OSD. In this
> >> case you wouldn't need to import anything. This just worked for me in
> >> my lab environment, so give it a shot.
> >>
> >>
> >>
> >> Zitat von Abdelillah Asraoui :
> >>
> >> > the /var/lib/ceph/osd/ceph-3/keyring is missing here ..
> >> > is there way to generate a keyring for osd.3 ?
> >> >
> >> >
> >> > thanks!
> >> >
> >> > On Thu, Sep 30, 2021 at 1:18 AM Eugen Block  wrote:
> >> >
> >> >> Is the content of OSD.3 still available in the filesystem? If the
> >> >> answer is yes you can get the OSD's keyring from
> >> >>
> >> >> /var/lib/ceph/osd/ceph-3/keyring
> >> >>
> >> >> Then update your osd.3.export file with the correct keyring and then
> >> >> import the correct back to ceph.
> >> >>
> >> >>
> >> >> Zitat von Abdelillah Asraoui :
> >> >>
> >> >> > I must have imported osd.2 key instead,  now osd.3 has the same
> key as
> >> >> osd.2
> >> >> >
> >> >> > ceph auth import -i osd.3.export
> >> >> >
> >> >> >
> >> >> > How do we update this ?
> >> >> >
> >> >> > thanks!
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Wed, Sep 29, 2021 at 2:13 AM Eugen Block  wrote:
> >> >> >
> >> >> >> Just to clarify, you didn't simply import the unchanged keyring
> but
> >> >> >> modified it to reflect the actual key of OSD.3, correct? If not,
> run
> >> >> >> 'ceph auth get osd.3' first and set the key in the osd.3.export
> file
> >> >> >> before importing it to ceph.
> >> >> >>
> >> >> >>
> >> >> >> Zitat von Abdelillah Asraoui :
> >> >> >>
> >> >> >> > i have created keyring for the osd3 but still pod is not booting
> >> up..
> >> >> >> >
> >> >> >> > As outlined:
> >> >> >> > https://access.redhat.com/solutions/3524771
> >> >> >> >
> >> >> >> > ceph auth export osd.2 -o osd.2.export
> >> >> >> > cp osd.2.export osd.3.export
> >> >> >> > ceph auth import -i osd.3.export
> >> >> >> > imported keyring
> >> >> >> >
> >> >> >> >
> >> >> >> > Any suggestions ?
> >> >> >> >
> >> >> >> > Thanks!
> >> >> >> >
> >> >> >> > On Tue, Sep 21, 2021 at 8:34 AM Abdelillah Asraoui <
> >> >> aasra...@gmail.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi,
> >> >> >> >>
> >> >> >> >> one of the osd in the cluster went down, is there a workaround
> to
> >> >> bring
> >> >> >> >> back this osd?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> logs from ceph osd pod shows the following:
> >> >> >> >>
> >> >> >> >> kubectl -n rook-ceph logs rook-ceph-osd-3-6497bdc65b-pn7mg
> >> >> >> >>
> >> >> >> >> debug 2021-09-20T14:32:46.388+ 7f930fe9cf00 -1 auth:
> unable to
> >> >> find
> >> >> >> a
> >> >> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
> >> denied
> >> >> >> >>
> >> >> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 auth:
> unable to
> >> >> find
> >> >> >> a
> >> >> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
> >> denied
> >> >> >> >>
> >> >> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 auth:
> unable to
> >> >> find
> >> >> >> a
> >> >> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
> >> denied
> >> >> >> >>
> >> >> >> >> debug 2021-09-2

[ceph-users] Re: osd marked down

[Eugen Block]

Did you put that file in the container?


Zitat von Abdelillah Asraoui :


I have create the keyring file: andvar/lib/ceph/osd/ceph-3/keyring and
chown to ceph but still getting these error on the osd pod log:

k -n rook-ceph logs rook-ceph-osd-3-6497bdc65b-5cvx3

debug 2021-10-04T16:06:38.287+ 7f8633cc1f00 -1 auth: unable to find a
keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied

debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 auth: unable to find a
keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied

debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 auth: unable to find a
keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied

debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 monclient: keyring not
found

failed to fetch mon config (--no-mon-config to skip)


thanks!


On Fri, Oct 1, 2021 at 2:02 AM Eugen Block  wrote:


I'm not sure if anything else could break, but since the OSD isn't
starting anyway... I guess you could delete osd.3 from ceph auth:

ceph auth del osd.3

And then recreate it with:

ceph auth get-or-create osd.3 mon 'allow profile osd' osd 'allow *'
mgr 'allow profile osd'
[osd.3]
 key = 

Then create a keyring file /var/lib/ceph/osd/ceph-3/keyring with the
respective content:

[osd.3]
 key = 
 caps mgr = "allow profile osd"
 caps mon = "allow profile osd"
 caps osd = "allow *"


Make sure the file owner is ceph and try to restart the OSD. In this
case you wouldn't need to import anything. This just worked for me in
my lab environment, so give it a shot.



Zitat von Abdelillah Asraoui :

> the /var/lib/ceph/osd/ceph-3/keyring is missing here ..
> is there way to generate a keyring for osd.3 ?
>
>
> thanks!
>
> On Thu, Sep 30, 2021 at 1:18 AM Eugen Block  wrote:
>
>> Is the content of OSD.3 still available in the filesystem? If the
>> answer is yes you can get the OSD's keyring from
>>
>> /var/lib/ceph/osd/ceph-3/keyring
>>
>> Then update your osd.3.export file with the correct keyring and then
>> import the correct back to ceph.
>>
>>
>> Zitat von Abdelillah Asraoui :
>>
>> > I must have imported osd.2 key instead,  now osd.3 has the same key as
>> osd.2
>> >
>> > ceph auth import -i osd.3.export
>> >
>> >
>> > How do we update this ?
>> >
>> > thanks!
>> >
>> >
>> >
>> > On Wed, Sep 29, 2021 at 2:13 AM Eugen Block  wrote:
>> >
>> >> Just to clarify, you didn't simply import the unchanged keyring but
>> >> modified it to reflect the actual key of OSD.3, correct? If not, run
>> >> 'ceph auth get osd.3' first and set the key in the osd.3.export file
>> >> before importing it to ceph.
>> >>
>> >>
>> >> Zitat von Abdelillah Asraoui :
>> >>
>> >> > i have created keyring for the osd3 but still pod is not booting
up..
>> >> >
>> >> > As outlined:
>> >> > https://access.redhat.com/solutions/3524771
>> >> >
>> >> > ceph auth export osd.2 -o osd.2.export
>> >> > cp osd.2.export osd.3.export
>> >> > ceph auth import -i osd.3.export
>> >> > imported keyring
>> >> >
>> >> >
>> >> > Any suggestions ?
>> >> >
>> >> > Thanks!
>> >> >
>> >> > On Tue, Sep 21, 2021 at 8:34 AM Abdelillah Asraoui <
>> aasra...@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> one of the osd in the cluster went down, is there a workaround to
>> bring
>> >> >> back this osd?
>> >> >>
>> >> >>
>> >> >> logs from ceph osd pod shows the following:
>> >> >>
>> >> >> kubectl -n rook-ceph logs rook-ceph-osd-3-6497bdc65b-pn7mg
>> >> >>
>> >> >> debug 2021-09-20T14:32:46.388+ 7f930fe9cf00 -1 auth: unable to
>> find
>> >> a
>> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
denied
>> >> >>
>> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 auth: unable to
>> find
>> >> a
>> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
denied
>> >> >>
>> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 auth: unable to
>> find
>> >> a
>> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
denied
>> >> >>
>> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 monclient:
keyring
>> >> not
>> >> >> found
>> >> >>
>> >> >> failed to fetch mon config (--no-mon-config to skip)
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> kubectl -n rook-ceph describe pod  rook-ceph-osd-3-64
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> Events:
>> >> >>
>> >> >>   Type Reason   Age  From Message
>> >> >>
>> >> >>    --     ---
>> >> >>
>> >> >>   Normal   Pulled   50m (x749 over 2d16h)kubelet  Container
image
>> >> >> "ceph/ceph:v15.2.13" already present on machine
>> >> >>
>> >> >>   Warning  BackOff  19s (x18433 over 2d16h)  kubelet  Back-off
>> >> restarting
>> >> >> failed container
>> >> >>
>> >> >>
>> >> >>
>> >> >> ceph health detail | more
>> >> >>
>> >> >> HEALTH_WARN noout flag(s) set; 1 osds down; 1 host (1 osds) down;
>> >> Degraded
>> >> >> data redundancy: 180969/542907 obj

[ceph-users] Re: [External Email] Re: ceph-objectstore-tool core dump

[Michael Thomas]

On 10/4/21 11:57 AM, Dave Hall wrote:
> I also had a delay on the start of the repair scrub when I was dealing with
> this issue.  I ultimately increased the number of simultaneous scrubs, but
> I think you could also temporarily disable scrubs and then re-issue the 'pg
> repair'.  (But I'm not one of the experts on this.)
> 
> My perception is that between EC pools, large HDDs, and the overall OSD
> count, there might need to be some tuning to assure that scrubs can get
> scheduled:  A large HDD contains pieces of more PGs.  Each PG in an EC pool
> is spread across more disks than a replication pool.  Thus, especially if
> the number of OSDs is not large, there is an increased chance that more
> than one scrub will want to read the same OSD.   Scheduling nightmare if
> the number of simultaneous scrubs is low and client traffic is given
> priority.
> 
> -Dave

That seemed to be the case.  After ~24 hours, 1 of the 8 repair tasks
had completed.  Unfortunately, it found another error that wasn't
present before.

After checking the SMART logs, it looks like this particular disk is
failing.  No sense in pursuing this any further; I'll be replacing it
with a spare instead.

I'll look into disabling scrubs the next time I need to schedule a
repair.  Hopefully it will run the repair jobs a bit sooner.

Regards,

--Mike
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: osd marked down

[Abdelillah Asraoui]

I have create the keyring file: andvar/lib/ceph/osd/ceph-3/keyring and
chown to ceph but still getting these error on the osd pod log:

k -n rook-ceph logs rook-ceph-osd-3-6497bdc65b-5cvx3

debug 2021-10-04T16:06:38.287+ 7f8633cc1f00 -1 auth: unable to find a
keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied

debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 auth: unable to find a
keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied

debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 auth: unable to find a
keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission denied

debug 2021-10-04T16:06:38.288+ 7f8633cc1f00 -1 monclient: keyring not
found

failed to fetch mon config (--no-mon-config to skip)


thanks!


On Fri, Oct 1, 2021 at 2:02 AM Eugen Block  wrote:

> I'm not sure if anything else could break, but since the OSD isn't
> starting anyway... I guess you could delete osd.3 from ceph auth:
>
> ceph auth del osd.3
>
> And then recreate it with:
>
> ceph auth get-or-create osd.3 mon 'allow profile osd' osd 'allow *'
> mgr 'allow profile osd'
> [osd.3]
>  key = 
>
> Then create a keyring file /var/lib/ceph/osd/ceph-3/keyring with the
> respective content:
>
> [osd.3]
>  key = 
>  caps mgr = "allow profile osd"
>  caps mon = "allow profile osd"
>  caps osd = "allow *"
>
>
> Make sure the file owner is ceph and try to restart the OSD. In this
> case you wouldn't need to import anything. This just worked for me in
> my lab environment, so give it a shot.
>
>
>
> Zitat von Abdelillah Asraoui :
>
> > the /var/lib/ceph/osd/ceph-3/keyring is missing here ..
> > is there way to generate a keyring for osd.3 ?
> >
> >
> > thanks!
> >
> > On Thu, Sep 30, 2021 at 1:18 AM Eugen Block  wrote:
> >
> >> Is the content of OSD.3 still available in the filesystem? If the
> >> answer is yes you can get the OSD's keyring from
> >>
> >> /var/lib/ceph/osd/ceph-3/keyring
> >>
> >> Then update your osd.3.export file with the correct keyring and then
> >> import the correct back to ceph.
> >>
> >>
> >> Zitat von Abdelillah Asraoui :
> >>
> >> > I must have imported osd.2 key instead,  now osd.3 has the same key as
> >> osd.2
> >> >
> >> > ceph auth import -i osd.3.export
> >> >
> >> >
> >> > How do we update this ?
> >> >
> >> > thanks!
> >> >
> >> >
> >> >
> >> > On Wed, Sep 29, 2021 at 2:13 AM Eugen Block  wrote:
> >> >
> >> >> Just to clarify, you didn't simply import the unchanged keyring but
> >> >> modified it to reflect the actual key of OSD.3, correct? If not, run
> >> >> 'ceph auth get osd.3' first and set the key in the osd.3.export file
> >> >> before importing it to ceph.
> >> >>
> >> >>
> >> >> Zitat von Abdelillah Asraoui :
> >> >>
> >> >> > i have created keyring for the osd3 but still pod is not booting
> up..
> >> >> >
> >> >> > As outlined:
> >> >> > https://access.redhat.com/solutions/3524771
> >> >> >
> >> >> > ceph auth export osd.2 -o osd.2.export
> >> >> > cp osd.2.export osd.3.export
> >> >> > ceph auth import -i osd.3.export
> >> >> > imported keyring
> >> >> >
> >> >> >
> >> >> > Any suggestions ?
> >> >> >
> >> >> > Thanks!
> >> >> >
> >> >> > On Tue, Sep 21, 2021 at 8:34 AM Abdelillah Asraoui <
> >> aasra...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> >> Hi,
> >> >> >>
> >> >> >> one of the osd in the cluster went down, is there a workaround to
> >> bring
> >> >> >> back this osd?
> >> >> >>
> >> >> >>
> >> >> >> logs from ceph osd pod shows the following:
> >> >> >>
> >> >> >> kubectl -n rook-ceph logs rook-ceph-osd-3-6497bdc65b-pn7mg
> >> >> >>
> >> >> >> debug 2021-09-20T14:32:46.388+ 7f930fe9cf00 -1 auth: unable to
> >> find
> >> >> a
> >> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
> denied
> >> >> >>
> >> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 auth: unable to
> >> find
> >> >> a
> >> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
> denied
> >> >> >>
> >> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 auth: unable to
> >> find
> >> >> a
> >> >> >> keyring on /var/lib/ceph/osd/ceph-3/keyring: (13) Permission
> denied
> >> >> >>
> >> >> >> debug 2021-09-20T14:32:46.389+ 7f930fe9cf00 -1 monclient:
> keyring
> >> >> not
> >> >> >> found
> >> >> >>
> >> >> >> failed to fetch mon config (--no-mon-config to skip)
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> kubectl -n rook-ceph describe pod  rook-ceph-osd-3-64
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Events:
> >> >> >>
> >> >> >>   Type Reason   Age  From Message
> >> >> >>
> >> >> >>    --     ---
> >> >> >>
> >> >> >>   Normal   Pulled   50m (x749 over 2d16h)kubelet  Container
> image
> >> >> >> "ceph/ceph:v15.2.13" already present on machine
> >> >> >>
> >> >> >>   Warning  BackOff  19s (x18433 over 2d16h)  kubelet  Back-off
> >> >> restarting
> >> >> >> failed co

[ceph-users] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input

[Stefan Kooman]

On 10/4/21 12:10, von Hoesslin, Volker wrote:
Here (see attachment) is a more verbose log output, perhaps someone can 
see more than I have already mentioned.



 -7633> 2021-10-04T11:27:17.434+0200 7f529f998700  4 mds.0.purge_queue 
operator():  data pool 4 not found in OSDMap


^^ What is this all about. What does a ceph osd dump | grep "pool 4" return?

 -7598> 2021-10-04T11:27:17.438+0200 7f529998c700 -1 mds.0.openfiles 
_load_finish: corrupted header/values: void 
Anchor::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past 
end  of struct encoding: Malformed input


^^ openfiles object(s) corrupted.

I don't get it why the Ceph client metrics are unhandled:

 -5040> 2021-10-04T11:27:19.446+0200 7f529f998700  0 
ms_deliver_dispatch: unhandled message 0x5651e7533760 client_metrics 
[client_metric_type: CAP_INFO cap_hits: 519404 cap_misses: 35742 nu
m_caps: 0][client_metric_type: READ_LATENCY latency: 
0.00][client_metric_type: WRITE_LATENCY latency: 
0.00][client_metric_type: METADATA_LATENCY latency: 
5978883.188000][client_metr
ic_type: DENTRY_LEASE dlease_hits: 288544 dlease_misses: 0 num_dentries: 
1] v1 from client.103629506 v1:100.64.0.48:0/1970528122


Support for client side metrics is supported since Pacific 
(https://docs.ceph.com/en/pacific/cephfs/cephfs-top/#cephfs-top). Or is 
this unhandled because you have not enabled the "stats" module?


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Multisite reshard stale instances

[Christian Rohmann]

On 04/10/2021 12:22, Christian Rohmann wrote:
So there is no reason those instances are still kept? How and when are 
those instances cleared up?
Also just like for the other reporters of this issue, in my case most 
buckets are deleted buckets, but not all of them.



I just hope somebody with a little more insight on the mechanisms at 
play here
joins this conversation. 


apparently this is known issue https://tracker.ceph.com/issues/20802, 
but does not cause any problems.

If I may bluntly quote Casey Bodley from our conversation on IRC:

17:20 < cbodley>  no crohmann, nothing cleans them up. 
https://tracker.ceph.com/issues/20802
17:27 < crohmann> A thanks cbodley for the pointer. Are there any 
side-effects of this to expect? Storage wise it's only a few kB for 
the bucket.instance I
  suppose. But what happens if a user creates another 
bucket with the same name at some point in the future?
17:28 < cbodley> new buckets with the same name don't conflict, they 
generate a different bucket instance





Regards


Christian

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Erasure coded pool chunk count k

[Etienne Menguy]

Hi,

It depends of hardware, failure domain, use case, overhead.

I don’t see an easy way to chose k and m values.

-
Etienne Menguy
etienne.men...@croit.io


> On 4 Oct 2021, at 16:57, Golasowski Martin  wrote:
> 
> Hello guys,
> how does one estimate number of chunks for erasure coded pool ( k = ? ) ? I 
> see that number of m chunks determines the pool’s resiliency, however I did 
> not find clear guideline how to determine k.
> 
> Red Hat states that they support only the following combinations:
> 
> k=8, m=3
> k=8, m=4
> k=4, m=2
> 
> without any rationale behind them.
> The table is taken from 
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/storage_strategies_guide/erasure_code_pools.
> 
> Thanks!
> 
> Regards,
> Martin
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input

[Stefan Kooman]

On 10/4/21 12:10, von Hoesslin, Volker wrote:
Here (see attachment) is a more verbose log output, perhaps someone can 
see more than I have already mentioned.


Just checking. Have you done "ceph osd require-osd-release pacific" 
after upgrading to pacific?


(make sure all your OSDs are upgraded to pacific: ceph osd versions).

Do you use snapshots in CephFS?

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Konstantin Shalygin]

This cluster isn't use cephx. ceph.conf global settings disable it


k

Sent from my iPhone

> On 4 Oct 2021, at 17:46, Stefan Kooman  wrote:
> 
> I'm missing the part where keyring is downloaded and used:
> 
> ceph auth get mon. -o /tmp/keyring
> ceph mon getmap -o /tmp/monmap
> chown -R ceph:ceph /var/lib/ceph/mon/ceph-mon2
> ceph-mon -i mon2 --mkfs --monmap /tmp/monmap --keyring /tmp/keyring --setuser 
> ceph --setgroup ceph

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Konstantin Shalygin]

After this I see only logs to stderr, what exactly I should looking for? Some 
grep keyword?


k

Sent from my iPhone

> On 4 Oct 2021, at 17:37, Vladimir Bashkirtsev  
> wrote:
> 
> I guess:
> 
> strace ceph-mon -d --id mon2 --setuser ceph --setgroup ceph
> 
> should do.
> 
> 
> 
> Try -f instead of -d if you are overwhelmed with output to get mon debug 
> output to log file.
> 
> 
> 
> Regards,
> 
> Vladimir
> 
> On 5/10/21 01:27, Konstantin Shalygin wrote:
>> 
>>> On 4 Oct 2021, at 17:07, Vladimir Bashkirtsev  
>>> wrote:
>>> 
>>> This line bothers me:
>>> 
>>> [v2:10.40.0.81:6898/2507925,v1:10.40.0.81:6899/2507925] conn(0x560287e4 
>>> 0x560287e56000 crc :-1 s=READY pgs=16872 cs=0 l=1 rev1=1 rx=0 
>>> tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) 
>>> Operation not permitted)
>>> 
>>> May be it is good idea to run mon under strace and see why your network 
>>> does not permit the frame read? msgr2 will show the message you have 
>>> referred to in case if no data is actually received from the network.
>> 
>> Do you know how exactly start strace do determine this?
>> 
>> 
>> 
>> k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Erasure coded pool chunk count k

[Golasowski Martin]

Hello guys,
how does one estimate number of chunks for erasure coded pool ( k = ? ) ? I see 
that number of m chunks determines the pool’s resiliency, however I did not 
find clear guideline how to determine k.

Red Hat states that they support only the following combinations:

k=8, m=3
k=8, m=4
k=4, m=2

without any rationale behind them.
The table is taken from 
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/storage_strategies_guide/erasure_code_pools.

Thanks!

Regards,
Martin


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Daemon Version Mismatch (But Not Really?) After Deleting/Recreating OSDs

[Edward R Huyer]

Over the summer, I upgraded my cluster from Nautilus to Pacific, and converted 
to use cephadm after doing so.  Over the past couple weeks, I've been 
converting my OSDs to use NVMe drives for db+wal storage.  Schedule a node's 
worth of OSDs to be removed, wait for that to happen, delete the PVs and zap 
the drives, let the orchestrator do its thing.

Over this past weekend, the cluster threw up a HEALTH_WARN due to mismatched 
daemon versions.  Apparently the recreated OSDs are reporting different version 
information from the old daemons.

New OSDs:

-  Container Image Name:  
docker.io/ceph/daemon-base:latest-pacific-devel

-  Container Image ID: d253896d959e

-  Version: 16.2.5-226-g7c9eb137

Old OSDs and other daemons:

-  Container Image Name: docker.io/ceph/ceph:v16

-  Container Image ID: 6933c2a0b7dd

-  Version: 16.2.5

I'm assuming this is not actually a problem and will go away when I next 
upgrade the cluster, but I figured I'd throw it out here in case someone with 
more knowledge than I thinks otherwise.  If it's not a problem, is there a way 
to silence it until I next run an upgrade?  Is there an explanation for why it 
happened?

-
Edward Huyer
Golisano College of Computing and Information Sciences
Rochester Institute of Technology
Golisano 70-2373
152 Lomb Memorial Drive
Rochester, NY 14623
585-475-6651
erh...@rit.edu

Obligatory Legalese:
The information transmitted, including attachments, is intended only for the 
person(s) or entity to which it is addressed and may contain confidential 
and/or privileged material. Any review, retransmission, dissemination or other 
use of, or taking of any action in reliance upon this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and destroy any copies of this information.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Vladimir Bashkirtsev]

I guess:

strace ceph-mon -d --id mon2 --setuser ceph --setgroup ceph

should do.


Try -f instead of -d if you are overwhelmed with output to get mon debug 
output to log file.



Regards,

Vladimir

On 5/10/21 01:27, Konstantin Shalygin wrote:


On 4 Oct 2021, at 17:07, Vladimir Bashkirtsev 
 wrote:


This line bothers me:

[v2:10.40.0.81:6898/2507925,v1:10.40.0.81:6899/2507925] 
conn(0x560287e4 0x560287e56000 crc :-1 s=READY pgs=16872 cs=0 l=1 
rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble 
failed r=-1 ((1) Operation not permitted)


May be it is good idea to run mon under strace and see why your 
network does not permit the frame read? msgr2 will show the message 
you have referred to in case if no data is actually received from the 
network.


Do you know how exactly start strace do determine this?



k

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Konstantin Shalygin]

> On 4 Oct 2021, at 17:07, Vladimir Bashkirtsev  
> wrote:
> 
> This line bothers me:
> 
> [v2:10.40.0.81:6898/2507925,v1:10.40.0.81:6899/2507925] conn(0x560287e4 
> 0x560287e56000 crc :-1 s=READY pgs=16872 cs=0 l=1 rev1=1 rx=0 
> tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) 
> Operation not permitted)
> 
> May be it is good idea to run mon under strace and see why your network does 
> not permit the frame read? msgr2 will show the message you have referred to 
> in case if no data is actually received from the network.

Do you know how exactly start strace do determine this?



k
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Vladimir Bashkirtsev]

This line bothers me:

[v2:10.40.0.81:6898/2507925,v1:10.40.0.81:6899/2507925] conn(0x560287e4 
0x560287e56000 crc :-1 s=READY pgs=16872 cs=0 l=1 rev1=1 rx=0 
tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) 
Operation not permitted)

May be it is good idea to run mon under strace and see why your network does 
not permit the frame read? msgr2 will show the message you have referred to in 
case if no data is actually received from the network.

Regards,
Vladimir

On 5 October 2021 12:27:10 am AEDT, Konstantin Shalygin  wrote:
>Hi,
>
>I was make a mkfs for new mon, but mon stuck on probing. On debug I see: fault 
>on lossy channel, failing. This is a bad (lossy) network (crc mismatch)?
>
>
>2021-10-04 16:22:24.707 7f5952761700 10 mon.mon2@-1(probing) e10 probing other 
>monitors
>2021-10-04 16:22:24.707 7f5952761700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
>[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_probe(probe 
>677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- ?+0 
>0x5602864cd480
>2021-10-04 16:22:24.707 7f5952761700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
>[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_probe(probe 
>677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- 
>0x5602864cd480 con 0x560285455600
>2021-10-04 16:22:24.707 7f5952761700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
>[v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0] -- mon_probe(probe 
>677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- ?+0 
>0x5602893ffc00
>2021-10-04 16:22:24.707 7f5952761700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
>[v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0] -- mon_probe(probe 
>677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- 
>0x5602893ffc00 con 0x560285455a80
>2021-10-04 16:22:24.707 7f5952761700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
>[v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0] -- mon_probe(probe 
>677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- ?+0 
>0x560288e98a00
>2021-10-04 16:22:24.707 7f5952761700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
>[v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0] -- mon_probe(probe 
>677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- 
>0x560288e98a00 con 0x5602862d8000
>2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] <== mon.1 v2:10.40.0.83:3300/0 581 
> mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-03 quorum 
>0,1,2 paxos( fc 127723108 lc 127723840 ) mon_release 14) v7  504+0+0 (crc 
>0 0 0) 0x560287a94f00 con 0x560285455a80
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10 handle_probe 
>mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-03 quorum 0,1,2 
>paxos( fc 127723108 lc 127723840 ) mon_release 14) v7
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10 
>handle_probe_reply mon.1 v2:10.40.0.83:3300/0 mon_probe(reply 
>677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-03 quorum 0,1,2 paxos( fc 
>127723108 lc 127723840 ) mon_release 14) v7
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  monmap is 
>e10: 3 mons at 
>{ceph-01=[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0],ceph-03=[v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0],ceph-06=[v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0]}
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  peer name is 
>ceph-03
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  existing 
>quorum 0,1,2
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  peer paxos 
>version 127723840 vs my version 127723835 (ok)
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  ready to 
>join, but i'm not in the monmap or my addr is blank, trying to join
>2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
>[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_join(mon2 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0]) v2 -- ?+0 0x5602864001c0
>2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
>[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_join(mon2 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0]) v2 -- 0x5602864001c0 con 
>0x560285455600
>2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
>[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] <== mon.2 v2:10.40.0.86:3300/0 574 
> mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-06 quorum 
>0,1,2 paxos( fc 127723108 lc 127723840 ) mon_release 14) v7  504+0+0 (crc 
>0 0 0) 0x56028aa25480 con 0x5602862d8000
>2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10 handle_probe 
>mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-06 quorum 0,1,2 
>paxos( fc 127723108 lc 127723840 ) mon_release 14) v7
>2021-10-04 16:22:24.707 7f594ff5c700 1

[ceph-users] Re: Can't join new mon - lossy channel, failing

[Konstantin Shalygin]

> On 4 Oct 2021, at 16:38, Stefan Kooman  wrote:
> 
> What procedure are you following to add the mon?

# ceph mon dump
epoch 10
fsid 677f4be1-cd98-496d-8b50-1f99df0df670
last_changed 2021-09-11 10:04:23.890922
created 2018-05-18 20:43:43.260897
min_mon_release 14 (nautilus)
0: [v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] mon.ceph-01
1: [v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0] mon.ceph-03
2: [v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0] mon.ceph-06
dumped monmap epoch 10


sudo -u ceph ceph mon getmap -o /tmp/monmap.map
got monmap epoch 10
sudo -u ceph ceph-mon -i mon2 --mkfs --monmap /tmp/monmap.map
chown -R ceph:ceph /var/lib/ceph/mon/ceph-mon2
systemctl start ceph-mon@mon2

> 
> Is this physical hardware? Or a (cloned) VM?

This is hardware

> 
> Have you checked that you can connect with the other monitors on port 6789 / 
> 3300 (and vice versa)?

Yes, of course:

root@mon2:/var/lib/ceph
# telnet 10.40.0.81 3300
Trying 10.40.0.81...
Connected to 10.40.0.81.
Escape character is '^]'.
ceph v2
Terminated
root@mon2:/var/lib/ceph
# telnet 10.40.0.83 3300
Trying 10.40.0.83...
Connected to 10.40.0.83.
Escape character is '^]'.
ceph v2
Terminated
root@mon2:/var/lib/ceph
# telnet 10.40.0.86 3300
Trying 10.40.0.86...
Connected to 10.40.0.86.
Escape character is '^]'.
ceph v2
Terminated




k

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: nfs and showmount

[Daniel Gryniewicz]

showmount uses the MNT protocol, which is only part of NFSv3.  NFSv4 
mounts a pseudoroot, under which actual exports are exposed, so the 
NFSv4 equivalent is to mount /, and then list it.


In general, NFSv4 should be used in preference to NFSv3 whenever possible.

Daniel

On 10/4/21 9:10 AM, Fyodor Ustinov wrote:

Hi!

Yes. You're right. Ganesha does. But ceph doesn't use all of ganesh's 
functionality.
In the ceph dashboard there is no way to enable nfs3, only nfs4

- Original Message -

From: "Marc" 
To: "Fyodor Ustinov" 
Cc: "ceph-users" 
Sent: Monday, 4 October, 2021 15:33:43
Subject: RE: nfs and showmount



Afaik ceph uses nfs-ganesha, and ganesha supports nfs 3 and 4 and other
protocols.


Hi!

I think ceph only supports nsf4?


- Original Message -

Sent: Monday, 4 October, 2021 12:44:38
Subject: RE: nfs and showmount



I can remember asking the same some time ago. I think it has to do

with the

version of nfs you are using.


-Original Message-
From: Fyodor Ustinov 
Sent: Monday, 4 October 2021 11:32
To: ceph-users 
Subject: [ceph-users] nfs and showmount

Hi!

As I understand it - the built-in NFS server does not support the
command "showmount -e"?

WBR,
 Fyodor.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Can't join new mon - lossy channel, failing

[Konstantin Shalygin]

Hi,

I was make a mkfs for new mon, but mon stuck on probing. On debug I see: fault 
on lossy channel, failing. This is a bad (lossy) network (crc mismatch)?


2021-10-04 16:22:24.707 7f5952761700 10 mon.mon2@-1(probing) e10 probing other 
monitors
2021-10-04 16:22:24.707 7f5952761700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_probe(probe 
677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- ?+0 
0x5602864cd480
2021-10-04 16:22:24.707 7f5952761700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_probe(probe 
677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- 
0x5602864cd480 con 0x560285455600
2021-10-04 16:22:24.707 7f5952761700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
[v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0] -- mon_probe(probe 
677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- ?+0 
0x5602893ffc00
2021-10-04 16:22:24.707 7f5952761700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
[v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0] -- mon_probe(probe 
677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- 
0x5602893ffc00 con 0x560285455a80
2021-10-04 16:22:24.707 7f5952761700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
[v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0] -- mon_probe(probe 
677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- ?+0 
0x560288e98a00
2021-10-04 16:22:24.707 7f5952761700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
[v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0] -- mon_probe(probe 
677f4be1-cd98-496d-8b50-1f99df0df670 name mon2 new mon_release 14) v7 -- 
0x560288e98a00 con 0x5602862d8000
2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] <== mon.1 v2:10.40.0.83:3300/0 581 
 mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-03 quorum 
0,1,2 paxos( fc 127723108 lc 127723840 ) mon_release 14) v7  504+0+0 (crc 0 
0 0) 0x560287a94f00 con 0x560285455a80
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10 handle_probe 
mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-03 quorum 0,1,2 
paxos( fc 127723108 lc 127723840 ) mon_release 14) v7
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10 
handle_probe_reply mon.1 v2:10.40.0.83:3300/0 mon_probe(reply 
677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-03 quorum 0,1,2 paxos( fc 
127723108 lc 127723840 ) mon_release 14) v7
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  monmap is 
e10: 3 mons at 
{ceph-01=[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0],ceph-03=[v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0],ceph-06=[v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0]}
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  peer name is 
ceph-03
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  existing 
quorum 0,1,2
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  peer paxos 
version 127723840 vs my version 127723835 (ok)
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  ready to 
join, but i'm not in the monmap or my addr is blank, trying to join
2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] send_to--> mon 
[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_join(mon2 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0]) v2 -- ?+0 0x5602864001c0
2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] --> 
[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0] -- mon_join(mon2 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0]) v2 -- 0x5602864001c0 con 
0x560285455600
2021-10-04 16:22:24.707 7f594ff5c700  1 -- 
[v2:10.40.0.82:3300/0,v1:10.40.0.82:6789/0] <== mon.2 v2:10.40.0.86:3300/0 574 
 mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-06 quorum 
0,1,2 paxos( fc 127723108 lc 127723840 ) mon_release 14) v7  504+0+0 (crc 0 
0 0) 0x56028aa25480 con 0x5602862d8000
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10 handle_probe 
mon_probe(reply 677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-06 quorum 0,1,2 
paxos( fc 127723108 lc 127723840 ) mon_release 14) v7
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10 
handle_probe_reply mon.2 v2:10.40.0.86:3300/0 mon_probe(reply 
677f4be1-cd98-496d-8b50-1f99df0df670 name ceph-06 quorum 0,1,2 paxos( fc 
127723108 lc 127723840 ) mon_release 14) v7
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  monmap is 
e10: 3 mons at 
{ceph-01=[v2:10.40.0.81:3300/0,v1:10.40.0.81:6789/0],ceph-03=[v2:10.40.0.83:3300/0,v1:10.40.0.83:6789/0],ceph-06=[v2:10.40.0.86:3300/0,v1:10.40.0.86:6789/0]}
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  peer name is 
ceph-06
2021-10-04 16:22:24.707 7f594ff5c700 10 mon.mon2@-1(probing) e10  existing 
quorum 0,1,2
2021

[ceph-users] Re: nfs and showmount

[Fyodor Ustinov]

Hi!

Yes. You're right. Ganesha does. But ceph doesn't use all of ganesh's 
functionality.
In the ceph dashboard there is no way to enable nfs3, only nfs4

- Original Message -
> From: "Marc" 
> To: "Fyodor Ustinov" 
> Cc: "ceph-users" 
> Sent: Monday, 4 October, 2021 15:33:43
> Subject: RE: nfs and showmount

> Afaik ceph uses nfs-ganesha, and ganesha supports nfs 3 and 4 and other
> protocols.
> 
>> Hi!
>> 
>> I think ceph only supports nsf4?
>> 
>> 
>> - Original Message -
>> > Sent: Monday, 4 October, 2021 12:44:38
>> > Subject: RE: nfs and showmount
>> 
>> > I can remember asking the same some time ago. I think it has to do
>> with the
>> > version of nfs you are using.
>> >
>> >> -Original Message-
>> >> From: Fyodor Ustinov 
>> >> Sent: Monday, 4 October 2021 11:32
>> >> To: ceph-users 
>> >> Subject: [ceph-users] nfs and showmount
>> >>
>> >> Hi!
>> >>
>> >> As I understand it - the built-in NFS server does not support the
>> >> command "showmount -e"?
>> >>
>> >> WBR,
>> >> Fyodor.
>> >> ___
>> >> ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: nfs and showmount

[Marc]

Afaik ceph uses nfs-ganesha, and ganesha supports nfs 3 and 4 and other 
protocols.

> Hi!
> 
> I think ceph only supports nsf4?
> 
> 
> - Original Message -
> > Sent: Monday, 4 October, 2021 12:44:38
> > Subject: RE: nfs and showmount
> 
> > I can remember asking the same some time ago. I think it has to do
> with the
> > version of nfs you are using.
> >
> >> -Original Message-
> >> From: Fyodor Ustinov 
> >> Sent: Monday, 4 October 2021 11:32
> >> To: ceph-users 
> >> Subject: [ceph-users] nfs and showmount
> >>
> >> Hi!
> >>
> >> As I understand it - the built-in NFS server does not support the
> >> command "showmount -e"?
> >>
> >> WBR,
> >> Fyodor.
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input

[von Hoesslin, Volker]

Von: Stefan Kooman 
Gesendet: Montag, 4. Oktober 2021 13:24
An: von Hoesslin, Volker; ceph-users@ceph.io
Betreff: [ceph-users] Re: MDS: corrupted header/values: decode past end of 
struct encoding: Malformed input

Externe E-Mail! Öffnen Sie nur Links oder Anhänge von vertrauenswürdigen 
Absendern!
On 10/4/21 12:10, von Hoesslin, Volker wrote:
> Here (see attachment) is a more verbose log output, perhaps someone can
> see more than I have already mentioned.

Just checking. Have you done "ceph osd require-osd-release pacific"
after upgrading to pacific?
<< YES

(make sure all your OSDs are upgraded to pacific: ceph osd versions).
<<
# ceph osd versions
{
"ceph version 16.2.6 (1a6b9a05546f335eeeddb460fdc89caadf80ac7a) pacific 
(stable)": 30
}


Do you use snapshots in CephFS?
<<< NO

Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input

[von Hoesslin, Volker]

Von: Stefan Kooman 
Gesendet: Montag, 4. Oktober 2021 13:57:45
An: von Hoesslin, Volker; ceph-users@ceph.io
Betreff: [URL wurde verändert] [ceph-users] Re: MDS: corrupted header/values: 
decode past end of struct encoding: Malformed input

Externe E-Mail! Öffnen Sie nur Links oder Anhänge von vertrauenswürdigen 
Absendern!
On 10/4/21 12:10, von Hoesslin, Volker wrote:
> Here (see attachment) is a more verbose log output, perhaps someone can
> see more than I have already mentioned.


  -7633> 2021-10-04T11:27:17.434+0200 7f529f998700  4 mds.0.purge_queue
operator():  data pool 4 not found in OSDMap

^^ What is this all about. What does a ceph osd dump | grep "pool 4" return?


<<
# ceph osd dump | grep "pool 4"
pool 4 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash 
rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 33804 lfor 
0/14119/33567 flags hashpspool stripe_width 0 application cephfs




  -7598> 2021-10-04T11:27:17.438+0200 7f529998c700 -1 mds.0.openfiles
_load_finish: corrupted header/values: void
Anchor::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past
end  of struct encoding: Malformed input

^^ openfiles object(s) corrupted.



<<
sounds bad! what does it mean? can we fixed it?



I don't get it why the Ceph client metrics are unhandled:

  -5040> 2021-10-04T11:27:19.446+0200 7f529f998700  0
ms_deliver_dispatch: unhandled message 0x5651e7533760 client_metrics
[client_metric_type: CAP_INFO cap_hits: 519404 cap_misses: 35742 nu
m_caps: 0][client_metric_type: READ_LATENCY latency:
0.00][client_metric_type: WRITE_LATENCY latency:
0.00][client_metric_type: METADATA_LATENCY latency:
5978883.188000][client_metr
ic_type: DENTRY_LEASE dlease_hits: 288544 dlease_misses: 0 num_dentries:
1] v1 from client.103629506 v1:100.64.0.48:0/1970528122

Support for client side metrics is supported since Pacific
(https://sis-schwerin.de/externer-link/?href=https://docs.ceph.com/en/pacific/cephfs/cephfs-top/#cephfs-top).
 Or is
this unhandled because you have not enabled the "stats" module?



<<<
i do not know what it mean, in deed, there is no cephfs-top installed:
# cephfs-top
-bash: cephfs-top: command not found




Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS not becoming active after migrating to cephadm

[胡 玮文]

By saying upgrade, I mean upgrade from the non-dockerized 16.2.5 to cephadm 
version 16.2.6. So I think you need to disable standby-replay and reduce the 
number of ranks to 1, then stop all the non-dockerized mds, deploy new mds with 
cephadm. Only scaling back up after you finish the migration. Did you also 
tried that?

In fact, similar issue has been reported several times on this list when 
upgrade mds to 16.2.6, e.g. [1]. I have faced that too. So I’m pretty confident 
that you are facing the same issue.

[1]: 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/KQ5A5OWRIUEOJBC7VILBGDIKPQGJQIWN/

在 2021年10月4日，19:00，Petr Belyaev  写道：

 Hi Weiwen,

Yes, we did that during the upgrade. In fact, we did that multiple times even 
after the upgrade to see if it will resolve the issue (disabling hot standby, 
scaling everything down to a single MDS, swapping it with the new one, scaling 
back up).

The upgrade itself went fine, problems started during the migration to cephadm 
(which was done after migrating everything to Pacific).
It only occurs when using dockerized MDS. Non-dockerized MDS nodes, also 
Pacific, everything runs fine.

Petr

On 4 Oct 2021, at 12:43, 胡 玮文 mailto:huw...@outlook.com>> 
wrote:

Hi Petr,

Please read https://docs.ceph.com/en/latest/cephfs/upgrading/ for MDS upgrade 
procedure.

In short, when upgrading to 16.2.6, you need to disable standby-replay and 
reduce the number of ranks to 1.

Weiwen Hu

从 Windows 版邮件发送

发件人: Petr Belyaev
发送时间: 2021年10月4日 18:00
收件人: ceph-users@ceph.io
主题: [ceph-users] MDS not becoming active after migrating to cephadm

Hi,

We’ve recently upgraded from Nautilus to Pacific, and tried moving our services 
to cephadm/ceph orch.
For some reason, MDS nodes deployed through orch never become active (or at 
least standby-replay). Non-dockerized MDS nodes can still be deployed and work 
fine. Non-dockerized mds version is 16.2.6, docker image version is 
16.2.5-387-g7282d81d (came as a default).

In the MDS log, the only related message is monitors assigning MDS as standby. 
Increasing the log level does not help much, it only adds beacon messages.
Monitor log also contains no differences compared to a non-dockerized MDS 
startup.
Mds metadata command output is identical to that of a non-dockerized MDS.

The only difference I can see in the log is the value in curly braces after the 
node name, e.g. mds.storage{0:1234ff}. For dockerized MDS, the first value is 
, for non-dockerized it’s zero. Compat flags are identical.

Could someone please advise me why the dockerized MDS is being stuck as a 
standby? Maybe some config values missing or smth?

Best regards,
Petr
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Tool to cancel pending backfills

[Peter Lieven]

Am 01.10.21 um 16:52 schrieb Josh Baergen:

Hi Peter,


When I check for circles I found that running the upmap balancer alone never 
seems to create
any kind of circle in the graph

By a circle, do you mean something like this?
pg 1.a: 1->2 (upmap to put a chunk on 2 instead of 1)
pg 1.b: 2->3
pg 1.c: 3->1



Exactly. The upmap balancer tries to remove upmap entries first as far as I 
understand so I would expect that there never will be a circle like, 1->2, 
2->1, but I don't

see why It would no accidently create a circle with more nodes involved.




If so, then it's not surprising that the upmap balancer wouldn't
create this situation by itself, since there's no reason for this set
of upmaps to exist purely for balance reasons. I don't think the
balancer needs any explicit code to avoid the situation because of
this.


Running pgremapper + balancer created circles with sometimes several dozen 
nodes. I would update the docs of the pgremapper
to warn about this fact and guide the users to use undo-upmap to slowly remove 
the upmaps create by cancel-backfill.

This is again not surprising, since cancel-backfill will do whatever's
necessary to undo a set of CRUSH changes (and some CRUSH changes
regularly lead to movement cycles like this), and then using the upmap
balancer will only make enough changes to achieve balance, not undo
everything that's there.


It might be a nice addition to pgremapper to add an option to optimze the upmap 
table.

What I'm still missing here is the value in this. Are there
demonstrable problems presented by a large upmap exception table (e.g.
performance or operational)?



I have no evidence, how expensive upmap table entries are. They need to be 
synched to every client and there have a certain overhead.

Maybe someone with more knowledge of the internals can give some insight here. 
The whole idea of crush is not carry a table with exact

mappings around and upmap entries are the exactly opposite of this idea. Bottom 
line, every circlic upmap definition is an overhead thats

unnecessary and i personally think it should be avoided.


Best,

Peter



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS not becoming active after migrating to cephadm

[Petr Belyaev]

Hi Weiwen,

Yes, we did that during the upgrade. In fact, we did that multiple times even 
after the upgrade to see if it will resolve the issue (disabling hot standby, 
scaling everything down to a single MDS, swapping it with the new one, scaling 
back up).

The upgrade itself went fine, problems started during the migration to cephadm 
(which was done after migrating everything to Pacific). 
It only occurs when using dockerized MDS. Non-dockerized MDS nodes, also 
Pacific, everything runs fine.

Petr

> On 4 Oct 2021, at 12:43, 胡 玮文  wrote:
> 
> Hi Petr,
>  
> Please read https://docs.ceph.com/en/latest/cephfs/upgrading/ 
>  for MDS upgrade procedure.
>  
> In short, when upgrading to 16.2.6, you need to disable standby-replay and 
> reduce the number of ranks to 1.
>  
> Weiwen Hu
>  
> 从 Windows 版邮件 发送
>  
> 发件人: Petr Belyaev 
> 发送时间: 2021年10月4日 18:00
> 收件人: ceph-users@ceph.io 
> 主题: [ceph-users] MDS not becoming active after migrating to cephadm
>  
> Hi,
> 
> We’ve recently upgraded from Nautilus to Pacific, and tried moving our 
> services to cephadm/ceph orch.
> For some reason, MDS nodes deployed through orch never become active (or at 
> least standby-replay). Non-dockerized MDS nodes can still be deployed and 
> work fine. Non-dockerized mds version is 16.2.6, docker image version is 
> 16.2.5-387-g7282d81d (came as a default).
> 
> In the MDS log, the only related message is monitors assigning MDS as 
> standby. Increasing the log level does not help much, it only adds beacon 
> messages.
> Monitor log also contains no differences compared to a non-dockerized MDS 
> startup.
> Mds metadata command output is identical to that of a non-dockerized MDS.
> 
> The only difference I can see in the log is the value in curly braces after 
> the node name, e.g. mds.storage{0:1234ff}. For dockerized MDS, the first 
> value is , for non-dockerized it’s zero. Compat flags are identical.
> 
> Could someone please advise me why the dockerized MDS is being stuck as a 
> standby? Maybe some config values missing or smth?
> 
> Best regards,
> Petr
> ___
> ceph-users mailing list -- ceph-users@ceph.io 
> To unsubscribe send an email to ceph-users-le...@ceph.io 
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 回复: MDS not becoming active after migrating to cephadm

[胡 玮文]

Hi Petr,

Please read https://docs.ceph.com/en/latest/cephfs/upgrading/ for MDS upgrade 
procedure.

In short, when upgrading to 16.2.6, you need to disable standby-replay and 
reduce the number of ranks to 1.

Weiwen Hu

从 Windows 版邮件发送

发件人: Petr Belyaev
发送时间: 2021年10月4日 18:00
收件人: ceph-users@ceph.io
主题: [ceph-users] MDS not becoming active after migrating to cephadm

Hi,

We’ve recently upgraded from Nautilus to Pacific, and tried moving our services 
to cephadm/ceph orch.
For some reason, MDS nodes deployed through orch never become active (or at 
least standby-replay). Non-dockerized MDS nodes can still be deployed and work 
fine. Non-dockerized mds version is 16.2.6, docker image version is 
16.2.5-387-g7282d81d (came as a default).

In the MDS log, the only related message is monitors assigning MDS as standby. 
Increasing the log level does not help much, it only adds beacon messages.
Monitor log also contains no differences compared to a non-dockerized MDS 
startup.
Mds metadata command output is identical to that of a non-dockerized MDS.

The only difference I can see in the log is the value in curly braces after the 
node name, e.g. mds.storage{0:1234ff}. For dockerized MDS, the first value is 
, for non-dockerized it’s zero. Compat flags are identical.

Could someone please advise me why the dockerized MDS is being stuck as a 
standby? Maybe some config values missing or smth?

Best regards,
Petr
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Multisite reshard stale instances

[Christian Rohmann]

Hey there again,

On 01/10/2021 17:35, Szabo, Istvan (Agoda) wrote:

In my setup I've disabled the sharding and preshard each bucket which needs 
more then 1.1 millions of objects.


I also use 11 shards as default, see my ML post 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/UFXPAINBV3DQXABSPY5XLMYFA3UGF5LF/#OK7XMNRFHTF3EQU6SAWPLKEVGVNV4XET




I don't think it's possible to cleanup, even if you run the command with the 
really-really mean it, it will not do anything, I've tried already.


Searching a little more through older ML posts it appears we are not he 
only ones and also that those "stale" instances are to be expected when 
deleting buckets in a multisite setup:


 * 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033575.html
 * 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/7CQZY6D2HLPLZAWKQPT4D74WLQ6GE3U5/#ZLAFDLS4MKOUPAIWRY73IBYJCVFYMECB


but even after running "data sync --init" again I still see stale 
instances, but both metadata and data are "caught up" on both sites.


So there is no reason those instances are still kept? How and when are 
those instances cleared up?
Also just like for the other reporters of this issue, in my case most 
buckets are deleted buckets, but not all of them.



I just hope somebody with a little more insight on the mechanisms at 
play here

joins this conversation.


Regards


Christian





___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: nfs and showmount

[Fyodor Ustinov]

Hi!

I think ceph only supports nsf4?


- Original Message -
> From: "Marc" 
> To: "Fyodor Ustinov" , "ceph-users" 
> Sent: Monday, 4 October, 2021 12:44:38
> Subject: RE: nfs and showmount

> I can remember asking the same some time ago. I think it has to do with the
> version of nfs you are using.
> 
>> -Original Message-
>> From: Fyodor Ustinov 
>> Sent: Monday, 4 October 2021 11:32
>> To: ceph-users 
>> Subject: [ceph-users] nfs and showmount
>> 
>> Hi!
>> 
>> As I understand it - the built-in NFS server does not support the
>> command "showmount -e"?
>> 
>> WBR,
>> Fyodor.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] MDS not becoming active after migrating to cephadm

[Petr Belyaev]

Hi,

We’ve recently upgraded from Nautilus to Pacific, and tried moving our services 
to cephadm/ceph orch.
For some reason, MDS nodes deployed through orch never become active (or at 
least standby-replay). Non-dockerized MDS nodes can still be deployed and work 
fine. Non-dockerized mds version is 16.2.6, docker image version is 
16.2.5-387-g7282d81d (came as a default).

In the MDS log, the only related message is monitors assigning MDS as standby. 
Increasing the log level does not help much, it only adds beacon messages.
Monitor log also contains no differences compared to a non-dockerized MDS 
startup.
Mds metadata command output is identical to that of a non-dockerized MDS.

The only difference I can see in the log is the value in curly braces after the 
node name, e.g. mds.storage{0:1234ff}. For dockerized MDS, the first value is 
, for non-dockerized it’s zero. Compat flags are identical.

Could someone please advise me why the dockerized MDS is being stuck as a 
standby? Maybe some config values missing or smth?

Best regards,
Petr
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: nfs and showmount

[Marc]

I can remember asking the same some time ago. I think it has to do with the 
version of nfs you are using.

> -Original Message-
> From: Fyodor Ustinov 
> Sent: Monday, 4 October 2021 11:32
> To: ceph-users 
> Subject: [ceph-users] nfs and showmount
> 
> Hi!
> 
> As I understand it - the built-in NFS server does not support the
> command "showmount -e"?
> 
> WBR,
> Fyodor.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] nfs and showmount

[Fyodor Ustinov]

Hi!

As I understand it - the built-in NFS server does not support the command 
"showmount -e"?

WBR,
Fyodor.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input

[Stefan Kooman]

On 10/4/21 11:09, von Hoesslin, Volker wrote:

hmmm... more and more PGs are broken:


:/

at the risk of making a fool of myself, but how do i check what data is 
in a PG?


https://docs.ceph.com/en/pacific/man/8/ceph-objectstore-tool/



i have already done a backup at the beginning using "cephfs-journal-tool 
journal export backup.bin", there is only a limited backup of the data 
itself from the ceph.


With a backup I meant the data outside of Ceph. Journal backup is of 
course also good to have.


I guess with "ceph health detail" you would get more information about 
the inconsistent PGs. And with the objectstore tool you can check what 
objects might be damaged.


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS: corrupted header/values: decode past end of struct encoding: Malformed input

[von Hoesslin, Volker]

hmmm... more and more PGs are broken:


# ceph health detail
HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds daemon; 1 
filesystem is offline; insufficient standby MDS daemons available; 46 scrub 
errors; Possible data damage: 32 pgs inconsistent; 2625 daemons have recently 
crashed
[WRN] FS_DEGRADED: 1 filesystem is degraded
fs cephfs is degraded
[WRN] FS_WITH_FAILED_MDS: 1 filesystem has a failed mds daemon
fs cephfs has 1 failed mds
[ERR] MDS_ALL_DOWN: 1 filesystem is offline
fs cephfs is offline because no MDS is active for it.
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
have 0; want 1 more
[ERR] OSD_SCRUB_ERRORS: 46 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 32 pgs inconsistent
pg 1.3 is active+clean+inconsistent, acting [4,15,25]
pg 1.5 is active+clean+inconsistent, acting [22,8,11]
pg 1.a is active+clean+inconsistent, acting [23,19,6]
pg 1.10 is active+clean+inconsistent, acting [18,22,0]
pg 1.1c is active+clean+inconsistent+failed_repair, acting [28,16,9]
pg 1.1e is active+clean+inconsistent, acting [22,10,6]
pg 1.26 is active+clean+inconsistent, acting [22,2,17]
pg 1.35 is active+clean+inconsistent, acting [27,7,11]
pg 1.37 is active+clean+inconsistent+failed_repair, acting [7,16,26]
pg 1.3d is active+clean+inconsistent, acting [0,17,22]
pg 5.47 is active+clean+inconsistent, acting [8,28,13]
pg 5.90 is active+clean+inconsistent+failed_repair, acting [13,9,21]
pg 5.a6 is active+clean+inconsistent, acting [20,19,8]
pg 5.b0 is active+clean+inconsistent, acting [20,3,17]
pg 5.b2 is active+clean+inconsistent, acting [24,11,9]
pg 5.b3 is active+clean+inconsistent, acting [1,23,18]
pg 5.d0 is active+clean+inconsistent, acting [27,4,14]
pg 5.d2 is active+clean+inconsistent, acting [15,24,0]
pg 11.5 is active+clean+inconsistent, acting [11,3,25]
pg 11.17 is active+clean+inconsistent, acting [24,19,8]
pg 16.0 is active+clean+inconsistent, acting [5,15,24]
pg 16.2 is active+clean+inconsistent, acting [12,1,27]
pg 16.15 is active+clean+inconsistent, acting [0,28,11]
pg 16.17 is active+clean+inconsistent, acting [2,21,13]
pg 16.1c is active+clean+inconsistent, acting [25,7,15]
pg 16.25 is active+clean+inconsistent, acting [15,4,25]
pg 16.2f is active+clean+inconsistent, acting [20,13,1]
pg 16.38 is active+clean+inconsistent, acting [2,18,22]
pg 16.3a is active+clean+inconsistent, acting [12,1,20]
pg 16.3d is active+clean+inconsistent, acting [21,19,6]
pg 16.3e is active+clean+inconsistent, acting [14,9,21]
pg 16.3f is active+clean+inconsistent, acting [23,5,15]
[WRN] RECENT_CRASH: 2625 daemons have recently crashed
client.admin crashed on host pve06 at 2021-09-30T05:08:19.213324Z
mds.pve05 crashed on host pve05 at 2021-09-30T06:09:49.543530Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:22.059405Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:26.077956Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:30.117664Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:34.149385Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:37.607766Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:41.639585Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:45.684791Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:49.711284Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:53.757538Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:10:57.622000Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:11:01.798656Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:11:05.821116Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:11:09.860788Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:11:13.903719Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:11:17.630383Z
mds.pve04 crashed on host pve04 at 2021-09-30T13:11:21.948918Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:25.979666Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:30.013149Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:34.044069Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:37.633660Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:41.664662Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:45.690034Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:49.735077Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:53.765387Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:11:57.655313Z
mds.pve05 crashed on host pve05 at 2021-09-30T13:12:01.812882Z
mds.pve06 crashed on host pve06 at 2021-09-30T13:12:05.838469Z
mds.pve06 crashed on host pve06 at 2021-09-30T13:12:09.874958Z
and 2595 more


for now, i have all three mds daemons are stoped.


at the risk of making a fool of myself, but how do i check what data is in a PG?

i have already done a backup at the beginning using "cephfs-journal-too