[ceph-users] Re: Next quincy point release 17.2.7

[Tobias Urdin]

Hello Yuri,

On the RGW side I would very much like to get this [1] patch in that release
that is already merged in reef [2] and pacific [3].

Perhaps Casey can approve and merge that so you can bring it into your
testing.

Thanks!

[1] https://github.com/ceph/ceph/pull/53414
[2] https://github.com/ceph/ceph/pull/53413
[3] https://github.com/ceph/ceph/pull/53416

On 4 Oct 2023, at 22:57, Yuri Weinstein  wrote:

Hello

We are getting very close to the next Quincy point release 17.2.7

Here is the list of must-have PRs https://pad.ceph.com/p/quincy_17.2.7_prs
We will start the release testing/review/approval process as soon as
all PRs from this list are merged.

If you see something missing please speak up and the dev leads will
make a decision on including it in this release.

TIA
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph 16.2.x excessive logging, how to reduce?

[Zakhar Kirpichenko]

Thank you for your response, Igor.

Currently debug_rocksdb is set to 4/5:

# ceph config get osd debug_rocksdb
4/5

This setting seems to be default. Is my understanding correct that you're
suggesting setting it to 3/5 or even 0/5? Would setting it to 0/5 have any
negative effects on the cluster?

/Z

On Wed, 4 Oct 2023 at 21:23, Igor Fedotov  wrote:

> Hi Zakhar,
>
> do reduce rocksdb logging verbosity you might want to set debug_rocksdb
> to 3 (or 0).
>
> I presume it produces a  significant part of the logging traffic.
>
>
> Thanks,
>
> Igor
>
> On 04/10/2023 20:51, Zakhar Kirpichenko wrote:
> > Any input from anyone, please?
> >
> > On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko 
> wrote:
> >
> >> Hi,
> >>
> >> Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very
> >> detailed messages, Ceph logs alone on hosts with monitors and several
> OSDs
> >> has already eaten through 50% of the endurance of the flash system
> drives
> >> over a couple of years.
> >>
> >> Cluster logging settings are default, and it seems that all daemons are
> >> writing lots and lots of debug information to the logs, such as for
> >> example: https://pastebin.com/ebZq8KZk (it's just a snippet, but
> there's
> >> lots and lots of various information).
> >>
> >> Is there a way to reduce the amount of logging and, for example, limit
> the
> >> logging to warnings or important messages so that it doesn't include
> every
> >> successful authentication attempt, compaction etc, etc, when the
> cluster is
> >> healthy and operating normally?
> >>
> >> I would very much appreciate your advice on this.
> >>
> >> Best regards,
> >> Zakhar
> >>
> >>
> >>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: outdated mds slow requests

[Ben]

Hi Eugen,

warnings continue to spam cluster log.Actually for the whole picture of the
issue please see:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VDL56J75FG5LO4ZECIWWGGBW4ULPZUIP/

I was thinking about the following options:
1, restart problematic nodes: 24,32,34,36: need to schedule with business
partners
2, restart problematic mds with trimming behind issue: 3,4,5: mds will
start up quickly, won't they? investigating...

mds 3,4,5 have logsegment item in segments list that are stuck in expiring
status, which break the trimming process. Growing segments lists draw
concerns overtime.
Any other ideas?

Thanks,
Ben


Eugen Block  于2023年10月4日周三 16:44写道：

> Hi,
>
> is this still an issue? If so, I would try to either evict the client
> via admin socket:
>
> ceph tell mds.5 client evict [...] --- Evict client
> session(s) based on a filter
>
> alternatively locally on the MDS:
> cephadm enter mds.
> ceph daemon mds. client evict 
>
> or restart the MDS which should also clear the client, I believe.
>
> Zitat von Ben :
>
> > Hi,
> > It is running 17.2.5. there are slow requests warnings in cluster log.
> >
> > ceph tell mds.5 dump_ops_in_flight,
> > get the following.
> >
> > These look like outdated and clients were k8s pods. There are warning of
> > the kind in other mds as well. How could they be cleaned from warnings
> > safely?
> >
> > Many thanks.
> >
> > {
> > "ops": [
> > {
> > "description": "peer_request(mds.3:5311742.0 authpin)",
> > "initiated_at": "2023-09-14T12:25:43.092201+",
> > "age": 926013.05098558997,
> > "duration": 926013.051015759,
> > "type_data": {
> > "flag_point": "dispatched",
> > "reqid": "mds.3:5311742",
> > "op_type": "peer_request",
> > "leader_info": {
> > "leader": "3"
> > },
> > "request_info": {
> > "attempt": 0,
> > "op_type": "authpin",
> > "lock_type": 0,
> > "object_info": "0x60001205d6d.head",
> > "srcdnpath": "",
> > "destdnpath": "",
> > "witnesses": "",
> > "has_inode_export": false,
> > "inode_export_v": 0,
> > "op_stamp": "0.00"
> > },
> > "events": [
> > {
> > "time": "2023-09-14T12:25:43.092201+",
> > "event": "initiated"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092202+",
> > "event": "throttled"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092201+",
> > "event": "header_read"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092207+",
> > "event": "all_read"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092218+",
> > "event": "dispatched"
> > }
> > ]
> > }
> > },
> > {
> > "description": "peer_request(mds.3:5311743.0 authpin)",
> > "initiated_at": "2023-09-14T12:25:43.092371+",
> > "age": 926013.05081614305,
> > "duration": 926013.05089185503,
> > "type_data": {
> > "flag_point": "dispatched",
> > "reqid": "mds.3:5311743",
> > "op_type": "peer_request",
> > "leader_info": {
> > "leader": "3"
> > },
> > "request_info": {
> > "attempt": 0,
> > "op_type": "authpin",
> > "lock_type": 0,
> > "object_info": "0x60001205d6d.head",
> > "srcdnpath": "",
> > "destdnpath": "",
> > "witnesses": "",
> > "has_inode_export": false,
> > "inode_export_v": 0,
> > "op_stamp": "0.00"
> > },
> > "events": [
> > {
> > "time": "2023-09-14T12:25:43.092371+",
> > "event": "initiated"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092371+",
> > "event": "throttled"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092371+",
> > "event": "header_read"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092374+",
> > "event": "all_read"
> > },
> > {
> > "time": "2023-09-14T12:25:43.092381+",
> > "event": "dispatched"
> > }
> > ]
> > }
> > },
> > {
> > "description": "peer_request(mds.4:4503615.0 authpin)",
> > "initiated_at": "2023-09-14T13:40:25.150040+",
> > "age": 921530.99314722,
> > "duration": 921530.99326053297,
> > "type_data": {
> > "flag_point": "dispatched",
> > "reqid": "mds.4:4503615",
> > "op_type": "peer_request",
> > "leader_info": {
> > "leader": "4"
> > },
> > "request_info": {
> > "attempt": 0,
> > "op_type": "authpin",
> > "lock_type": 0,
> > "object_info": "0x60001205c4f.head",
> > "srcdnpath": "",
> > "destdnpath": "",
> > "witnesses": "",
> > "has_inode_export": false,
> > "inode_export_v": 0,
> > "op_stamp": "0.00"
> > },
> > "events": [
> > {
> > "time": "2023-09-14T13:40:25.150040+",
> > "event": "initiated"
> > },
> > {
> > "time": "2023-09-14T13:40:25.150040+",
> > "event": "throttled"
> > },
> > {
> > "time": "2023-09-14T13:40:25.150040+",
> > "event": "header_read"
> > },
> > {
> > "time": "2023-09-14T13:40:25.150045+",
> > "event": "all_read"
> > },
> > {
> > "time": "2023-09-14T13:40:25.150053+",
> > "event": "dispatched"
> > }
> > ]
> > }
> > },
> > {
> > "description": "client_request(client.460983:5731820 getattr pAsLsXsFs
> > #0x60001205c4f 2023-09-14T13:40:25.144336+ caller_uid=0,
> > caller_gid=0{})",
> > "initiated_at": "2023-09-14T13:40:25.150176+",
> > "age": 921530.99301089498,
> > "duration": 921530.99316312897,
> > "type_data": {
> > 

[ceph-users] Re: Next quincy point release 17.2.7

[Satoru Takeuchi]

Hi Yuri (resend it because I forgot to add ccs to MLs.)

2023年10月5日(木) 5:57 Yuri Weinstein :

Hello

We are getting very close to the next Quincy point release 17.2.7

Here is the list of must-have PRs https://pad.ceph.com/p/quincy_17.2.7_prs
We will start the release testing/review/approval process as soon as
all PRs from this list are merged.

If you see something missing please speak up and the dev leads will
make a decision on including it in this release.



IMO, the following PR should also be merged into v17.2.7.

rgw: fix bug with PUT request failing during versioned bucket reshard #52097
https://github.com/ceph/ceph/pull/52097

This is to fix the potential data loss if PUT and index resharding happens
simultaneously for versioned bucket.
This is marked as "Urgent" in the corresponding issue.

Although the title is not prefixed by "quincy:", it is actually for quincy.

Best,
Satoru
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rgw: disallowing bucket creation for specific users?

[Matthias Ferdinand]

On Tue, Oct 03, 2023 at 06:10:17PM +0200, Matthias Ferdinand wrote:
> On Sun, Oct 01, 2023 at 12:00:58PM +0200, Peter Goron wrote:
> > Hi Matthias,
> > 
> > One possible way to achieve your need is to set a quota on number of
> > buckets  at user level (see
> > https://docs.ceph.com/en/reef/radosgw/admin/#quota-management). Quotas are
> > under admin control.
> 
> thanks a lot, rather an elegant solution.

sadly, bucket quotas are not really as effective and elegant as I first
thought, since "--max-buckets=0" means "unlimited", not "no buckets".

Setting and enabling per-user bucket-scoped quota:

# radosgw-admin quota set --uid=rgw_user_03 --quota-scope=bucket 
--max-objects=1 --max-size=1 --max-buckets=1

# radosgw-admin quota enable --quota-scope=bucket --uid=rgw_user_03

# radosgw-admin user info --uid=rgw_user_03 | jq 
'.max_buckets,.bucket_quota'
1
{
  "enabled": true,
  "check_on_raw": false,
  "max_size": 1024,
  "max_size_kb": 1,
  "max_objects": 1
}


"--max-buckets=0": number of buckets seems to be effectively unlimited

"--max-buckets=1": the user can create exactly 1 bucket, further bucket
   creation attempts get a "TooManyBuckets" HTTP 400
   response.

Tried a negative number ("--max-buckets=-1"), but that had no effect at
all (not even an error message).

I might pre-create a bucket for each user, e.g. a bucket named
"dead-end-bucket-for-rgw_user_03", so they are already at their maximum
bucket number when they first get their account credentials.
But can I also keep the user from simply deleting this pre-created
bucket and creating a new one with a name we intended for some other
use?

Buckets of reserved names can't be pre-created (and pre-owned by some
special users) here, as the list of reserved names is not fully known,
only a few name prefixes are known so far (i.e. something like
"-.*"), but even with these prefixes we do not have an
exhaustive list.


Regards
Matthias

> > Rgds,
> > Peter
> > 
> > 
> > Le dim. 1 oct. 2023, 10:51, Matthias Ferdinand  a
> > écrit :
> > 
> > > Hi,
> > >
> > > I am still evaluating ceph rgw for specific use cases.
> > >
> > > My question is about keeping the realm of bucket names under control of
> > > rgw admins.
> > >
> > > Normal S3 users have the ability to create new buckets as they see fit.
> > > This opens opportunities for creating excessive amounts of buckets, or
> > > for blocking nice bucket names for other uses, or even using
> > > bucketname-typosquatting as an attack vector.
> > >
> > > In AWS, I can create some IAM users and provide per-bucket access to
> > > them via bucket or IAM user policies. These IAM users can't create new
> > > buckets on their own. Giving out only those IAM credentials to users and
> > > applications, I can ensure no bucket namespace pollution occurs.
> > >
> > > Ceph rgw does not have IAM users (yet?). What could I use here to not
> > > allow certain S3 users to create buckets on their own?
> > >
> > >
> > > Regards
> > > Matthias
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Next quincy point release 17.2.7

[Yuri Weinstein]

Hello

We are getting very close to the next Quincy point release 17.2.7

Here is the list of must-have PRs https://pad.ceph.com/p/quincy_17.2.7_prs
We will start the release testing/review/approval process as soon as
all PRs from this list are merged.

If you see something missing please speak up and the dev leads will
make a decision on including it in this release.

TIA
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Question about RGW S3 Select

[Dave S]

Hi Everyone,
I've been trying to get S3 Select working on our system and whenever I
send a query I get the following in the Payload (Result 200 from RGW):

# aws --endpoint-url http://cephtest1 s3api select-object-content
--bucket test1 --expression-type SQL --input-serialization  '{"CSV":
{"Fieldter": "\"" , "RecordDelimiter" : "\n" , "QuoteEscapeCharacter"
: "\\" , "FileHeaderInfo": "USE" }, "CompressionType": "NONE"}'
--output-serialization '{"CSV": {"FieldDelimiter": ":",
"RecordDelimiter":"\t", "QuoteFields": "ALWAYS"}}' --key
sample_data.csv --expression 'SELECT * from s3object' /dev/stderr




failure -->SELECT * from s3object<---


I also get the same behaviour when accessing via boto3/python.The
same command/code works when accessing other S3 services.  Am I
missing some config or something?

The test1/sample_data.csv file is there and the account is able to get
the sample_data.csv data.  I've tried uploading a version with unix
endings and Dos endings for the data file.  This is the data file from
the AWS example.

 We're running Pacific and the RGWs are deployed as podman containers
on Rocky 8.8:
ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)


Thanks,
-Dave
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph 16.2.x excessive logging, how to reduce?

[Igor Fedotov]

Hi Zakhar,

do reduce rocksdb logging verbosity you might want to set debug_rocksdb 
to 3 (or 0).


I presume it produces a  significant part of the logging traffic.


Thanks,

Igor

On 04/10/2023 20:51, Zakhar Kirpichenko wrote:

Any input from anyone, please?

On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko  wrote:


Hi,

Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very
detailed messages, Ceph logs alone on hosts with monitors and several OSDs
has already eaten through 50% of the endurance of the flash system drives
over a couple of years.

Cluster logging settings are default, and it seems that all daemons are
writing lots and lots of debug information to the logs, such as for
example: https://pastebin.com/ebZq8KZk (it's just a snippet, but there's
lots and lots of various information).

Is there a way to reduce the amount of logging and, for example, limit the
logging to warnings or important messages so that it doesn't include every
successful authentication attempt, compaction etc, etc, when the cluster is
healthy and operating normally?

I would very much appreciate your advice on this.

Best regards,
Zakhar




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph 16.2.x excessive logging, how to reduce?

[Zakhar Kirpichenko]

Any input from anyone, please?

On Tue, 19 Sept 2023 at 09:01, Zakhar Kirpichenko  wrote:

> Hi,
>
> Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very
> detailed messages, Ceph logs alone on hosts with monitors and several OSDs
> has already eaten through 50% of the endurance of the flash system drives
> over a couple of years.
>
> Cluster logging settings are default, and it seems that all daemons are
> writing lots and lots of debug information to the logs, such as for
> example: https://pastebin.com/ebZq8KZk (it's just a snippet, but there's
> lots and lots of various information).
>
> Is there a way to reduce the amount of logging and, for example, limit the
> logging to warnings or important messages so that it doesn't include every
> successful authentication attempt, compaction etc, etc, when the cluster is
> healthy and operating normally?
>
> I would very much appreciate your advice on this.
>
> Best regards,
> Zakhar
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Manual resharding with multisite

[Yixin Jin]

Hi folks,

I am aware that dynamic resharding isn't supported before Reef with multisite. 
However, does manual resharding work? It doesn't seem to be so, either. First 
of all, running "bucket reshard" has to be in the master zone. But if the 
objects of that bucket isn't in the master zone, resharding in the master zone 
seems to render those objects inaccessible in the zone that actually has them. 
So, what is recommended practice of resharding with multiste? No resharding at 
all?

Thanks,
Yixin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rgw: disallowing bucket creation for specific users?

[Matthias Ferdinand]

> Tried a negative number ("--max-buckets=-1"), but that had no effect at
> all (not even an error message).

must have mistyped the command; trying again with "-max-buckets=-1", it
shows the wanted effect: user cannot create any bucket.

So, an effective and elegant method indeed :-)

Matthias

PS: answering my own question below "how can I keep users from deleting
their own bucket?": bucket policy.

$ cat denypolicy.txt
{
  "Version": "2012-10-17",
  "Statement": [{
"Effect": "Deny",
"Principal": {"AWS": ["*"]},
"Action": [ "s3:DeleteBucket", "s3:DeleteBucketPolicy", 
"s3:PutBucketPolicy" ],
"Resource": [
  "arn:aws:s3:::*"
]
  }]
}

Bucket cannot be deleted any more even by the bucket owner, and also the
bucket policy can't be modified/deleted anymore.

This closes the loopholes I could come up with so far; there might still
be some left I am currently not aware of :-)


On Wed, Oct 04, 2023 at 06:20:09PM +0200, Matthias Ferdinand wrote:
> On Tue, Oct 03, 2023 at 06:10:17PM +0200, Matthias Ferdinand wrote:
> > On Sun, Oct 01, 2023 at 12:00:58PM +0200, Peter Goron wrote:
> > > Hi Matthias,
> > > 
> > > One possible way to achieve your need is to set a quota on number of
> > > buckets  at user level (see
> > > https://docs.ceph.com/en/reef/radosgw/admin/#quota-management). Quotas are
> > > under admin control.
> > 
> > thanks a lot, rather an elegant solution.
> 
> sadly, bucket quotas are not really as effective and elegant as I first
> thought, since "--max-buckets=0" means "unlimited", not "no buckets".
> 
> Setting and enabling per-user bucket-scoped quota:
> 
> # radosgw-admin quota set --uid=rgw_user_03 --quota-scope=bucket 
> --max-objects=1 --max-size=1 --max-buckets=1
> 
> # radosgw-admin quota enable --quota-scope=bucket --uid=rgw_user_03
> 
> # radosgw-admin user info --uid=rgw_user_03 | jq 
> '.max_buckets,.bucket_quota'
> 1
> {
>   "enabled": true,
>   "check_on_raw": false,
>   "max_size": 1024,
>   "max_size_kb": 1,
>   "max_objects": 1
> }
> 
> 
> "--max-buckets=0": number of buckets seems to be effectively unlimited
> 
> "--max-buckets=1": the user can create exactly 1 bucket, further bucket
>  creation attempts get a "TooManyBuckets" HTTP 400
>response.
> 
> Tried a negative number ("--max-buckets=-1"), but that had no effect at
> all (not even an error message).
> 
> I might pre-create a bucket for each user, e.g. a bucket named
> "dead-end-bucket-for-rgw_user_03", so they are already at their maximum
> bucket number when they first get their account credentials.
> But can I also keep the user from simply deleting this pre-created
> bucket and creating a new one with a name we intended for some other
> use?
> 
> Buckets of reserved names can't be pre-created (and pre-owned by some
> special users) here, as the list of reserved names is not fully known,
> only a few name prefixes are known so far (i.e. something like
> "-.*"), but even with these prefixes we do not have an
> exhaustive list.
> 
> 
> Regards
> Matthias
> 
> > > Rgds,
> > > Peter
> > > 
> > > 
> > > Le dim. 1 oct. 2023, 10:51, Matthias Ferdinand  a
> > > écrit :
> > > 
> > > > Hi,
> > > >
> > > > I am still evaluating ceph rgw for specific use cases.
> > > >
> > > > My question is about keeping the realm of bucket names under control of
> > > > rgw admins.
> > > >
> > > > Normal S3 users have the ability to create new buckets as they see fit.
> > > > This opens opportunities for creating excessive amounts of buckets, or
> > > > for blocking nice bucket names for other uses, or even using
> > > > bucketname-typosquatting as an attack vector.
> > > >
> > > > In AWS, I can create some IAM users and provide per-bucket access to
> > > > them via bucket or IAM user policies. These IAM users can't create new
> > > > buckets on their own. Giving out only those IAM credentials to users and
> > > > applications, I can ensure no bucket namespace pollution occurs.
> > > >
> > > > Ceph rgw does not have IAM users (yet?). What could I use here to not
> > > > allow certain S3 users to create buckets on their own?
> > > >
> > > >
> > > > Regards
> > > > Matthias
> > > > ___
> > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rgw: disallowing bucket creation for specific users?

[Matthias Ferdinand]

On Tue, Oct 03, 2023 at 06:10:17PM +0200, Matthias Ferdinand wrote:
> On Sun, Oct 01, 2023 at 12:00:58PM +0200, Peter Goron wrote:
> > Hi Matthias,
> > 
> > One possible way to achieve your need is to set a quota on number of
> > buckets  at user level (see
> > https://docs.ceph.com/en/reef/radosgw/admin/#quota-management). Quotas are
> > under admin control.
> 
> thanks a lot, rather an elegant solution.

sadly, bucket quotas are not really as effective and elegant as I first
thought, since "--max-buckets=0" means "unlimited", not "no buckets".

Setting and enabling per-user bucket-scoped quota:

# radosgw-admin quota set --uid=rgw_user_03 --quota-scope=bucket 
--max-objects=1 --max-size=1 --max-buckets=1

# radosgw-admin quota enable --quota-scope=bucket --uid=rgw_user_03

# radosgw-admin user info --uid=rgw_user_03 | jq 
'.max_buckets,.bucket_quota'
1
{
  "enabled": true,
  "check_on_raw": false,
  "max_size": 1024,
  "max_size_kb": 1,
  "max_objects": 1
}


"--max-buckets=0": number of buckets seems to be effectively unlimited

"--max-buckets=1": the user can create exactly 1 bucket, further bucket
   creation attempts get a "TooManyBuckets" HTTP 400
   response.

Tried a negative number ("--max-buckets=-1"), but that had no effect at
all (not even an error message).

I might pre-create a bucket for each user, e.g. a bucket named
"dead-end-bucket-for-rgw_user_03", so they are already at their maximum
bucket number when they first get their account credentials.
But can I also keep the user from simply deleting this pre-created
bucket and creating a new one with a name we intended for some other
use?

Buckets of reserved names can't be pre-created (and pre-owned by some
special users) here, as the list of reserved names is not fully known,
only a few name prefixes are known so far (i.e. something like
"-.*"), but even with these prefixes we do not have an
exhaustive list.


Regards
Matthias

> > Rgds,
> > Peter
> > 
> > 
> > Le dim. 1 oct. 2023, 10:51, Matthias Ferdinand  a
> > écrit :
> > 
> > > Hi,
> > >
> > > I am still evaluating ceph rgw for specific use cases.
> > >
> > > My question is about keeping the realm of bucket names under control of
> > > rgw admins.
> > >
> > > Normal S3 users have the ability to create new buckets as they see fit.
> > > This opens opportunities for creating excessive amounts of buckets, or
> > > for blocking nice bucket names for other uses, or even using
> > > bucketname-typosquatting as an attack vector.
> > >
> > > In AWS, I can create some IAM users and provide per-bucket access to
> > > them via bucket or IAM user policies. These IAM users can't create new
> > > buckets on their own. Giving out only those IAM credentials to users and
> > > applications, I can ensure no bucket namespace pollution occurs.
> > >
> > > Ceph rgw does not have IAM users (yet?). What could I use here to not
> > > allow certain S3 users to create buckets on their own?
> > >
> > >
> > > Regards
> > > Matthias
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Calling all Ceph users and developers! Submit a topic for the next User + Dev Meeting!

[Laura Flores]

Hi Ceph users and developers,

We are gearing up for the next User + Developer Monthly Meeting, happening
October 19th at 10am EST.

If you are interested in being a guest speaker, you are invited to submit a
focus topic to this Google form:
https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4vJDGBrp6d-D3-BlQ/viewform?usp=sf_link

Examples of what we're looking for in a focus topic include:

   - Feature requests / RFEs
   - User feedback
   - Knowledge sharing (upgrades, workloads, etc.)
   - Ideas for longterm improvement (user-facing)
   - Share the use case of your cluster

Any Ceph user or developer is eligible to submit! Reach out to me with any
questions.

- Laura Flores

-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: snap_schedule works after 1 hour of scheduling

[Milind Changire]

On Wed, Oct 4, 2023 at 7:19 PM Kushagr Gupta
 wrote:
>
> Hi Milind,
>
> Thank you for your swift response.
>
> >>How many hours did you wait after the "start time" and decide to restart 
> >>mgr ?
> We waited for ~3 days before restarting the mgr-service.

The only thing I can think of is a stale mgr that wasn't restarted
after an upgrade.
Was an upgrade performed lately ?

Did the dir exist at the time the snapshot was scheduled to take place.
If it didn't then the schedule gets disabled until explicitly enabled.

>
> There was one more instance where we waited for 2 hours and then re-started 
> and in the third hour the schedule started working.
>
> Could you please guide us if we are doing anything wrong.
> Kindly let us know if any logs are required.
>
> Thanks and Regards,
> Kushagra Gupta
>
> On Wed, Oct 4, 2023 at 5:39 PM Milind Changire  wrote:
>>
>> On Wed, Oct 4, 2023 at 3:40 PM Kushagr Gupta
>>  wrote:
>> >
>> > Hi Team,Milind
>> >
>> > Ceph-version: Quincy, Reef
>> > OS: Almalinux 8
>> >
>> > Issue: snap_schedule works after 1 hour of schedule
>> >
>> > Description:
>> >
>> > We are currently working in a 3-node ceph cluster.
>> > We are currently exploring the scheduled snapshot capability of the 
>> > ceph-mgr module.
>> > To enable/configure scheduled snapshots, we followed the following link:
>> >
>> >
>> >
>> > https://docs.ceph.com/en/quincy/cephfs/snap-schedule/
>> >
>> >
>> >
>> > We were able to create snap schedules for the subvolumes as suggested.
>> > But we have observed a two very strange behaviour:
>> > 1. The snap_schedules only work when we restart the ceph-mgr service on 
>> > the mgr node:
>> > We then restarted the mgr-service on the active mgr node, and after 1 hour 
>> > it started getting created. I am attaching the log file for the same after 
>> > restart. Thre behaviour looks abnormal.
>>
>> A mgr restart is not required for the schedule to get triggered.
>> How many hours did you wait after the "start time" and decide to restart mgr 
>> ?
>>
>> >
>> > So,  for eg consider the below output:
>> > ```
>> > [root@storagenode-1 ~]# ceph fs snap-schedule status 
>> > /volumes/subvolgrp/test3
>> > {"fs": "cephfs", "subvol": null, "path": "/volumes/subvolgrp/test3", 
>> > "rel_path": "/volumes/subvolgrp/test3", "schedule": "1h", "retention": {}, 
>> > "start": "2023-10-04T07:20:00", "created": "2023-10-04T07:18:41", "first": 
>> > "2023-10-04T08:20:00", "last": "2023-10-04T09:20:00", "last_pruned": null, 
>> > "created_count": 2, "pruned_count": 0, "active": true}
>> > [root@storagenode-1 ~]#
>> > ```
>> > As we can see in the above o/p, we created the schedule at 
>> > 2023-10-04T07:18:41. The schedule was suppose to start at 
>> > 2023-10-04T07:20:00 but it started at 2023-10-04T08:20:00
>>
>> seems normal behavior to me
>> the schedule starts countdown for 1h from 2023-10-04T07:20:00 and
>> created first snapshot at 2023-10-04T08:20:00
>>
>> >
>> > Any input w.r.t the same will be of great help.
>> >
>> > Thanks and Regards
>> > Kushagra Gupta
>>
>>
>>
>> --
>> Milind
>>


-- 
Milind
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Autoscaler problems in pacific

[Boris Behrens]

Also found what the 2nd problem was:
When there are pools using the default replicated_ruleset while there are
multiple rulesets with differenct device classes, the autoscaler does not
produce any output.

Should I open a bug for that?

Am Mi., 4. Okt. 2023 um 14:36 Uhr schrieb Boris Behrens :

> Found the bug for the TOO_MANY_PGS: https://tracker.ceph.com/issues/62986
> But I am still not sure, why I don't have any output on that one cluster.
>
> Am Mi., 4. Okt. 2023 um 14:08 Uhr schrieb Boris Behrens :
>
>> Hi,
>> I've just upgraded to our object storages to the latest pacific version
>> (16.2.14) and the autscaler is acting weird.
>> On one cluster it just shows nothing:
>> ~# ceph osd pool autoscale-status
>> ~#
>>
>> On the other clusters it shows this when it is set to warn:
>> ~# ceph health detail
>> ...
>> [WRN] POOL_TOO_MANY_PGS: 2 pools have too many placement groups
>> Pool .rgw.buckets.data has 1024 placement groups, should have 1024
>> Pool device_health_metrics has 1 placement groups, should have 1
>>
>> Version 16.2.13 seems to act normal.
>> Is this a known bug?
>> --
>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
>> groüen Saal.
>>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Remove empty orphaned PGs not mapped to a pool

[Malte Stroem]

Hello Eugen,

yes, we followed the documentation and everything worked fine. The cache 
is gone.


Removing the pool worked well. Everything is clean.

The PGs are empty active+clean.

Possible solutions:

1.

ceph pg {pg-id} mark_unfound_lost delete

I do not think this is the right way since it is for PGs with status 
unfound. But it could work also.


2.

Set the following for the three disk:

ceph osd lost {osd-id}

I am not sure how the cluster will react to this.

3.

ceph-objectstore-tool --data-path /path/to/osd --op remove --pgid 3.0 
--force


Now, will the cluster accept the removed PG status?

4.

The three disks are still presented in the crush rule, class ssd, each 
single OSD under one host entry.


What if I remove them from crush?

Do you have a better idea, Eugen?

Best,
Malte

Am 04.10.23 um 09:21 schrieb Eugen Block:

Hi,

just for clarity, you're actually talking about the cache tier as 
described in the docs [1]? And you followed the steps until 'ceph osd 
tier remove cold-storage hot-storage' successfully? And the pool has 
been really deleted successfully ('ceph osd pool ls detail')?


[1] 
https://docs.ceph.com/en/latest/rados/operations/cache-tiering/#removing-a-cache-tier


Zitat von Malte Stroem :


Hello,

we removed an SSD cache tier and its pool.

The PGs for the pool do still exist.

The cluster is healthy.

The PGs are empty and they reside on the cache tier pool's SSDs.

We like to take out the disks but it is not possible. The cluster sees 
the PGs and answers with a HEALTH_WARN.


Because of the replication of three there are still 128 PGs on three 
of the 24 OSDs. We were able to remove the other OSDs.


Summary:

- pool removed
- 3 x 128 empty PGs still exist
- 3 of 24 OSDs still exist

How is it possible to remove these empty and healthy PGs?

The only way I found was something like:

ceph pg {pg-id} mark_unfound_lost delete

Is that the right way?

Some output of:

ceph pg ls-by-osd 23

PG  OBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES  OMAP_BYTES* 
OMAP_KEYS*  LOG   STATE SINCE  VERSION REPORTED UP   
  ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
3.0  0 0  0    0  0    0   0 
    0  active+clean    27h 0'0    2627265:196316 
[15,6,23]p15  [15,6,23]p15  2023-09-28T12:41:52.982955+0200 
2023-09-27T06:48:23.265838+0200
3.1  0 0  0    0  0    0   0 
    0  active+clean 9h 0'0    2627266:19330 
[6,23,15]p6  [6,23,15]p6  2023-09-29T06:30:57.630016+0200 
2023-09-27T22:58:21.992451+0200
3.2  0 0  0    0  0    0   0 
    0  active+clean 2h 0'0   2627265:1135185 
[23,15,6]p23  [23,15,6]p23  2023-09-29T13:42:07.346658+0200 
2023-09-24T14:31:52.844427+0200
3.3  0 0  0    0  0    0   0 
    0  active+clean    13h 0'0    2627266:193170 
[6,15,23]p6  [6,15,23]p6  2023-09-29T01:56:54.517337+0200 
2023-09-27T17:47:24.961279+0200
3.4  0 0  0    0  0    0   0 
    0  active+clean    14h 0'0   2627265:2343551 
[23,6,15]p23  [23,6,15]p23  2023-09-29T00:47:47.548860+0200 
2023-09-25T09:39:51.259304+0200
3.5  0 0  0    0  0    0   0 
    0  active+clean 2h 0'0    2627265:194111 
[15,6,23]p15  [15,6,23]p15  2023-09-29T13:28:48.879959+0200 
2023-09-26T15:35:44.217302+0200
3.6  0 0  0    0  0    0   0 
    0  active+clean 6h 0'0   2627265:2345717 
[23,15,6]p23  [23,15,6]p23  2023-09-29T09:26:02.534825+0200 
2023-09-27T21:56:57.500126+0200


Best regards,
Malte
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: snap_schedule works after 1 hour of scheduling

[Kushagr Gupta]

Hi Milind,

Thank you for your swift response.

>>How many hours did you wait after the "start time" and decide to restart
mgr ?
We waited for ~3 days before restarting the mgr-service.

There was one more instance where we waited for 2 hours and then re-started
and in the third hour the schedule started working.

Could you please guide us if we are doing anything wrong.
Kindly let us know if any logs are required.

Thanks and Regards,
Kushagra Gupta

On Wed, Oct 4, 2023 at 5:39 PM Milind Changire  wrote:

> On Wed, Oct 4, 2023 at 3:40 PM Kushagr Gupta
>  wrote:
> >
> > Hi Team,Milind
> >
> > Ceph-version: Quincy, Reef
> > OS: Almalinux 8
> >
> > Issue: snap_schedule works after 1 hour of schedule
> >
> > Description:
> >
> > We are currently working in a 3-node ceph cluster.
> > We are currently exploring the scheduled snapshot capability of the
> ceph-mgr module.
> > To enable/configure scheduled snapshots, we followed the following link:
> >
> >
> >
> > https://docs.ceph.com/en/quincy/cephfs/snap-schedule/
> >
> >
> >
> > We were able to create snap schedules for the subvolumes as suggested.
> > But we have observed a two very strange behaviour:
> > 1. The snap_schedules only work when we restart the ceph-mgr service on
> the mgr node:
> > We then restarted the mgr-service on the active mgr node, and after 1
> hour it started getting created. I am attaching the log file for the same
> after restart. Thre behaviour looks abnormal.
>
> A mgr restart is not required for the schedule to get triggered.
> How many hours did you wait after the "start time" and decide to restart
> mgr ?
>
> >
> > So,  for eg consider the below output:
> > ```
> > [root@storagenode-1 ~]# ceph fs snap-schedule status
> /volumes/subvolgrp/test3
> > {"fs": "cephfs", "subvol": null, "path": "/volumes/subvolgrp/test3",
> "rel_path": "/volumes/subvolgrp/test3", "schedule": "1h", "retention": {},
> "start": "2023-10-04T07:20:00", "created": "2023-10-04T07:18:41", "first":
> "2023-10-04T08:20:00", "last": "2023-10-04T09:20:00", "last_pruned": null,
> "created_count": 2, "pruned_count": 0, "active": true}
> > [root@storagenode-1 ~]#
> > ```
> > As we can see in the above o/p, we created the schedule at
> 2023-10-04T07:18:41. The schedule was suppose to start at
> 2023-10-04T07:20:00 but it started at 2023-10-04T08:20:00
>
> seems normal behavior to me
> the schedule starts countdown for 1h from 2023-10-04T07:20:00 and
> created first snapshot at 2023-10-04T08:20:00
>
> >
> > Any input w.r.t the same will be of great help.
> >
> > Thanks and Regards
> > Kushagra Gupta
>
>
>
> --
> Milind
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Issue with radosgw-admin reshard when bucket belongs to user with tenant on ceph quincy (17.2.6)

[christoph . weber+cephmailinglist]

Hi everybody,

I tried to reshard a bucket belonging to the tenant "test-tenant", but got an 
"No such file or directory" error.

$ radosgw-admin reshard add --bucket test-tenant/test-bucket --num-shards 40

$ radosgw-admin reshard process
2023-10-04T12:12:52.470+0200 7f654237afc0  0 process_single_logshard: Error 
during resharding bucket test-tenant/test-bucket:(2) No such file or directory

$ radosgw-admin reshard list
[
{
"time": "2023-10-04T10:12:46.528741Z",
"tenant": "",
"bucket_name": "test-tenant/test-bucket",
"bucket_id": "43e570fd-7573-403f-ab86-a75e12e60146.24142.3",
"new_instance_id": "",
"old_num_shards": 23,
"tentative_new_num_shards": 40
}
]

(I also tried --bucket test-bucket)

I then tried to remove the incomplete reshard-job

$ radosgw-admin reshard cancel --bucket  test-tenant/test-bucket
$ radosgw-admin reshard list
[]

$ radosgw-admin reshard process
2023-10-04T12:33:04.251+0200 7fb728e16fc0  0 INFO: RGWReshardLock::lock found 
lock on reshard.00 to be held by another RGW process; skipping for now

With

$ radosgw-admin lc reshard fix --bucket  test-tenant/test-bucket

and restarting the rgw containers, I could get rid of the reshard.00 
lock.

Unfortunately updating to the latest 17.2 version did not help.

$ ceph orch upgrade start --ceph-version 17.2.6

$ ceph --version
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)


But after transferring the bucket to a different RGW user without tenant, 
resharding worked fine:

$ reshard add --bucket test-bucket-without-tenant-user --num-shards 40

$ radosgw-admin reshard process
2023-10-04T12:44:43.599+0200 7f75c908ffc0  1 execute INFO: reshard of bucket 
"test-bucket-without-tenant-user" from 
"test-bucket-without-tenant-user:43e570fd-7573-403f-ab86-a75e12e60146.74748.1" 
to 
"test-bucket-without-tenant-user:43e570fd-7573-403f-ab86-a75e12e60146.74819.1" 
completed successfully


This solved my problem, but I wanted to report it, as it may affect other 
installations too.

Is this problem known? I could not find information for this.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] snap_schedule works after 1 hour of scheduling

[Kushagr Gupta]

Hi Team,Milind

*Ceph-version:* Quincy, Reef
*OS:* Almalinux 8

*Issue:* snap_schedule works after 1 hour of schedule

*Description:*

We are currently working in a 3-node ceph cluster.
We are currently exploring the scheduled snapshot capability of the
ceph-mgr module.
To enable/configure scheduled snapshots, we followed the following link:



https://docs.ceph.com/en/quincy/cephfs/snap-schedule/



We were able to create snap schedules for the subvolumes as suggested.
But we have observed a two very strange behaviour:
1. The snap_schedules only work when we restart the ceph-mgr service on the
mgr node:
We then restarted the mgr-service on the active mgr node, and after 1 hour
it started getting created. I am attaching the log file for the same after
restart. Thre behaviour looks abnormal.

So,  for eg consider the below output:
```
[root@storagenode-1 ~]# ceph fs snap-schedule status
/volumes/subvolgrp/test3
{"fs": "cephfs", "subvol": null, "path": "/volumes/subvolgrp/test3",
"rel_path": "/volumes/subvolgrp/test3", "schedule": "1h", "retention": {},
"start": "2023-10-04T07:20:00", "created": "2023-10-04T07:18:41", "first":
"2023-10-04T08:20:00", "last": "2023-10-04T09:20:00", "last_pruned": null,
"created_count": 2, "pruned_count": 0, "active": true}
[root@storagenode-1 ~]#
```
As we can see in the above o/p, we created the schedule at
2023-10-04T07:18:41. The schedule was suppose to start at
2023-10-04T07:20:00 but it started at 2023-10-04T08:20:00

Any input w.r.t the same will be of great help.

Thanks and Regards
Kushagra Gupta
2023-10-03T03:50:15.857+ 7f6971c0e700 -1 mgr handle_mgr_signal  *** Got 
signal Terminated ***
2023-10-03T03:50:16.659+ 7f9d9da97200  0 set uid:gid to 167:167 (ceph:ceph)
2023-10-03T03:50:16.659+ 7f9d9da97200  0 ceph version 18.2.0 
(5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable), process ceph-mgr, pid 
781047
2023-10-03T03:50:16.660+ 7f9d9da97200  0 pidfile_write: ignore empty 
--pid-file
2023-10-03T03:50:16.726+ 7f9d9da97200  1 mgr[py] Loading python module 
'alerts'
2023-10-03T03:50:16.975+ 7f9d9da97200 -1 mgr[py] Module alerts has missing 
NOTIFY_TYPES member
2023-10-03T03:50:16.975+ 7f9d9da97200  1 mgr[py] Loading python module 
'balancer'
2023-10-03T03:50:17.126+ 7f9d9da97200 -1 mgr[py] Module balancer has 
missing NOTIFY_TYPES member
2023-10-03T03:50:17.126+ 7f9d9da97200  1 mgr[py] Loading python module 
'crash'
2023-10-03T03:50:17.291+ 7f9d9da97200 -1 mgr[py] Module crash has missing 
NOTIFY_TYPES member
2023-10-03T03:50:17.291+ 7f9d9da97200  1 mgr[py] Loading python module 
'devicehealth'
2023-10-03T03:50:17.435+ 7f9d9da97200 -1 mgr[py] Module devicehealth has 
missing NOTIFY_TYPES member
2023-10-03T03:50:17.435+ 7f9d9da97200  1 mgr[py] Loading python module 
'influx'
2023-10-03T03:50:17.580+ 7f9d9da97200 -1 mgr[py] Module influx has missing 
NOTIFY_TYPES member
2023-10-03T03:50:17.580+ 7f9d9da97200  1 mgr[py] Loading python module 
'insights'
2023-10-03T03:50:17.697+ 7f9d9da97200  1 mgr[py] Loading python module 
'iostat'
2023-10-03T03:50:17.830+ 7f9d9da97200 -1 mgr[py] Module iostat has missing 
NOTIFY_TYPES member
2023-10-03T03:50:17.830+ 7f9d9da97200  1 mgr[py] Loading python module 
'localpool'
2023-10-03T03:50:17.933+ 7f9d9da97200  1 mgr[py] Loading python module 
'mds_autoscaler'
2023-10-03T03:50:18.118+ 7f9d9da97200  1 mgr[py] Loading python module 
'mirroring'
2023-10-03T03:50:18.287+ 7f9d9da97200  1 mgr[py] Loading python module 'nfs'
2023-10-03T03:50:18.491+ 7f9d9da97200 -1 mgr[py] Module nfs has missing 
NOTIFY_TYPES member
2023-10-03T03:50:18.491+ 7f9d9da97200  1 mgr[py] Loading python module 
'orchestrator'
2023-10-03T03:50:18.673+ 7f9d9da97200 -1 mgr[py] Module orchestrator has 
missing NOTIFY_TYPES member
2023-10-03T03:50:18.673+ 7f9d9da97200  1 mgr[py] Loading python module 
'osd_perf_query'
2023-10-03T03:50:18.859+ 7f9d9da97200 -1 mgr[py] Module osd_perf_query has 
missing NOTIFY_TYPES member
2023-10-03T03:50:18.859+ 7f9d9da97200  1 mgr[py] Loading python module 
'osd_support'
2023-10-03T03:50:18.964+ 7f9d9da97200 -1 mgr[py] Module osd_support has 
missing NOTIFY_TYPES member
2023-10-03T03:50:18.964+ 7f9d9da97200  1 mgr[py] Loading python module 
'pg_autoscaler'
2023-10-03T03:50:19.083+ 7f9d9da97200 -1 mgr[py] Module pg_autoscaler has 
missing NOTIFY_TYPES member
2023-10-03T03:50:19.083+ 7f9d9da97200  1 mgr[py] Loading python module 
'progress'
2023-10-03T03:50:19.285+ 7f9d9da97200 -1 mgr[py] Module progress has 
missing NOTIFY_TYPES member
2023-10-03T03:50:19.285+ 7f9d9da97200  1 mgr[py] Loading python module 
'prometheus'
2023-10-03T03:50:19.680+ 7f9d9da97200 -1 mgr[py] Module prometheus has 
missing NOTIFY_TYPES member
2023-10-03T03:50:19.680+ 7f9d9da97200  1 mgr[py] Loading python module 
'rbd_support'
2023-10-03T03:50:19.821+ 7f9d9da97200 -1 mgr[py] Module rbd_support has 
missing NOTIFY_TYPES 

[ceph-users] Re: Autoscaler problems in pacific

[Boris Behrens]

Found the bug for the TOO_MANY_PGS: https://tracker.ceph.com/issues/62986
But I am still not sure, why I don't have any output on that one cluster.

Am Mi., 4. Okt. 2023 um 14:08 Uhr schrieb Boris Behrens :

> Hi,
> I've just upgraded to our object storages to the latest pacific version
> (16.2.14) and the autscaler is acting weird.
> On one cluster it just shows nothing:
> ~# ceph osd pool autoscale-status
> ~#
>
> On the other clusters it shows this when it is set to warn:
> ~# ceph health detail
> ...
> [WRN] POOL_TOO_MANY_PGS: 2 pools have too many placement groups
> Pool .rgw.buckets.data has 1024 placement groups, should have 1024
> Pool device_health_metrics has 1 placement groups, should have 1
>
> Version 16.2.13 seems to act normal.
> Is this a known bug?
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: snap_schedule works after 1 hour of scheduling

[Milind Changire]

On Wed, Oct 4, 2023 at 3:40 PM Kushagr Gupta
 wrote:
>
> Hi Team,Milind
>
> Ceph-version: Quincy, Reef
> OS: Almalinux 8
>
> Issue: snap_schedule works after 1 hour of schedule
>
> Description:
>
> We are currently working in a 3-node ceph cluster.
> We are currently exploring the scheduled snapshot capability of the ceph-mgr 
> module.
> To enable/configure scheduled snapshots, we followed the following link:
>
>
>
> https://docs.ceph.com/en/quincy/cephfs/snap-schedule/
>
>
>
> We were able to create snap schedules for the subvolumes as suggested.
> But we have observed a two very strange behaviour:
> 1. The snap_schedules only work when we restart the ceph-mgr service on the 
> mgr node:
>   We then restarted the mgr-service on the active mgr node, and after 1 hour 
> it started getting created. I am attaching the log file for the same after 
> restart. Thre behaviour looks abnormal.

A mgr restart is not required for the schedule to get triggered.
How many hours did you wait after the "start time" and decide to restart mgr ?

>
>   So,  for eg consider the below output:
> ```
> [root@storagenode-1 ~]# ceph fs snap-schedule status 
> /volumes/subvolgrp/test3
> {"fs": "cephfs", "subvol": null, "path": "/volumes/subvolgrp/test3", 
> "rel_path": "/volumes/subvolgrp/test3", "schedule": "1h", "retention": {}, 
> "start": "2023-10-04T07:20:00", "created": "2023-10-04T07:18:41", "first": 
> "2023-10-04T08:20:00", "last": "2023-10-04T09:20:00", "last_pruned": null, 
> "created_count": 2, "pruned_count": 0, "active": true}
> [root@storagenode-1 ~]#
> ```
>   As we can see in the above o/p, we created the schedule at 
> 2023-10-04T07:18:41. The schedule was suppose to start at 2023-10-04T07:20:00 
> but it started at 2023-10-04T08:20:00

seems normal behavior to me
the schedule starts countdown for 1h from 2023-10-04T07:20:00 and
created first snapshot at 2023-10-04T08:20:00

>
> Any input w.r.t the same will be of great help.
>
> Thanks and Regards
> Kushagra Gupta



-- 
Milind
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Autoscaler problems in pacific

[Boris Behrens]

Hi,
I've just upgraded to our object storages to the latest pacific version
(16.2.14) and the autscaler is acting weird.
On one cluster it just shows nothing:
~# ceph osd pool autoscale-status
~#

On the other clusters it shows this when it is set to warn:
~# ceph health detail
...
[WRN] POOL_TOO_MANY_PGS: 2 pools have too many placement groups
Pool .rgw.buckets.data has 1024 placement groups, should have 1024
Pool device_health_metrics has 1 placement groups, should have 1

Version 16.2.13 seems to act normal.
Is this a known bug?
-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW multisite - requesting help for fixing error_code: 125

[Eugen Block]

Hi,

I just did this successfully on a test Reef cluster (no multi-site)

$ radosgw-admin object rewrite --bucket=bucket1 --object="myfile.txt"

where "--object" is the object name. The epoch and the tag have been  
updated, so I guess it worked.
But I also got a segfault on a Octopus test cluster so I'm not sure if  
it will work. Maybe try with a test object first and see if it works  
in general.


In your case the object name should be  
"b1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:2897" then (I guess),  
or is this just a directory? Any chance to reqrite the object from the  
client side?



Zitat von Jayanth Reddy :


Hello Users,

We're running 2 Ceph clusters with v17.2.6 and noticing the error message
in # radosgw-admin sync error list

*"message": "failed to sync bucket instance: (125) Operation canceled"*



We've the output as below,


[

{

"shard_id": 0,

"entries": [

{

"id": "1_1690711173.869335_133603.1",

"section": "data",

"name":
"b1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:2897",

"timestamp": "2023-07-30T09:59:33.869335Z",

"info": {

"source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d",

"error_code": 125,

"message": "failed to sync bucket instance: (125)
Operation canceled"

}

},

{

"id": "1_1690711175.505687_133683.1",

"section": "data",

"name":
"b1:d09d3d16-8601-448b-bf3d-609b8a29647d.38987.1:1719",

"timestamp": "2023-07-30T09:59:35.505687Z",

"info": {

"source_zone": "d09d3d16-8601-448b-bf3d-609b8a29647d",

"error_code": 125,

"message": "failed to sync bucket instance: (125)
Operation canceled"

}

},


and with around 26236 errors


# radosgw-admin sync error list | grep -i "(125) Operation canceled" | wc -l

26236


I'm trying to fix these by rewriting objects but I'm having trouble finding
the exact object names and the procedure. Any help is really appreciated!



Thanks,

Jayanth
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph luminous client connect to ceph reef always permission denied

[Eugen Block]

Hi,

I suspect the auth_allow_insecure_global_id_reclaim config option. If  
you really need this to work you can set


$ ceph config set mon auth_allow_insecure_global_id_reclaim true

and the client should be able to connect. You will get a warning though:


mon is allowing insecure global_id reclaim


You can disable the warning if you need to:

$ ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false

Regards,
Eugen

Zitat von Pureewat Kaewpoi :


Hi All !

We have a new installed cluster with ceph reef. but our old client  
still using ceph luminous.
The problem is when using any command to ceph cluster It will hang  
and no any output.


This is a output from command ceph osd pool ls --debug-ms 1
2023-10-02 23:35:22.727089 7fc93807c700  1  Processor -- start
2023-10-02 23:35:22.729256 7fc93807c700  1 -- - start start
2023-10-02 23:35:22.729790 7fc93807c700  1 -- - --> MON-1:6789/0 --  
auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc930174cb0 con 0
2023-10-02 23:35:22.730724 7fc935e72700  1 -- CLIENT:0/187462963  
learned_addr learned my addr CLIENT:0/187462963
2023-10-02 23:35:22.732091 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.0 MON-1:6789/0 1  auth_reply(proto 2 0 (0) Success) v1   
33+0+0 (2762451217 0 0) 0x7fc920002310 con 0x7fc93017d0f0
2023-10-02 23:35:22.732228 7fc927fff700  1 -- CLIENT:0/187462963 -->  
MON-1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x7fc914000fc0  
con 0
2023-10-02 23:35:22.733237 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.0 MON-1:6789/0 2  auth_reply(proto 2 0 (0) Success) v1   
206+0+0 (3693167043 0 0) 0x7fc920002830 con 0x7fc93017d0f0
2023-10-02 23:35:22.733428 7fc927fff700  1 -- CLIENT:0/187462963 -->  
MON-1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc914002e10  
con 0
2023-10-02 23:35:22.733451 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.0 MON-1:6789/0 3  mon_map magic: 0 v1  532+0+0  
(3038142027 0 0) 0x7fc92e50 con 0x7fc93017d0f0
2023-10-02 23:35:22.734365 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.0 MON-1:6789/0 4  auth_reply(proto 2 0 (0) Success) v1   
580+0+0 (3147563293 0 0) 0x7fc920001640 con 0x7fc93017d0f0
2023-10-02 23:35:22.734597 7fc927fff700  1 -- CLIENT:0/187462963 -->  
MON-1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- 0x7fc9301755e0 con 0
2023-10-02 23:35:22.734678 7fc93807c700  1 -- CLIENT:0/187462963 -->  
MON-1:6789/0 -- mon_subscribe({mgrmap=0+}) v2 -- 0x7fc930180750 con 0
2023-10-02 23:35:22.734805 7fc93807c700  1 -- CLIENT:0/187462963 -->  
MON-1:6789/0 -- mon_subscribe({osdmap=0}) v2 -- 0x7fc930180f00 con 0
2023-10-02 23:35:22.734891 7fc935e72700  1 -- CLIENT:0/187462963 >>  
MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1  
l=1).read_bulk peer close file descriptor 13
2023-10-02 23:35:22.734917 7fc935e72700  1 -- CLIENT:0/187462963 >>  
MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1  
l=1).read_until read failed
2023-10-02 23:35:22.734922 7fc935e72700  1 -- CLIENT:0/187462963 >>  
MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1  
l=1).process read tag failed
2023-10-02 23:35:22.734926 7fc935e72700  1 -- CLIENT:0/187462963 >>  
MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_OPEN pgs=754 cs=1  
l=1).fault on lossy channel, failing
2023-10-02 23:35:22.734966 7fc927fff700  1 -- CLIENT:0/187462963 >>  
MON-1:6789/0 conn(0x7fc93017d0f0 :-1 s=STATE_CLOSED pgs=754 cs=1  
l=1).mark_down
2023-10-02 23:35:22.735062 7fc927fff700  1 -- CLIENT:0/187462963 -->  
MON-2:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fc914005580  
con 0
2023-10-02 23:35:22.735077 7fc927fff700  1 -- CLIENT:0/187462963 -->  
MON-3:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fc914005910  
con 0
2023-10-02 23:35:22.737246 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.2 MON-3:6789/0 1  auth_reply(proto 2 0 (0) Success) v1   
33+0+0 (2138308960 0 0) 0x7fc920002fd0 con 0x7fc91400b0c0
2023-10-02 23:35:22.737443 7fc927fff700  1 -- CLIENT:0/187462963 -->  
MON-3:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x7fc914014f10  
con 0
2023-10-02 23:35:22.737765 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.1 MON-2:6789/0 1  auth_reply(proto 2 0 (0) Success) v1   
33+0+0 (3855879565 0 0) 0x7fc928002390 con 0x7fc91400f730
2023-10-02 23:35:22.737799 7fc927fff700  1 -- CLIENT:0/187462963 -->  
MON-2:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x7fc914015850  
con 0
2023-10-02 23:35:22.737966 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.2 MON-3:6789/0 2  auth_reply(proto 2 -13 (13) Permission  
denied) v1  24+0+0 (2583972696 0 0) 0x7fc920003240 con  
0x7fc91400b0c0
2023-10-02 23:35:22.737981 7fc927fff700  1 -- CLIENT:0/187462963 >>  
MON-3:6789/0 conn(0x7fc91400b0c0 :-1 s=STATE_OPEN pgs=464 cs=1  
l=1).mark_down
2023-10-02 23:35:22.738096 7fc927fff700  1 -- CLIENT:0/187462963 <==  
mon.1 MON-2:6789/0 2  auth_reply(proto 2 -13 (13) Permission  
denied) v1  24+0+0 (2583972696 0 0) 0x7fc928002650 con  
0x7fc91400f730

[ceph-users] Re: outdated mds slow requests

[Eugen Block]

Hi,

is this still an issue? If so, I would try to either evict the client  
via admin socket:


ceph tell mds.5 client evict [...] --- Evict client  
session(s) based on a filter


alternatively locally on the MDS:
cephadm enter mds.
ceph daemon mds. client evict 

or restart the MDS which should also clear the client, I believe.

Zitat von Ben :


Hi,
It is running 17.2.5. there are slow requests warnings in cluster log.

ceph tell mds.5 dump_ops_in_flight,
get the following.

These look like outdated and clients were k8s pods. There are warning of
the kind in other mds as well. How could they be cleaned from warnings
safely?

Many thanks.

{
"ops": [
{
"description": "peer_request(mds.3:5311742.0 authpin)",
"initiated_at": "2023-09-14T12:25:43.092201+",
"age": 926013.05098558997,
"duration": 926013.051015759,
"type_data": {
"flag_point": "dispatched",
"reqid": "mds.3:5311742",
"op_type": "peer_request",
"leader_info": {
"leader": "3"
},
"request_info": {
"attempt": 0,
"op_type": "authpin",
"lock_type": 0,
"object_info": "0x60001205d6d.head",
"srcdnpath": "",
"destdnpath": "",
"witnesses": "",
"has_inode_export": false,
"inode_export_v": 0,
"op_stamp": "0.00"
},
"events": [
{
"time": "2023-09-14T12:25:43.092201+",
"event": "initiated"
},
{
"time": "2023-09-14T12:25:43.092202+",
"event": "throttled"
},
{
"time": "2023-09-14T12:25:43.092201+",
"event": "header_read"
},
{
"time": "2023-09-14T12:25:43.092207+",
"event": "all_read"
},
{
"time": "2023-09-14T12:25:43.092218+",
"event": "dispatched"
}
]
}
},
{
"description": "peer_request(mds.3:5311743.0 authpin)",
"initiated_at": "2023-09-14T12:25:43.092371+",
"age": 926013.05081614305,
"duration": 926013.05089185503,
"type_data": {
"flag_point": "dispatched",
"reqid": "mds.3:5311743",
"op_type": "peer_request",
"leader_info": {
"leader": "3"
},
"request_info": {
"attempt": 0,
"op_type": "authpin",
"lock_type": 0,
"object_info": "0x60001205d6d.head",
"srcdnpath": "",
"destdnpath": "",
"witnesses": "",
"has_inode_export": false,
"inode_export_v": 0,
"op_stamp": "0.00"
},
"events": [
{
"time": "2023-09-14T12:25:43.092371+",
"event": "initiated"
},
{
"time": "2023-09-14T12:25:43.092371+",
"event": "throttled"
},
{
"time": "2023-09-14T12:25:43.092371+",
"event": "header_read"
},
{
"time": "2023-09-14T12:25:43.092374+",
"event": "all_read"
},
{
"time": "2023-09-14T12:25:43.092381+",
"event": "dispatched"
}
]
}
},
{
"description": "peer_request(mds.4:4503615.0 authpin)",
"initiated_at": "2023-09-14T13:40:25.150040+",
"age": 921530.99314722,
"duration": 921530.99326053297,
"type_data": {
"flag_point": "dispatched",
"reqid": "mds.4:4503615",
"op_type": "peer_request",
"leader_info": {
"leader": "4"
},
"request_info": {
"attempt": 0,
"op_type": "authpin",
"lock_type": 0,
"object_info": "0x60001205c4f.head",
"srcdnpath": "",
"destdnpath": "",
"witnesses": "",
"has_inode_export": false,
"inode_export_v": 0,
"op_stamp": "0.00"
},
"events": [
{
"time": "2023-09-14T13:40:25.150040+",
"event": "initiated"
},
{
"time": "2023-09-14T13:40:25.150040+",
"event": "throttled"
},
{
"time": "2023-09-14T13:40:25.150040+",
"event": "header_read"
},
{
"time": "2023-09-14T13:40:25.150045+",
"event": "all_read"
},
{
"time": "2023-09-14T13:40:25.150053+",
"event": "dispatched"
}
]
}
},
{
"description": "client_request(client.460983:5731820 getattr pAsLsXsFs
#0x60001205c4f 2023-09-14T13:40:25.144336+ caller_uid=0,
caller_gid=0{})",
"initiated_at": "2023-09-14T13:40:25.150176+",
"age": 921530.99301089498,
"duration": 921530.99316312897,
"type_data": {
"flag_point": "failed to authpin, inode is being exported",
"reqid": "client.460983:5731820",
"op_type": "client_request",
"client_info": {
"client": "client.460983",
"tid": 5731820
},
"events": [
{
"time": "2023-09-14T13:40:25.150176+",
"event": "initiated"
},
{
"time": "2023-09-14T13:40:25.150177+",
"event": "throttled"
},
{
"time": "2023-09-14T13:40:25.150176+",
"event": "header_read"
},
{
"time": "2023-09-14T13:40:25.150180+",
"event": "all_read"
},
{
"time": "2023-09-14T13:40:25.150186+",
"event": "dispatched"
},
{
"time": "2023-09-14T13:40:25.150195+",
"event": "failed to authpin, inode is being exported"
}
]
}
}
],
"num_ops": 4
}
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Balancer blocked as autoscaler not acting on scaling change

[Joachim Kraftmayer - ceph ambassador]

Hi,
we have often seen strange behavior and also interesting pg targets from 
pg_autoscaler in the last years.

That's why we disable it globally.

The commands:
ceph osd reweight-by-utilization
ceph osd test-reweight-by-utilization
are from the time before the upmap balancer was introduced and did not 
solve the problem in the long run in an active cluster.


That the balancer skips the pool with the rebalancing, I've seen more 
than once.


Why pg_autoscaler behaves this way would have to be analyzed in more 
detail. As mentioned above, we normally turn it off.


The idea of Eugen would help when the pool rebalancing is done.

Joachim

___
ceph ambassador DACH
ceph consultant since 2012

Clyso GmbH - Premier Ceph Foundation Member

https://www.clyso.com/

Am 22.09.23 um 11:22 schrieb b...@sanger.ac.uk:

Hi Folks,

We are currently running with one nearfull OSD and 15 nearfull pools. The most 
full OSD is about 86% full but the average is 58% full. However, the balancer 
is skipping a pool on which the autoscaler is trying to complete a pg_num 
reduction from 131,072 to 32,768 (default.rgw.buckets.data pool). However, the 
autoscaler has been working on this for the last 20 days, it works through a 
list of objects that are misplaced but when it gets close to the end, more 
objects get added to the list.

This morning I observed the list get down to c. 7,000 objects misplaced with 2 
PGs active+remapped+backfilling, one PG completed the backfilling then the list 
shot up to c. 70,000 objects misplaced with 3 PGs active+remapped+backfilling.

Has anyone come across this behaviour before? If so, what was your remediation?

Thanks in advance for sharing.
Bruno

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Balancer blocked as autoscaler not acting on scaling change

[Eugen Block]

Hi,

you could change the target_max_misplaced_ratio to 1, the balancer has  
a default 5% ratio of misplaced objects, see [1] for more information:


ceph config get mgr target_max_misplaced_ratio
0.05

[1] https://docs.ceph.com/en/latest/rados/operations/balancer/#throttling

Zitat von b...@sanger.ac.uk:


Hi Folks,

We are currently running with one nearfull OSD and 15 nearfull  
pools. The most full OSD is about 86% full but the average is 58%  
full. However, the balancer is skipping a pool on which the  
autoscaler is trying to complete a pg_num reduction from 131,072 to  
32,768 (default.rgw.buckets.data pool). However, the autoscaler has  
been working on this for the last 20 days, it works through a list  
of objects that are misplaced but when it gets close to the end,  
more objects get added to the list.


This morning I observed the list get down to c. 7,000 objects  
misplaced with 2 PGs active+remapped+backfilling, one PG completed  
the backfilling then the list shot up to c. 70,000 objects misplaced  
with 3 PGs active+remapped+backfilling.


Has anyone come across this behaviour before? If so, what was your  
remediation?


Thanks in advance for sharing.
Bruno

Cluster details:
3,068 OSDs when all running, c. 60 per storage node
OS: Ubuntu 20.04
Ceph: Pacific 16.2.13 from Ubuntu Cloud Archive

Use case:
S3 storage and OpenStack backend, all pools three-way replicated
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: VM hangs when overwriting a file on erasure coded RBD

[Peter Linder]

Hi all,

I would like to follow up on this, it turns out that overwriting the 
file doesn't actually hang, but is just super slow, like several 
minutes. The process is busy in a syscall reading large amounts of what 
I'm assuming is filesystem metadata until the operation finally completes.


The thing is this VM has been randomly creating, deleting, overwriting, 
reading etc random files on a 25TB image for almost 2 years. The image 
has been around 50% full. I have copied all the data to a new image (new 
fs) and things are working fast again, so I don't think this is ceph's 
fault any more. Instead it looks like something that over time broke 
with ext4.


Sending this to the list in case someone else has a similar problem in 
the future.


/Peter


Den 2023-09-29 kl. 19:02, skrev peter.lin...@fiberdirekt.se:
Yes, this is all set up. It was working fine until after the problem 
with the osd host that lost the cluster/sync network occured.


There are a few other VMs that keep running along fine without this 
issue. I've restarted the problematic VM without success (that is, 
creating a file works, but overwriting it still hangs right away). 
fsck runs fine so reading the whole image works.


I'm kind of stumped as to what can cause this.

Because of the lengthy recovery, and then pg autoscaler currenty doing 
things there are currently lots of PGs that haven't been scrubbed, but 
I doubt that is an issue here.



Den 2023-09-29 kl. 18:52, skrev Anthony D'Atri:
EC for RBD wasn't possible until Luminous IIRC, so I had to ask.  You 
have a replicated metadata pool defined?  Does proxmox know that this 
is an EC pool?  When connecting it needs to know both the metata and 
data pools.



On Sep 29, 2023, at 12:49, peter.lin...@fiberdirekt.se wrote:

(sorry for duplicate emails)

This turns out to be a good question actually.

The cluster is running Quincy, 17.2.6.

The compute node that is running the VM is proxmox, version 7.4-3. 
Supposedly this is fairly new, but the version of librbd1 claims to 
be 14.2.21 when I check with "apt list". We are not using proxmox's 
own ceph cluster release. However we haven't had any issues with 
this setup before, but we haven't been using neither erasure coded 
pools nor had the node-half-dead problem for such a long time.


The VM is configured using proxmox which is not libvirt but similar, 
and krbd is not enabled. I don't know for sure if proxmox has its 
own librbd linked in qemu/kvm.


"ceph features" looks like this:

{
 "mon": [
 {
 "features": "0x3f01cfbf7ffd",
 "release": "luminous",
 "num": 5
 }
 ],
 "osd": [
 {
 "features": "0x3f01cfbf7ffd",
 "release": "luminous",
 "num": 24
 }
 ],
 "client": [
 {
 "features": "0x3f01cfb87fec",
 "release": "luminous",
 "num": 4
 },
 {
 "features": "0x3f01cfbf7ffd",
 "release": "luminous",
 "num": 12
 }
 ],
 "mgr": [
 {
 "features": "0x3f01cfbf7ffd",
 "release": "luminous",
 "num": 2
 }
 ]
}

Regards,

Peter


Den 2023-09-29 kl. 17:55, skrev Anthony D'Atri:
Which Ceph releases are installed on the VM and the back end?  Is 
the VM using librbd through libvirt, or krbd?


On Sep 29, 2023, at 09:09, Peter Linder 
 wrote:


Dear all,

I have a problem that after an OSD host lost connection to the 
sync/cluster rear network for many hours (the public network was 
online), a test VM using RBD cant overwrite its files. I can 
create a new file inside it just fine, but not overwrite it, the 
process just hangs.


The VM's disk is on an erasure coded data pool and a replicated 
pool in front of it. EC overwrites is on for the pool.


The custer consists of 5 hosts and 4 OSDs on each, and separate 
hosts for compute. There is a public and separate cluster network, 
separated. In this case, the AOC cable to the cluster network went 
link down on a host and it had to be replaced and the host was 
rebooted. Recovery took about a week to complete. The host was 
half-down for about 12 hours like this.


I have some other VMs as well with images in the same pool 
(totally 4), and they seem to work fine, it is just this one that 
cant overwrite.


I'm thinking there is somehow something wrong with just this image?

Regards,

Peter
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___

[ceph-users] Re: Remove empty orphaned PGs not mapped to a pool

[Eugen Block]

Hi,

just for clarity, you're actually talking about the cache tier as  
described in the docs [1]? And you followed the steps until 'ceph osd  
tier remove cold-storage hot-storage' successfully? And the pool has  
been really deleted successfully ('ceph osd pool ls detail')?


[1]  
https://docs.ceph.com/en/latest/rados/operations/cache-tiering/#removing-a-cache-tier


Zitat von Malte Stroem :


Hello,

we removed an SSD cache tier and its pool.

The PGs for the pool do still exist.

The cluster is healthy.

The PGs are empty and they reside on the cache tier pool's SSDs.

We like to take out the disks but it is not possible. The cluster  
sees the PGs and answers with a HEALTH_WARN.


Because of the replication of three there are still 128 PGs on three  
of the 24 OSDs. We were able to remove the other OSDs.


Summary:

- pool removed
- 3 x 128 empty PGs still exist
- 3 of 24 OSDs still exist

How is it possible to remove these empty and healthy PGs?

The only way I found was something like:

ceph pg {pg-id} mark_unfound_lost delete

Is that the right way?

Some output of:

ceph pg ls-by-osd 23

PG  OBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES  OMAP_BYTES*  
OMAP_KEYS*  LOG   STATE SINCE  VERSION REPORTED UP
  ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
3.0  0 0  00  00   0  
0  active+clean27h 0'02627265:196316  
[15,6,23]p15  [15,6,23]p15  2023-09-28T12:41:52.982955+0200  
2023-09-27T06:48:23.265838+0200
3.1  0 0  00  00   0  
0  active+clean 9h 0'02627266:19330  
[6,23,15]p6  [6,23,15]p6  2023-09-29T06:30:57.630016+0200  
2023-09-27T22:58:21.992451+0200
3.2  0 0  00  00   0  
0  active+clean 2h 0'0   2627265:1135185  
[23,15,6]p23  [23,15,6]p23  2023-09-29T13:42:07.346658+0200  
2023-09-24T14:31:52.844427+0200
3.3  0 0  00  00   0  
0  active+clean13h 0'02627266:193170  
[6,15,23]p6  [6,15,23]p6  2023-09-29T01:56:54.517337+0200  
2023-09-27T17:47:24.961279+0200
3.4  0 0  00  00   0  
0  active+clean14h 0'0   2627265:2343551  
[23,6,15]p23  [23,6,15]p23  2023-09-29T00:47:47.548860+0200  
2023-09-25T09:39:51.259304+0200
3.5  0 0  00  00   0  
0  active+clean 2h 0'02627265:194111  
[15,6,23]p15  [15,6,23]p15  2023-09-29T13:28:48.879959+0200  
2023-09-26T15:35:44.217302+0200
3.6  0 0  00  00   0  
0  active+clean 6h 0'0   2627265:2345717  
[23,15,6]p23  [23,15,6]p23  2023-09-29T09:26:02.534825+0200  
2023-09-27T21:56:57.500126+0200


Best regards,
Malte
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery and inaccurate recovery figures since Quincy upgrade

[Sake]

Hi,Please take a look at the following thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PWHG6QJ6N2TJEYD2U4AXJAJ23CRPJG4E/#7ZMBM23GXYFIGY52ZWJDY5NUSYSDSYL6In short, the value for "osd_mclock_cost_per_byte_usec_hdd" isn't correct. With the release of 17.2.7 this option will be gone, but the recovery speed will be fixed :)Best regards, Sake ___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: set proxy for ceph installation

[Eugen Block]

Did you apply the changes to the containers.conf file on all hosts?  
The MGR daemon is issuing the cephadm commands on the remote hosts, so  
it would need that as well. That setup works for me quite well for  
years now. What distro is your host running on? We mostly use openSUSE  
or SLES, but I also had that successfully running on Ubuntu VMs the  
same way.


Zitat von Majid Varzideh :


Thanks for your responses.
locally it is ok and i can get images without any issue. Also I set proxy
parameters in containers.conf but no result.
it seems that env proxy parameters are not known to user issuing get
commands,


On Wed, Sep 27, 2023 at 11:00 AM Eugen Block  wrote:


You'll need a containers.conf file:

# cat /etc/containers/containers.conf
[engine]
env = ["http_proxy=:", "https_proxy=:",
"no_proxy=localhost"]

Restarting the container should apply the change. Make sure you also
have the correct no_proxy settings, for example so the ceph servers
don't communicate via proxy with each other.

Zitat von Majid Varzideh :

> hi friends
> i have deployed my first node in cluster. we dont have direct internet on
> my server so i have to set proxy for that.i set it /etc/environment
> /etc/profile but i get bellow error
> 2023-09-26 17:09:38,254 7f04058b4b80 DEBUG
>

> cephadm ['--image', 'quay.io/ceph/ceph:v17', 'pull']
> 2023-09-26 17:09:38,302 7f04058b4b80 INFO Pulling container image
> quay.io/ceph/ceph:v17...
> 2023-09-26 17:09:42,083 7f04058b4b80 INFO Non-zero exit code 125 from
> /usr/bin/podman pull quay.io/ceph/ceph:v17
> 2023-09-26 17:09:42,083 7f04058b4b80 INFO /usr/bin/podman: stderr Trying
to
> pull quay.io/ceph/ceph:v17...
> 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr
> time="2023-09-26T17:09:38+03:30" level=warning msg="Failed, retrying in
1s
> ... (1/3). Error: initializing source docker://quay.io/ceph/ceph:v17:
> pinging container registry quay.io: Get \"https://quay.io/v2/\": dial
tcp
> 34.228.154.221:443: connect: connection refused"
> 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr
> time="2023-09-26T17:09:39+03:30" level=warning msg="Failed, retrying in
1s
> ... (2/3). Error: initializing source docker://quay.io/ceph/ceph:v17:
> pinging container registry quay.io: Get \"https://quay.io/v2/\": dial
tcp
> 3.220.246.53:443: connect: connection refused"
> 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr
> time="2023-09-26T17:09:40+03:30" level=warning msg="Failed, retrying in
1s
> ... (3/3). Error: initializing source docker://quay.io/ceph/ceph:v17:
> pinging container registry quay.io: Get \"https://quay.io/v2/\": dial
tcp
> 18.213.60.205:443: connect: connection refused"
> 2023-09-26 17:09:42,084 7f04058b4b80 INFO /usr/bin/podman: stderr Error:
> initializing source docker://quay.io/ceph/ceph:v17: pinging container
> registry quay.io: Get "https://quay.io/v2/": dial tcp 34.231.182.47:443:
> connect: connection refused
> 2023-09-26 17:09:42,084 7f04058b4b80 ERROR ERROR: Failed command:
> /usr/bin/podman pull quay.io/ceph/ceph:v17
>
>
> would you please help.
> thanks,
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow recovery and inaccurate recovery figures since Quincy upgrade

[Sridhar Seshasayee]

To help complete the recovery, you can temporarily try disabling scrub and
deep scrub
operations by running:

ceph osd set noscrub
ceph osd set nodeep-scrub

This should help speed up the recovery process. Once the recovery is done,
you
can unset the above scrub flags and revert the mClock profile back to
'high_client_ops'.

If the above doesn't help, then there's something else that's causing the
slow recovery.
-Sridhar
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs health warn

[Ben]

Hi Venky,

thanks for help on this. Will change to multimds with subtree pinning.

For the moment, it needs to get the segments list items go by loop of
expiring -> expired -> trimmed. It is observed that each problematic mds
has a few expiring segment stuck in the road of trimming. the segment list
continually grows overtime. Any ideas for having the segment list to be
processed well normal again?

The issue has been around for weeks and haven't seen complaints from
storage client side so far.

Best wishes,
Ben

Venky Shankar  于2023年10月4日周三 13:31写道：

> Hi Ben,
>
> On Tue, Oct 3, 2023 at 8:56 PM Ben  wrote:
> >
> > Yes, I am. 8 active + 2 standby, no subtree pinning. What if I restart
> the mds with trimming issues? Trying to figure out what happens with
> restarting.
>
> We have come across instances in the past where multimds without
> subtree pinning can lead to accumulation of log segments which then
> leada to trim warnings. This happens due to the default mds balancer
> misbehaving. We have a change that's pending merge (and backport)
> which switches off the default balancer for this very reason.
>
> https://github.com/ceph/ceph/pull/52196
>
> Suggest using single active mds or multimds with subtree pinning.
>
> >
> > Venky Shankar  于2023年10月3日周二 12:39写道：
> >>
> >> Hi Ben,
> >>
> >> Are you using multimds without subtree pinning?
> >>
> >> On Tue, Oct 3, 2023 at 10:00 AM Ben  wrote:
> >> >
> >> > Dear cephers:
> >> > more log captures(see below) show the full segments list(more than
> 3 to
> >> > be trimmed stuck, growing over time). any ideas to get out of this?
> >> >
> >> > Thanks,
> >> > Ben
> >> >
> >> >
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195341004/893374309813, 180 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195341184/893386318445, 145 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195341329/893386757388, 1024 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195342353/893388361174, 1024 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195343377/893389870480, 790 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195344167/893390955408, 1024 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195345191/893392321470, 1024 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195346215/893393752928, 1024 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195347239/893395131457, 2 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195347241/893395212055, 1024 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195348265/893396582755, 1024 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195349289/893398132191, 860 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expiring segment 195350149/893399338619, 42 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195350192/893408004655, 33 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195350226/893412331017, 23 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195350249/893416563419, 20 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195350269/893420702085, 244 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195350513/893424694857, 74 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195350587/893428947395, 843 events
> >> > debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim
> already
> >> > expired segment 195351430/893432893900, 1019 events
> >> > .
> >> > . (all expired items abbreviated)
> >> > .
> >> > debug 2023-09-30T15:10:56.521+ 7fc752167700 5 mds.3.log trim
> already
> >> > expired segment 216605661/827226016068, 100 events
> >> > debug 2023-09-30T15:10:56.521+ 7fc752167700 5 mds.3.log trim
> already
> >> > expired segment 216605761/827230263164, 153 events
> >> > debug 2023-09-30T15:10:56.521+ 7fc752167700 5 mds.3.log trim
> already
> >> > expired segment 216605914/827234408294, 35 events
> >> > debug 2023-09-30T15:10:56.521+