[ceph-users] Re: After a huge amount of snaphot delete many snaptrim+snaptrim_wait pgs

[Szabo, Istvan (Agoda)]

Yes, but before I update need to have a healthy cluster, don't really want to 
update if it is not healthy to carry issue over.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Konstantin Shalygin 
Sent: Sunday, May 16, 2021 4:59 PM
To: Szabo, Istvan (Agoda) 
Cc: Ceph Users 
Subject: Re: [ceph-users] After a huge amount of snaphot delete many 
snaptrim+snaptrim_wait pgs

Hi,


On 16 May 2021, at 04:22, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:

The cluster has 3 servers, running on luminous 12.2.8.

Again, this is old and unsupported version of Ceph. Please, upgrade at least to 
12.2.13



k


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Pool has been deleted before snaptrim finished

[Szabo, Istvan (Agoda)]

Hi,

We decided to delete the pool before the snaptrim finished after 4 days waiting.
Now we have bigger issue, many osd started to flap, 2 of them cannot even 
restart due after.

Did some bluestore fsck on the not started osds and has many messages like this 
inside:

2021-05-17 18:37:07.176203 7f416d20bec0 10 stupidalloc 0x0x564e4e804f50 
init_add_free 0x482d0778000~4000
2021-05-17 18:37:07.176204 7f416d20bec0 10 freelist enumerate_next 
0x482d0784000~4000
2021-05-17 18:37:07.176204 7f416d20bec0 10 stupidalloc 0x0x564e4e804f50 
init_add_free 0x482d0784000~4000
2021-05-17 18:37:07.176205 7f416d20bec0 10 freelist enumerate_next 
0x482d078c000~c000
2021-05-17 18:37:07.176206 7f416d20bec0 10 stupidalloc 0x0x564e4e804f50 
init_add_free 0x482d078c000~c000
[root@hk-cephosd-2002 ~]# tail -f /tmp/ceph-osd-44-fsck.log
2021-05-17 18:39:16.466967 7f416d20bec0 20 bluefs _read_random read buffered 
0x2cd6e8f~ed6 of 1:0x372e070+420
2021-05-17 18:39:16.467154 7f416d20bec0 20 bluefs _read_random got 3798
2021-05-17 18:39:16.467179 7f416d20bec0 10 bluefs _read_random h 0x564e4e658500 
0x24d6e35~ee2 from file(ino 216551 size 0x43a382d mtime 2021-05-17 
13:21:19.839668 bdev 1 allocated 440 extents [1:0x35bc7c0+440])
2021-05-17 18:39:16.467186 7f416d20bec0 20 bluefs _read_random read buffered 
0x24d6e35~ee2 of 1:0x35bc7c0+440
2021-05-17 18:39:16.467409 7f416d20bec0 20 bluefs _read_random got 3810

and

uh oh, missing shared_blob

I've set back buffered_io to false back because when restart the osds always 
had to wait to fix degraded pgs.
Many of the SSDs are smashing at the moment on 100% and don't really know what 
to do to stop the process and bring back the 2 ssds :/

Some paste: https://justpaste.it/9bj3a

Some metric (each column is 1 server metric, total 3 servers):
How it is smashing the ssds: https://i.ibb.co/x3xm0Rj/ssds.png
IOWAIT Super high due to ssd utilization: https://i.ibb.co/683TR9y/iowait.png
Capacity seems coming back: https://i.ibb.co/mz4Lq2r/space.png

Thank you the help.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

What happens if we are using buffered_io and the machine restared due to some 
power failure? Everything that was in the cache will be lost or how ceph handle 
this?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Szabo, Istvan (Agoda)
Sent: Friday, May 14, 2021 3:21 PM
To: 'Konstantin Shalygin' 
Cc: ceph-users@ceph.io
Subject: RE: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn 
to true

When this stop  ? When died … :D

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Konstantin Shalygin mailto:k0...@k0ste.ru>>
Sent: Friday, May 14, 2021 3:00 PM
To: Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn 
to true


On 14 May 2021, at 10:50, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:

Is it also normal if this buffered_ioturned on, it eats all the memory on the 
system? Hmmm.

This is what actually do this option - eat all free memory as cached for bluefs 
speedups



k


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] After a huge amount of snaphot delete many snaptrim+snaptrim_wait pgs

[Szabo, Istvan (Agoda)]

Hi,

The user deleted 20-30 snapshots and clones from the cluster and it seems like 
slows down the whole system.

I’ve set the snaptrim parameters to the lowest as possible, set bufferred_io to 
true so at least have some speed for the user, but I can see the objects 
removal from the cluster is still happening, the beginning was 45 millioms, now 
19millions but what I don’t understand many osds getting more full :( ?

And the snaptrim is super slow, have 195 snaptrim wait and 36 snaptrim but in 
every 5 hours only 1 done :/?

What can I do? One of the osd has been 62%, now it is 75% in 2 days and still 
growing. Set back the snap options or?

The cluster has 3 servers, running on luminous 12.2.8.

Some paste: https://jpst.it/2vw4H

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

Yup, I just saw, should have 3GB :/ I will wait until the system goes back to 
normal and will increase.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Irek Fasikhov 
Sent: Friday, May 14, 2021 6:28 PM
To: Szabo, Istvan (Agoda) 
Cc: Konstantin Shalygin ; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io 
turn to true

Hi.

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ceph_object_gateway_for_production/deploying_a_cluster#reserving_free_memory_for_osds

пт, 14 мая 2021 г. в 14:21, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>:
Howmuch is yours? Mine is vm.min_free_kbytes = 90112.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: 
istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com><mailto:istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>>
---

From: Konstantin Shalygin mailto:k0...@k0ste.ru>>
Sent: Friday, May 14, 2021 6:07 PM
To: Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn 
to true

I suggest to look into vm.min_free_kbytes kernel option, and increase it twice


k


On 14 May 2021, at 13:45, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com><mailto:istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>>>
 wrote:

Is there anything that should be set just to be sure oom kill not happen? Or 
nothing?



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to 
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

Howmuch is yours? Mine is vm.min_free_kbytes = 90112.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Konstantin Shalygin 
Sent: Friday, May 14, 2021 6:07 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn 
to true

I suggest to look into vm.min_free_kbytes kernel option, and increase it twice


k


On 14 May 2021, at 13:45, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:

Is there anything that should be set just to be sure oom kill not happen? Or 
nothing?



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

Is there anything that should be set just to be sure oom kill not happen? Or 
nothing?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Konstantin Shalygin 
Sent: Friday, May 14, 2021 5:32 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn 
to true

It's enough, should be true now...


k


On 14 May 2021, at 12:51, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:

Did I do something wrong?
I set in the global config the bluefs option, and restarted ceph.target on the 
osd node :/ ?

Doe this need some special thing to apply?



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

Ok, seems like it doesn’t go below 600MB out of the 256GB, let’s wait until the 
pg_degradation healed.

Did I do something wrong?
I set in the global config the bluefs option, and restarted ceph.target on the 
osd node :/ ?

Doe this need some special thing to apply?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Konstantin Shalygin 
Sent: Friday, May 14, 2021 3:26 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn 
to true

Nope, kernel reserves enough memory to free on pressure, for example 36OSD 
0.5TiB RAM host:

  totalusedfree  shared  buff/cache   available
Mem:   502G168G2.9G 18M331G472G
Swap:  952M248M704M


k


On 14 May 2021, at 11:20, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:

When this stop  ? When died … :D



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

When this stop  ? When died … :D

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Konstantin Shalygin 
Sent: Friday, May 14, 2021 3:00 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn 
to true


On 14 May 2021, at 10:50, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:

Is it also normal if this buffered_ioturned on, it eats all the memory on the 
system? Hmmm.

This is what actually do this option - eat all free memory as cached for bluefs 
speedups



k


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

Is it also normal if this buffered_ioturned on, it eats all the memory on the 
system? Hmmm.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Konstantin Shalygin 
Sent: Friday, May 14, 2021 2:12 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn 
to true

I recommend to upgrade at least to 12.2.13, for luminous even .12 and .13 is 
significant difference in code.



k

On 14 May 2021, at 09:22, Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:

It is quite an older cluster, luminous 12.2.8.



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

Hi,

It is quite an older cluster, luminous 12.2.8.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Konstantin Shalygin  
Sent: Friday, May 14, 2021 1:12 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to 
true

Hi,

This is not a normal, It's something different I think, like a crush changes on 
restart. This option will be enabled by default again in Nautilus next, so you 
can use it now with 14.2.19-20


k

Sent from my iPhone

> On 14 May 2021, at 08:21, Szabo, Istvan (Agoda)  
> wrote:
> 
> Hi,
> 
> I had issue with the snaptrim after a hug amount of deleted data, it slows 
> down the team operations due to the snaptrim and snaptrim_wait pgs.
> 
> I've changed couple of things:
> 
> debug_ms = 0/0 #default 0/5
> osd_snap_trim_priority = 1 # default 5 
> osd_pg_max_concurrent_snap_trims = 1 # default 2
> 
> But didn't help.
> 
> I've found this thread about buffered io and seems like it helped to them:
> https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-
> night-slowing-down-vms.71573/
> 
> I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it 
> caused basically the complete node's pg-s are degraded. Is it normal? I hope 
> it will not rebalance the complete node because I don't have space for that. 
> I changed it back but still slowly decreasing, so not sure this settings is 
> correct or not or this behavior is good or not?
> 
> 2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 
> 3353/91976715 objects misplaced (0.004%) (OBJECT_MISPLACED)
> 2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: 
> Degraded data redundancy: 33078466/91976715 objects degraded 
> (35.964%), 254 pgs degraded, 253 pgs undersized (PG_DEGRADED)
> 
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
> ---
> 
> 
> 
> This message is confidential and is for the sole use of the intended 
> recipient(s). It may also be privileged or otherwise protected by copyright 
> or other legal rules. If you have received it by mistake please let us know 
> by reply email and delete it from your system. It is prohibited to copy this 
> message or disclose its content to anyone. Any confidentiality or privilege 
> is not waived or lost by any mistaken delivery or unauthorized disclosure of 
> the message. All messages sent to and from Agoda may be monitored to ensure 
> compliance with company policies, to protect the company's interests and to 
> remove potential malware. Electronic messages may be intercepted, amended, 
> lost or deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
> email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] bluefs_buffered_io turn to true

[Szabo, Istvan (Agoda)]

Hi,

I had issue with the snaptrim after a hug amount of deleted data, it slows down 
the team operations due to the snaptrim and snaptrim_wait pgs.

I've changed couple of things:

debug_ms = 0/0 #default 0/5
osd_snap_trim_priority = 1 # default 5
osd_pg_max_concurrent_snap_trims = 1 # default 2

But didn't help.

I've found this thread about buffered io and seems like it helped to them:
https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-night-slowing-down-vms.71573/

I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused 
basically the complete node's pg-s are degraded. Is it normal? I hope it will 
not rebalance the complete node because I don't have space for that. I changed 
it back but still slowly decreasing, so not sure this settings is correct or 
not or this behavior is good or not?

2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 3353/91976715 
objects misplaced (0.004%) (OBJECT_MISPLACED)
2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: Degraded data 
redundancy: 33078466/91976715 objects degraded (35.964%), 254 pgs degraded, 253 
pgs undersized (PG_DEGRADED)

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Which EC-code for 6 servers?

[Szabo, Istvan (Agoda)]

Ok, will stay with 2:2 or 3:2 so once 1 host goes down it can go to the other 
active host.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

On 2021. May 11., at 14:44, Frank Schilder  wrote:

For performance reasons stay with powers of 2 for k. Any of 2+2 or 4+2 will 
work with your set-up and tolerate one (!) host failure with continued RW 
access and two host failures with RO (!) access.

To tolerate 2 host failures with RW access, you need m=3, which is probably a 
bit much with 6 hosts.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Szabo, Istvan (Agoda) 
Sent: 10 May 2021 10:35:34
To: ceph-users
Subject: [ceph-users] Which EC-code for 6 servers?

Hi,

Thinking to have 2:2 so I can tolerate 2 hosts loss, but if I just want to 
tolerate 1 host loss, which one better, 3:2  or 4:1?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data

[Szabo, Istvan (Agoda)]

Ok, will be challenging with an 800 millions object bucket  But I might give a 
try.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: 特木勒 
Sent: Monday, May 10, 2021 6:53 PM
To: Szabo, Istvan (Agoda) 
Cc: Jean-Sebastien Landry ; 
ceph-users@ceph.io; Amit Ghadge 
Subject: Re: [ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does 
not sync olds data

Hi Istvan:

Thanks for your help.

After we rewrite all the objects that in buckets, the sync seems to work again.

We are using this command to rewrite all the objects in specific bucket:
`radosgw-admin bucket rewrite —bucket=BUCKET_NAME --min-rewrite-size 0`

You can try to run this on 1 bucket and see if it could help you fix the 
problem.

Thank you~

Szabo, Istvan (Agoda) mailto:istvan.sz...@agoda.com>> 
于2021年5月10日周一 下午12:16写道：
So how is your multisite things going at the moment? Seems like with this 
rewrite you’ve moved further than me  Is it working properly now? If yes, what 
is the steps to make it work? Where is the magic  ?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: 特木勒 mailto:twl...@gmail.com>>
Sent: Thursday, May 6, 2021 11:27 AM
To: Jean-Sebastien Landry 
mailto:jean-sebastien.landr...@ulaval.ca>>
Cc: Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>; 
ceph-users@ceph.io<mailto:ceph-users@ceph.io>; Amit Ghadge 
mailto:amitg@gmail.com>>
Subject: Re: [ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does 
not sync olds data

Hi Jean:

Thanks for your info.

Unfortunately I check the secondary cluster and non-objects had been synced. 
The only way I have is to force rewrite objects for whole buckets.

I have tried to set up multiple site between Nautilus and octopus. It works 
pretty well. But after I upgrade primary cluster to octopus, we have this 
issue. :(

Here is the issue: https://tracker.ceph.com/issues/49542#change-193975

Thanks

Jean-Sebastien Landry 
mailto:jean-sebastien.landr...@ulaval.ca>> 
于2021年4月27日周二 下午7:52写道：
Hi, I hit the same errors when doing multisite sync between luminous and
octopus, but what I founded is that my sync errors was mainly on old
multipart and shadow objects, at the "rados level" if I might say.
(leftovers from luminous bugs)

So check at the "user level", using s3cmd/awscli and the objects md5,
you will probably find that your pretty much in sync. Hopefully.

Cheers!

On 4/25/21 11:29 PM, 特木勒 wrote:
> [Externe UL*]
>
> Another problem I notice for a new bucket, the first object in the bucket
> will not be sync. the sync will start with the second object. I tried to
> fix the index on the bucket and manually rerun bucket sync, but the first
> object still does not sync with secondary cluster.
>
> Do you have any ideas for this issue?
>
> Thanks
>
> 特木勒 mailto:twl...@gmail.com>> 于2021年4月26日周一 上午11:16写道：
>
>> Hi Istvan:
>>
>> Thanks Amit's suggestion.
>>
>> I followed his suggestion to fix bucket index and re-do sync on buckets,
>> but it still did not work for me.
>>
>> Then I tried to use bucket rewrite command to rewrite all the objects in
>> buckets and it works for me. I think the reason is there's something wrong
>> with bucket index and rewrite has rebuilt the index.
>>
>> Here's the command I use:
>> `sudo radosgw-admin bucket rewrite -b BUCKET-NAME --min-rewrite-size 0`
>>
>> Maybe you can try this to fix the sync issues.
>>
>> @Amit Ghadge mailto:amitg@gmail.com>> Thanks for 
>> your suggestions. Without
>> your suggestions, I will not notice something wrong with index part.
>>
>> Thanks :)
>>
>> Szabo, Istvan (Agoda) 
>> mailto:istvan.sz...@agoda.com>> 于2021年4月26日周一 
>> 上午9:57写道：
>>
>>> Hi,
>>>
>>>
>>>
>>> No, doesn’t work, now we will write our own sync app for ceph, I gave up.
>>>
>>>
>>>
>>> Istvan Szabo
>>> Senior Infrastructure Engineer
>>> ---
>>> Agoda Services Co., Ltd.
>>> e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
>>> ---
>>>
>>>
>>>
>>> *From:* 特木勒 mailto:twl...@gmail.com>>
>>> *Sent:* Friday, April 23, 2021 7:50 PM
>>> *To:* Szabo, Istvan (Agoda) 
>>> mailto:istvan.sz...@agoda.com>&

[ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data

[Szabo, Istvan (Agoda)]

So how is your multisite things going at the moment? Seems like with this 
rewrite you’ve moved further than me  Is it working properly now? If yes, what 
is the steps to make it work? Where is the magic  ?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: 特木勒 
Sent: Thursday, May 6, 2021 11:27 AM
To: Jean-Sebastien Landry 
Cc: Szabo, Istvan (Agoda) ; ceph-users@ceph.io; Amit 
Ghadge 
Subject: Re: [ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does 
not sync olds data

Hi Jean:

Thanks for your info.

Unfortunately I check the secondary cluster and non-objects had been synced. 
The only way I have is to force rewrite objects for whole buckets.

I have tried to set up multiple site between Nautilus and octopus. It works 
pretty well. But after I upgrade primary cluster to octopus, we have this 
issue. :(

Here is the issue: https://tracker.ceph.com/issues/49542#change-193975

Thanks

Jean-Sebastien Landry 
mailto:jean-sebastien.landr...@ulaval.ca>> 
于2021年4月27日周二 下午7:52写道：
Hi, I hit the same errors when doing multisite sync between luminous and
octopus, but what I founded is that my sync errors was mainly on old
multipart and shadow objects, at the "rados level" if I might say.
(leftovers from luminous bugs)

So check at the "user level", using s3cmd/awscli and the objects md5,
you will probably find that your pretty much in sync. Hopefully.

Cheers!

On 4/25/21 11:29 PM, 特木勒 wrote:
> [Externe UL*]
>
> Another problem I notice for a new bucket, the first object in the bucket
> will not be sync. the sync will start with the second object. I tried to
> fix the index on the bucket and manually rerun bucket sync, but the first
> object still does not sync with secondary cluster.
>
> Do you have any ideas for this issue?
>
> Thanks
>
> 特木勒 mailto:twl...@gmail.com>> 于2021年4月26日周一 上午11:16写道：
>
>> Hi Istvan:
>>
>> Thanks Amit's suggestion.
>>
>> I followed his suggestion to fix bucket index and re-do sync on buckets,
>> but it still did not work for me.
>>
>> Then I tried to use bucket rewrite command to rewrite all the objects in
>> buckets and it works for me. I think the reason is there's something wrong
>> with bucket index and rewrite has rebuilt the index.
>>
>> Here's the command I use:
>> `sudo radosgw-admin bucket rewrite -b BUCKET-NAME --min-rewrite-size 0`
>>
>> Maybe you can try this to fix the sync issues.
>>
>> @Amit Ghadge mailto:amitg@gmail.com>> Thanks for 
>> your suggestions. Without
>> your suggestions, I will not notice something wrong with index part.
>>
>> Thanks :)
>>
>> Szabo, Istvan (Agoda) 
>> mailto:istvan.sz...@agoda.com>> 于2021年4月26日周一 
>> 上午9:57写道：
>>
>>> Hi,
>>>
>>>
>>>
>>> No, doesn’t work, now we will write our own sync app for ceph, I gave up.
>>>
>>>
>>>
>>> Istvan Szabo
>>> Senior Infrastructure Engineer
>>> ---
>>> Agoda Services Co., Ltd.
>>> e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
>>> ---
>>>
>>>
>>>
>>> *From:* 特木勒 mailto:twl...@gmail.com>>
>>> *Sent:* Friday, April 23, 2021 7:50 PM
>>> *To:* Szabo, Istvan (Agoda) 
>>> mailto:istvan.sz...@agoda.com>>
>>> *Cc:* ceph-users@ceph.io<mailto:ceph-users@ceph.io>
>>> *Subject:* Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site
>>> does not sync olds data
>>>
>>>
>>>
>>> Hi Istvan:
>>>
>>>
>>>
>>> We just upgraded whole cluster to 15.2.10 and the multiple site still
>>> cannot sync whole objects to secondary cluster. 
>>>
>>>
>>>
>>> Do you have any suggestions on this? And I open another issues in ceph
>>> tracker site:
>>>
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftracker.ceph.com%2Fissues%2F50474data=04%7C01%7Cjean-sebastien.landry%40dti.ulaval.ca%7C7758dcb00aa0481e098408d90863da13%7C56778bd56a3f4bd3a26593163e4d5bfe%7C1%7C0%7C637550047218082332%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=flvOwiaSDmDRQsbhzZsw5Q16hTmzUj9RXhcBrL6wsjo%3Dreserved=0
>>>
>>>
>>>
>>> Hope someone could go to check this issue.
>>>
>>>
>>>
>>> Thanks
>>

[ceph-users] Which EC-code for 6 servers?

[Szabo, Istvan (Agoda)]

Hi,

Thinking to have 2:2 so I can tolerate 2 hosts loss, but if I just want to 
tolerate 1 host loss, which one better, 3:2  or 4:1?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Building ceph clusters with 8TB SSD drives?

[Szabo, Istvan (Agoda)]

We are using in our objectstore 15TB SSDs.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Matt Larson 
Sent: Saturday, May 8, 2021 3:11 AM
To: ceph-users 
Subject: [Suspicious newsletter] [ceph-users] Building ceph clusters with 8TB 
SSD drives?

Is anyone trying Ceph clusters containing larger (4-8TB) SSD drives?

8TB SSDs are described here (
https://www.anandtech.com/show/16136/qlc-8tb-ssd-review-samsung-870-qvo-sabrent-rocket-q
) and make use QLC NAND flash memory to reach the costs and capacity.
Currently, the 8TB Samsung 870 SSD is $800/ea at some online retail stores.

SATA form-factor SSDs can reach read/write rates of 560/520 MB/s, while not as 
great as nVME drives is still a multiple faster than 7200 RPM drives.
SSDs now appear to have much lower failure rates than HDs in 2021 ( 
https://www.techspot.com/news/89590-backblaze-latest-storage-reliability-figures-add-ssd-boot.html
).

Are there any major caveats to considering working with larger SSDs for data 
pools?

Thanks,
  Matt

--
Matt Larson, PhD
Madison, WI  53705 U.S.A.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint

[Szabo, Istvan (Agoda)]

Hi,

What we have found seems like it is a blocking issue when I terminate https on 
a loadbalancer and between the loadbalancer and rgw http is the mode. So seems 
liket he ssl termination has to be done on the rgw and can't be done on the 
loadbalancer? Or how we can workaround it any idea?

Here are the debug logs:

With loadbalancer https endpoint: https://justpaste.it/5d93w
Directly with rgw ip without loadbalancer: https://justpaste.it/9rn28

Both case the issue is like this: "endpoint validation error: sending password 
over insecure transport"

To be honest I want to do the ssl on loadbalancer don't want to do on rgw. 
Maybe you can suggest something.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Yuval Lifshitz 
Sent: Tuesday, April 27, 2021 11:49 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io; Raveendran, Vigneshwaran (Agoda) 

Subject: [Suspicious newsletter] [ceph-users] Re: Getting `InvalidInput` when 
trying to create a notification topic with Kafka endpoint

On Tue, Apr 27, 2021 at 1:59 PM Szabo, Istvan (Agoda) < istvan.sz...@agoda.com> 
wrote:

> Hello,
>
> Thank you very much to pickup the question and sorry for the late response.
>
> Yes, we are sending in cleartext also using HTTPS, but how it should
> be send if not like this?
>
>
if you send the user/password using HTTPS connection between the client and the 
RGW there should be no error. could you please provide the RGW debug log, to 
see why "invalid argument" was replied?


> Also connected to this issue a bit, when we subscribe a bucket to a
> topic with non-ACL kafka topic, any operations (PUT or DELETE) is
> simply blocking and not returning. Not even any error response.
>
> this would be the case when the kafka broker is down (or the
> parameters
you provided to the topic were incorrect). a workaround for this issue is to 
mark the endpoint with "kafka-ack-level=none", this will not block for the 
reply, but note that if the broker is down or misconfigured, the notification 
will be lost.
a better option (if you are using "pacific" and up) is to mark the topic with 
the "persistent" flag. this would mean that even if the broker is down or 
misconfigured, the notification will be retired until successful, and, in 
addition, will not block the request.



> $ s3cmd -c ~/.s3cfg put --add-header x-amz-meta-foo:bar3
> certificate.pdf s3://vig-test
> WARNING: certificate.pdf: Owner groupname not known. Storing
> GID=1354917867 instead.
> WARNING: Module python-magic is not available. Guessing MIME types
> based on file extensions.
> upload: 'certificate.pdf' -> 's3://vig-test/certificate.pdf'  [1 of 1]
>  65536 of 9122471% in0s   291.17 KB/s
>
>
>
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
>
>
> *From:* Yuval Lifshitz 
> *Sent:* Wednesday, April 21, 2021 10:34 PM
> *To:* Szabo, Istvan (Agoda) 
> *Cc:* ceph-users@ceph.io
> *Subject:* Re: [ceph-users] Getting `InvalidInput` when trying to
> create a notification topic with Kafka endpoint
>
>
>
> Hi Istvan,
>
> Can you please share the relevant part for the radosgw log, indicating
> which input was invalid?
>
> The only way I managed to reproduce that error is by sending the
> request to a non-HTTPS radosgw (which does not seem to be your case).
> In such a case it replies with "InvalidInput" because we are trying to
> send user/password in cleartext.
>
> I used curl, similarly to what you did against a vstart cluster based
> off of master: https://paste.sh/SQ_8IrB5#BxBYbh1kTh15n7OKvjB5wEOM
>
>
>
> Yuval
>
>
>
> On Wed, Apr 21, 2021 at 11:23 AM Szabo, Istvan (Agoda) <
> istvan.sz...@agoda.com> wrote:
>
> Hi Ceph Users,
> Here is the latest request I tried but still not working
>
> curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +' -H 'Authorization:
> AWS :' -L -H 'content-type:
> application/x-www-form-urlencoded' -k -X POST https://servername -d
> Action=CreateTopic=test-ceph-event-replication
> .key=push-endpoint=kafka://: rd>@servername2:9093=use-ssl
> .value=true
>
> And the response I get is still Invalid Input  encoding="UTF-8"?>InvalidInputtx000
> 0007993081-00607efbdd-1c7e96b-hkg1c7e96b-hkg-d
> ata
> Can someone please help with this?
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> 

[ceph-users] Re: Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint

[Szabo, Istvan (Agoda)]

Hello,

Thank you very much to pickup the question and sorry for the late response.

Yes, we are sending in cleartext also using HTTPS, but how it should be send if 
not like this?

Also connected to this issue a bit, when we subscribe a bucket to a topic with 
non-ACL kafka topic, any operations (PUT or DELETE) is simply blocking and not 
returning. Not even any error response.

$ s3cmd -c ~/.s3cfg put --add-header x-amz-meta-foo:bar3 certificate.pdf 
s3://vig-test
WARNING: certificate.pdf: Owner groupname not known. Storing GID=1354917867 
instead.
WARNING: Module python-magic is not available. Guessing MIME types based on 
file extensions.
upload: 'certificate.pdf' -> 's3://vig-test/certificate.pdf'  [1 of 1]
 65536 of 9122471% in0s   291.17 KB/s


Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: Yuval Lifshitz 
Sent: Wednesday, April 21, 2021 10:34 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [ceph-users] Getting `InvalidInput` when trying to create a 
notification topic with Kafka endpoint

Hi Istvan,
Can you please share the relevant part for the radosgw log, indicating which 
input was invalid?
The only way I managed to reproduce that error is by sending the request to a 
non-HTTPS radosgw (which does not seem to be your case). In such a case it 
replies with "InvalidInput" because we are trying to send user/password in 
cleartext.
I used curl, similarly to what you did against a vstart cluster based off of 
master: https://paste.sh/SQ_8IrB5#BxBYbh1kTh15n7OKvjB5wEOM

Yuval

On Wed, Apr 21, 2021 at 11:23 AM Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>> wrote:
Hi Ceph Users,
Here is the latest request I tried but still not working

curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +' -H 'Authorization: AWS 
:' -L -H 'content-type: application/x-www-form-urlencoded' 
-k -X POST https://servername -d 
Action=CreateTopic=test-ceph-event-replication=push-endpoint=kafka://:@servername2:9093=use-ssl=true

And the response I get is still Invalid Input
InvalidInputtx007993081-00607efbdd-1c7e96b-hkg1c7e96b-hkg-data
Can someone please help with this?
Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: 
istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com><mailto:istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>>
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to 
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data

[Szabo, Istvan (Agoda)]

Hi,

No, doesn’t work, now we will write our own sync app for ceph, I gave up.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: 特木勒 
Sent: Friday, April 23, 2021 7:50 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site does not 
sync olds data

Hi Istvan:

We just upgraded whole cluster to 15.2.10 and the multiple site still cannot 
sync whole objects to secondary cluster. 

Do you have any suggestions on this? And I open another issues in ceph tracker 
site:
https://tracker.ceph.com/issues/50474

Hope someone could go to check this issue.

Thanks

特木勒 mailto:twl...@gmail.com>>于2021年3月22日 周一下午9:08写道：
Thank you~

I will try to upgrade cluster too. Seem like this is the only way for now. 

I will let you know once I complete testing. :)

Have a good day

Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>于2021年3月22日 周一下午3:38写道：
Yeah, doesn't work. Last week they fixed my problem ticket which caused the 
crashes, and due to the crashes stopped the replication I'll give a try this 
week again after the update if the daemon doesn't crash, maybe it will work, 
because if crash hasn't happened, the data was synced. Fingers crossed ;) Don't 
give up 

From: 特木勒 mailto:twl...@gmail.com>>
Sent: Monday, March 22, 2021 1:38 PM
To: Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> 
mailto:ceph-users@ceph.io>>

Subject: Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site does not 
sync olds data

Hi Istvan:

Do you have any update on directional sync?

I am trying to upgrade cluster to 15.2.10 to see if the problem is solved. :(

Thanks

Szabo, Istvan (Agoda) mailto:istvan.sz...@agoda.com>> 
于2021年3月1日周一 上午10:01写道：

So-so. I had some interruption so it failed on one site, but the other is kind 
of working. This is the first time when I saw data caught up in the 
radosgw-admin data sync status on 1 side.

Today will finish the other problematic site, I’ll let you know the result is 
it working or not.



Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---



From: 特木勒 mailto:twl...@gmail.com>>
Sent: Sunday, February 28, 2021 1:34 PM
To: Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site does not 
sync olds data



Email received from outside the company. If in doubt don't click links nor open 
attachments!



Hi Istvan:



Thanks for your reply.



Does directional sync solve the problem? I tried to run `radosgw-admin sync 
init`, bit it still did not work. :(



Thanks



Szabo, Istvan (Agoda) mailto:istvan.sz...@agoda.com>> 
于2021年2月26日周五 上午7:47写道：

Same for me, 15.2.8 also.
I’m trying directional sync now, looks like symmetrical has issue.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: 
istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com><mailto:istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>>
---

On 2021. Feb 26., at 1:03, 特木勒 mailto:twl...@gmail.com>> 
wrote:

Email received from outside the company. If in doubt don't click links nor 
open attachments!


Hi all:

ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8)

I have a strange question, I just create a multiple site for Ceph cluster.
But I notice the old data of source cluster is not synced. Only new data
will be synced into second zone cluster.

Is there anything I need to do to enable full sync for bucket or this is a
bug?

Thanks
___
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to 
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitore

[ceph-users] PG can't deep and simple scrub after unfound data delete

[Szabo, Istvan (Agoda)]

Hi,

I have a pg where it has been run the following command:
ceph pg 44.1aa mark_unfound_lost delete

After the cluster never report the unknown pgs which was actually the goal to 
ran this.

However this pg is now inconsistent and can't be deepscrubbed.

ceph health detail
HEALTH_ERR 214275 scrub errors; Possible data damage: 1 pg inconsistent; 1 pgs 
not deep-scrubbed in time
[ERR] OSD_SCRUB_ERRORS: 214275 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 44.1aa is active+clean+inconsistent, acting [59,128,127,43]
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 44.1aa not deep-scrubbed since 2021-01-14T05:50:23.852626+0100

ceph pg dump pgs_brief|grep 'ACTING_PRIMARY\|44.1aa'
dumped pgs_brief
PG_STAT  STATE   UPUP_PRIMARY  
ACTINGACTING_PRIMARY
44.1aaactive+clean+inconsistent   [59,128,127,43]  59   
[59,128,127,43]  59

Any idea what to do with it?



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: cephadm: how to create more than 1 rgw per host

[Szabo, Istvan (Agoda)]

I don't use cephadm, I'm using the not dockerised deployment, but how you 
create another one on the same host with this config?

This is my RGW section:

[client.rgw.xyz-cephmon-2s01.rgw0]
host = xyz-cephmon-2s01
keyring = /var/lib/ceph/radosgw/ceph-rgw.xyz-cephmon-2s01.rgw0/keyring
log file = /var/log/ceph/ceph-rgw-xyz-cephmon-2s01.rgw0.log
rgw frontends = beast endpoint=123.456.199.1:8080
rgw thread pool size = 512
rgw_zone=FRT

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: by morphin 
Sent: Thursday, April 22, 2021 6:30 PM
To: i...@z1storage.com
Cc: Sebastian Wagner ; Ceph Users 
Subject: [Suspicious newsletter] [ceph-users] Re: cephadm: how to create more 
than 1 rgw per host

Hello.

Its easy. In ceph.conf copy the rgw fields and change 3 things.
1- name
2- log path name
3- client port.


After that feel free to start rgw service with systemctl. Check service status 
and Tail the rgw log file. Try to read or write and check the logs.
If everything works as expected then you are ready to add the new service to 
loadbalancer if you have one.



22 Nis 2021 Per 14:00 tarihinde i...@z1storage.com  şunu 
yazdı:

> Does anyone know how to create more than 1 rgw per host? Surely it's
> not a rare configuration.
>
> On 2021/04/19 17:09, i...@z1storage.com wrote:
> >
> > Hi Sebastian,
> >
> > Thank you. Is there a way to create more than 1 rgw per host until
> > this new feature is released?
> >
> > On 2021/04/19 11:39, Sebastian Wagner wrote:
> >> Hi Ivan,
> >>
> >> this is a feature that is not yet released in Pacific. It seems the
> >> documentation is a bit ahead of time right now.
> >>
> >> Sebastian
> >>
> >> On Fri, Apr 16, 2021 at 10:58 PM i...@z1storage.com
> >>   >> > wrote:
> >>
> >> Hello,
> >>
> >> According to the documentation, there's count-per-host key to 'ceph
> >> orch', but it does not work for me:
> >>
> >> :~# ceph orch apply rgw z1 sa-1 --placement='label:rgw
> >> count-per-host:2'
> >> --port=8000 --dry-run
> >> Error EINVAL: Host and label are mutually exclusive
> >>
> >> Why it says anything about Host if I don't specify any hosts,
> >> just labels?
> >>
> >> ~# ceph orch host ls
> >> HOST  ADDR  LABELS   STATUS
> >> s101  s101  mon rgw
> >> s102  s102  mgr mon rgw
> >> s103  s103  mon rgw
> >> s104  s104  mgr mon rgw
> >> s105  s105  mgr mon rgw
> >> s106  s106  mon rgw
> >> s107  s107  mon rgw
> >>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> 
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >> 
> >>
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Metrics for object sizes

[Szabo, Istvan (Agoda)]

Objects inside RGW buckets like in couch base software they have their own 
metrics and has this information.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

On 2021. Apr 22., at 14:00, Wido den Hollander  wrote:



On 21/04/2021 11:46, Szabo, Istvan (Agoda) wrote:
Hi,
Is there any clusterwise metric regarding object sizes?
I'd like to collect some information about the users what is the object sizes 
in their buckets.

Are you talking about RADOS objects or objects inside RGW buckets?

I think you are talking about RGW, but I just wanted to check.

Afaik this information is not available for both RADOS and RGW.

Do keep in mind that small objects are much more expensive then large objects. 
The metadata overhead becomes costly and can even become problematic if you 
have millions of tiny (few kb) objects.

Wido


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: cephadm: how to create more than 1 rgw per host

[Szabo, Istvan (Agoda)]

Have you ever tried this? Did it work for you?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2021. Apr 22., at 18:30, by morphin  wrote:

Hello.

Its easy. In ceph.conf copy the rgw fields and change 3 things.
1- name
2- log path name
3- client port.


After that feel free to start rgw service with systemctl. Check service
status and Tail the rgw log file. Try to read or write and check the logs.
If everything works as expected then you are ready to add the new service
to loadbalancer if you have one.



22 Nis 2021 Per 14:00 tarihinde i...@z1storage.com 
şunu yazdı:

Does anyone know how to create more than 1 rgw per host? Surely it's not
a rare configuration.

On 2021/04/19 17:09, i...@z1storage.com wrote:

Hi Sebastian,

Thank you. Is there a way to create more than 1 rgw per host until
this new feature is released?

On 2021/04/19 11:39, Sebastian Wagner wrote:
Hi Ivan,

this is a feature that is not yet released in Pacific. It seems the
documentation is a bit ahead of time right now.

Sebastian

On Fri, Apr 16, 2021 at 10:58 PM i...@z1storage.com
 mailto:i...@z1storage.com>> wrote:

   Hello,

   According to the documentation, there's count-per-host key to 'ceph
   orch', but it does not work for me:

   :~# ceph orch apply rgw z1 sa-1 --placement='label:rgw
   count-per-host:2'
   --port=8000 --dry-run
   Error EINVAL: Host and label are mutually exclusive

   Why it says anything about Host if I don't specify any hosts,
   just labels?

   ~# ceph orch host ls
   HOST  ADDR  LABELS   STATUS
   s101  s101  mon rgw
   s102  s102  mgr mon rgw
   s103  s103  mon rgw
   s104  s104  mgr mon rgw
   s105  s105  mgr mon rgw
   s106  s106  mon rgw
   s107  s107  mon rgw

   ___
   ceph-users mailing list -- ceph-users@ceph.io
   
   To unsubscribe send an email to ceph-users-le...@ceph.io
   

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Metrics for object sizes

[Szabo, Istvan (Agoda)]

Hi,

Is there any clusterwise metric regarding object sizes?

I'd like to collect some information about the users what is the object sizes 
in their buckets.




This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint

[Szabo, Istvan (Agoda)]

Hi Ceph Users,
Here is the latest request I tried but still not working

curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +' -H 'Authorization: AWS 
:' -L -H 'content-type: application/x-www-form-urlencoded' 
-k -X POST https://servername -d 
Action=CreateTopic=test-ceph-event-replication=push-endpoint=kafka://:@servername2:9093=use-ssl=true

And the response I get is still Invalid Input
InvalidInputtx007993081-00607efbdd-1c7e96b-hkg1c7e96b-hkg-data
Can someone please help with this?
Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] cleanup multipart in radosgw

[Szabo, Istvan (Agoda)]

Hi,

You have 2 ways:

First is using s3vrowser app and in the menu select the multipart uploads and 
clean it up.
The other is like this:

Set lifecycle policy
On the client:
vim lifecyclepolicy

http://s3.amazonaws.com/doc/2006-03-01/;>

Incomplete Multipart 
Uploads

Enabled


1





/bin/s3cmd setlifecycle lifecyclepolicy  s3://bucketname
On mon node process manually
radosgw-admin lc list
radosgw-admin lc process

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Boris Behrens 
Sent: Monday, April 19, 2021 4:10 PM
To: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] cleanup multipart in radosgw

Hi,
is there a way to remove multipart uploads that are older than X days?

It doesn't need to be build into ceph or is automated to the end. Just 
something I don't need to build on my own.

I currently try to debug a problem where ceph reports a lot more used space 
than it actually requires ( 
https://www.mail-archive.com/ceph-users@ceph.io/msg09810.html).

I came across a lot of old _multipart_ files in some buckets and now I want to 
clean them up.
I don't know if this will fix my problem but I would love to rule that out.

radosgw-admin bucket check --bucket=bucket --check-objects --fix does not work 
because it is a shareded bucket.

I have also some buckets that look like this, and contain 100% _multipart_ 
files which are >2 years old:
"buckets": [
{
"bucket": "ncprod",
"tenant": "",
"num_objects": -482,
"num_shards": 0,
"objects_per_shard": -482,
"fill_status": "OVER 180143985094819%"
}
]

--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im 
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: what-does-nosuchkey-error-mean-while-subscribing-for-notification-in-ceph

[Szabo, Istvan (Agoda)]

This one:

http://s3.amazonaws.com/doc/2006-03-01/;>

id1
arn:aws:sns:data::testcephevent





Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

On 2021. Apr 16., at 14:58, David Caro  wrote:


What does notif.xml have in it?

Looking at the docs you linked, I say that it does not find the `S3Key` from 
that xml for whatever reason.

On 04/16 06:54, Szabo, Istvan (Agoda) wrote:
Hi,


I am trying to follow this url 
https://docs.ceph.com/en/latest/radosgw/s3/bucketops/#create-notification

to create a publisher for my bucket into a topic.

My curl:

curl -v -H 'Date: Fri, 16 Apr 2021 05:21:14 +' -H 'Authorization: AWS 
accessid:secretkey' -L -H 'content-type: text/xml' -H 'Content-MD5: 
pBRX39Oo7aAUYbilIYMoAw==' -T notif.xml http://ceph:8080/vig-test?notification

and it returns me this error





 NoSuchKey

 vig-test

 tx0016ac570-0060791ecb-1c7e96b-hkg

 1c7e96b-hkg-data




Does anybody know what does this error mean in Ceph? How can I proceed?


Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE  1171 4071 C7E1 D262 69C3

"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] what-does-nosuchkey-error-mean-while-subscribing-for-notification-in-ceph

[Szabo, Istvan (Agoda)]

Hi,


I am trying to follow this url 
https://docs.ceph.com/en/latest/radosgw/s3/bucketops/#create-notification

to create a publisher for my bucket into a topic.

My curl:

curl -v -H 'Date: Fri, 16 Apr 2021 05:21:14 +' -H 'Authorization: AWS 
accessid:secretkey' -L -H 'content-type: text/xml' -H 'Content-MD5: 
pBRX39Oo7aAUYbilIYMoAw==' -T notif.xml http://ceph:8080/vig-test?notification

and it returns me this error





  NoSuchKey

  vig-test

  tx0016ac570-0060791ecb-1c7e96b-hkg

  1c7e96b-hkg-data




Does anybody know what does this error mean in Ceph? How can I proceed?


Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?

[Szabo, Istvan (Agoda)]

Hi,

So finally how did you solve it? Which method out of the three?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Joshua West 
Sent: Friday, April 9, 2021 5:41 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Abandon incomplete (damaged EC) pgs - How to manage the 
impact on cephfs?

Hey everyone.

Inside of cephfs, I have a directory which I setup a directory layout field to 
use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs 
is using normal replication.

Fast forward some time, and the EC directory has been used pretty extensively, 
and through some bad luck and poor timing, ~200pgs are in an incomplete state, 
and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 
34, not that it matters at this point)

# ceph pg ls incomplete --> is attached for reference.

Fortunately, it's primarily (only) my on-site backups, and other replaceable 
data inside of

I tried for a few days to recover the PGs:
 - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs)
 - Deep Scrub
 - osd_find_best_info_ignore_history_les = true (`pg query` was showing related 
error) etc.

I've finally just accepted this pool to be a lesson learned, and want to get 
the rest of my cephfs back to normal.

My questions:

 -- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 
objects
 -- Deleting the pool seems like an appropriate step, but as I am using an 
xattr within cephfs, which is otherwise on another pool, I am not confident 
that this approach is safe?
 -- cephfs currently blocks when attemping to impact every third file in the EC 
directory. Once I delete the pool, how will I remove the files if even `rm` is 
blocking?

Thank you for your time,

Joshua West
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [ERR] OSD_SCRUB_ERRORS: 2 scrub errors

[Szabo, Istvan (Agoda)]

Forgot the very minimum entries after the scrub done:

2021-04-01T11:37:43.559539+0700 osd.39 (osd.39) 50 : cluster [DBG] 20.19 repair 
starts
2021-04-01T11:37:43.889909+0700 osd.39 (osd.39) 51 : cluster [ERR] 20.19 soid 
20:990258ea:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.17263260.1.237:head : 
omap_digest 0x775cd866 != omap_digest 0xda11ecd0 from shard 39
2021-04-01T11:37:43.950318+0700 osd.39 (osd.39) 52 : cluster [ERR] 20.19 soid 
20:994159a0:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.17263260.1.35:head : 
omap_digest 0xb61affda != omap_digest 0xb3467a38 from shard 39
2021-04-01T11:37:45.397338+0700 mgr.sg-cephmon-6s01 (mgr.25028786) 81795 : 
cluster [DBG] pgmap v81983: 225 pgs: 1 
active+clean+scrubbing+deep+inconsistent+repair, 224 active+clean; 4.8 TiB 
data, 25 TiB used, 506 TiB / 531 TiB avail; 14 MiB/s rd, 5.0 MiB/s wr, 14.28k 
op/s
2021-04-01T11:37:45.690930+0700 osd.39 (osd.39) 53 : cluster [ERR] 20.19 repair 
0 missing, 2 inconsistent objects
2021-04-01T11:37:45.690951+0700 osd.39 (osd.39) 54 : cluster [ERR] 20.19 repair 
2 errors, 0 fixed
2021-04-01T11:37:45.762565+0700 osd.39 (osd.39) 55 : cluster [DBG] 20.19 
deep-scrub starts

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Szabo, Istvan (Agoda)  
Sent: Thursday, April 1, 2021 11:38 AM
To: ceph-users 
Subject: [ceph-users] [ERR] OSD_SCRUB_ERRORS: 2 scrub errors

Hi,

I’m continuously getting scrub errors in my index pool and log pool that I need 
to repair always.
HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] 
OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg 
inconsistent
pg 20.19 is active+clean+inconsistent, acting [39,41,37]

Why is this?
I have no cue at all, no log entry no anything ☹


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [ERR] OSD_SCRUB_ERRORS: 2 scrub errors

[Szabo, Istvan (Agoda)]

Hi,

I’m continuously getting scrub errors in my index pool and log pool that I need 
to repair always.
HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent
[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 20.19 is active+clean+inconsistent, acting [39,41,37]

Why is this?
I have no cue at all, no log entry no anything ☹


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD Crash During Deep-Scrub

[Szabo, Istvan (Agoda)]

I have this deep scrub issue in the index pool's pg's almost every week which 
made the cluster health error so I always need to repair that pg :/

Any solution that you have found so far?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Dave Hall 
Sent: Tuesday, March 30, 2021 3:30 AM
To: ceph-users 
Subject: [ceph-users] OSD Crash During Deep-Scrub

Hello,

A while back, I was having an issue with an OSD repeatedly crashing.  I 
ultimately reweighted it to zero and then marked out 'Out'.  Since I found that 
the logs for thoses crashes match https://tracker.ceph.com/issues/46490
.

Since the OSD is in a 'Safe-to-Destroy' state, I'm wondering the best course of 
action - should I just mark it back in?  Or should I destroy and rebuild it.  
If clearing it in the way I have, in combination with updating to 14.2.16, will 
prevent it from misbehaving, why go through the trouble of destroying and 
rebuilding?

Thanks.

-Dave

--
Dave Hall
Binghamton University
kdh...@binghamton.edu
607-760-2328 (Cell)
607-777-4641 (Office)
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: How to clear Health Warning status?

[Szabo, Istvan (Agoda)]

Restart the osd.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: jinguk.k...@ungleich.ch 
Sent: Monday, March 29, 2021 10:41 AM
To: Anthony D'Atri 
Cc: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: How to clear Health Warning 
status?

Hello there,

Thank you for your response.
There is no error at syslog, dmesg, or SMART.

# ceph health detail
HEALTH_WARN Too many repaired reads on 2 OSDs OSD_TOO_MANY_REPAIRS Too many 
repaired reads on 2 OSDs
osd.29 had 38 reads repaired
osd.16 had 17 reads repaired

How can i clear this waning ?
My ceph is version 14.2.9(clear_shards_repaired is not supported.)



/dev/sdh1 on /var/lib/ceph/osd/ceph-16 type xfs 
(rw,relatime,attr2,inode64,noquota)

# cat dmesg | grep sdh
[   12.990728] sd 5:2:3:0: [sdh] 19531825152 512-byte logical blocks: (10.0 
TB/9.09 TiB)
[   12.990728] sd 5:2:3:0: [sdh] Write Protect is off
[   12.990728] sd 5:2:3:0: [sdh] Mode Sense: 1f 00 00 08
[   12.990728] sd 5:2:3:0: [sdh] Write cache: enabled, read cache: enabled, 
doesn't support DPO or FUA
[   13.016616]  sdh: sdh1 sdh2
[   13.017780] sd 5:2:3:0: [sdh] Attached SCSI disk

# ceph tell osd.29 bench
{
"bytes_written": 1073741824,
"blocksize": 4194304,
"elapsed_sec": 6.464404,
"bytes_per_sec": 166100668.21318716,
"iops": 39.60148530320815
}
# ceph tell osd.16 bench
{
"bytes_written": 1073741824,
"blocksize": 4194304,
"elapsed_sec": 9.61689458,
"bytes_per_sec": 111651617.26584397,
"iops": 26.619819942914003
}

Thank you


> On 26 Mar 2021, at 16:04, Anthony D'Atri  wrote:
>
> Did you look at syslog, dmesg, or SMART?  Mostly likely the drives are 
> failing.
>
>
>> On Mar 25, 2021, at 9:55 PM, jinguk.k...@ungleich.ch wrote:
>>
>> Hello there,
>>
>> Thank you for advanced.
>> My ceph is ceph version 14.2.9
>> I have a repair issue too.
>>
>> ceph health detail
>> HEALTH_WARN Too many repaired reads on 2 OSDs OSD_TOO_MANY_REPAIRS
>> Too many repaired reads on 2 OSDs
>>   osd.29 had 38 reads repaired
>>   osd.16 had 17 reads repaired
>>
>> ~# ceph tell osd.16 bench
>> {
>>   "bytes_written": 1073741824,
>>   "blocksize": 4194304,
>>   "elapsed_sec": 7.148673815996,
>>   "bytes_per_sec": 150201541.10217974,
>>   "iops": 35.81083800844663
>> }
>> ~# ceph tell osd.29 bench
>> {
>>   "bytes_written": 1073741824,
>>   "blocksize": 4194304,
>>   "elapsed_sec": 6.924432750002,
>>   "bytes_per_sec": 155065672.9246161,
>>   "iops": 36.970537406114602
>> }
>>
>> But it looks like those osds are ok. how can i clear this warning ?
>>
>> Best regards
>> JG
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
>> email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: [Suspicious newsletter] bucket index and WAL/DB

[Szabo, Istvan (Agoda)]

Makes sense what you are talking about, I had the same confusing like you, 
finally went with redhat setup:

https://hubb.blob.core.windows.net/c2511cea-81c5-4386-8731-cc444ff806df-public/resources/Optimize%20Ceph%20object%20storage%20for%20production%20in%20multisite%20clouds.pdf

Slide 27.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Marcelo 
Sent: Friday, March 26, 2021 7:35 PM
Cc: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: [Suspicious newsletter] 
bucket index and WAL/DB

This is exactly the problem, so we thought about not risking losing the entire 
host by using both NVMes.

From what I understand the bucket index data is stored in the omap, which is 
stored in the block.db, making it unnecessary to create a separate OSD for the 
bucket index. But I didn't find anything in the documentation about it.
It is also unclear whether, if it is necessary to create a separate index pool, 
it would be recommended to place the OSD that serves that pool with wal / DB.

Em qui., 25 de mar. de 2021 às 22:42, Szabo, Istvan (Agoda) < 
istvan.sz...@agoda.com> escreveu:

> In couple of documentation that I've read I finally made the decision
> to separate index from wal+db.
> However don't you think that the density is a bit high with 12HDD for
> 1 nvme? So if you loose nvme you actually loose your complete host and
> a lot of data movements will happen.
>
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
> -Original Message-
> From: Marcelo 
> Sent: Thursday, March 25, 2021 11:15 PM
> To: ceph-users@ceph.io
> Subject: [Suspicious newsletter] [ceph-users] bucket index and WAL/DB
>
> Hello everybody.
>
> I searched in several places and I couldn't find any information about
> what the best bucket index and WAL / DB organization would be.
>
> I have several hosts consisting of 12 HDDs and 2 NVMes, and currently
> one of the NVMes serves as WAL / DB for the 10 OSDs and the other NVMe
> is partitioned in two, serving as 2 OSDs to serve the S3 index pool.
>
> I saw in ceph-ansible a playbook (infrastructure-playbooks /
> lv-create.yml) that creates a division where we have an OSD living
> with a journal on the same NVMe. The problem is that in lv-vars.yaml
> used by lv-create.yml it is said that this only applies to the
> filestore. Is this correct or can I use this same structure with bluestore?
>
> Thank you all,
> Marcelo.
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io
>
> 
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by
> copyright or other legal rules. If you have received it by mistake
> please let us know by reply email and delete it from your system. It
> is prohibited to copy this message or disclose its content to anyone.
> Any confidentiality or privilege is not waived or lost by any mistaken
> delivery or unauthorized disclosure of the message. All messages sent
> to and from Agoda may be monitored to ensure compliance with company
> policies, to protect the company's interests and to remove potential
> malware. Electronic messages may be intercepted, amended, lost or deleted, or 
> contain viruses.
>
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Possible to update from luminous 12.2.8 to nautilus latest?

[Szabo, Istvan (Agoda)]

Hi,

Is it possible to do a big jump or needs to go slower to luminous latest, then 
mimic latest, then nautilus latest?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How ceph sees when the pool is getting full?

[Szabo, Istvan (Agoda)]

Hi,

I want to set alert on the user's pool before it got's full but in nautilus I 
still haven't found the way which is the value of their data usage based on 
ceph detail df?

POOL ID  STORED   OBJECTS   
  USED %USED   MAX AVAILQUOTA OBJECTS   
  QUOTA BYTES  DIRTY  USED COMPRUNDER COMPR
k8s-dbss-w-mdb 16  1.8 TiB 488.61k  
 2.6 TiB   2.87 45 TiB  N/A 
   1.8 TiB  488.61k 
 0 B   0 B

This output is still confusing :/


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] bucket index and WAL/DB

[Szabo, Istvan (Agoda)]

In couple of documentation that I've read I finally made the decision to 
separate index from wal+db.
However don't you think that the density is a bit high with 12HDD for 1 nvme? 
So if you loose nvme you actually loose your complete host and a lot of data 
movements will happen.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Marcelo 
Sent: Thursday, March 25, 2021 11:15 PM
To: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] bucket index and WAL/DB

Hello everybody.

I searched in several places and I couldn't find any information about what the 
best bucket index and WAL / DB organization would be.

I have several hosts consisting of 12 HDDs and 2 NVMes, and currently one of 
the NVMes serves as WAL / DB for the 10 OSDs and the other NVMe is partitioned 
in two, serving as 2 OSDs to serve the S3 index pool.

I saw in ceph-ansible a playbook (infrastructure-playbooks / lv-create.yml) 
that creates a division where we have an OSD living with a journal on the same 
NVMe. The problem is that in lv-vars.yaml used by lv-create.yml it is said that 
this only applies to the filestore. Is this correct or can I use this same 
structure with bluestore?

Thank you all,
Marcelo.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Incomplete pg , any chance to to make it survive or data loss :( ?

[Szabo, Istvan (Agoda)]

Some news, due to the ceph pg inactive list command gave back that 0 objects 
are in this pg, I've marked complete on the primary osd, now it is unfound. Now 
I've stucked again 

[WRN] OBJECT_UNFOUND: 4/58369044 objects unfound (0.000%)
pg 44.1aa has 4 unfound objects
[ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
pg 44.1aa is active+recovery_unfound+undersized+degraded+remapped, acting 
[59,2147483647,127,43], 4 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 4/232017450 objects degraded 
(0.000%), 1 pg degraded, 1 pg undersized
pg 44.1aa is stuck undersized for 6m, current state 
active+recovery_unfound+undersized+degraded+remapped, last acting 
[59,2147483647,127,43]
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 44.1aa not deep-scrubbed since 2021-01-14T05:50:23.852626+0100
[WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
pg 44.1aa not scrubbed since 2021-01-14T05:50:23.852626+0100


From: Szabo, Istvan (Agoda) 
Sent: Monday, March 22, 2021 3:00 PM
To: Ceph Users 
Subject: [ceph-users] Re: Incomplete pg , any chance to to make it survive or 
data loss :( ?

Forgot to say, this is an octopus 15.2.9 cluster, there isn't any 
force_create_pg option that has couple of thread to make it work.
https://tracker.ceph.com/issues/10411
https://www.oreilly.com/library/view/mastering-proxmox-/9781788397605/42d80c67-10aa-4cf2-8812-e38c861cdc5d.xhtml
[https://www.oreilly.com/library/cover/9781788397605/]<https://www.oreilly.com/library/view/mastering-proxmox-/9781788397605/42d80c67-10aa-4cf2-8812-e38c861cdc5d.xhtml>
Stuck inactive incomplete PGs in Ceph - Mastering Proxmox - Third Edition 
[Book] - O'Reilly 
Media<https://www.oreilly.com/library/view/mastering-proxmox-/9781788397605/42d80c67-10aa-4cf2-8812-e38c861cdc5d.xhtml>
Stuck inactive incomplete PGs in Ceph If any PG is stuck due to OSD or node 
failure and becomes unhealthy, resulting in the cluster becoming inaccessible 
due to a blocked … - Selection from Mastering Proxmox - Third Edition [Book]
www.oreilly.com<http://www.oreilly.com>

Bug #10411: PG stuck incomplete after failed node - Ceph - 
Ceph<https://tracker.ceph.com/issues/10411>
Yesterday, I was in the process of expanding the number of PGs in one of our 
pools. While I was doing this, one of the disks in an OSD failed (probably due 
to the high load of the cluster at that point).
tracker.ceph.com

____________
From: Szabo, Istvan (Agoda) 
Sent: Monday, March 22, 2021 2:33 PM
To: Ceph Users 
Subject: [ceph-users] Incomplete pg , any chance to to make it survive or data 
loss :( ?

Hi,


What can I do with this pg to make it work?
We lost and don't have the osds  61,122 but we have the 32,33,70. I've exported 
the pg chunk from them, but they are very small and when I imported back to 
another osd that osd never started again so I had to remove that chunk 
(44.1aas2, 44.1aas3) to be able to start the osd.


[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg incomplete
pg 44.1aa is incomplete, acting [59,128,127,43] (reducing pool 
cephfs1-data01-pool min_size from 3 may help; search ceph.com/docs for 
'incomplete')
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 44.1aa not deep-scrubbed since 2021-01-14T05:50:23.852626+0100
[WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
pg 44.1aa not scrubbed since 2021-01-14T05:50:23.852626+0100
[WRN] SLOW_OPS: 96 slow ops, oldest one blocked for 228287 sec, osd.59 has slow 
ops

This is the pg query and pg map important parts:

"probing_osds": [
"29(3)",
"34(3)",
"43(3)",
"56(1)",
"59(0)",
"72(2)",
"73(2)",
"74(2)",
"127(2)",
"128(1)",
"131(2)"
],
"down_osds_we_would_probe": [
32,
33,
61,
70,
122
],
"peering_blocked_by": [],
"peering_blocked_by_detail": [
{
"detail": "peering_blocked_by_history_les_bound"


osdmap e5666778 pg 44.1aa (44.1aa) -> up [59,128,127,43] acting [59,128,127,43]



Internal


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthori

[ceph-users] Incomplete pg , any chance to to make it survive or data loss :( ?

[Szabo, Istvan (Agoda)]

Hi,


What can I do with this pg to make it work?
We lost and don't have the osds  61,122 but we have the 32,33,70. I've exported 
the pg chunk from them, but they are very small and when I imported back to 
another osd that osd never started again so I had to remove that chunk 
(44.1aas2, 44.1aas3) to be able to start the osd.


[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg incomplete
pg 44.1aa is incomplete, acting [59,128,127,43] (reducing pool 
cephfs1-data01-pool min_size from 3 may help; search ceph.com/docs for 
'incomplete')
[WRN] PG_NOT_DEEP_SCRUBBED: 1 pgs not deep-scrubbed in time
pg 44.1aa not deep-scrubbed since 2021-01-14T05:50:23.852626+0100
[WRN] PG_NOT_SCRUBBED: 1 pgs not scrubbed in time
pg 44.1aa not scrubbed since 2021-01-14T05:50:23.852626+0100
[WRN] SLOW_OPS: 96 slow ops, oldest one blocked for 228287 sec, osd.59 has slow 
ops

This is the pg query and pg map important parts:

"probing_osds": [
"29(3)",
"34(3)",
"43(3)",
"56(1)",
"59(0)",
"72(2)",
"73(2)",
"74(2)",
"127(2)",
"128(1)",
"131(2)"
],
"down_osds_we_would_probe": [
32,
33,
61,
70,
122
],
"peering_blocked_by": [],
"peering_blocked_by_detail": [
{
"detail": "peering_blocked_by_history_les_bound"


osdmap e5666778 pg 44.1aa (44.1aa) -> up [59,128,127,43] acting [59,128,127,43]



Internal


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] v15.2.10 Octopus released

[Szabo, Istvan (Agoda)]

Hi David,

I guess this one fixes the non-containerized deployment too, isn't it?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: David Galloway 
Sent: Thursday, March 18, 2021 9:10 PM
To: ceph-annou...@ceph.io; ceph-users@ceph.io; d...@ceph.io; 
ceph-maintain...@ceph.io
Subject: [Suspicious newsletter] [ceph-users] v15.2.10 Octopus released

We're happy to announce the 10th backport release in the Octopus series.
We recommend users to update to this release. For a detailed release notes with 
links & changelog please refer to the official blog entry at 
https://ceph.io/releases/v15-2-10-octopus-released

Notable Changes
---

* The containers include an updated tcmalloc that avoids crashes seen on 
15.2.9.  See `issue#49618 `_ for details.

* RADOS: BlueStore handling of huge(>4GB) writes from RocksDB to BlueFS has 
been fixed.

* When upgrading from a previous cephadm release, systemctl may hang when 
trying to start or restart the monitoring containers. (This is caused by a 
change in the systemd unit to use `type=forking`.) After the upgrade, please 
run::

ceph orch redeploy nfs
ceph orch redeploy iscsi
ceph orch redeploy node-exporter
ceph orch redeploy prometheus
ceph orch redeploy grafana
ceph orch redeploy alertmanager


Getting Ceph

* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-15.2.10.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 27917a557cca91e4da407489bbaa64ad4352cc02
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: PG export import

[Szabo, Istvan (Agoda)]

Yeah, finally started just super slow.
Currently I want to export import the pgs from the died OSDs make the cluster 
be able to start cephfs and save the data. Also looking for some space to be 
able to export the pg because it's quite big, 100s of gb.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Frank Schilder  
Sent: Thursday, March 18, 2021 6:16 PM
To: Szabo, Istvan (Agoda) ; Ceph Users 

Subject: Re: PG export import

It sounds like there is a general problem on this cluster with OSDs not 
starting. You probably need to go back to the logs and try to find out why the 
MONs don't allow the OSDs to join. MON IPs, cluster ID, network config in 
ceph.conf and on host, cluster name, authentication, ports, messenger version 
etc.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Szabo, Istvan (Agoda) 
Sent: 18 March 2021 10:48:05
To: Ceph Users
Subject: [ceph-users] PG export import

Hi,

I’ve tried to save some pg from a dead osd, I made this:

Picked on the same server an osd which is not really used and stopped that osd 
and import the exported one from the dead one.

root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-33 
--no-mon-config --pgid 44.c0s0 --op export --file ./pg44c0s0 Exporting 44.c0s0 
info 44.c0s0( empty local-lis/les=0/0 n=0 ec=192123/175799 
lis/c=4865474/4851556 les/c/f=4865475/4851557/0 sis=4865493) Export successful

root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-34 
--no-mon-config --op import --file ./pg44c0s0 get_pg_num_history pg_num_history 
pg_num_history(e5583546 pg_nums 
{20={173213=256},21={219434=64},22={220991=64},24={219240=32},25={1446965=128},42={175793=32},43={197388=64},44={192123=512}}
 deleted_pools ) Importing pgid 44.c0s0 write_pg epoch 4865498 info 44.c0s0( 
empty local-lis/les=0/0 n=0 ec=192123/175799 lis/c=4865474/4851556 
les/c/f=4865475/4851557/0 sis=4865493) Import successful

Started back 34 and it says the osd is running but in the cluster map it is 
down :/

root@server:~# systemctl status ceph-osd@34 -l ● 
ceph-osd@34.service<mailto:ceph-osd@34.service> - Ceph object storage daemon 
osd.34
 Loaded: loaded 
(/lib/systemd/system/ceph-osd@.service<mailto:/lib/systemd/system/ceph-osd@.service>;
 enabled-runtime; vendor preset: enabled)
 Active: active (running) since Thu 2021-03-18 10:38:00 CET; 8min ago
Process: 45388 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster 
${CLUSTER} --id 34 (code=exited, sta>
   Main PID: 45392 (ceph-osd)
  Tasks: 60
 Memory: 856.2M
 CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@34.service
 └─45392 /usr/bin/ceph-osd -f --cluster ceph --id 34 --setuser ceph 
--setgroup ceph

Mar 18 10:38:00 server systemd[1]: Starting Ceph object storage daemon osd.34...
Mar 18 10:38:00 server systemd[1]: Started Ceph object storage daemon osd.34.
Mar 18 10:38:21 server ceph-osd[45392]: 2021-03-18T10:38:21.817+0100 
7f41738d5dc0 -1 osd.34 5583546 log_to_mon> Mar 18 10:38:21 server 
ceph-osd[45392]: 2021-03-18T10:38:21.825+0100 7f41738d5dc0 -1 osd.34 5583546 
mon_cmd_ma>


Any idea?


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] PG export import

[Szabo, Istvan (Agoda)]

Hi,

I’ve tried to save some pg from a dead osd, I made this:

Picked on the same server an osd which is not really used and stopped that osd 
and import the exported one from the dead one.

root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-33 
--no-mon-config --pgid 44.c0s0 --op export --file ./pg44c0s0
Exporting 44.c0s0 info 44.c0s0( empty local-lis/les=0/0 n=0 ec=192123/175799 
lis/c=4865474/4851556 les/c/f=4865475/4851557/0 sis=4865493)
Export successful

root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-34 
--no-mon-config --op import --file ./pg44c0s0
get_pg_num_history pg_num_history pg_num_history(e5583546 pg_nums 
{20={173213=256},21={219434=64},22={220991=64},24={219240=32},25={1446965=128},42={175793=32},43={197388=64},44={192123=512}}
 deleted_pools )
Importing pgid 44.c0s0
write_pg epoch 4865498 info 44.c0s0( empty local-lis/les=0/0 n=0 
ec=192123/175799 lis/c=4865474/4851556 les/c/f=4865475/4851557/0 sis=4865493)
Import successful

Started back 34 and it says the osd is running but in the cluster map it is 
down :/

root@server:~# systemctl status ceph-osd@34 -l
● ceph-osd@34.service - Ceph object storage daemon 
osd.34
 Loaded: loaded 
(/lib/systemd/system/ceph-osd@.service;
 enabled-runtime; vendor preset: enabled)
 Active: active (running) since Thu 2021-03-18 10:38:00 CET; 8min ago
Process: 45388 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster 
${CLUSTER} --id 34 (code=exited, sta>
   Main PID: 45392 (ceph-osd)
  Tasks: 60
 Memory: 856.2M
 CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@34.service
 └─45392 /usr/bin/ceph-osd -f --cluster ceph --id 34 --setuser ceph 
--setgroup ceph

Mar 18 10:38:00 server systemd[1]: Starting Ceph object storage daemon osd.34...
Mar 18 10:38:00 server systemd[1]: Started Ceph object storage daemon osd.34.
Mar 18 10:38:21 server ceph-osd[45392]: 2021-03-18T10:38:21.817+0100 
7f41738d5dc0 -1 osd.34 5583546 log_to_mon>
Mar 18 10:38:21 server ceph-osd[45392]: 2021-03-18T10:38:21.825+0100 
7f41738d5dc0 -1 osd.34 5583546 mon_cmd_ma>


Any idea?


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Inactive pg, how to make it active / or delete

[Szabo, Istvan (Agoda)]

Can’t bring back, now trying to make the cluster hosts equal.

This signal error bot going away from the osd start.

I think I’ll try the ceph-objectstore tool pg export import on the died osds 
and put back to another. Let’s see.

> On 2021. Mar 16., at 18:54, Frank Schilder  wrote:
> 
> The PG says blocked_by at least 2 of your down-OSDs. When you look at the 
> history (past_intervals), it needs to backfill from the down OSDs 
> (down_osds_we_would_probe). Since its more than 1, it can't proceed. You need 
> to get the OSDs up.
> 
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> 
> ____________
> From: Szabo, Istvan (Agoda) 
> Sent: 16 March 2021 10:44:10
> To: Ceph Users
> Subject: [ceph-users] Inactive pg, how to make it active / or delete
> 
> Hi,
> 
> I have 4 inactive pg in my cluster, the osds are dies where it was before. 
> How can I make it work again? Maybe just threw away because last backfill=max?
> Based on the pg query it is totally up on other osds.
> It is an EC 3+1.
> 
> This is an example inactive pg:
> 
> ceph pg 44.1f0 query
> {
>"snap_trimq": "[]",
>"snap_trimq_len": 0,
>"state": "incomplete",
>"epoch": 5541839,
>"up": [
>46,
>34,
>62,
>74
>],
>"acting": [
>46,
>34,
>62,
>74
>],
>"info": {
>"pgid": "44.1f0s0",
>"last_update": "4863820'2109288",
>"last_complete": "4863820'2109288",
>"log_tail": "3881944'2103139",
>"last_user_version": 11189093,
>"last_backfill": "MAX",
>"purged_snaps": [],
>"history": {
>"epoch_created": 192123,
>"epoch_pool_created": 175799,
>"last_epoch_started": 4865266,
>"last_interval_started": 4865265,
>"last_epoch_clean": 4865007,
>"last_interval_clean": 4865006,
>"last_epoch_split": 192123,
>"last_epoch_marked_full": 0,
>"same_up_since": 5541572,
>"same_interval_since": 5541572,
>"same_primary_since": 5520915,
>"last_scrub": "4863820'2109288",
>"last_scrub_stamp": "2021-01-19T23:44:40.885414+0100",
>"last_deep_scrub": "4808731'2109261",
>"last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100",
>"last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100",
>"prior_readable_until_ub": 0
>},
>"stats": {
>"version": "4863820'2109288",
>"reported_seq": "12355046",
>"reported_epoch": "5541839",
>"state": "incomplete",
>"last_fresh": "2021-03-16T10:39:21.058569+0100",
>"last_change": "2021-03-16T10:39:21.058569+0100",
>"last_active": "2021-01-20T01:07:18.246158+0100",
>"last_peered": "2021-01-20T01:07:13.931842+0100",
>"last_clean": "2021-01-20T01:07:07.392736+0100",
>"last_became_active": "2021-01-20T01:07:09.187047+0100",
>"last_became_peered": "2021-01-20T01:07:09.187047+0100",
>"last_unstale": "2021-03-16T10:39:21.058569+0100",
>"last_undegraded": "2021-03-16T10:39:21.058569+0100",
>"last_fullsized": "2021-03-16T10:39:21.058569+0100",
>"mapping_epoch": 5541572,
>"log_start": "3881944'2103139",
>"ondisk_log_start": "3881944'2103139",
>"created": 192123,
>"last_epoch_clean": 4865007,
>"parent": "0.0",
>"parent_split_bits": 9,
>"last_scrub": "4863820'2109288",
>"last_scrub_stamp": "2021-01-19T23:44:40.885414+0100",
>"last_deep_scrub": "4808731'2109261&

[ceph-users] Inactive pg, how to make it active / or delete

[Szabo, Istvan (Agoda)]

Hi,

I have 4 inactive pg in my cluster, the osds are dies where it was before. How 
can I make it work again? Maybe just threw away because last backfill=max?
Based on the pg query it is totally up on other osds.
It is an EC 3+1.

This is an example inactive pg:

ceph pg 44.1f0 query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "incomplete",
"epoch": 5541839,
"up": [
46,
34,
62,
74
],
"acting": [
46,
34,
62,
74
],
"info": {
"pgid": "44.1f0s0",
"last_update": "4863820'2109288",
"last_complete": "4863820'2109288",
"log_tail": "3881944'2103139",
"last_user_version": 11189093,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 192123,
"epoch_pool_created": 175799,
"last_epoch_started": 4865266,
"last_interval_started": 4865265,
"last_epoch_clean": 4865007,
"last_interval_clean": 4865006,
"last_epoch_split": 192123,
"last_epoch_marked_full": 0,
"same_up_since": 5541572,
"same_interval_since": 5541572,
"same_primary_since": 5520915,
"last_scrub": "4863820'2109288",
"last_scrub_stamp": "2021-01-19T23:44:40.885414+0100",
"last_deep_scrub": "4808731'2109261",
"last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100",
"last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "4863820'2109288",
"reported_seq": "12355046",
"reported_epoch": "5541839",
"state": "incomplete",
"last_fresh": "2021-03-16T10:39:21.058569+0100",
"last_change": "2021-03-16T10:39:21.058569+0100",
"last_active": "2021-01-20T01:07:18.246158+0100",
"last_peered": "2021-01-20T01:07:13.931842+0100",
"last_clean": "2021-01-20T01:07:07.392736+0100",
"last_became_active": "2021-01-20T01:07:09.187047+0100",
"last_became_peered": "2021-01-20T01:07:09.187047+0100",
"last_unstale": "2021-03-16T10:39:21.058569+0100",
"last_undegraded": "2021-03-16T10:39:21.058569+0100",
"last_fullsized": "2021-03-16T10:39:21.058569+0100",
"mapping_epoch": 5541572,
"log_start": "3881944'2103139",
"ondisk_log_start": "3881944'2103139",
"created": 192123,
"last_epoch_clean": 4865007,
"parent": "0.0",
"parent_split_bits": 9,
"last_scrub": "4863820'2109288",
"last_scrub_stamp": "2021-01-19T23:44:40.885414+0100",
"last_deep_scrub": "4808731'2109261",
"last_deep_scrub_stamp": "2021-01-15T21:17:56.729962+0100",
"last_clean_scrub_stamp": "2021-01-19T23:44:40.885414+0100",
"log_size": 6149,
"ondisk_log_size": 6149,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"stat_sum": {
"num_bytes": 356705195545,
"num_objects": 98594,
"num_object_clones": 758,
"num_object_copies": 394376,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 98594,
"num_whiteouts": 750,
"num_read": 30767,
"num_read_kb": 37109881,
"num_write": 31023,
"num_write_kb": 66655410,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 1000136,
"num_bytes_recovered": 3631523354253,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
   

[ceph-users] Re: Safe to remove osd or not? Which statement is correct?

[Szabo, Istvan (Agoda)]

Hi Boris,

Yeah, this is the reason:

   -1> 2021-03-15T16:21:35.307+0100 7f8b1fd8d700  5 prioritycache tune_memory 
target: 4294967296mapped: 454098944 unmapped: 
8560640 heap: 462659584 old mem: 
2845415832 new mem: 2845415832
0> 2021-03-15T16:21:35.311+0100 7f8b11570700 -1 *** Caught signal (Aborted) 
**
in thread 7f8b11570700 thread_name:tp_osd_tp


On 2021. Mar 14., at 15:10, Boris Behrens  wrote:


Email received from outside the company. If in doubt don't click links nor open 
attachments!

Hi,
do you know why the OSDs are not starting?

When I had the problem that a start does not work, I tried the 'ceph-volume lvm 
activate --all' on the host, which brought the OSDs back up.

But I can't tell you if it is safe to remove the OSD.

Cheers
 Boris

Am So., 14. März 2021 um 02:38 Uhr schrieb Szabo, Istvan (Agoda) 
mailto:istvan.sz...@agoda.com>>:
Hi Gents,

There is a cluster with 14 hosts in this state:

https://i.ibb.co/HPF3Pdr/6-ACB2-C5-B-6-B54-476-B-835-D-227-E9-BFB1247.jpg

There is a host based crush rule ec 3:1 and there are 3 hosts where are osds 
down.
Unfortunately there are pools with 3 replicas also which is host based.

2 hosts have 2 osds down,1 host has 1 osd down, which means I guess if we don’t 
bring it back, 0.441% data loss is going to happen. Am I right?
I hope I’m not right because there isn’t any missplaced objects, only degraded.

The problem is that the osds are not starting so somehow these 5 osds should be 
removed, but I’m curious which is my statement is correct.

Thank you in advance.



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to 
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>


--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im 
groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: millions slow ops on a cluster without load

[Szabo, Istvan (Agoda)]

Yeah, the mtu is on the cluster network’s nic cards are 8982, the ping works 
with 8954 packets between interfaces.

On 2021. Mar 15., at 23:40, Matthew H  wrote:


Might be an MTU problem, have you checked your network and MTU settings?




From: Szabo, Istvan (Agoda) 
Sent: Monday, March 15, 2021 12:08 PM
To: Ceph Users 
Subject: [ceph-users] millions slow ops on a cluster without load

We have a cluster with a huge amount of  warnings like this even if nothing is 
going on in the cluster.

It makes mgr physical memory full, mon db maxed out 5 osds can't start :/

[WRN] slow request osd_op(mds.0.537792:26453 43.38 
43:1d6c5587:::1fe56a6.:head [create,setxattr parent (367) 
in=373b,setxattr layout (30) in=36b] snapc 0=[] RETRY=131 
ondisk+retry+write+known_if_redirected+full_force e5532574) initiated 
2021-03-15T16:44:18.428585+0100 currently delayed

Is there a way to somehow stop it?

MDS service stopped also.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] millions slow ops on a cluster without load

[Szabo, Istvan (Agoda)]

We have a cluster with a huge amount of  warnings like this even if nothing is 
going on in the cluster.

It makes mgr physical memory full, mon db maxed out 5 osds can't start :/

[WRN] slow request osd_op(mds.0.537792:26453 43.38 
43:1d6c5587:::1fe56a6.:head [create,setxattr parent (367) 
in=373b,setxattr layout (30) in=36b] snapc 0=[] RETRY=131 
ondisk+retry+write+known_if_redirected+full_force e5532574) initiated 
2021-03-15T16:44:18.428585+0100 currently delayed

Is there a way to somehow stop it?

MDS service stopped also.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Safe to remove osd or not? Which statement is correct?

[Szabo, Istvan (Agoda)]

Hi Gents,

There is a cluster with 14 hosts in this state:

https://i.ibb.co/HPF3Pdr/6-ACB2-C5-B-6-B54-476-B-835-D-227-E9-BFB1247.jpg

There is a host based crush rule ec 3:1 and there are 3 hosts where are osds 
down.
Unfortunately there are pools with 3 replicas also which is host based.

2 hosts have 2 osds down,1 host has 1 osd down, which means I guess if we don’t 
bring it back, 0.441% data loss is going to happen. Am I right?
I hope I’m not right because there isn’t any missplaced objects, only degraded.

The problem is that the osds are not starting so somehow these 5 osds should be 
removed, but I’m curious which is my statement is correct.

Thank you in advance.



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ERROR: S3 error: 403 (SignatureDoesNotMatch)

[Szabo, Istvan (Agoda)]

Fixed, need to remove this entry ... ehhh, spent 3 days on it 

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Szabo, Istvan (Agoda) 
Sent: Friday, March 12, 2021 11:57 AM
To: 'Szabo, Istvan (Agoda)' ; Ceph Users 

Subject: RE: ERROR: S3 error: 403 (SignatureDoesNotMatch)

Seems like the issue is this line in the radosgw-configuration:

rgw_dns_name = 

It is only binded to the one which is listed there and ignore the cname totally 
and haproxy ...
Is there a way to have 2 rgw_dns_name?

When I've pleayed around to put 2 names or 2 complete entries doesn't work.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Szabo, Istvan (Agoda)  
Sent: Friday, March 12, 2021 10:33 AM
To: Ceph Users 
Subject: [ceph-users] ERROR: S3 error: 403 (SignatureDoesNotMatch)

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

I'm struggling with my old cluster cnamed address.
The s3 and curl commands are working properly with the not cnamed address, but 
with the cnamed one, I got this in the ciwetweb log:

2021-03-12 10:24:18.812329 7f6b0c527700  1 == starting new request 
req=0x7f6b0c520f90 =
2021-03-12 10:24:18.812387 7f6b0c527700  2 req 10:0.58::HEAD 
/::initializing for trans_id = txa-00604adee2-8e4fc3-default
2021-03-12 10:24:18.812412 7f6b0c527700 10 rgw api priority: s3=5 s3website=4
2021-03-12 10:24:18.812417 7f6b0c527700 10 host=cnamedhostname
2021-03-12 10:24:18.812484 7f6b0c527700 10 handler=25RGWHandler_REST_Bucket_S3
2021-03-12 10:24:18.812490 7f6b0c527700  2 req 10:0.000163:s3:HEAD /::getting 
op 3
2021-03-12 10:24:18.812499 7f6b0c527700 10 op=25RGWStatBucket_ObjStore_S3
2021-03-12 10:24:18.812503 7f6b0c527700  2 req 10:0.000176:s3:HEAD 
/:stat_bucket:verifying requester
2021-03-12 10:24:18.812541 7f6b0c527700  2 req 10:0.000214:s3:HEAD 
/:stat_bucket:normalizing buckets and tenants
2021-03-12 10:24:18.812548 7f6b0c527700 10 s->object= s->bucket= 
cnamedhostname
2021-03-12 10:24:18.812556 7f6b0c527700  2 req 10:0.000229:s3:HEAD 
/:stat_bucket:init permissions
2021-03-12 10:24:18.812594 7f6b0c527700 10 cache get: 
name=default.rgw.meta+root+ cnamedhostname : type miss (requested=0x16, 
cached=0x0)
2021-03-12 10:24:18.813525 7f6b0c527700 10 cache put: 
name=default.rgw.meta+root+ cnamedhostname info.flags=0x0
2021-03-12 10:24:18.813554 7f6b0c527700 10 moving default.rgw.meta+root+ 
cnamedhostname to cache LRU end
2021-03-12 10:24:18.813664 7f6b0c527700 10 read_permissions on cnamedhostname 
[] ret=-2002
2021-03-12 10:24:18.813833 7f6b0c527700  2 req 10:0.001506:s3:HEAD 
/:stat_bucket:op status=0
2021-03-12 10:24:18.813848 7f6b0c527700  2 req 10:0.001520:s3:HEAD 
/:stat_bucket:http status=404
2021-03-12 10:24:18.813855 7f6b0c527700  1 == req done req=0x7f6b0c520f90 
op status=0 http_status=404 ==
2021-03-12 10:24:18.813962 7f6b0c527700  1 civetweb: 0x557d45468000: 
10.118.199.248 - - [12/Mar/2021:10:24:18 +0700] "HEAD / HTTP/1.1" 404 0 - 
curl/7.29.0

And I got this on the s3cmd verbose output:
DEBUG: s3cmd version 2.1.0
DEBUG: ConfigParser: Reading file '.s3cfg-testuser-http'
DEBUG: ConfigParser: access_key->29...17_chars...J
DEBUG: ConfigParser: secret_key->fK...37_chars...R
DEBUG: ConfigParser: host_base->cnamedhostname:80
DEBUG: ConfigParser: host_bucket->cnamedhostname:80/%(bucket)
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: signature_v2->True
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Command: ls
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: u'GET\n\n\n\nx-amz-date:Fri, 12 Mar 2021 03:31:39 +\n/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): cnamedhostname
DEBUG: ConnMan.get(): creating new connection: http://cnamedhostname
DEBUG: non-proxied HTTPConnection(cnamedhostname, None)
DEBUG: format_uri(): /
DEBUG: Sending request method_string='GET', uri=u'/', headers={'Authorization': 
u'AWS 293WEU2ADWGIUO4RN39J:Q7kh7kzWXWSqMvUqqWwLOY6QKUE=', 'x-amz-date': 'Fri, 
12 Mar 2021 03:31:39 +'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://cnamedhostname#1)
DEBUG: Response:
{'data': 'SignatureDoesNotMatchtxb-00604ae09b-8e4fbd-default8e4fbd-default-default',
'headers': {'accept-ranges': 'bytes',
 'content-length': '198',
 'content-type': 'application/xml',
 'date': 'Fri, 12 Mar 2021 03

[ceph-users] ERROR: S3 error: 403 (SignatureDoesNotMatch)

[Szabo, Istvan (Agoda)]

Hi,

I'm struggling with my old cluster cnamed address.
The s3 and curl commands are working properly with the not cnamed address, but 
with the cnamed one, I got this in the ciwetweb log:

2021-03-12 10:24:18.812329 7f6b0c527700  1 == starting new request 
req=0x7f6b0c520f90 =
2021-03-12 10:24:18.812387 7f6b0c527700  2 req 10:0.58::HEAD 
/::initializing for trans_id = txa-00604adee2-8e4fc3-default
2021-03-12 10:24:18.812412 7f6b0c527700 10 rgw api priority: s3=5 s3website=4
2021-03-12 10:24:18.812417 7f6b0c527700 10 host=cnamedhostname
2021-03-12 10:24:18.812484 7f6b0c527700 10 handler=25RGWHandler_REST_Bucket_S3
2021-03-12 10:24:18.812490 7f6b0c527700  2 req 10:0.000163:s3:HEAD /::getting 
op 3
2021-03-12 10:24:18.812499 7f6b0c527700 10 op=25RGWStatBucket_ObjStore_S3
2021-03-12 10:24:18.812503 7f6b0c527700  2 req 10:0.000176:s3:HEAD 
/:stat_bucket:verifying requester
2021-03-12 10:24:18.812541 7f6b0c527700  2 req 10:0.000214:s3:HEAD 
/:stat_bucket:normalizing buckets and tenants
2021-03-12 10:24:18.812548 7f6b0c527700 10 s->object= s->bucket= 
cnamedhostname
2021-03-12 10:24:18.812556 7f6b0c527700  2 req 10:0.000229:s3:HEAD 
/:stat_bucket:init permissions
2021-03-12 10:24:18.812594 7f6b0c527700 10 cache get: 
name=default.rgw.meta+root+ cnamedhostname : type miss (requested=0x16, 
cached=0x0)
2021-03-12 10:24:18.813525 7f6b0c527700 10 cache put: 
name=default.rgw.meta+root+ cnamedhostname info.flags=0x0
2021-03-12 10:24:18.813554 7f6b0c527700 10 moving default.rgw.meta+root+ 
cnamedhostname to cache LRU end
2021-03-12 10:24:18.813664 7f6b0c527700 10 read_permissions on cnamedhostname 
[] ret=-2002
2021-03-12 10:24:18.813833 7f6b0c527700  2 req 10:0.001506:s3:HEAD 
/:stat_bucket:op status=0
2021-03-12 10:24:18.813848 7f6b0c527700  2 req 10:0.001520:s3:HEAD 
/:stat_bucket:http status=404
2021-03-12 10:24:18.813855 7f6b0c527700  1 == req done req=0x7f6b0c520f90 
op status=0 http_status=404 ==
2021-03-12 10:24:18.813962 7f6b0c527700  1 civetweb: 0x557d45468000: 
10.118.199.248 - - [12/Mar/2021:10:24:18 +0700] "HEAD / HTTP/1.1" 404 0 - 
curl/7.29.0

And I got this on the s3cmd verbose output:
DEBUG: s3cmd version 2.1.0
DEBUG: ConfigParser: Reading file '.s3cfg-testuser-http'
DEBUG: ConfigParser: access_key->29...17_chars...J
DEBUG: ConfigParser: secret_key->fK...37_chars...R
DEBUG: ConfigParser: host_base->cnamedhostname:80
DEBUG: ConfigParser: host_bucket->cnamedhostname:80/%(bucket)
DEBUG: ConfigParser: use_https->False
DEBUG: ConfigParser: signature_v2->True
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Command: ls
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: u'GET\n\n\n\nx-amz-date:Fri, 12 Mar 2021 03:31:39 +\n/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): cnamedhostname
DEBUG: ConnMan.get(): creating new connection: http://cnamedhostname
DEBUG: non-proxied HTTPConnection(cnamedhostname, None)
DEBUG: format_uri(): /
DEBUG: Sending request method_string='GET', uri=u'/', headers={'Authorization': 
u'AWS 293WEU2ADWGIUO4RN39J:Q7kh7kzWXWSqMvUqqWwLOY6QKUE=', 'x-amz-date': 'Fri, 
12 Mar 2021 03:31:39 +'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://cnamedhostname#1)
DEBUG: Response:
{'data': 'SignatureDoesNotMatchtxb-00604ae09b-8e4fbd-default8e4fbd-default-default',
'headers': {'accept-ranges': 'bytes',
 'content-length': '198',
 'content-type': 'application/xml',
 'date': 'Fri, 12 Mar 2021 03:31:39 GMT',
 'x-amz-request-id': 
'txb-00604ae09b-8e4fbd-default'},
'reason': 'Forbidden',
'status': 403}
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: date: Fri, 12 Mar 2021 03:31:39 GMT
DEBUG: HttpHeader: content-length: 198
DEBUG: HttpHeader: x-amz-request-id: 
txb-00604ae09b-8e4fbd-default
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: accept-ranges: bytes
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: RequestId: 'txb-00604ae09b-8e4fbd-default'
DEBUG: ErrorXML: HostId: '8e4fbd-default-default'
ERROR: S3 error: 403 (SignatureDoesNotMatch)

Any idea?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the 

[ceph-users] Re: A practical approach to efficiently store 100 billions small objects in Ceph

[Szabo, Istvan (Agoda)]

Yeah, makes sense and sounds a good idea :) I’ve never thought about this, will 
think in case of object store in our clusters.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

On 2021. Mar 11., at 15:06, Loïc Dachary  wrote:

Thanks for clarifying, I think I understand. The idea is that 1,000 ~4KB 
objects are packed together in RBD which stores them in a single 4MB RADOS 
object. Does that answer your question?

On 11/03/2021 08:22, Szabo, Istvan (Agoda) wrote:
Hi,

It relates to this sentence:
"The median object size is ~4KB, written in RBD images using the default 4MB[0] 
object size. That will be ~100 millions RADOS objects instead of 100 billions."

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Loïc Dachary 
Sent: Thursday, March 11, 2021 2:10 PM
To: Szabo, Istvan (Agoda) 
Cc: Ceph Users 
Subject: [ceph-users] Re: A practical approach to efficiently store 100 
billions small objects in Ceph

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

On 11/03/2021 04:38, Szabo, Istvan (Agoda) wrote:
Does this mean that even in an object store the files which is smaller than 4MB 
 will be packed in one 4 MB object?
I'm not sure I understand the question. Would you be so kind as to rephrase it?

Cheers
-Original Message-
From: Loïc Dachary 
Sent: Thursday, March 11, 2021 2:13 AM
To: Konstantin Shalygin 
Cc: Ceph Users ; swh-de...@inria.fr
Subject: [ceph-users] Re: A practical approach to efficiently store 100 
billions small objects in Ceph

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi Konstantin,

Thanks for the advice. Luckily objects are packed together and Ceph will only 
see larger objects. The median object size is ~4KB, written in RBD images using 
the default 4MB[0] object size. That will be ~100 millions RADOS objects 
instead of 100 billions.

Cheers

[0] https://docs.ceph.com/en/latest/man/8/rbd/#cmdoption-rbd-object-size

On 10/03/2021 17:44, Konstantin Shalygin wrote:
Loic, please wait (or use shaman builds) for 14.2.17 cause for clusters with 
billions of injects code was not optimal [1] at objects delete step


[1] https://tracker.ceph.com/issues/47044 
<https://tracker.ceph.com/issues/47044>
k

Sent from my iPhone

On 10 Mar 2021, at 17:55, Loïc Dachary  wrote:

The next step will be to write and run benchmarks
--
Loïc Dachary, Artisan Logiciel Libre




This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
--
Loïc Dachary, Artisan Logiciel Libre



--
Loïc Dachary, Artisan Logiciel Libre


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: A practical approach to efficiently store 100 billions small objects in Ceph

[Szabo, Istvan (Agoda)]

Hi,

It relates to this sentence:
"The median object size is ~4KB, written in RBD images using the default 4MB[0] 
object size. That will be ~100 millions RADOS objects instead of 100 billions."

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Loïc Dachary  
Sent: Thursday, March 11, 2021 2:10 PM
To: Szabo, Istvan (Agoda) 
Cc: Ceph Users 
Subject: [ceph-users] Re: A practical approach to efficiently store 100 
billions small objects in Ceph

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

On 11/03/2021 04:38, Szabo, Istvan (Agoda) wrote:
> Does this mean that even in an object store the files which is smaller than 
> 4MB  will be packed in one 4 MB object?
I'm not sure I understand the question. Would you be so kind as to rephrase it?

Cheers
>
> -Original Message-
> From: Loïc Dachary 
> Sent: Thursday, March 11, 2021 2:13 AM
> To: Konstantin Shalygin 
> Cc: Ceph Users ; swh-de...@inria.fr
> Subject: [ceph-users] Re: A practical approach to efficiently store 100 
> billions small objects in Ceph
>
> Email received from outside the company. If in doubt don't click links nor 
> open attachments!
> 
>
> Hi Konstantin,
>
> Thanks for the advice. Luckily objects are packed together and Ceph will only 
> see larger objects. The median object size is ~4KB, written in RBD images 
> using the default 4MB[0] object size. That will be ~100 millions RADOS 
> objects instead of 100 billions.
>
> Cheers
>
> [0] https://docs.ceph.com/en/latest/man/8/rbd/#cmdoption-rbd-object-size
>
> On 10/03/2021 17:44, Konstantin Shalygin wrote:
>> Loic, please wait (or use shaman builds) for 14.2.17 cause for clusters with 
>> billions of injects code was not optimal [1] at objects delete step
>>
>>
>> [1] https://tracker.ceph.com/issues/47044 
>> <https://tracker.ceph.com/issues/47044>
>> k
>>
>> Sent from my iPhone
>>
>>> On 10 Mar 2021, at 17:55, Loïc Dachary  wrote:
>>>
>>> The next step will be to write and run benchmarks
> --
> Loïc Dachary, Artisan Logiciel Libre
>
>
>
> 
> This message is confidential and is for the sole use of the intended 
> recipient(s). It may also be privileged or otherwise protected by copyright 
> or other legal rules. If you have received it by mistake please let us know 
> by reply email and delete it from your system. It is prohibited to copy this 
> message or disclose its content to anyone. Any confidentiality or privilege 
> is not waived or lost by any mistaken delivery or unauthorized disclosure of 
> the message. All messages sent to and from Agoda may be monitored to ensure 
> compliance with company policies, to protect the company's interests and to 
> remove potential malware. Electronic messages may be intercepted, amended, 
> lost or deleted, or contain viruses.

--
Loïc Dachary, Artisan Logiciel Libre


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bluestore OSD Layout - WAL, DB, Journal

[Szabo, Istvan (Agoda)]

Hi,

If you don't specifiy WAL it will be located on the same drive as the rocksdb. 
You need to specify wal if you have faster disk the your rocksdb, like DATA on 
HDD, Rocksdb on SSD, wal on nvme/optane.

In the past they suggested like this:
300GB data 30GB rocksdb, 3GB wal. Not sure is this still true.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Dave Hall 
Sent: Wednesday, March 10, 2021 11:27 PM
To: ceph-users 
Subject: [ceph-users] Bluestore OSD Layout - WAL, DB, Journal

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hello,

I'm in the process of doubling the number of OSD nodes in my Nautilus cluster - 
 from 3 to 6.  Based on answers receive from earlier posts to this list, the 
new nodes have more NVMe that the old nodes.  More to the point, on the 
original nodes the amount of NVMe allocated to each OSD was about 120GB, so the 
RocksDB was limited to 30GB.  However, for my workload 300GB is probably 
recommended.

As I prepare to lay out the NVMe on these new nodes, I'm still trying to 
understand how to size the DB and WAL for my OSDs and whether Journal is even 
needed.

According to https://docs.ceph.com/en/nautilus/ceph-volume/lvm/prepare/


> Bluestore supports the following configurations:
>
>   * A block device, a block.wal, and a block.db device
>   * A block device and a block.wal device
>   * A block device and a block.db device
>   * A single block device
>
First question:  On my first nodes I managed to get a DB, but no WAL.
My current perception is that WAL and DB occupy separate physical/logical 
partitions.  By specifying a WAL size and a DB size, ceph-volume will create 
the corresponding logical volumes on the NVMe.
Is this correct?  It is also possible to lay these out as basic logical 
partitions?

Second question:   How do I decide whether I need WAL, DB, or both?

Third question:  Once I answer the above WAL/DB question, what are the 
guidelines for sizing them?

Thanks.

-Dave

--
Dave Hall
Binghamton University

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Failure Domain = NVMe?

[Szabo, Istvan (Agoda)]

Don't forget if you have server failure you might loose many objects. If the 
failure domain is osd, it means let's say you have 12 drives in each server, 
8+2 EC in an unlucky situation can be located in 1 server also.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Dave Hall 
Sent: Wednesday, March 10, 2021 11:42 PM
To: ceph-users 
Subject: [ceph-users] Failure Domain = NVMe?

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hello,

In some documentation I was reading last night about laying out OSDs, it was 
suggested that if more that one OSD uses the same NVMe drive, the 
failure-domain should probably be set to node. However, for a small cluster the 
inclination is to use EC-pools and failure-domain = OSD.

I was wondering if there is a middle ground - could we define failure-domain = 
NVMe?  I think the map would need to be defined manually in the same way that 
failure-domain = rack requires information about which nodes are in each rack.

Example:  My latest OSD nodes have 8 HDDs and 3 U.2 NVMe.  I'd set up the 
WAL/DB for with HDDs per OSD  (wasted space on the 3rd NVMe).
Across all my OSD nodes I will have 8 HDDs and either 2 or 3 NVMe
devices per node - 15 total NVMe devices.   My preferred EC-pool profile
is 8+2.  It seems that this profile could be safely dispersed across 15 failure 
domains, resulting in protection against NVMe failure.

Please let me know if this is worth pursuing.

Thanks.

-Dave

--
Dave Hall
Binghamton University
kdh...@binghamton.edu
607-760-2328 (Cell)
607-777-4641 (Office)
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: A practical approach to efficiently store 100 billions small objects in Ceph

[Szabo, Istvan (Agoda)]

Does this mean that even in an object store the files which is smaller than 4MB 
 will be packed in one 4 MB object?

-Original Message-
From: Loïc Dachary 
Sent: Thursday, March 11, 2021 2:13 AM
To: Konstantin Shalygin 
Cc: Ceph Users ; swh-de...@inria.fr
Subject: [ceph-users] Re: A practical approach to efficiently store 100 
billions small objects in Ceph

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi Konstantin,

Thanks for the advice. Luckily objects are packed together and Ceph will only 
see larger objects. The median object size is ~4KB, written in RBD images using 
the default 4MB[0] object size. That will be ~100 millions RADOS objects 
instead of 100 billions.

Cheers

[0] https://docs.ceph.com/en/latest/man/8/rbd/#cmdoption-rbd-object-size

On 10/03/2021 17:44, Konstantin Shalygin wrote:
> Loic, please wait (or use shaman builds) for 14.2.17 cause for clusters with 
> billions of injects code was not optimal [1] at objects delete step
>
>
> [1] https://tracker.ceph.com/issues/47044 
> 
> k
>
> Sent from my iPhone
>
>> On 10 Mar 2021, at 17:55, Loïc Dachary  wrote:
>>
>> The next step will be to write and run benchmarks

--
Loïc Dachary, Artisan Logiciel Libre




This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Bluestore OSD crash with tcmalloc::allocate_full_cpp_throw_oom in multisite setup with PG_DAMAGED cluster error

[Szabo, Istvan (Agoda)]

Hi,

I have a 3 DC multisite setup.

The replication is directional like HKG->SGP->US so the bucket is replicated 
from HKG to SGP and the same bucket is replicated further from SGP to US.

The HKG > SGP connection is pretty fast 12.5millions objects (600GB) 
transferred in 6.5 hours. Once the OSD crashed in SGP, it stopped the complete 
chain replication and made PG_DAMAGED cluster error.
The pg can be repaired but the sync never started back only bucket sync 
disable/enable helped.
I got OSD crash also in HKG BUT in ASH. ASH no any error, the replication speed 
is 2 millions objects in 6.5 hours which is like 90GB of data.

This is the crash of the osd:

{
"backtrace": [
"(()+0x12b20) [0x7f597d3fbb20]",
"(gsignal()+0x10f) [0x7f597c0667ff]",
"(abort()+0x127) [0x7f597c050c35]",
"(()+0x9009b) [0x7f597ca1c09b]",
"(()+0x9653c) [0x7f597ca2253c]",
"(()+0x96597) [0x7f597ca22597]",
"(()+0x967f8) [0x7f597ca227f8]",
"(()+0x19d24) [0x7f597e168d24]",
"(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) 
[0x7f597e18b0d6]",
"(rocksdb::Arena::AllocateNewBlock(unsigned long)+0x43) 
[0x5632f08ccb93]",
"(rocksdb::Arena::AllocateFallback(unsigned long, bool)+0x4b) 
[0x5632f08ccc3b]",
"(rocksdb::ConcurrentArena::AllocateAligned(unsigned long, unsigned 
long, rocksdb::Logger*)+0xb4) [0x5632f07fae94]",
"(()+0x1103e7e) [0x5632f085ae7e]",
"(rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, 
rocksdb::Slice const&, rocksdb::Slice const&, bool, 
rocksdb::MemTablePostProcessInfo*)+0xcf) [0x5632f07f6f8f]",
"(rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice 
const&, rocksdb::Slice const&, rocksdb::ValueType)+0x452) [0x5632f08520e2]",
"(rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&, 
rocksdb::Slice const&)+0x17) [0x5632f0852e97]",
"(rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) 
const+0x480) [0x5632f084ac20]",

"(rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&, 
unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, 
unsigned long, rocksdb::DB*, bool, bool, bool)+0x149) [0x5632f084ebe9]",
"(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, 
rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, 
bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1acd) 
[0x5632f078a03d]",
"(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, 
rocksdb::WriteBatch*)+0x21) [0x5632f078ac11]",
"(RocksDBStore::submit_common(rocksdb::WriteOptions&, 
std::shared_ptr)+0x8c) [0x5632f074180c]",

"(RocksDBStore::submit_transaction(std::shared_ptr)+0x87)
 [0x5632f0742027]",
"(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x426) 
[0x5632f0226376]",
"(BlueStore::_kv_sync_thread()+0x176f) [0x5632f024bc1f]",
"(BlueStore::KVSyncThread::entry()+0x11) [0x5632f0273791]",
"(()+0x814a) [0x7f597d3f114a]",
"(clone()+0x43) [0x7f597c12bf23]"
],
"ceph_version": "15.2.9",
"crash_id": 
"2021-03-04T14:55:45.094048Z_3d481fd3-7573-4cb7-9b22-20784b418e64",
"entity_name": "osd.5",
"os_id": "centos",
"os_name": "CentOS Linux",
"os_version": "8",
"os_version_id": "8",
"process_name": "ceph-osd",
"stack_sig": 
"9643c370a20c0d34f5e8965ae4461e2a7cf709ab4183929239bc263d0e1eef94",
"timestamp": "2021-03-04T14:55:45.094048Z",
"utsname_hostname": "hostname",
"utsname_machine": "x86_64",
"utsname_release": "4.18.0-240.10.1.el8_3.x86_64",
   "utsname_sysname": "Linux",
"utsname_version": "#1 SMP Mon Jan 18 17:05:51 UTC 2021"
}

Any idea what I should tune?

Thank you.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Questions RE: Ceph/CentOS/IBM

[Szabo, Istvan (Agoda)]

But are you using kernel 4 then with centos 7?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Marc 
Sent: Wednesday, March 3, 2021 11:40 PM
To: Alexander E. Patrakov ; Drew Weaver 

Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: Questions RE: Ceph/CentOS/IBM

Email received from outside the company. If in doubt don't click links nor open 
attachments!


> This is wrong. Ceph 15 runs on CentOS 7 just fine, but without the
> dashboard.
>

I also hope that ceph is keeping support for el7 till it is eol in 2024. So I 
have enough time to figure out what OS to choose.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Multiple radosgw on the same server

[Szabo, Istvan (Agoda)]

Hi,

I've heard many time that to install multiple rados-gateway on the same server 
is possible, just need to create on a different port.
However I've never managed to make it work.
Today I gave another try like this:


  1.  Created a new keyring: ceph auth get-or-create client.rgw.servername.rgw1 
mon 'allow rw' osd 'allow rwx'
  2.  Created the keyring file: 
/var/lib/ceph/radosgw/ceph-rgw.servername.rgw1/keyring
  3.  Added another entry in the ceph octopus configuration with different port:
[client.rgw.servername.rgw1]

host = servername

keyring = /var/lib/ceph/radosgw/ceph-rgw.servername.rgw1/keyring

log file = /var/log/ceph/ceph-rgw-servername.rgw1.log

rgw frontends = beast endpoint=10.104.198.101:8081

rgw thread pool size = 512

rgw_zone=zone

  1.  Copied another RGW system file in centos 8: cp -pr 
/etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw\@rgw.servername.rgw0.service
 
/etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw\@rgw.servername.rgw1.service
  2.  Restarted ceph.target.
  3.  Result is the same number of rados-gateways.

So how is this actually should have been done?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] RBD clone on Bluestore

[Szabo, Istvan (Agoda)]

I might be wrong but the object size is 4M in ceph so this might be the case 
also. But yeah, on the lower level of data storing I'm not familiar.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Pawel S 
Sent: Tuesday, March 2, 2021 1:34 AM
To: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] RBD clone on Bluestore

Email received from outside the company. If in doubt don't click links nor open 
attachments!


hello!

I'm trying to understand how Bluestore cooperates with RBD image clones, so my 
test is simple

1. create an image (2G) and fill with data 2. create a snapshot 3. protect it 
4. create a clone of the image 5. write a small portion of data (4K) to clone 
6. check how it changed and if just 4K are used to prove CoW allocated new 
extent instead of copying out snapped data.

Unfortunately it occurs that at least rbd du reports that 4M was changed and 
the clone consumes 4M of data instead of expected 4K...
'''
rbd du rbd/clone1
NAME PROVISIONED USED
clone1 2 GiB 4 MiB
'''

How can I trace/prove Bluestore CoW really works in this case, and prevent 
copying the rest of the 4M stripe like Filestore did ?

p.s tested on Luminous/Octopus, ssd devices, min_alloc_size: 16k,
block_size: 4k

best regards!
--
Pawel S.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data

[Szabo, Istvan (Agoda)]

So-so. I had some interruption so it failed on one site, but the other is kind 
of working. This is the first time when I saw data caught up in the 
radosgw-admin data sync status on 1 side.
Today will finish the other problematic site, I’ll let you know the result is 
it working or not.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

From: 特木勒 
Sent: Sunday, February 28, 2021 1:34 PM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users@ceph.io
Subject: Re: [Suspicious newsletter] [ceph-users] RGW: Multiple Site does not 
sync olds data

Email received from outside the company. If in doubt don't click links nor open 
attachments!

Hi Istvan:

Thanks for your reply.

Does directional sync solve the problem? I tried to run `radosgw-admin sync 
init`, bit it still did not work. :(

Thanks

Szabo, Istvan (Agoda) mailto:istvan.sz...@agoda.com>> 
于2021年2月26日周五 上午7:47写道：
Same for me, 15.2.8 also.
I’m trying directional sync now, looks like symmetrical has issue.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: 
istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com><mailto:istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>>
---

On 2021. Feb 26., at 1:03, 特木勒 mailto:twl...@gmail.com>> 
wrote:

Email received from outside the company. If in doubt don't click links nor 
open attachments!


Hi all:

ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8)

I have a strange question, I just create a multiple site for Ceph cluster.
But I notice the old data of source cluster is not synced. Only new data
will be synced into second zone cluster.

Is there anything I need to do to enable full sync for bucket or this is a
bug?

Thanks
___
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to 
ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data

[Szabo, Istvan (Agoda)]

Same for me, 15.2.8 also.
I’m trying directional sync now, looks like symmetrical has issue.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2021. Feb 26., at 1:03, 特木勒  wrote:

Email received from outside the company. If in doubt don't click links nor 
open attachments!


Hi all:

ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8)

I have a strange question, I just create a multiple site for Ceph cluster.
But I notice the old data of source cluster is not synced. Only new data
will be synced into second zone cluster.

Is there anything I need to do to enable full sync for bucket or this is a
bug?

Thanks
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Erasure coded calculation

[Szabo, Istvan (Agoda)]

109 is 81.75 yes the rest of them some bluestore stuffs I guess.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Simon Sutter 
Sent: Thursday, February 25, 2021 5:55 PM
To: ceph-users@ceph.io
Subject: [ceph-users] Erasure coded calculation

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hello everyone!

I'm trying to calculate the theoretical usable storage of a ceph cluster with 
erasure coded pools.

I have 8 nodes and the profile for all data pools will be k=6 m=2.
If every node has 6 x 1TB wouldn't the calculation be like this:
RAW capacity: 8Nodes x 6Disks x 1TB = 48TB Loss to m=2: 48TB / 8Nodes x 2m = 
12TB EC capacity: 48TB - 12TB = 36TB

At the moment I have one cluster with 8 nodes and different disks than the 
sample (but every node has the same amount of disks and the same sized disks).
The output of ceph df detail is:
--- RAW STORAGE ---
CLASS  SIZE AVAILUSED RAW USED  %RAW USED
hdd109 TiB  103 TiB  5.8 TiB   5.9 TiB   5.41
TOTAL  109 TiB  103 TiB  5.8 TiB   5.9 TiB   5.41

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  %USED  MAX AVAIL
device_health_metrics   11   51 MiB   48  0 30 TiB
rep_data_fs 2   32   14 KiB3.41k  0 30 TiB
rep_meta_fs 3   32  227 MiB1.72k  0 30 TiB
ec_bkp14   32  4.2 TiB1.10M   6.11 67 TiB

So ec_bkp1 uses 4.2TiB an there are 67TiB free usable Storage.
This means total EC usable storage would be 71.2TiB.
But calculating with the 109TiB RAW storage, shouldn't it be  81.75?
Are the 10TiB just some overhead (that would be much overhead) or is the 
calculation not correct?

And what If I want to expand the cluster in the first sample above by three 
nodes with 6 x 2TB, which means not the same sized disks as the others.
Will the calculation with the same EC profile still be the same?
RAW capacity: 8Nodes x 6Disks x 1TB + 3Nodes x 6Disks x 2TB = 84TB Loss to m=2: 
84TB / 11Nodes x 2m = 15.27TB EC capacity: 84TB - 15.27TB = 68.72TB


Thanks in advance,
Simon
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: List number of buckets owned per user

[Szabo, Istvan (Agoda)]

Maybe this one?

radosgw-admin bucket list --uid=

And after linux commands, grep, wc -l.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Konstantin Shalygin 
Sent: Thursday, February 25, 2021 12:49 PM
To: Marcelo 
Cc: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: List number of buckets owned 
per user

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Or you can achieve users from bucket usage, consult with code of 
radosgw_usage_exporter [1] maybe if enough to just start exporter and work with 
data in Grafana


Cheers,
k

[1] https://github.com/blemmenes/radosgw_usage_exporter

> On 24 Feb 2021, at 16:08, Marcelo  wrote:
>
> I'm trying to list the number of buckets that users have for
> monitoring purposes, but I need to list and count the number of
> buckets per user. Is it possible to get this information somewhere else?

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Multisite sync shards cleanup

[Szabo, Istvan (Agoda)]

Hi,

Is there a way to cleanup the sync shards and start from scratch?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Multisite cluster sync latency

[Szabo, Istvan (Agoda)]

Hi Gents,

Can you tell me how much is your latency in your multisite cluster? Multisite 
should be latency sensitive and I scare this is my sync issue, but I don't 
really know what means "low latency".

Here is mine, I wonder is it good or not.

In HKG:
"data-sync-from-ash": {
  "fetch_bytes": {
"avgcount": 0,
"sum": 0
  },
  "fetch_not_modified": 425395,
  "fetch_errors": 1,
  "poll_latency": {
"avgcount": 890,
"sum": 47481.671537512,
"avgtime": 53.350192738
  },
  "poll_errors": 0
},
"data-sync-from-sin": {
  "fetch_bytes": {
"avgcount": 0,
"sum": 0
  },
  "fetch_not_modified": 484757,
  "fetch_errors": 0,
  "poll_latency": {
"avgcount": 21686,
"sum": 135649.750753768,
"avgtime": 6.255176185
  },
  "poll_errors": 3

In ASH:
"data-sync-from-hkg": {
  "fetch_bytes": {
"avgcount": 7904,
"sum": 497898243
  },
  "fetch_not_modified": 7383973,
  "fetch_errors": 654,
  "poll_latency": {
"avgcount": 6586,
"sum": 2568055.690045521,
"avgtime": 389.926463717
  },
  "poll_errors": 3
},
"data-sync-from-sin": {
  "fetch_bytes": {
"avgcount": 13362,
"sum": 800114616
  },
  "fetch_not_modified": 7326406,
  "fetch_errors": 558,
  "poll_latency": {
"avgcount": 10137,
"sum": 3145053.032619919,
"avgtime": 310.254812333
  },
  "poll_errors": 5
},

In SGP:
"data-sync-from-ash": {
  "fetch_bytes": {
"avgcount": 0,
"sum": 0
  },
  "fetch_not_modified": 2057839,
  "fetch_errors": 1,
  "poll_latency": {
"avgcount": 8874,
"sum": 682176.718044618,
"avgtime": 76.873644133
  },
  "poll_errors": 0
},
"data-sync-from-hkg": {
  "fetch_bytes": {
"avgcount": 114,
"sum": 1097512
  },
  "fetch_not_modified": 1939198,
  "fetch_errors": 823,
  "poll_latency": {
"avgcount": 2123,
"sum": 60947.760976996,
"avgtime": 28.708318877
  },
  "poll_errors": 1


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Data sync init vs bucket sync init

[Szabo, Istvan (Agoda)]

Hi,

What’s the difference between data sync init and bucket sync init? Data 
initialise the complete cluster? Bucket only bucket?

I see when initialise finished, have shards behind but doesn’t do anything with 
it?

What is the proper steps to bring things back to sync?

Init
Run
Restart ??


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Data sync init vs bucket sync init

[Szabo, Istvan (Agoda)]

Hi,

What’s the difference between data sync init and bucket sync init? Data 
initialise the complete cluster? Bucket only bucket?

I see when initialise finished, have shards behind but doesn’t do anything with 
it?

What is the proper steps to bring things back to sync?

Init
Run
Restart


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Multisite reshard stale instances

[Szabo, Istvan (Agoda)]

Hi,

I found 6-700 stale instances with the reshard stale instances list command.
Is there a way to clean it up (or actually should I clean it up)?
The stale instance rm doesn't work in multisite.

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] log_meta log_data was turned off in multisite and deleted

[Szabo, Istvan (Agoda)]

Hi,

Is there a way to reinitialize the stored data and make it sync from the logs?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Bucket synchronization works only after disable/enable, once finished, some operation maxes out SSDs/nvmes and sync degraded.

[Szabo, Istvan (Agoda)]

Hello,

We have a 3 geo locational freshly installed multisite setup with an upgraded 
octopus from 15.2.5 to 15.2.7.
We have 6 osd nodes, 3 mon/mgr/rgw in each dc, full SSD, 3 ssd is using 1 nvme 
for journaling. Each zone backed with 3 RGW, one on each mon/mgr node.
The goal is to replicate 2 (currently) big buckets in the zonegroup but it only 
works if I disable and reenable the bucket sync.
Big buckets means, one bucket is presharded for 9000 shards (9 billions 
objects), the 2nd bucket that I'm detailing in this ticket 24000 (24 billions 
objects) shards.

Once picked up the objects (not all, only the ones that was on the source site 
at that given time when it was enabled) it will slows down a lot from 100.000 
objects / 15 minutes in and 10GB/15 minutes to 50 objects/4 hours.
Once it synchronized after enabled/disabled, it maxing out the osd nodes with 
NVME/SSD drives with some operation which I don't know what is it. Let me show 
you the symptoms below.

Let me summarize as much as I can.

We have 1 realm, in this realm we have 1 zonegroup (please help me to check if 
the sync policies are ok) and in this zonegroup we have 1 cluster in US, 1 in 
Hong Kong (master) and 1 in Singapore.

Here is the realm, zonegroup and zones definition: 
https://pastebin.com/raw/pu66tqcf

Let me show you one enable/disable operation when I've disabled on the HKG 
master site the pix-bucket and enabled it.

In this screenshot: https://i.ibb.co/WNC0gNQ/6nodes6day.png
the highlighted area is when the data sync is running after disable enable. You 
can see almost no operation. You can see also when sync is not running, the 
green and yellow is the NVME drive rocksdb+wal drives. The screenshot 
represents the 6 Singapore nodes SSD/NVME disk utilizations. The first node you 
can see in the last hours no green and yellow, it's because I've reinstalled in 
that nodes all the osds to not use NVME.

In the following 1st screenshot you can see the HKG object usage where the user 
is uploading the objects. 2nd screenshot the SGP one where you can see the 
highlighted area is the disable/enable operation.
HKG where user upload: https://i.ibb.co/vj2VFYP/pixhkg6d.png
SGP where sync happened: https://i.ibb.co/w41rmQT/pixsgp6d.png

Let me show you some troubleshooting logs regarding bucket sync status, cluster 
sync status, reshard list (which might be because of previous testing), sync 
error list

https://pastebin.com/raw/TdwiZFC1

The issue might be very similar to this issue:
https://tracker.ceph.com/issues/21591

Where I should move forward or how can I help you to provide more logs to help 
me please?

Thank you in advance


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: Multisite recovering shards

[Szabo, Istvan (Agoda)]

Sorry for the late response and thank you to pickup my question.
I wanted to create some detailed information, here it is, please have a look 
and if you could help me I'd very appreciate it.

https://tracker.ceph.com/issues/49075


-Original Message-
From: Eugen Block 
Sent: Monday, February 1, 2021 4:59 PM
To: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: Multisite recovering shards

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

> We are using octopus 15.2.7 for bucket sync with symmetrical replication.

replication is asynchronous with both CephFS and RGW, so if your clients keep 
writing new data into the cluster as you state the sync status will always stay 
behind a little bit. I have two one-node test clusters with no client traffic 
where the sync status is actually
up-to-date:

siteb:~ # radosgw-admin sync status
   realm c7d5fd30-9c06-46a1-baf4-497f95bf3abc (masterrealm)
   zonegroup 68adec15-aace-403d-bd63-f5182a6437b1 (master-zonegroup)
zone 69329911-c3b0-48c3-a359-7f6214e0480c (siteb-zone)
   metadata sync syncing
 full sync: 0/64 shards
 incremental sync: 64/64 shards
 metadata is caught up with master
   data sync source: 0fb33fa1-8110-4179-ae45-acf5f5f825c5 (sitea-zone)
 syncing
 full sync: 0/128 shards
 incremental sync: 128/128 shards
 data is caught up with source



Zitat von "Szabo, Istvan (Agoda)" :

> Hi,
>
> I’ve never seen in our multisite sync status healthy output, almost
> all the sync shards are recovering.
>
> What can I do with recovering shards?
>
> We have 1 realm, 1 zonegroup and inside the zonegroup we have 3 zones
> in 3 different geo location.
>
> We are using octopus 15.2.7 for bucket sync with symmetrical replication.
>
> The user is at the moment migrating their data and the sites are
> always behind which is replicated from the place where it was
> uploaded.
>
> I’ve restarted all rgw and disable / enable bucket sync, it started to
> work, but I think when it comes to close sync it will stop again due
> to the recovering shards.
>
> Any idea?
>
> Thank you
>
> 
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by
> copyright or other legal rules. If you have received it by mistake
> please let us know by reply email and delete it from your system. It
> is prohibited to copy this message or disclose its content to anyone.
> Any confidentiality or privilege is not waived or lost by any mistaken
> delivery or unauthorized disclosure of the message. All messages sent
> to and from Agoda may be monitored to ensure compliance with company
> policies, to protect the company's interests and to remove potential
> malware. Electronic messages may be intercepted, amended, lost or
> deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Multisite recovering shards

[Szabo, Istvan (Agoda)]

2 things I forgot to mention which might be interesting, we have only 2 bucket 
at the moment, one is presharded to 9000 shards, the other presharded to 24000 
shards (different users)

> On 2021. Jan 30., at 10:02, Szabo, Istvan (Agoda)  
> wrote:
>
> Hi,
>
> I’ve never seen in our multisite sync status healthy output, almost all the 
> sync shards are recovering.
>
> What can I do with recovering shards?
>
> We have 1 realm, 1 zonegroup and inside the zonegroup we have 3 zones in 3 
> different geo location.
>
> We are using octopus 15.2.7 for bucket sync with symmetrical replication.
>
> The user is at the moment migrating their data and the sites are always 
> behind which is replicated from the place where it was uploaded.
>
> I’ve restarted all rgw and disable / enable bucket sync, it started to work, 
> but I think when it comes to close sync it will stop again due to the 
> recovering shards.
>
> Any idea?
>
> Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Multisite recovering shards

[Szabo, Istvan (Agoda)]

Hi,

I’ve never seen in our multisite sync status healthy output, almost all the 
sync shards are recovering.

What can I do with recovering shards?

We have 1 realm, 1 zonegroup and inside the zonegroup we have 3 zones in 3 
different geo location.

We are using octopus 15.2.7 for bucket sync with symmetrical replication.

The user is at the moment migrating their data and the sites are always behind 
which is replicated from the place where it was uploaded.

I’ve restarted all rgw and disable / enable bucket sync, it started to work, 
but I think when it comes to close sync it will stop again due to the 
recovering shards.

Any idea?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: Rbd pool shows 458GB USED but the image is empty

[Szabo, Istvan (Agoda)]

Does this mean the space is allocated, but actually empty so can let’s say 
overwrite?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

On 2021. Jan 28., at 19:51, Burkhard Linke 
 wrote:

Email received from outside the company. If in doubt don't click links nor 
open attachments!


Hi,

On 28.01.21 13:21, Szabo, Istvan (Agoda) wrote:
I mean the image hasn’t been deleted, but the content from the image.


RBD is (as the name implies) is a block device layer. Block devices do
not have a concept of content, file, directories or even allocated or
unallocated space. They are just continuous byte streams with a certain
size.


The _filesystem_ on top of the block device implements these concepts,
and also tracks free / unallocated space. And a filesystem is able to
inform the block layer about allocations using TRIM and DISCARD calls
(e.g. tools like fstrim or corresponding mount options). RBD usually
supports trim operation in recent kernels (krbd) or with certain
configuration options (librbd)


Regards,

Burkhard

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: Rbd pool shows 458GB USED but the image is empty

[Szabo, Istvan (Agoda)]

I mean the image hasn’t been deleted, but the content from the image.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

On 2021. Jan 28., at 18:21, Eugen Block  wrote:

Email received from outside the company. If in doubt don't click links nor 
open attachments!


The image is probably still in the trash, I assume.

rbd -p  trash ls

Zitat von "Szabo, Istvan (Agoda)" :

Hi,

We have a pool where the user has 2 image.
They cleaned up the images, no snaphot in it, but when I see ceph df
detail it still shows 458GB in the first column.
Why?

Thanks


This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to
anyone. Any confidentiality or privilege is not waived or lost by
any mistaken delivery or unauthorized disclosure of the message. All
messages sent to and from Agoda may be monitored to ensure
compliance with company policies, to protect the company's interests
and to remove potential malware. Electronic messages may be
intercepted, amended, lost or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Rbd pool shows 458GB USED but the image is empty

[Szabo, Istvan (Agoda)]

Hi,

We have a pool where the user has 2 image.
They cleaned up the images, no snaphot in it, but when I see ceph df detail it 
still shows 458GB in the first column.
Why?

Thanks


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Running ceph cluster on different os

[Szabo, Istvan (Agoda)]

Hi,

Is there anybody running a cluster with different os?
Due to the centos 8 change I might try to add ubuntu osd nodes to centos 
cluster and decommission the centos slowly but I'm not sure this is possible or 
not.

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Multisite bucket data inconsistency

[Szabo, Istvan (Agoda)]

Debugging a bit more it shows in all sites many stale instances which can't be 
removed due to multisite limitation ☹ in octopus 15.2.7.




-Original Message-
From: Szabo, Istvan (Agoda)  
Sent: Monday, January 25, 2021 11:51 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: Multisite bucket data inconsistency

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hmm,

Looks like attached screenshots not allowed, so in HKG we have 19 millions 
objects, in ash we have 32millions.


-Original Message-
From: Szabo, Istvan (Agoda) 
Sent: Monday, January 25, 2021 11:44 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Multisite bucket data inconsistency

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

We have bucket sync enabled and seems like it is inconsistent ☹

This is the master zone sync status on that specific bucket:
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
  metadata sync no sync (zone is master)
  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 126 shards
behind shards: 
[0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-25T11:32:57.726042+0700 [62]
104 shards are recovering
recovering shards: 
[0,2,3,4,5,7,8,9,10,11,12,13,15,16,17,18,19,20,21,22,24,25,26,27,28,29,31,32,33,36,37,38,39,40,42,43,44,45,47,50,51,52,53,54,55,57,58,61,63,65,66,67,68,69,70,71,72,73,74,75,76,78,80,81,82,83,84,85,87,88,90,92,93,95,96,97,98,99,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,123,124,125,126,127]

This is the secondary zone where the data has been uploaded:
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
  metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 125 shards
behind shards: 
[0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,18,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-25T11:29:32.450031+0700 [61]
126 shards are recovering
recovering shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,104,105,106,107,108,109,110,111,112,114,115,116,117,118,119,120,121,122,123,124,125,126,127]

The pipes are already there:
"id": "seo-2",
"data_flow": {
  "symmetrical": [
{
  "id": "seo-2-flow",
  "zones": [
"9213182a-14ba-48a

[ceph-users] Re: Multisite bucket data inconsistency

[Szabo, Istvan (Agoda)]

Hmm,

Looks like attached screenshots not allowed, so in HKG we have 19 millions 
objects, in ash we have 32millions.


-Original Message-
From: Szabo, Istvan (Agoda)  
Sent: Monday, January 25, 2021 11:44 AM
To: ceph-users@ceph.io
Subject: [ceph-users] Multisite bucket data inconsistency

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

We have bucket sync enabled and seems like it is inconsistent ☹

This is the master zone sync status on that specific bucket:
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
  metadata sync no sync (zone is master)
  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 126 shards
behind shards: 
[0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-25T11:32:57.726042+0700 [62]
104 shards are recovering
recovering shards: 
[0,2,3,4,5,7,8,9,10,11,12,13,15,16,17,18,19,20,21,22,24,25,26,27,28,29,31,32,33,36,37,38,39,40,42,43,44,45,47,50,51,52,53,54,55,57,58,61,63,65,66,67,68,69,70,71,72,73,74,75,76,78,80,81,82,83,84,85,87,88,90,92,93,95,96,97,98,99,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,123,124,125,126,127]

This is the secondary zone where the data has been uploaded:
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
  metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 125 shards
behind shards: 
[0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,18,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-25T11:29:32.450031+0700 [61]
126 shards are recovering
recovering shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,104,105,106,107,108,109,110,111,112,114,115,116,117,118,119,120,121,122,123,124,125,126,127]

The pipes are already there:
"id": "seo-2",
"data_flow": {
  "symmetrical": [
{
  "id": "seo-2-flow",
  "zones": [
"9213182a-14ba-48ad-bde9-289a1c0c0de8",
"f20ddd64-924b-4f78-8d2d-dd6c65f98ba9"
  ]
}
  ]
},
"pipes": [
  {
"id": "seo-2-hkg-ash-pipe",
"source": {
  "bucket": "seo..prerender",
  "zones": [
"9213182a-14ba-48ad-bde9-289a1c0c0de8"
  ]
},
"dest&q

[ceph-users] Multisite bucket data inconsistency

[Szabo, Istvan (Agoda)]

Hi,

We have bucket sync enabled and seems like it is inconsistent ☹

This is the master zone sync status on that specific bucket:
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
  metadata sync no sync (zone is master)
  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 126 shards
behind shards: 
[0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-25T11:32:57.726042+0700 [62]
104 shards are recovering
recovering shards: 
[0,2,3,4,5,7,8,9,10,11,12,13,15,16,17,18,19,20,21,22,24,25,26,27,28,29,31,32,33,36,37,38,39,40,42,43,44,45,47,50,51,52,53,54,55,57,58,61,63,65,66,67,68,69,70,71,72,73,74,75,76,78,80,81,82,83,84,85,87,88,90,92,93,95,96,97,98,99,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,123,124,125,126,127]

This is the secondary zone where the data has been uploaded:
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
  metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source
source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 125 shards
behind shards: 
[0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,18,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-25T11:29:32.450031+0700 [61]
126 shards are recovering
recovering shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,104,105,106,107,108,109,110,111,112,114,115,116,117,118,119,120,121,122,123,124,125,126,127]

The pipes are already there:
"id": "seo-2",
"data_flow": {
  "symmetrical": [
{
  "id": "seo-2-flow",
  "zones": [
"9213182a-14ba-48ad-bde9-289a1c0c0de8",
"f20ddd64-924b-4f78-8d2d-dd6c65f98ba9"
  ]
}
  ]
},
"pipes": [
  {
"id": "seo-2-hkg-ash-pipe",
"source": {
  "bucket": "seo..prerender",
  "zones": [
"9213182a-14ba-48ad-bde9-289a1c0c0de8"
  ]
},
"dest": {
  "bucket": "seo..prerender",
  "zones": [
"f20ddd64-924b-4f78-8d2d-dd6c65f98ba9"
  ]
},
"params": {
  "source": {
"filter": {
  "tags": []
}
  },
  "dest": {},
  "priority": 0,
  "mode": "system",
  "user": ""
}
  },
  {
"id": "seo-2-ash-hkg-pipe",
"source": {
  "bucket": "seo..prerender",
  "zones": [

[ceph-users] Re: [Suspicious newsletter] radosgw-admin realm pull from the secondary site fails "(13) Permission denied"

[Szabo, Istvan (Agoda)]

Hi,

If you are using proxy, try to disable it.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

On 2021. Jan 23., at 1:45, Hayashida, Mami  wrote:

Email received from outside the company. If in doubt don't click links nor 
open attachments!


I have been trying to create two virtual test clusters to learn about the
RGW multisite setting.  So far, I have set up two small Nautilus
(v.14.2.16) clusters, designated one of them as the "master zone site" and
followed every step outlined in the doc (
https://docs.ceph.com/en/nautilus/radosgw/multisite/), including create a
system user, updating the period, and restarting the rgw daemon.  (For the
sake of simplicity, there is only one RGW daemon running on each site.)

Once I installed the RGW daemon on the secondary zone site, I tried pulling
the realm from the master zone cluster, but ended up with this:

```
$ radosgw-admin realm pull --url=http://:80
--access-key=  --secret=
request failed: (13) Permission denied
If the realm has been changed on the master zone, the master zone's gateway
may need to be restarted to recognize this user.
```
I tried adding the --rgw-realm=, but the
result was the same.   I restarted the rgw daemon on both sides -- that did
not help, either.

The output of all of the following on the master zone side, as far as I
could tell, seems correct -- the realm, zonegroup, zone I created are the
only ones and set to default.
```
radosgw-admin zone/zonegroup/realm list
radosgw-admin zone/zonegroup/realm get
```

On the "master zone" side, the rgw log shows
```
2021-01-22 13:34:48.404 7fb9ca89e700  1 == starting new request
req=0x7fb9ca897740 =
2021-01-22 13:34:48.428 7fb9ca89e700  1 == req done req=0x7fb9ca897740
op status=0 http_status=403 latency=0.0240002s ==
2021-01-22 13:34:48.428 7fb9ca89e700  1 civetweb: 0x559d6509a000:
10.33.30.55 - - [22/Jan/2021:13:34:48 -0500] "GET /admin/realm HTTP/1.1"
403 318 - -
```

I am using Ubuntu 18.04, Ceph v.14.2.16, deployed using `ceph-deploy`.

*Mami Hayashida*
*Research Computing Associate*
Univ. of Kentucky ITS Research Computing Infrastructure
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RBD on windows

[Szabo, Istvan (Agoda)]

Hi,

I'm looking the suse documentation regarding their option to have rbd on win.
I want to try on windows server 2019 vm, but I got this error:

PS C:\Users\$admin$> rbd create image01 --size 4096 --pool windowstest -m 
10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring 
C:/ProgramData/ceph/keyring
2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file 
C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input
2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load 
C:/ProgramData/ceph/keyring: (5) Input/output error
2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file 
C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input
2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load 
C:/ProgramData/ceph/keyring: (5) Input/output error
2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: error parsing file 
C:/ProgramData/ceph/keyring: cannot parse buffer: Malformed input
rbd: couldn't connect to the cluster!
2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 auth: failed to load 
C:/ProgramData/ceph/keyring: (5) Input/output error
2021-01-20T11:15:29.066SE Asia Standard Time 1 -1 monclient: keyring not found

This is the keyring file:

[client.windowstest]
key = AQBJ7wdgdWLIMhAAle+/pg+26XvWsDv8PyPcvw==
caps mon = "allow rw"
caps osd = "allow rwx pool=windowstest"

And this is the ceph.conf file on the windows client:
[global]
 log to stderr = true
 run dir = C:/ProgramData/ceph
 crash dir = C:/ProgramData/ceph
[client]
 keyring = C:/ProgramData/ceph/keyring
 log file = C:/ProgramData/ceph/$name.$pid.log
 admin socket = C:/ProgramData/ceph/$name.$pid.asok
[global]
 mon host = [v2:10.118.199.231:3300,v1:10.118.199.231:6789] 
[v2:10.118.199.232:3300,v1:10.118.199.232:6789] 
[v2:10.118.199.233:3300,v1:10.118.199.233:6789]

Commands I've tried:
rbd create image01 --size 4096 --pool windowstest -m 
10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring 
C:/ProgramData/ceph/keyring
rbd create image01 --size 4096 --pool windowstest -m 
10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring 
C:\ProgramData\ceph\keyring
rbd create image01 --size 4096 --pool windowstest -m 
10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring 
"C:/ProgramData/ceph/keyring"
rbd create image01 --size 4096 --pool windowstest -m 
10.118.199.248,10.118.199.249,10.118.199.250 --id windowstest --keyring 
"C:\ProgramData\ceph\keyring"
rbd create blank_image --size=1G

The ceph version is luminous 12.2.8.

Don't know why they don't find mon keyring.

Thank you.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: .rgw.root was created wit a lot of PG

[Szabo, Istvan (Agoda)]

Luminous

I've done actually, now monitoring is there any issue, if not, I'll remove the 
old pool.

-Original Message-
From: Eugen Block 
Sent: Friday, January 15, 2021 3:47 PM
To: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: .rgw.root was created wit a 
lot of PG

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Which ceph version is this? Since Nautilus you can decrease pg numbers (or let 
pg-autoscaler do that for you).


Zitat von "Szabo, Istvan (Agoda)" :

> Hi,
>
> Originally this pool was created with 512PG which makes couple of OSDs
> having 500PG 
> What is the safe steps to copy over this pool?
>
> These are the files in this pool:
> default.realm
> period_config.f320e60d-8cff-4824-878e-c316423cc519
> periods.18d63a25-8a50-4e17-9561-d452621f62fa.latest_epoch
> default.zonegroup.f320e60d-8cff-4824-878e-c316423cc519
> zone_info.ba16656f-2191-40bb-bc39-9f19448d215d
> periods.6605bb4c-2226-4509-a3be-d5c95300fe14.1
> default.zone.f320e60d-8cff-4824-878e-c316423cc519
> zonegroup_info.f47a81ba-b214-4b8d-9b0e-84c14bc153cf
> periods.f320e60d-8cff-4824-878e-c316423cc519:staging
> realms.f320e60d-8cff-4824-878e-c316423cc519.control
> periods.6605bb4c-2226-4509-a3be-d5c95300fe14.latest_epoch
> realms.f320e60d-8cff-4824-878e-c316423cc519
> realms_names.default
> periods.18d63a25-8a50-4e17-9561-d452621f62fa.1
> zone_names.default
> zonegroups_names.default
>
> It’s single site cluster.
>
> Do I need to stop rados gateway?
>
> After I would follow the following steps:
>
> ceph osd pool create .rgw.root.new 8
> rados cppool .rgw.root .rgw.root.new
> ceph osd pool delete .rgw.root .rgw.root --yes-i-really-really-mean-it
> ceph osd pool rename .rgw.root.new .rgw.root ceph osd pool application
> enable .rgw.root rgw
>
> Just not sure do I need to create outage request or not due to the
> rados stop.
>
> Thank you.
>
>
> 
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by
> copyright or other legal rules. If you have received it by mistake
> please let us know by reply email and delete it from your system. It
> is prohibited to copy this message or disclose its content to anyone.
> Any confidentiality or privilege is not waived or lost by any mistaken
> delivery or unauthorized disclosure of the message. All messages sent
> to and from Agoda may be monitored to ensure compliance with company
> policies, to protect the company's interests and to remove potential
> malware. Electronic messages may be intercepted, amended, lost or
> deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] .rgw.root was created wit a lot of PG

[Szabo, Istvan (Agoda)]

Hi,

Originally this pool was created with 512PG which makes couple of OSDs having 
500PG 
What is the safe steps to copy over this pool?

These are the files in this pool:
default.realm
period_config.f320e60d-8cff-4824-878e-c316423cc519
periods.18d63a25-8a50-4e17-9561-d452621f62fa.latest_epoch
default.zonegroup.f320e60d-8cff-4824-878e-c316423cc519
zone_info.ba16656f-2191-40bb-bc39-9f19448d215d
periods.6605bb4c-2226-4509-a3be-d5c95300fe14.1
default.zone.f320e60d-8cff-4824-878e-c316423cc519
zonegroup_info.f47a81ba-b214-4b8d-9b0e-84c14bc153cf
periods.f320e60d-8cff-4824-878e-c316423cc519:staging
realms.f320e60d-8cff-4824-878e-c316423cc519.control
periods.6605bb4c-2226-4509-a3be-d5c95300fe14.latest_epoch
realms.f320e60d-8cff-4824-878e-c316423cc519
realms_names.default
periods.18d63a25-8a50-4e17-9561-d452621f62fa.1
zone_names.default
zonegroups_names.default

It’s single site cluster.

Do I need to stop rados gateway?

After I would follow the following steps:

ceph osd pool create .rgw.root.new 8
rados cppool .rgw.root .rgw.root.new
ceph osd pool delete .rgw.root .rgw.root --yes-i-really-really-mean-it
ceph osd pool rename .rgw.root.new .rgw.root
ceph osd pool application enable .rgw.root rgw

Just not sure do I need to create outage request or not due to the rados stop.

Thank you.



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw-admin sync status takes ages to print output

[Szabo, Istvan (Agoda)]

Hello,

I see, do you know how much I should increase and how? Haven’t found much 
documentation about it :(


> On 2021. Jan 14., at 22:36, dhils...@performair.com wrote:
> 
> Email received from outside the company. If in doubt don't click links nor 
> open attachments!
> 
> 
> Istvan;
> 
> What version of Ceph are you running?  Another email chain indicates you're 
> running on CentOS 8, which suggests Octopus (15).
> 
> We're running multisite replicated radosgw on Nautilus.  I don't see the long 
> running time that you are suggesting, though we only have ~35k objects.
> 
> I generally don't worry about sync unless the "oldest incremental change not 
> applied" is several minutes or more in the past.  Our work day has just 
> started, so use isn't very high yet.  This afternoon, when anticipated use 
> peaks, I'll set a watch to see how behind the clusters get.
> 
> According to the command output, you have 64 shards in the metadata, and 128 
> shards in the data.  That seems low, as that's the same number of shards 
> we're running, with our significantly lower object count.
> 
> Thank you,
> 
> Dominic L. Hilsbos, MBA
> Director – Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
> 
> 
> -Original Message-
> From: Szabo, Istvan (Agoda) [mailto:istvan.sz...@agoda.com]
> Sent: Wednesday, January 13, 2021 11:18 PM
> To: ceph-users@ceph.io
> Subject: [ceph-users] Re: radosgw-admin sync status takes ages to print output
> 
> UPDATE: Finally got back the master sync command output:
> 
> radosgw-admin sync status
>  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 realm)
>  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
>   zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
>  metadata sync no sync (zone is master)
>  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
>syncing
>full sync: 0/128 shards
>incremental sync: 128/128 shards
>source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
>syncing
>full sync: 0/128 shards
>incremental sync: 128/128 shards
>data is behind on 128 shards
>behind shards: 
> [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
>oldest incremental change not applied: 
> 2021-01-14T13:03:17.807529+0700 [20]
>45 shards are recovering
>recovering shards: 
> [5,14,23,25,26,34,36,37,38,45,46,47,49,50,51,52,54,55,57,58,60,61,62,67,68,69,71,77,79,80,88,89,90,95,97,100,108,110,111,117,118,120,121,125,126]
> 
> Sorry for the 2 email.
> 
> 
> -Original Message-
> From: Szabo, Istvan (Agoda) 
> Sent: Thursday, January 14, 2021 12:57 PM
> To: ceph-users@ceph.io
> Subject: [ceph-users] radosgw-admin sync status takes ages to print output
> 
> Email received from outside the company. If in doubt don't click links nor 
> open attachments!
> 
> 
> Hello,
> 
> I have a 3 DC octopus Multisite setup with bucket sync policy applied.
> 
> I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 
> because they want to use 1 bucket but with a huge amount of objects 
> (2.400.000.000 and 900.000.000) and in case of multisite we need to preshard 
> the buckets as it is in the documentation.
> 
> Do I need to fine tune something on the syncing to make this query faster?
> This is the output after 5-10 minutes query time not sure is it healthy or 
> good or not to be honest, haven’t really find any good explanation about the 
> output in the ceph documentation.
> 
> From the master zone I can’r reallt even query because timed out, but in 
> secondary zone can see this:
> 
> 
> radosgw-admin sync status
>  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
>  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
>   zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
>  metadata sync syncing
>full sync: 0/64 shards
>incremental sync: 64/64 shards
>metadata is caught up

[ceph-users] Re: [Suspicious newsletter] Re: Centos 8 2021 with ceph, how to move forward?

[Szabo, Istvan (Agoda)]

Thank you guys, so we might should use ububtu based then as it has good driver 
support and the lts sounds like a working solution.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com>
---

On 2021. Jan 14., at 19:50, Martin Verges  wrote:

Email received from outside the company. If in doubt don't click links nor 
open attachments!


Hello,

we from croit use Ceph on Debian and deploy all our clusters with it.
It works like a charm and I personally have quite good experience with
it since ~20 years. It is a fantastic solid OS for Servers.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx

Am Do., 14. Jan. 2021 um 11:12 Uhr schrieb David Majchrzak, ODERLAND
Webbhotell AB :

One of our providers (cloudlinux)  released a 1:1 binary compatible
redhat fork due to the changes with Centos 8.

Could be worth looking at.

https://almalinux.org/

In our case we're using ceph on debian 10.

--

David Majchrzak
CTO
Oderland Webbhotell AB
Östra Hamngatan 50B, 411 09 Göteborg, SWEDEN

Den 2021-01-14 kl. 09:04, skrev Szabo, Istvan (Agoda):
Hi,

Just curious how you guys move forward with this Centos 8 change.

We just finished installing our full multisite cluster and looks like we need 
to change the operating system.

So curious if you are using centos 8 with ceph, where you are going to move 
forward.

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Centos 8 2021 with ceph, how to move forward?

[Szabo, Istvan (Agoda)]

Hi,

Just curious how you guys move forward with this Centos 8 change.

We just finished installing our full multisite cluster and looks like we need 
to change the operating system.

So curious if you are using centos 8 with ceph, where you are going to move 
forward.

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw-admin sync status takes ages to print output

[Szabo, Istvan (Agoda)]

UPDATE: Finally got back the master sync command output:

radosgw-admin sync status
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
  metadata sync no sync (zone is master)
  data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 128 shards
behind shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-14T13:03:17.807529+0700 [20]
45 shards are recovering
recovering shards: 
[5,14,23,25,26,34,36,37,38,45,46,47,49,50,51,52,54,55,57,58,60,61,62,67,68,69,71,77,79,80,88,89,90,95,97,100,108,110,111,117,118,120,121,125,126]

Sorry for the 2 email.


-Original Message-
From: Szabo, Istvan (Agoda)  
Sent: Thursday, January 14, 2021 12:57 PM
To: ceph-users@ceph.io
Subject: [ceph-users] radosgw-admin sync status takes ages to print output

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hello,

I have a 3 DC octopus Multisite setup with bucket sync policy applied.

I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because 
they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 
900.000.000) and in case of multisite we need to preshard the buckets as it is 
in the documentation.

Do I need to fine tune something on the syncing to make this query faster?
This is the output after 5-10 minutes query time not sure is it healthy or good 
or not to be honest, haven’t really find any good explanation about the output 
in the ceph documentation.

From the master zone I can’r reallt even query because timed out, but in 
secondary zone can see this:


radosgw-admin sync status
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
  metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
  data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 128 shards
behind shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-14T12:01:00.131104+0700 [11]
source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 128 shards
behind shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-14T12:05:26.879014+0700 [98]



Hope I can find some expert in the multisite area 

Thank you in advance.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy

[ceph-users] radosgw-admin sync status takes ages to print output

[Szabo, Istvan (Agoda)]

Hello,

I have a 3 DC octopus Multisite setup with bucket sync policy applied.

I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because 
they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 
900.000.000) and in case of multisite we need to preshard the buckets as it is 
in the documentation.

Do I need to fine tune something on the syncing to make this query faster?
This is the output after 5-10 minutes query time not sure is it healthy or good 
or not to be honest, haven’t really find any good explanation about the output 
in the ceph documentation.

From the master zone I can’r reallt even query because timed out, but in 
secondary zone can see this:


radosgw-admin sync status
  realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm)
  zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data)
   zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin)
  metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
  data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 128 shards
behind shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-14T12:01:00.131104+0700 [11]
source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is behind on 128 shards
behind shards: 
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
oldest incremental change not applied: 
2021-01-14T12:05:26.879014+0700 [98]



Hope I can find some expert in the multisite area 

Thank you in advance.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Data migration between clusters

[Szabo, Istvan (Agoda)]

Hmmm, doesn’t seems smooth :(
How about copying bucket with s3 browser :D?

So the actual migration steps are the steps which went smoothly right?

> On 2020. Dec 22., at 20:00, Kalle Happonen  wrote:
>
> Email received from outside the company. If in doubt don't click links nor 
> open attachments!
> 
>
> Hi Istvan,
> I'm not sure it helps, but here's at least some pitfalls we faced when 
> migrating radosgws between clusters.
>
> https://cloud.blog.csc.fi/2019/12/ceph-object-storage-migraine-i-mean.html
>
> Cheers,
> Kalle
>
> - Original Message -
>> From: "Szabo, Istvan (Agoda)" 
>> To: "ceph-users" 
>> Sent: Thursday, 17 December, 2020 12:11:19
>> Subject: [ceph-users] Data migration between clusters
>
>> What is the easiest and best way to migrate bucket from an old cluster to a 
>> new
>> one?
>>
>> Luminous to octopus not sure does it matter from the data perspective.
>>
>> 
>> This message is confidential and is for the sole use of the intended
>> recipient(s). It may also be privileged or otherwise protected by copyright 
>> or
>> other legal rules. If you have received it by mistake please let us know by
>> reply email and delete it from your system. It is prohibited to copy this
>> message or disclose its content to anyone. Any confidentiality or privilege 
>> is
>> not waived or lost by any mistaken delivery or unauthorized disclosure of the
>> message. All messages sent to and from Agoda may be monitored to ensure
>> compliance with company policies, to protect the company's interests and to
>> remove potential malware. Electronic messages may be intercepted, amended, 
>> lost
>> or deleted, or contain viruses.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Data migration between clusters

[Szabo, Istvan (Agoda)]

What is the easiest and best way to migrate bucket from an old cluster to a new 
one?

Luminous to octopus not sure does it matter from the data perspective.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Weird ceph df

[Szabo, Istvan (Agoda)]

Hi,

It is a nautilus 14.2.13 ceph.

The quota on the pool is 745GiB, how can be the stored data 788GiB? (2 replicas 
pool).
Based on the used column it means just 334GiB is used because the pool has 2 
replicas only. I don't understand.

POOLS:
POOLID STORED  OBJECTS USED%USED 
MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY   USED COMPR 
UNDER COMPR
k8s-dbss-w-mdc  12 788 GiB 202.42k 668 GiB  0.75
43 TiB N/A   745 GiB 202.42k0 B 
0 B

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: PG_DAMAGED

[Szabo, Istvan (Agoda)]

This is a completely new cluster with full ssd and nvme :/


-Original Message-
From: Eugen Block 
Sent: Friday, December 4, 2020 4:32 PM
To: ceph-users@ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: PG_DAMAGED

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Hi,

this is not necessarily but most likely a hint to a (slowly) failing disk. 
Check all OSDs for this PG for disk errors in dmesg and smartctl.

Regards,
Eugen


Zitat von "Szabo, Istvan (Agoda)" :

> Hi,
>
> Not sure is it related to my 15.2.7 update, but today I got many time
> this issue:
>
> 2020-12-04T15:14:23.910799+0700 osd.40 (osd.40) 11 : cluster [DBG]
> 11.2 deep-scrub starts
> 2020-12-04T15:14:23.947255+0700 osd.40 (osd.40) 12 : cluster [ERR]
> 11.2 soid
> 11:434f049b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.372.1:head :
> omap_digest 0x48532c00 != omap_digest 0x8a18f5d7 from shard 40
> 2020-12-04T15:14:23.977138+0700 mgr.hk-cephmon-2s02 (mgr.2120884)
> 4330 : cluster [DBG] pgmap v4338: 209 pgs: 209 active+clean; 2.8 GiB
> data, 21 TiB used, 513 TiB / 534 TiB avail; 32 KiB/s rd, 32 op/s
> 2020-12-04T15:14:24.030888+0700 osd.40 (osd.40) 13 : cluster [ERR]
> 11.2 soid
> 11:4b86603b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.197.3:head :
> omap_digest 0xcb62779b != omap_digest 0xefef7471 from shard 40
> 2020-12-04T15:14:24.229000+0700 osd.40 (osd.40) 14 : cluster [ERR]
> 11.2 deep-scrub 0 missing, 2 inconsistent objects
> 2020-12-04T15:14:24.229003+0700 osd.40 (osd.40) 15 : cluster [ERR]
> 11.2 deep-scrub 2 errors
> 2020-12-04T15:14:25.978189+0700 mgr.hk-cephmon-2s02 (mgr.2120884)
> 4331 : cluster [DBG] pgmap v4339: 209 pgs: 1
> active+clean+scrubbing+deep, 208 active+clean; 2.8 GiB data, 21 TiB
> used, 513 TiB / 534 TiB avail; 55 KiB/s rd, 0 B/s wr, 61 op/s
> 2020-12-04T15:14:27.978588+0700 mgr.hk-cephmon-2s02 (mgr.2120884)
> 4332 : cluster [DBG] pgmap v4340: 209 pgs: 1
> active+clean+scrubbing+deep, 208 active+clean; 2.8 GiB data, 21 TiB
> used, 513 TiB / 534 TiB avail; 43 KiB/s rd, 0 B/s wr, 49 op/s
> 2020-12-04T15:14:30.293180+0700 mon.hk-cephmon-2s01 (mon.0) 4475 :
> cluster [ERR] Health check failed: 2 scrub errors (OSD_SCRUB_ERRORS)
> 2020-12-04T15:14:30.293196+0700 mon.hk-cephmon-2s01 (mon.0) 4476 :
> cluster [ERR] Health check failed: Possible data damage: 1 pg
> inconsistent (PG_DAMAGED)
>
> I had to repair pg and it worked fine, but not sure where this come
> from. I have this in the log only :/
>
> Thank you.
>
> 
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by
> copyright or other legal rules. If you have received it by mistake
> please let us know by reply email and delete it from your system. It
> is prohibited to copy this message or disclose its content to anyone.
> Any confidentiality or privilege is not waived or lost by any mistaken
> delivery or unauthorized disclosure of the message. All messages sent
> to and from Agoda may be monitored to ensure compliance with company
> policies, to protect the company's interests and to remove potential
> malware. Electronic messages may be intercepted, amended, lost or
> deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] PG_DAMAGED

[Szabo, Istvan (Agoda)]

Hi,

Not sure is it related to my 15.2.7 update, but today I got many time this 
issue:

2020-12-04T15:14:23.910799+0700 osd.40 (osd.40) 11 : cluster [DBG] 11.2 
deep-scrub starts
2020-12-04T15:14:23.947255+0700 osd.40 (osd.40) 12 : cluster [ERR] 11.2 soid 
11:434f049b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.372.1:head : 
omap_digest 0x48532c00 != omap_digest 0x8a18f5d7 from shard 40
2020-12-04T15:14:23.977138+0700 mgr.hk-cephmon-2s02 (mgr.2120884) 4330 : 
cluster [DBG] pgmap v4338: 209 pgs: 209 active+clean; 2.8 GiB data, 21 TiB 
used, 513 TiB / 534 TiB avail; 32 KiB/s rd, 32 op/s
2020-12-04T15:14:24.030888+0700 osd.40 (osd.40) 13 : cluster [ERR] 11.2 soid 
11:4b86603b:::.dir.75333f99-93d0-4238-91a4-ba833a0edd24.1744118.197.3:head : 
omap_digest 0xcb62779b != omap_digest 0xefef7471 from shard 40
2020-12-04T15:14:24.229000+0700 osd.40 (osd.40) 14 : cluster [ERR] 11.2 
deep-scrub 0 missing, 2 inconsistent objects
2020-12-04T15:14:24.229003+0700 osd.40 (osd.40) 15 : cluster [ERR] 11.2 
deep-scrub 2 errors
2020-12-04T15:14:25.978189+0700 mgr.hk-cephmon-2s02 (mgr.2120884) 4331 : 
cluster [DBG] pgmap v4339: 209 pgs: 1 active+clean+scrubbing+deep, 208 
active+clean; 2.8 GiB data, 21 TiB used, 513 TiB / 534 TiB avail; 55 KiB/s rd, 
0 B/s wr, 61 op/s
2020-12-04T15:14:27.978588+0700 mgr.hk-cephmon-2s02 (mgr.2120884) 4332 : 
cluster [DBG] pgmap v4340: 209 pgs: 1 active+clean+scrubbing+deep, 208 
active+clean; 2.8 GiB data, 21 TiB used, 513 TiB / 534 TiB avail; 43 KiB/s rd, 
0 B/s wr, 49 op/s
2020-12-04T15:14:30.293180+0700 mon.hk-cephmon-2s01 (mon.0) 4475 : cluster 
[ERR] Health check failed: 2 scrub errors (OSD_SCRUB_ERRORS)
2020-12-04T15:14:30.293196+0700 mon.hk-cephmon-2s01 (mon.0) 4476 : cluster 
[ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED)

I had to repair pg and it worked fine, but not sure where this come from. I 
have this in the log only :/

Thank you.


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: Unable to reshard bucket

[Szabo, Istvan (Agoda)]

Hi Eric,

Thank you to pickup my question.
Correct me if I'm wrong please regarding sharding and indexes.
The flow when the user put an object to the cluster, it will create 1 object in 
the index pool that will hold the let's say location of the file in the data 
pool.
1 index entry is for 1 bucket so if the bucket objects number is growing the 
index object will grow too.
Here is where sharding come into picture, with sharding we can make smaller 
chunks of this 1 big index object. Document says we can calculate the shard 
numbers with 100.000, so 1 shard is for 100.000 objects which means if the 
bucket has 100 shards, it can hold let's say 10 millions objects.

Now I have the situation, there is 100 shards and 100.000 objects/shard set. 
Have a bucket which crossed the 10 millions of objects and to be honest I don't 
know what is happening at the moment, they are at 11.5 millions objects, no 
issue, I just don't understand what is happening.

So if we don't know at the beginning of the bucket creation what is the planned 
number of objects in the future, it's better to set the sharding to a high 
number. And as the documentation says, 64k is the max shards bucket, so why not 
set this number to avoid any limitation.

And now we have a new cluster with multisite enabled, here dynamic bucket 
sharding is not even possible, so I don't know at the moment, what I should set 
as a basic before put it into production.

Thank you in advance your clarification.


-Original Message-
From: Eric Ivancich 
Sent: Wednesday, November 25, 2020 5:37 AM
To: Szabo, Istvan (Agoda) 
Cc: ceph-users 
Subject: Re: [Suspicious newsletter] [ceph-users] Re: Unable to reshard bucket

Email received from outside the company. If in doubt don't click links nor open 
attachments!


Can you clarify, Istvan, what you plan on setting to 64K? If it’s the number of 
shards for a bucket, that would be a mistake.

> On Nov 21, 2020, at 2:09 AM, Szabo, Istvan (Agoda)  
> wrote:
>
> Seems like this sharding we need to be plan carefully since the beginning. 
> I'm thinking to set the shard number by default to the maximum which is 64k 
> and leave it as is so we will never reach the limit only if we reach the 
> maximum number of objects.
>
> Would be interesting to know what is the side effect if I set the shards to 
> 64k by default.
>
> Istvan Szabo
> Senior Infrastructure Engineer

--
J. Eric Ivancich
he / him / his
Red Hat Storage
Ann Arbor, Michigan, USA



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] HA_proxy setup

[Szabo, Istvan (Agoda)]

Hi,

I wonder is there anybody have a setup like I want to setup?

1st subnet: 10.118.170.0/24 (FE users)
2nd subnet: 10.192.150.0/24 (BE users)

The users are coming from these subnets, and I want that the FE users will come 
on the 1st interface on the loadbalancer, the BE users will come one the 2nd 
interface of the HA_Proxy loadbalancer, so somehow need to create 2 backends 
maybe in the HA_Proxy config?

Both users would go to the same rados gateways and to the same ceph cluster.

Somehow I want to create static routes on the loadbalancer, but not sure how 
can I define in the HA_Proxy config, that go to that specific interface ?

Thanks



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Sizing radosgw and monitor

[Szabo, Istvan (Agoda)]

Hi,

I haven't really find any documentation about how to size radosgw.
One redhat doc says we need to decide the ratio like 1:50 or 1:100 osd / rgw.
I had an issue earlier where I had a user who source loadbalanced so always 
went to the same radosgateway and 1 time just maxed out.

So the question is, how to monitor rgw, what kind of values or ... ?
How to size RGW?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Re: Unable to reshard bucket

[Szabo, Istvan (Agoda)]

Seems like this sharding we need to be plan carefully since the beginning. I'm 
thinking to set the shard number by default to the maximum which is 64k and 
leave it as is so we will never reach the limit only if we reach the maximum 
number of objects.

Would be interesting to know what is the side effect if I set the shards to 64k 
by default.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Timothy Geier 
Sent: Saturday, November 21, 2020 12:22 AM
To: Eric Ivancich ; ceph-users 
Subject: [Suspicious newsletter] [ceph-users] Re: Unable to reshard bucket

Email received from outside the company. If in doubt don't click links nor open 
attachments!


On Thu, 2020-11-19 at 13:38 -0500, Eric Ivancich wrote:
> Hey Timothy,
>
> Did you ever resolve this issue, and if so, how?

Unfortunately, I was never able to resolve it; the bucket(s) in question had to 
be recreated and then removed.

>
> > Thank you..I looked through both logs and noticed this in the cancel
> > one:
> >
> > osd_op(unknown.0.0:4164 41.2
> > 41:55b0279d:reshard::reshard.09:head [call
> > rgw.reshard_remove] snapc 0=[] ondisk+write+known_if_redirected
> > e24984) v8 --
> > 0x7fe9b3625710 con 0
> > osd_op_reply(4164 reshard.09 [call] v24984'105796943
> > uv105796922 ondisk = -2
> > ((2) No such file or directory)) v8  162+0+0 (203651653 0 0)
> > 0x7fe9880044a0 con
> > 0x7fe9b3625b70
> > ERROR: failed to remove entry from reshard log,
> > oid=reshard.09 tenant= bucket=foo
> >
> > Is there anything else that I should look for?  It looks like the
> > cancel process thinks that reshard.09 is present (and
> > probably blocking my attempts at resharding) but it's not actually
> > there and thus can't be removed.
>
> Eric
> --
> J. Eric Ivancich
> he / him / his
> Red Hat Storage
> Ann Arbor, Michigan, USA
>

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Weird ceph use case, is there any unknown bucket limitation?

[Szabo, Istvan (Agoda)]

Hi,

I have a use case where the user would like to have 5 Buckets.
Is it normal for ceph just too much for me?


The reason they want this level of granularity is because they might need to 
clean buckets for a specific subset and not affect others.



Bucket format is this:

PR_PAGETPYE-_DEVICE-_WHITELABEL-_LANGUAGE-_SUBTYPE-/



Thank you



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io