[ceph-users] Best practice regarding rgw scaling

Hi, Wonder what is the best practice to scale RGW, increase the thread numbers or spin up more gateways? * Let's say I have 21000 connections on my haproxy * I have 3 physical gateway servers so let's say each of them need to server 7000 connections This means with 512 thread pool size

[ceph-users] Multisite: metadata behind on shards

Hi, Wonder what is the mechanism behind the sync mechanism because I need to restart all the gateways every 2 days on the remote sites to keep those it in sync. (Octopus 15.2.7) Thank you This message is confidential and is for the sole use of the intended

[ceph-users] Numa pinning best practices

Hi, Haven't really found a proper descripton in case of 2 socket how to pin osds to numa node, only this: https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployments#Ceph-Storage-Node-NUMA-Tuning Tuning for All Flash Deployments - Ceph - Ceph

[ceph-users] NVME node disks maxed out during rebalance after adding to existing cluster

Hi, I have slow heartbeat in front and back with the extra node added to the cluster and this occasionally causing slow ops and failed osd reports. I'm extending our cluster with +3 relatively differently configured servers compared to the original 12. Our cluster (latest octopus) is an

[ceph-users] Re: RBD image metric

lto:istvan.sz...@agoda.com> --- From: Anthony D'Atri Sent: Thursday, April 4, 2024 3:19 AM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: [ceph-users] Re: RBD image metric Email received from the inter

[ceph-users] RBD image metric

Hi, Trying to pull out some metrics from ceph about the rbd images sizes but haven't found anything only pool related metrics. Wonder is there any metric about images or I need to create by myself to collect it with some third party tool? Thank you This

[ceph-users] 1x port from bond down causes all osd down in a single machine

Hi, Wonder what we are missing from the netplan configuration on ubuntu which ceph needs to tolerate properly. We are using this bond configuration on ubuntu 20.04 with octopus ceph: bond1: macaddress: x.x.x.x.x.50 dhcp4: no dhcp6: no addresses: -

[ceph-users] Robust cephfs design/best practice

Hi, I'd like to add cephfs to our production objectstore/block storage cluster so I'd like to collect hands on experiences like, good to know/be careful/avoid etc ... other than ceph documentation. Thank you This message is confidential and is for the sole use

[ceph-users] Re: Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?

Hi Goetz, Which method you finally choose? We've done a successful migration from Centos 8 to ubuntu 20.04 but we have a centos 7 nautilus cluster which we'd like to move to Ubuntu 20.04 octopus same as you. Wonder any of you tried to skip Rocky 8 from the flow? Thank you

[ceph-users] Re: About ceph disk slowops effect to cluster

Is it better? Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- From: Phong Tran

[ceph-users] Re: RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures

Hi, I'm using in the frontend https config on haproxy like this, it works so far good: stick-table type ip size 1m expire 10s store http_req_rate(10s) tcp-request inspect-delay 10s tcp-request content track-sc0 src http-request deny deny_status 429 if { sc_http_req_rate(0) gt 1 } Istvan

[ceph-users] Re: increasing number of (deep) scrubs

Hi, You are on octopus right? Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Re: Space reclaim doesn't happening in nautilus RBD pool

3 6:10 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] Space reclaim doesn't happening in nautilus RBD pool Email received from the internet. If in doubt, don't click any link nor open any attachment ! Hi Istvan, The number of objects

[ceph-users] Re: How to identify the index pool real usage?

istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: David C. Sent: Monday, December 4, 2023 3:44 PM To: Szabo, Istvan (Agoda) Cc: Anthony D'Atri ; Ceph Users Subject: Re: [ceph-users] How to identify t

[ceph-users] Re: How to identify the index pool real usage?

With the nodes that has some free space on that namespace, we don't have issue, only with this which is weird. From: Anthony D'Atri Sent: Friday, December 1, 2023 10:53 PM To: David C. Cc: Szabo, Istvan (Agoda) ; Ceph Users Subject: Re: [ceph-users] How

[ceph-users] How to identify the index pool real usage?

Hi, Today we had a big issue with slow ops on the nvme drives which holding the index pool. Why the nvme shows full if on ceph is barely utilized? Which one I should belive? When I check the ceph osd df it shows 10% usage of the osds (1x 2TB nvme drive has 4x osds on it): ID CLASS WEIGHT

[ceph-users] Re: Space reclaim doesn't happening in nautilus RBD pool

: Ilya Dryomov Sent: Thursday, November 30, 2023 6:27 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] Space reclaim doesn't happening in nautilus RBD pool Email received from the internet. If in doubt, don't click any link nor open any atta

[ceph-users] Space reclaim doesn't happening in nautilus RBD pool

Hi, Is there any config on Ceph that block/not perform space reclaim? I test on one pool which has only one image 1.8 TiB in used. rbd $p du im/root warning: fast-diff map is not enabled for root. operation may be slow. NAMEPROVISIONED USED root 2.2 TiB 1.8 TiB I already

[ceph-users] Experience with deduplication

Hi Developers, What is the status of the deduplication for objectsore? I see it under the dev area only since octopus even with the latest release. https://docs.ceph.com/en/octopus/dev/deduplication/ Is it something that can be used in production? Thank you

[ceph-users] Previously synced bucket resharded after sync removed

Hi, I had a multisite bucket which I've removed from sync completely and resharded on the master zone the bucket which was successful. On the 2nd site (which was expected) can't list anything inside that bucket anymore which is okay, the issue is how I can delete the data somehow? It was 50TB

[ceph-users] Re: radosgw bucket usage metrics gone after created in a loop 64K buckets

I think this is related to my radosgw-exporter, not related to ceph, I'll report it in git, sorry for the noise. From: Szabo, Istvan (Agoda) Sent: Monday, September 18, 2023 1:58 PM To: Ceph Users Subject: [ceph-users] radosgw bucket usage metrics gone after

[ceph-users] radosgw bucket usage metrics gone after created in a loop 64K buckets

Hi, Last week we've created for a user 64K buckets to be able to properly shard their huge amount of objects and I can see that the "radosgw_usage_bucket" metrics disappeared from 10pm that day when the mass creation happened in our octopus 15.2.17 cluster. In the logs I don't really see

[ceph-users] Re: Is it safe to add different OS but same ceph version to the existing cluster?

god with these kernel parameter values, do you see something that might be related to the high disk utilization? Thank you [https://i.ibb.co/Tk5Srk6/image-2023-09-04-09-55-52-311.png] From: Milind Changire Sent: Monday, August 7, 2023 11:38 PM To: Szabo, Istvan (Ag

[ceph-users] Re: Is there any way to fine tune peering/pg relocation/rebalance?

Seems like tested on nautilus but I still see commits last month so I guess it is good with octopus. From: Matt Vandermeulen Sent: Wednesday, August 30, 2023 12:44 AM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] Is there any way to fine

[ceph-users] Re: Is there any way to fine tune peering/pg relocation/rebalance?

I'm using upmap with max deviation 1, maybe is it too aggressive? From: Louis Koo Sent: Wednesday, August 30, 2023 4:17 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: Is there any way to fine tune peering/pg relocation/rebalance? Email received from the

[ceph-users] Is there any way to fine tune peering/pg relocation/rebalance?

Hello, Is there a way to somehow fine tune the rebalance even further than basic tuning steps when adding new osds? Today I've added some osd to the index pool and it generated many slow ops due to OSD op latency increase + read operation latency increase = high put get latency.

[ceph-users] 64k buckets for 1 user

Hi, We are in a transition where I'd like to ask my user who stores 2B objects in 1 bucket to split it some way. Thinking for the future we identified to make it future proof and don't store huge amount of objects in 1 bucket, we would need to create 65xxx buckets. Is there anybody aware of

[ceph-users] Is it safe to add different OS but same ceph version to the existing cluster?

Hi, I have an octopus cluster on the latest octopus version with mgr/mon/rgw/osds on centos 8. Is it safe to add an ubuntu osd host with the same octopus version? Thank you This message is confidential and is for the sole use of the intended recipient(s). It

[ceph-users] Multisite sync - zone permission denied

Hi, Have you had the issue with zones are permission denied? failed to retrieve sync info: (13) Permission denied It's a newly added zone, uses the same sync user and credentials but it shows permission denied and I don't see any reason behind. Thank you This

[ceph-users] Re: RGW dynamic resharding blocks write ops

so add that they have their index pool on HDDs (with rocksdb on SSD), not sure how big the impact is during resharding though. Zitat von "Szabo, Istvan (Agoda)" : I turned off :) Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., L

[ceph-users] Re: RGW dynamic resharding blocks write ops

I turned off :) Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- On 2023. Jul 7., at 17:35, Eugen Block wrote:

[ceph-users] Re: radosgw hang under pressure

Hi, Can you check the read and write latency of your osds? Maybe it hangs because it’s waiting for pg’s but maybe the pg are under scrub or something else. Also with many small objects don’t rely on pg autoscaler, it might not tell to increase pg but maybe it should be. Istvan Szabo Staff

[ceph-users] Transmit rate metric based per bucket

Hello, I'd like to know is there a way to query some metrics/logs in octopus (or if has newer version I'm interested for the future too) about the bandwidth used in the bucket for put/get operations? Thank you This message is confidential and is for the sole

[ceph-users] Re: Bottleneck between loadbalancer and rgws

--- -Original Message- From: Kai Stian Olstad Sent: Wednesday, June 14, 2023 9:02 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] Bottleneck between loadbalancer and rgws Email received from the internet. If in doubt

[ceph-users] Bottleneck between loadbalancer and rgws

Hi, I have a dedicated loadbalancer pairs separated on 2x baremetal servers and behind the haproxy balancers I have 3 mon/mgr/rgw nodes. Each rgw node has 2rgw on it so in the cluster altogether 6, (now I just added one more so currently 9). Today I see pretty high GET latency in the cluster

[ceph-users] Non cephadm cluster upgrade from octopus to quincy

Hi, I don't find any documentation for this upgrade process. Is there anybody who has already done it yet? Is the normal apt-get update method works? Thank you This message is confidential and is for the sole use of the intended recipient(s). It may also be

[ceph-users] Re: Deleting millions of objects

If it works I’d be amazed. We have this slow and limited delete issue also. What we’ve done to run on the same bucket multiple delete from multiple servers via s3cmd. Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e:

[ceph-users] Re: Octopus on Ubuntu 20.04.6 LTS with kernel 5

: Ilya Dryomov Sent: Thursday, May 11, 2023 3:39 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] Re: Octopus on Ubuntu 20.04.6 LTS with kernel 5 Email received from the internet. If in doubt, don't click any link nor open any attachment

[ceph-users] Re: Octopus on Ubuntu 20.04.6 LTS with kernel 5

I can answer my question, even in the official ubuntu repo they are using by default the octopus version so for sure it works with kernel 5. https://packages.ubuntu.com/focal/allpackages -Original Message- From: Szabo, Istvan (Agoda) Sent: Thursday, May 11, 2023 11:20 AM To: Ceph

[ceph-users] Octopus on Ubuntu 20.04.6 LTS with kernel 5

Hi, In octopus documentation we can see kernel 4 as recommended, however we've changed our test cluster yesterday from centos 7 / 8 to Ubuntu 20.04.6 LTS with kernel 5.4.0-148 and seems working, I just want to make sure before I move to prod there isn't any caveats. Thank you

[ceph-users] Os changed to Ubuntu, device class not shown

Hi, We have an octopus cluster where we want to move from centos to Ubuntu, after activate all the osd, class is not shown in ceph osd tree. However ceph-volume list shows the crush device class :/ Should I just add it or? This message is confidential and is

[ceph-users] Re: Bucket notification

elated errors there? Yuval On Tue, Apr 25, 2023 at 5:48 PM Szabo, Istvan (Agoda) mailto:istvan.sz...@agoda.com>> wrote: Hi, I'm trying to set a kafka endpoint for bucket object create operation notifications but the notification is not created in kafka endpoint. Settings seems to be fine b

[ceph-users] Bucket notification

Hi, I'm trying to set a kafka endpoint for bucket object create operation notifications but the notification is not created in kafka endpoint. Settings seems to be fine because I can upload to the bucket objects when these settings are applied: NotificationConfiguration> bulknotif

[ceph-users] Re: RGW access logs with bucket name

It has the full url begins with the bucket name in the beast logs http requests, hasn’t it? Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Re: Changing os to ubuntu from centos 8

ehrens Sent: Tuesday, March 21, 2023 4:29 PM To: Szabo, Istvan (Agoda) ; Ceph Users Cc: dietr...@internet-sicherheit.de; ji...@spets.org Subject: Re: [ceph-users] Changing os to ubuntu from centos 8 Email received from the internet. If in doubt, don't click any link nor open any atta

[ceph-users] Changing os to ubuntu from centos 8

Hi, I'd like to change the os to ubuntu 20.04.5 from my bare metal deployed octopus 15.2.14 on centos 8. On the first run I would go with octopus 15.2.17 just to not make big changes in the cluster. I've found couple of threads on the mailing list but those were containerized (like: Re:

[ceph-users] Re: Adding osds to each nodes

c, e.g. [1]. Regards, Eugen [1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg36475.html Zitat von "Szabo, Istvan (Agoda)" : Hi, What is the safest way to add disk(s) to each of the node in the cluster? Should it be done 1 by 1 or can add all of them at once and let i

[ceph-users] Adding osds to each nodes

Hi, What is the safest way to add disk(s) to each of the node in the cluster? Should it be done 1 by 1 or can add all of them at once and let it rebalance? My concern is that if add all in one due to host based EC code it will block all the host. The other side if I add 1 by 1, one node will

[ceph-users] PG increase / data movement fine tuning

Hi, I've increased the placement group in my octopus cluster firstly in the index pool and I gave almost 2.5 hours bad performance for the user. I'm planning to increase the data pool also, but first I'd like to know is there any way to make it smoother or not. At the moment I have these

[ceph-users] Real memory usage of the osd(s)

Hello, If buffered_io is enabled, is there a way to know what is the exactly used physical memory from each osd? What I've found is the dump_mempools which last entries are the following, but this bytes would be the real physical memory usage? "total": { "items": 60005205,

[ceph-users] Re: Very slow snaptrim operations blocking client I/O

How is your pg distribution on your osd devices? Do you have enough assigned pgs? Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Snap trimming best practice

Hi, Wonder have you ever faced issue with snaptrimming if you follow ceph pg allocation recommendation (100pg/osd)? We have a nautilus cluster and we scare to increase the pg-s of the pools because seems like even if we have 4osd/nvme, if the pg number is higher = the snaptrimming is slower.

[ceph-users] User migration between clusters

Hi, Normally I use rclone to migrate buckets across clusters. However this time the user has close to 1000 buckets so I wonder what would be the best approach to do this rather buckets by buckets, any idea? Thank you This message is confidential and is for the

[ceph-users] Ceph mgr rgw module missing in quincy

Hi, When I want to enable this module it is missing: https://docs.ceph.com/en/quincy/mgr/rgw.html Looked in the mgr module list but nothing there. What is the reason? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e:

[ceph-users] Multi site alternative

Hi, Due to the lack of documentation and issues with multisite bucket sync I’m looking for an alternative solution where I can put some sla around the sync like I can guarantee that the file will be available in x minutes. Which solution you guys are using which works fine with huge amount of

[ceph-users] Re: Monitor server move across cages

kl 08:00 skrev Szabo, Istvan (Agoda) : Hi, I have 3 mons in my cluster and I need to move to another cage one of them. I guess it is not an issue to have one mon down for an hour, is it? If nothing else changes (ie, no OSDs,rgws,mds's fall off the net), no new mounts from new clients, you

[ceph-users] Monitor server move across cages

Hi, I have 3 mons in my cluster and I need to move to another cage one of them. I guess it is not an issue to have one mon down for an hour, is it? Thank you This message is confidential and is for the sole use of the intended recipient(s). It may also be

[ceph-users] What is the reason of the rgw_user_quota_bucket_sync_interval and rgw_bucket_quota_ttl values?

Hi, One of my user told me that they can upload bigger files to the bucket than the limit. My question is to the developers mainly what’s the reason to set the rgw_bucket_quota_ttl=600 and rgw_user_quota_bucket_sync_interval=180? I don’t want to set to 0 before I know the reason  With this

[ceph-users] Re: Strange 50K slow ops incident

Are those connected to the same switches? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- On 2022. Nov 3., at

[ceph-users] Does Ceph support presigned url (like s3) for uploading?

Hi, I found this long time back tracker https://tracker.ceph.com/issues/23470 which I guess some way show that it is possible but haven't really found any documentation in ceph, how to do properly. This is how it works with minio:

[ceph-users] Rgw compression any experience?

Hi, I’m looking in ceph octopus in my existing cluster to have object compression. Any feedback/experience appreciated. Also I’m curious is it possible to set after cluster setup or need to setup at the beginning? Thank you This message is confidential and is

[ceph-users] Re: Low space hindering backfill and 2 backfillfull osd(s)

--- -Original Message- From: Janne Johansson Sent: Friday, October 14, 2022 5:26 PM To: Szabo, Istvan (Agoda) Cc: Ceph Users Subject: Re: [ceph-users] Low space hindering backfill and 2 backfillfull osd(s) Email received from the internet. If in doubt, don't click any link nor open

[ceph-users] Low space hindering backfill and 2 backfillfull osd(s)

Hi, I've added 5 more nodes to my cluster and got this issue. HEALTH_WARN 2 backfillfull osd(s); 17 pool(s) backfillfull; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull OSD_BACKFILLFULL 2 backfillfull osd(s) osd.150 is backfill full

[ceph-users] Re: OSD crashes during upgrade mimic->octopus

Finally how is your pg distribution? How many pg/disk? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Frank

[ceph-users] one pg periodically got inconsistent

Hi, I have a pg which is periodically got inconsistent. Normally pg-repair helps but is there a way to avoid it? In my opinion is comes from a bigger data delete on the weekend that cannot be handled. Thank you This message is confidential and is for the sole

[ceph-users] Re: Any disadvantage to go above the 100pg/osd or 4osd/disk?

of recovery. Otherwise we don't see any real issue with 200 or 400 PGs/OSD if the nodes can handle it. Regards, Eugen Zitat von "Szabo, Istvan (Agoda)" : > Hi, > > My question is, is there any technical limit to have 8osd/ssd and on > each of them 100pg if the memory and

[ceph-users] Re: Any disadvantage to go above the 100pg/osd or 4osd/disk?

Sorry, 96vcore is a typo, 2vcore/osd but can be 4 also. > > On 2022. Sep 19., at 19:50, Szabo, Istvan (Agoda) > wrote: > > Hi, > > My question is, is there any technical limit to have 8osd/ssd and on each of > them 100pg if the memory and cpu resource a

[ceph-users] Any disadvantage to go above the 100pg/osd or 4osd/disk?

Hi, My question is, is there any technical limit to have 8osd/ssd and on each of them 100pg if the memory and cpu resource available (8gb memory/osd and 96vcore)? The iops and bandwidth on the disks are very low so I don’t see any issue to go with this. In my cluster I’m using 15.3TB ssds. We

[ceph-users] Re: ceph-osd iodepth for high-performance SSD OSDs

Isn’t it too much for ssd 4 osd? Normally nvme is suitable for 4osd isn’t it? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Re: Ceph performance optimization with SSDs

Be careful when you are designing, if you are planning to have billions of objects, you need to have more than 2-4% for rocksdb+wal to avoid spillover. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e:

[ceph-users] Re: RGW/multisite sync traffic rps

I see the same issue (45k GET requests constantly as admin), what my guess is, the primary site is putting to the datalog the changes and the secondary sites are pulling these logs as it changes. Do you have user who constantly uploading, deleting? Istvan Szabo Senior Infrastructure Engineer

[ceph-users] Re: inconsistent pg after upgrade nautilus to octopus

Have you tried to repair pg? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- On 2021. Oct 20., at 9:04, Glaza

[ceph-users] Re: ceph IO are interrupted when OSD goes down

Octopus 15.2.14? I have totally the same issue and it makes me prod issue. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Limit scrub impact

Hi, During scrub I see slow ops like this: osd.31 [WRN] slow request osd_op(client.115442393.0:263257613728.76s0 28:6ed54dc8:::9213182a-14ba-48ad-bde9-289a1c0c0de8.6034919.1_%2fWHITELABEL-1%2fPAGETPYE-7%2fDEVICE-4%2fLANGUAGE-46%2fSUBTYPE-0%2f492210:head [create,setxattr user.rgw.idtag (57)

[ceph-users] Re: Metrics for object sizes

! On 23/04/2021 03:53, Szabo, Istvan (Agoda) wrote: > Objects inside RGW buckets like in couch base software they have their own > metrics and has this information. Not as detailed as you would like, but how about using the bucket stats on bucket size and number of objects? $ radosgw

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

--- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Igor Fedotov Sent: Tuesday, October 12, 2021 7:15 PM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; 胡 玮文 Subject: Re: [ceph-use

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Szabo, Istvan (Agoda) Sent: Tuesday, October 12, 2021 6:54 PM To: Igor Fedotov Cc: ceph-users@ceph.io; 胡 玮文 Subject: RE: [ceph-users] Re: is it possible to remove the db+wal from an external device

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Igor Fedotov Sent: Tuesday, October 12, 2021 6:45 PM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; 胡 玮文 Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an ex

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

10/12/2021 12:13 PM, Szabo, Istvan (Agoda) wrote: Hi Igor, I’ve attached here, thank you in advance. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com<mailto:istvan.sz...@

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

: Igor Fedotov Sent: Monday, October 11, 2021 10:40 PM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; Eugen Block ; 胡 玮文 Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in doubt, don't click any link nor ope

[ceph-users] Re: Metrics for object sizes

...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Yuval Lifshitz Sent: Tuesday, September 14, 2021 7:38 PM To: Szabo, Istvan (Agoda) Cc: Wido den Hollander ; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Metrics for object sizes Email receive

[ceph-users] Re: Where is my free space?

I see, I'm using ssds so it shouldn't be a problem I guess, because the : "bluestore_min_alloc_size": "0" is overwritten with the: "bluestore_min_alloc_size_ssd": "4096" ? -Original Message- From: Stefan Kooman Sent: Tuesday, October 1

[ceph-users] get_health_metrics reporting slow ops and gw outage

Hi, Many of my osds having this issue which causes 10-15ms osd write operation latency and more than 60ms read operation latency. This causes rgw wait for operations and after a while rgw just restarted (all of them in my cluster) and only available after slow ops disappeared. I see similar

[ceph-users] Where is my free space?

Hi, 377TiB is the total cluster size, data pool 4:2 ec, stored 66TiB, how can be the data pool on 60% used??!! Some output: ceph df detail --- RAW STORAGE --- CLASS SIZE AVAILUSED RAW USED %RAW USED nvme12 TiB 11 TiB 128 MiB 1.2 TiB 9.81 ssd377 TiB 269 TiB

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- From: Igor Fedotov Sent: Tuesday, October 5, 2021 10:02 PM To: Szabo, Istvan (Agoda) ; 胡 玮文 Cc: ceph-users@ceph.io; Eugen Block Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email receive

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

stvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> ----------- On 2021. Oct 5., at 17:19, Szabo, Istvan (Agoda) wrote:  Hmm, I’ve removed from the cluster, now data rebalance, I’ll do with the next one ☹ Istvan Szabo Senior Infrastructure Engineer

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- From: Igor Fedotov Sent: Tuesday, October 5, 2021 10:02 PM To: Szabo, Istvan (Agoda) ; 胡 玮文 Cc: ceph-users@ceph.io; Eugen Block Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

--- -Original Message- From: Eugen Block Sent: Tuesday, October 5, 2021 9:52 PM To: Szabo, Istvan (Agoda) Cc: 胡 玮文 ; Igor Fedotov ; ceph-users@ceph.io Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

": "ceph-osd", "stack_sig": "6a43b6c219adac393b239fbea4a53ff87c4185bcd213724f0d721b452b81ddbf", "timestamp": "2021-10-05T13:31:28.513463Z", "utsname_hostname": "server-2s07", "utsname_machine":

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- From: Igor Fedotov Sent: Saturday, October 2, 2021 5:22 AM To: Szabo, Istvan (Agoda) Cc: ceph-users@ceph.io; Eugen Block ; Christian Wuerdig Subject: Re: [ceph-users] Re: is it possible to remove the db+wal from an external device (nvme) Email received from the in

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

click any link nor open any attachment ! Hi Istvan, yeah both db and wal to slow migration are supported. And spillover state isn't a show stopper for that. On 10/2/2021 1:16 AM, Szabo, Istvan (Agoda) wrote: Dear Igor, Is the ceph-volume lvm migrate command

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

com> --- On 2021. Oct 1., at 11:47, Szabo, Istvan (Agoda) wrote: 3x SSD osd /nvme Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Or

[ceph-users] Re: Multisite reshard stale instances

Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Christian Rohmann Sent: Friday, October 1, 2021 10:14 PM To: Szabo, Istvan (Agoda) ; ceph-users@ceph.io

[ceph-users] Re: Multisite reshard stale instances

--- -Original Message- From: Christian Rohmann Sent: Friday, October 1, 2021 9:37 PM To: Szabo, Istvan (Agoda) ; ceph-users@ceph.io Subject: Re: [ceph-users] Multisite reshard stale instances Email received from the internet. If in doubt, don't click

[ceph-users] Re: dealing with unfound pg in 4:2 ec pool

"Szabo, Istvan (Agoda)" : > Hi, > > If I set the min size of the pool to 4, will this pg be recovered? > Or how I can take out the cluster from health error like this? > Mark as lost seems risky based on some maillist experience, even if > marked lost after you still

[ceph-users] Re: dealing with unfound pg in 4:2 ec pool

Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Szabo, Istvan (Agoda) Sent: Friday, October 1, 2021 2:50 PM To: Eugen Block ; ceph-users

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

have? On 10/1/2021 9:55 AM, Szabo, Istvan (Agoda) wrote: > I have my dashboards and I can see that the db nvmes are always running on > 100% utilization (you can monitor with iostat -x 1) and it generates all the > time iowaits which is between 1-3. > > I’m using nvme in fr

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

--- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com<mailto:istvan.sz...@agoda.com> --- From: Victor Hooi Sent: Friday, October 1, 2021 5:30 AM To: Eugen Block Cc: Szabo, Istvan (Agoda) ; 胡 玮文 ; ceph-users Subje

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

tember 30, 2021 1:10 PM To: Szabo, Istvan (Agoda) Cc: 胡 玮文 ; Igor Fedotov ; ceph-users@ceph.io Subject: Re: is it possible to remove the db+wal from an external device (nvme) Email received from the internet. If in doubt, don't click any link nor open any atta

[ceph-users] Re: osd_memory_target=level0 ?

5.2.14 becauase that one has buffered_io enabled by default. ty From: Frédéric Nass Sent: Thursday, September 30, 2021 4:43 PM To: Szabo, Istvan (Agoda) ; Christian Wuerdig Cc: Ceph Users Subject: Re: [ceph-users] Re: osd_memory_target=level0 ? Email received from the internet

  1   2   3   4   >