[ceph-users] Re: MacOS Ceph Filesystem client

Hi Duncan Great, thank you for the tip. I never open the graphical user interface as we use this as a server. When logging in with SSH you sometimes miss popups on the desktop. Current status: Mac Mini Intel Catalina - Connected and working fine. Mac Mini M1 BigSur - Can't compile brew cask, no

[ceph-users] Billions of objects upload with bluefs spillover cause osds down?

Hi, One of our user is migrating 1.2 billions of objects to one bucket from another system (cassandra) and we are facing in our clusters bluefs spillovers on 50% of the osds. We have 600-900GB dbs bit seems like can’t fit. Also the cluster is very unstable, I can’t really set recovery operation

[ceph-users] New Ceph cluster in PRODUCTION

Hello Team I am new in CEPH . I am going to deploy ceph in production at my first time which will be integrated with openstack . below is my ceph.conf configurations and my ansible inventory set up. Please if I miss something important please let me know and advise on changes I have to make. I de

[ceph-users] Re: Billions of objects upload with bluefs spillover cause osds down?

Regarding point 2, how can it spillover if I wouldn’t use db device just block. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com

[ceph-users] Re: "Partitioning" in RGW

Hi, Your DMZ is S3 protocol. Access to buckets will be provided via S3 keys Just create as much users as much you need If you need definitely different "fake S3", I think create another pools and RGW instances is a way to achieve "real DMZ" Cheers, k Sent from my iPhone > On 23 Sep 2021, at 2

[ceph-users] Re: "Partitioning" in RGW

Hi, thanks for the answer. After studying the documentation a bit more (Ceph has a pretty steep learning curve) and reading your email, I figured out that it's probably best to have a separate "dmz" realm with separate RGWs on different ports and using a separate load balancing IP to partition the

[ceph-users] Re: Cephadm set rgw SSL port

Hi, 1. I believe the field is called 'rgw_frontend_port' 2. I don't think something like that exists but probably should -Daniel Pivonka On Mon, Sep 27, 2021 at 4:40 PM Sergei Genchev wrote: > Hi, I need to deploy RGW with SSL and was looking at the page > https://docs.ceph.com/en/pacific/ce

[ceph-users] Re: 16.2.6 CEPHADM_REFRESH_FAILED New Cluster

Thank you very much Adam, Really appreciate it. On Mon, Sep 27, 2021 at 3:02 PM Adam King wrote: > Unfortunately, I can't think of a workaround that doesn't involve a code > change. I've created a tracker (https://tracker.ceph.com/issues/52745) > and am working towards a fix for this, but I'm n

[ceph-users] Re: Cephadm set rgw SSL port

Am 28.09.21 um 15:12 schrieb Daniel Pivonka: > Hi, > > 1. I believe the field is called 'rgw_frontend_port' > 2. I don't think something like that exists but probably should At least for RGWs, we have: https://docs.ceph.com/en/pacific/cephadm/rgw/#service-specification > > -Daniel Pivonka > >

[ceph-users] 16.2.6: clients being incorrectly directed to the OSDs cluster_network address

Hi all, I am trying to understand a issue with ceph directing clients to connect to OSDs through their cluster_network address instead of their public_network address. I have a configured a ceph cluster with a public and cluster network: >> ceph config dump|grep network global advanced cluste

[ceph-users] Re: 16.2.6: clients being incorrectly directed to the OSDs cluster_network address

Interestingly enough this happens for some pools and not for others. For example I have just realized that when trying to connect to another pool the client is correctly directed to the OSD public_network address: >> strace -f -e trace=network -s 1 rbd ls --pool cinder-volumes --name client.c

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

Gave a try of it, so all the 3 osds finally failed :/ Not sure what went wrong. Do the normal maintenance things, ceph osd set noout, ceph osd set norebalance, stop the osd and run this command: ceph-bluestore-tool bluefs-bdev-migrate --dev-target /var/lib/ceph/osd/ceph-0/block --devs-source /va

[ceph-users] Re: Billions of objects upload with bluefs spillover cause osds down?

RGW stores a lot of metadata in the db of OSDs [1], so I would expect to see extensive usage on db device if you store billions of objects through RGW. Anyway, splitting over should not cause OSDs to reboot, and should still work better than not having dedicated db device, unless the db device i

[ceph-users] Re: 16.2.6: clients being incorrectly directed to the OSDs cluster_network address

Just curious, does it always happen with the same OSDs? On 09/28 16:14, Javier Cacheiro wrote: > Interestingly enough this happens for some pools and not for others. > > For example I have just realized that when trying to connect to another > pool the client is correctly directed to the OSD pub

[ceph-users] Re: is it possible to remove the db+wal from an external device (nvme)

I tried this in my lab again with Nautilus and it worked as expected, I could start the new OSD immediately. I'll try with Octopus tomorrow again. Zitat von "Szabo, Istvan (Agoda)" : Gave a try of it, so all the 3 osds finally failed :/ Not sure what went wrong. Do the normal maintenanc

[ceph-users] DAEMON_OLD_VERSION for 16.2.5-387-g7282d81d

Hello. After upgrade from Nautilus to Pacific my Dashboard shows "DAEMON_OLD_VERSION" warning. 'sudo ceph orch ps' shows versions like this: haproxy.nfs.nfs-cluster.crgw1.zbmnoi2.3.13-ce43e1d nfs.nfs-cluster.0.0.crgw1.lnvlwf3.5 keepalived.nfs.nfs-cluste

[ceph-users] Re: osd marked down

i have created keyring for the osd3 but still pod is not booting up.. As outlined: https://access.redhat.com/solutions/3524771 ceph auth export osd.2 -o osd.2.export cp osd.2.export osd.3.export ceph auth import -i osd.3.export imported keyring Any suggestions ? Thanks! On Tue, Sep 21, 2021 a

[ceph-users] 回复: Re: is it possible to remove the db+wal from an external device (nvme)

You may need to use `ceph-volume lvm migrate’ [1] instead of ceph-bluestore-tool. If I recall correctly, this is a pretty new feature, I’m not sure whether it is available to your version. If you use ceph-bluestore-tool, then you need to modify the LVM tags manually. Please refer to the previou

[ceph-users] prometheus - figure out which mgr (metrics endpoint) that is active

Hi, I am running ceph 16.2.6 installed with cephadm. I have enabled prometheus to be able scrape metrics from an external promethus server. I have 3 nodes with mgr daeamon all reply to the query against node:9283/metrics 2 is returning a empty reply - the none active mgr's. Is there a node:9283/

[ceph-users] Re: Billions of objects upload with bluefs spillover cause osds down?

Yes, I understand you but that time you wouldn't have issue because it can fit on the OSD or it would use the space on the osd which is ssd. Or I don't know then what's the secret of storing billions of objects, the osds are not even used 20%, if I calculate as of the current situation, even if

[ceph-users] Re: prometheus - figure out which mgr (metrics endpoint) that is active

We scrape all mgr endpoints since we use external Prometheus clusters, as well. The query results will have {instance=activemgrhost}. The dashboards in upstream don't have multiple cluster support, so we have to modify them to work with our deployments since we have multiple ceph clusters being pol

[ceph-users] Limiting osd or buffer/cache memory with Pacific/cephadm?

Hello, I'm running ceph pacific OSD servers that are orchestratedy by cephadm (on docker.io v20.10.8 on CentOS 7.9). The servers are a bit ... low equipped than others when it comes to memory per OSD. Earlier, we were able to accommodate for this by using the following /etc/ceph/ceph.conf setting

[ceph-users] Re: Limiting osd or buffer/cache memory with Pacific/cephadm?

buff/cache is the Linux kernel buffer and page cache which is unrelated to the ceph bluestore cache. Check the memory consumption of your individual OSD processes to confirm. Top also says 132GB available (since buffers and page cache entries will be dropped automatically if processes need more RAM

[ceph-users] Set some but not all drives as 'autoreplace'?

Hi all, I know Ceph offers a way to 'automatically' cause blank drives it detects to be spun up into osds, but I think that's an 'all or nothing' situation if I read the docs properly. Is there a way to specify which slots, or even better, a way to specify not specific slots?  It sure would

[ceph-users] Re: Limiting osd or buffer/cache memory with Pacific/cephadm?

Dear Christian, thank you for your reply. I think I found the cause of my problem (a leader election loop with the leader being evicted from quorum and returning every 5s) and could resolve it by rebooting the host. I don't know whether that's the root cause but I'm happy enough with the situation

[ceph-users] SSD partitioned for HDD wal+db plus SSD osd

Hi, Is there any way of using "ceph orch apply osd" to partition an SSD as wal+db for a HDD OSD, with the rest of the SSD as a separate OSD? E.g. on a machine (here called 'k1') with a small boot drive and a single HDD and SSD, this will create an OSD on the HDD, with wal+db on a 60G logical