[ceph-users] Re: 15.2.17: RGW deploy through cephadm exits immediately with exit code 5/NOTINSTALLED

2022-09-28 Thread Michel Jouvin
One additional information that may be relevant for the problem: the server hosting the RGW has 2 networks configured on the same interface and the default one is not the Ceph public network but another network not related to Ceph. I suspect a Podman configuration issue where the default networ

[ceph-users] Re: Questions about the QA process and the data format of both OSD and MON

2022-09-28 Thread Satoru Takeuchi
Hi Junior, >> >- a.3 How to run Teuthology in my local environment? >> >> At this point, we have the ability to run some tests locally using >> teuthology, Junior (cc'ed here) did a presentation on this topic, >> which was recorded here: https://www.youtube.com/watch?v=wZHcg0oVzhY. > > Thank y

[ceph-users] Traffic between public and cluster network

2022-09-28 Thread Murilo Morais
Good evening everyone. I setup a cluster with three machines, each with two network interfaces, one for the public network and one for the cluster network (172.25.50.0/24 for public and 10.10.10.0/24 for cluster). All machines see each other and are communicable in their respective networks. So fa

[ceph-users] Re: Low read/write rate

2022-09-28 Thread Murilo Morais
Thanks a lot for the explanation. It makes the most sense. After testing using the FIO and ETHTOOL tools I discovered that my problem was the network interface. After replacing it I managed to reach the mark of 1150 Megabytes per second. Em seg., 26 de set. de 2022 às 03:58, Janne Johansson escr

[ceph-users] Re: HA cluster

2022-09-28 Thread Murilo Morais
Thank you very much for the clarifications. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] strange osd error during add disk

2022-09-28 Thread Satish Patel
Folks, I have 15 nodes for ceph and each node has a 160TB disk attached. I am using cephadm quincy release and all 14 nodes have been added except one node which is giving a very strange error during adding it. I have put all logs here https://paste.opendev.org/show/bbSKwlSLyANMbrlhwzXL/ In short

[ceph-users] 15.2.17: RGW deploy through cephadm exits immediately with exit code 5/NOTINSTALLED

2022-09-28 Thread Michel Jouvin
Hi, We have a cephadm-based Octopus (upgraded to 15.2.17 today but the problem started with 15.2.16) cluster where we try to deploy a RGW in multisite configuration. We followed the documentation at https://docs.ceph.com/en/octopus/radosgw/multisite/ to do the basic realm, zonegroup, zone and

[ceph-users] Re: CLT meeting summary 2022-09-28

2022-09-28 Thread Gregory Farnum
On Wed, Sep 28, 2022 at 9:15 AM Adam King wrote: > Budget Discussion > >- Going to investigate current resources being used, see if any costs >can be cut >- What can be moved from virtual environments to internal ones? >- Need to take inventory of what resources we currently have

[ceph-users] CLT meeting summary 2022-09-28

2022-09-28 Thread Adam King
Budget Discussion - Going to investigate current resources being used, see if any costs can be cut - What can be moved from virtual environments to internal ones? - Need to take inventory of what resources we currently have and what their costs are 17.2.4 - Gibba and LRC cluste

[ceph-users] Re: RGW multi site replication performance

2022-09-28 Thread Steven Goodliff
Hi, >From what I've discovered so far with one bucket and one topic max out on our >system around ~1k second notifications but multiple buckets with multiple >topics (even if the topics all point to the same push endpoint gives more >performance), still digging. Steven Goodliff Global Relay

[ceph-users] Re: Fstab entry for mounting specific ceph fs?

2022-09-28 Thread Ken Dreyer
On Sun, Sep 25, 2022 at 11:24 PM Sagittarius-A Black Hole wrote: > > No, I actually included the ceph fstype, just not in my example (the > initial post), but the key is really mds_namespace for specifying the > filesystem, this should be included in the documentation. We removed mds_namespace fr

[ceph-users] rgw txt file access denied error

2022-09-28 Thread Marc
I have a strange situation where I have uploaded a txt file to a folder set the permissions correctly. Yet still I am getting an access denied message[1]. If I put some zip file in the same folder, apply the same permissions I have no problems accessing it[2]. Is there anything known about such

[ceph-users] Re: Upgrade from Octopus to Quiny fails on third ceph-mon

2022-09-28 Thread Ackermann, Christoph
Josh, only one monitor was on kv_backend leveldb by mistake. Initial cluster version was Mimic as far as i know. Maybe I forgot this one to convert to rocksdb in transit of Luminous or so... ;-) Cheers, Christoph Am Mi., 28. Sept. 2022 um 15:18 Uhr schrieb Josh Baergen < jbaer...@digitalocean

[ceph-users] Re: Upgrade from Octopus to Quiny fails on third ceph-mon

2022-09-28 Thread Josh Baergen
FWIW, this is what the Quincy release notes say: LevelDB support has been removed. WITH_LEVELDB is no longer a supported build option. Users should migrate their monitors and OSDs to RocksDB before upgrading to Quincy. Josh On Wed, Sep 28, 2022 at 4:20 AM Eugen Block wrote: > > Hi, > > there was

[ceph-users] Re: Upgrade from Octopus to Quiny fails on third ceph-mon

2022-09-28 Thread Eugen Block
Sorry, I was busy and couldn't reply. But it's great that the issue is resolved! Zitat von "Ackermann, Christoph" : Eugen, i had to restart all five monitors from scratch, so now i have a valid quorum 5/5 and all mons are v17.2.3. This was really strange. The main problem seemingly was: mo

[ceph-users] Re: Upgrade from Octopus to Quiny fails on third ceph-mon

2022-09-28 Thread Ackermann, Christoph
Eugen, i had to restart all five monitors from scratch, so now i have a valid quorum 5/5 and all mons are v17.2.3. This was really strange. The main problem seemingly was: mon.ceph1n011@0(electing) e46 handle_auth_request failed to assign global_id and so forth... The one Mon with levelb i did

[ceph-users] Re: Upgrade from Octopus to Quiny fails on third ceph-mon

2022-09-28 Thread Ackermann, Christoph
Eugen, thank you very much. This (ceph1n021) is indeed the one using leveldb (in kv_backend) Other mons have kv_backend "rocksdb" but unfortunately after reinstalling ceph-mon@ceph1n021 we get no ceph status anymore and our mon logs get filed with: 2022-09-28T13:10:54.822+0200 7fbc6a863700 0 l

[ceph-users] Re: waiting for the monitor(s) to form the quorum.

2022-09-28 Thread Eugen Block
Hi, ceph requires the "public network" (including the MONs) because clients contact MONs to request the osdmap so they can contact the OSDs directly (also via public network). If required you can separate the replication traffic between OSDs and introduce the "cluster network". That's it.

[ceph-users] Re: Upgrade from Octopus to Quiny fails on third ceph-mon

2022-09-28 Thread Eugen Block
Hi, there was a thread about deprecating leveldb [1], but I didn't get the impression that it already has been deprecated. But the thread mentions that it's not tested anymore, so that might explain it. To confirm that you use leveldb you can run: cat /var/lib/ceph/mon/ceph-/kv_backend S

[ceph-users] Re: External RGW always down

2022-09-28 Thread Eugen Block
To turn on debug logs on one daemon you can try this: host1:~ # ceph orch ps --daemon-type rgw NAME HOSTPORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID rgw.ebl-rgw.host3.pjlrvr host3 running (12s) 7s a

[ceph-users] Re: External RGW always down

2022-09-28 Thread Monish Selvaraj
Hi Eugen, Kindly tell me the command to enable debug log for rgw. Also after all the pg inactive problems were resolved the rgw wont start.The ceph health becomes ok and in the pg status there is no working, warning and unknown pgs are there. On Wed, Sep 28, 2022 at 2:37 PM Eugen Block wrote:

[ceph-users] Upgrade from Octopus to Quiny fails on third ceph-mon

2022-09-28 Thread Ackermann, Christoph
Hello List, i'm on the way to upgrade our "non cephadm" from Octopus to Quiny. It fails/stuck on third ceph-mon ceph1n021 with an strange error: 2022-09-28T11:04:27.691+0200 7f8681543880 -1 _open error initializing leveldb db back storage in /var/lib/ceph/mon/ceph-ceph1n021/store.db This monito

[ceph-users] Re: External RGW always down

2022-09-28 Thread Eugen Block
As I already said, it's possible that your inactive PGs prevent the RGWs from starting. You can turn on debug logs for the RGWs, maybe they reveal more. Zitat von Monish Selvaraj : Hi Eugen, The OSD fails because of RAM/CPU overloaded whatever it is.After Osd fails it starts again. That's

[ceph-users] Re: How to remove remaining bucket index shard objects

2022-09-28 Thread 伊藤 祐司
Hi, Eric Thank you for your reply. > I don’t believe there is any tooling to find and clean orphaned bucket index > shards. So if you’re certain they’re no longer needed, you can use `rados` > commands to remove the objects. I'll delete the bucket index shard object using the rados command as

[ceph-users] Re: 2-Layer CRUSH Map Rule?

2022-09-28 Thread Janne Johansson
> > ceph osd setcrushmap -i /tmp/crush.new > > > > Note: If you are overwriting your current rule, your data will need to > > rebalance as soon as your set the crushmap, close to 100% of your > > objects will move. If you create a new rule, you can set your pool to > > use the new pool id anytime y