date:20230830

[ceph-users] Re: Is there any way to fine tune peering/pg relocation/rebalance?

2023-08-30 Thread Louis Koo

maybe the default value is ok, I think set it to 1 is too aggressive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Reef - what happened to OSD spec?

2023-08-30 Thread Eugen Block

Hi, just a few days ago I replied to a thread [2] with some
explanations for destroy, delete and purge.

So if you "destroy" an OSD it is meant to be replaced, reusing the ID.
A failed drive may not be responsive at all so an automated wipe might
fail as well. If the db/wal is located on a different device you'll
have to clean that up manually. I don't think cephadm is able to do
that for you (yet). I just checked with a virtual Reef cluster.
So in a real world scenario if an OSD fails, you destroy it (which
marks it as "destroyed" in the crush tree), then you wipe the LV
containing db/wal manually. Unfortunately, 'ceph orch device zap'
can't deal with VG/LV syntax:

$ ceph orch device zap ceph01
/dev/ceph-0a339b77-072a-4a03-92d2-a2eda00bd12c/osd-db-5e64f473-07a3-432b-96e4-643e3bdbf2c0
--force
Error EINVAL: Device path
'/dev/ceph-0a339b77-072a-4a03-92d2-a2eda00bd12c/osd-db-5e64f473-07a3-432b-96e4-643e3bdbf2c0' not found on host
'ceph01'

So you'll have to wipe it locally with ceph-volume:

cephadm ceph-volume lvm zap --destroy /dev/ceph-{VG}/osd-db-{LV}

When the failed drive has been replaced, cephadm will redeploy the
OSD(s). You might want to pause the orchestrator (ceph orch pause)
before wiping the drive(s) as it might deploy OSDs if multiple specs
apply to the configuration.

[2]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/74FHIF73PTIGEC2T7JBPZ2BUIEMNNMEF/

Zitat von Nigel Williams :

Thanks Eugen for following up. Sorry my second response was incomplete. I
can confirm that it works as expected too. My confusion was that the
section from the online documentation seemed to be missing/moved, and when
it initially failed I wrongly thought that the OSD-add process had changed
in the Reef release.

There might still need to be a way that "destroy" does additional clean-up
to clear remnants of LVM fingerprints on the devices as this tripped me up
when the OSDspec apply failed due to "filesystem on device" checks.

Documentation has been improved and OSD spec is now under this heading for
Reef:

https://docs.ceph.com/en/reef/cephadm/services/osd/#advanced-osd-service-specifications
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: lack of RGW_API_HOST in ceph dashboard, 17.2.6, causes ceph mgr dashboard problems

2023-08-30 Thread Eugen Block


Hi,

there have been multiple discussions on this list without any  
satisfying solution for all possible configurations. One of the  
changes [1] made in Pacific was to use hostname instead of IP, but it  
only uses the shortname (you can check the "hostname" in 'ceph service  
dump' output. But this seems to only impact the dashboard access if  
you have ssl-verify set to true. I'm still waiting for a solution as  
well for a customer cluster which uses wildcard certificates only,  
until then we let ssl-verify disabled. But I didn't check the tracker  
for any pending tickets, so someone might be working on it.


Regards,
Eugen

[1] https://github.com/ceph/ceph/pull/47207/files

Zitat von Christopher Durham :


Hi,
I am using 17.2.6 on Rocky Linux 8
The ceph mgr dashboard, in my situation, (bare metal install,  
upgraded from 15->16-> 17.2.6), can no longer hit the  
ObjectStore->(Daemons,Users,Buckets) pages.


When I try to hit those pages, it gives an error:
RGW REST API failed request with status code 403 {"Code":  
"AccessDenied", RequestId: "xxx", HostId: "-"}


The log of the rgw server it hit has:

"GET /admin/metadata/user?myself HTTP/1.1" 403 125

It appears that the mgr dashboard setting RGW_API_HOST is no longer  
an option that can be set, nor does that name exist anywhere under  
/usr/share/ceph/mgr/dashboard, and:


# ceph dashboard set-rgw-api-host 

is no longer in existence in 17.2.6

However, since my situation is an upgrade, the config value still  
exists in my config, and I can retrieve it with:


# ceph dashboard get-rgw-api-host

To get the  to work in my situation, I have modified  
/usr/share/ceph/mgr/dashboard/settings.py and re-added RGW_API_HOST  
to the Options class using


RGW_API_HOST = Settings('', [dict,str])

I then modified  
/usr/share/ceph/mgr/dashboard/services/rgw_request.py such that each  
rgw daemon retrieved has its 'host' member set to  
Settings.RGW_API_HOST.


Then after restarting the mgr, I was able to access the  
Objectstore->(Daemons,Users,Buckets) pages in the dashboard.


HOWEVER, I know this is NOT the right way to fix this, it is a hack.  
It seems like the dashboard is trying to contact an rgw server  
individually. For us, the RGW_API_HOST is
a name in DNS: s3.my.dom, that has multiple A records, one for each  
of our rgw servers, each of which have the *same* SSL cert with CN  
and SubjectAltNames that allow
the cert to present itself as both s3.my.dom as well as the  
individual host name (SubjectAltName has ALL the rgw servers in it).  
This works well for us and has
done so since 15.x.y, The endpoint for the zone is set to s3.my.dom.  
Thus my users only have a single endpoint to care about, unless  
there is a failure situation onan rgw server. (We have other ways of  
handling that).
Any thoughts on the CORRECT way to handle this so I can have the  
ceph dashboard work with the ObjectStore->(Daemons,Users,Buckets)  
pages? Thanks.

-Chris

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: A couple OSDs not starting after host reboot

2023-08-30 Thread Alison Peisker

Hi,
It looks like Igor is right, it does appear to be a corruption.

ls /var/lib/ceph/252fcf9a-b169-11ed-87be-3cecef623f33/osd.665/
ceph_fsid config fsid keyring ready require_osd_release type unit.configured 
unit.created unit.image unit.meta unit.poststop unit.run unit.stop whoami

head -c 4096 
/dev/ceph-febad5a5-ba44-41aa-a39e-b9897f757752/osd-block-87e548f4-b9b5-4ed8-aca8-de703a341a50
 | hexdump -C
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ||
*
1000

What could have caused this to happen? The ceph-bluestore-tool isn’t able to 
repair it. Do I need to remove the OSD and create a new one?
Thanks,
Alison
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Reed Dier

It looks like 16.2.14 was released, but it looks like in an incomplete way in 
the debian repo?

I first noticed it because my nightly mirror snapshot picked it up, and showed 
that the majority of packages were removed, and only a handful had a new 
version.
> focal-ceph-pacific 230829 to 230830
>   Arch   | Package  | Version in A
>  | Version in B
> - all| ceph-grafana-dashboards  | 16.2.13-1focal  
>  | -   
> - all| ceph-mgr-cephadm | 16.2.13-1focal  
>  | -   
> ! all| ceph-mgr-dashboard   | 16.2.13-1focal  
>  | 16.2.14-1focal  
> - all| ceph-mgr-diskprediction-local| 16.2.13-1focal  
>  | -   
> - all| ceph-mgr-k8sevents   | 16.2.13-1focal  
>  | -   
> - all| ceph-mgr-modules-core| 16.2.13-1focal  
>  | -   
> - all| ceph-mgr-rook| 16.2.13-1focal  
>  | -   
> ! all| ceph-prometheus-alerts   | 16.2.13-1focal  
>  | 16.2.14-1focal  
> - all| cephfs-shell | 16.2.13-1focal  
>  | -   
> - all| cephfs-top   | 16.2.13-1focal  
>  | -   
> - all| libcephfs-java   | 16.2.13-1focal  
>  | -   
> ! all| python3-ceph-argparse| 16.2.13-1focal  
>  | 16.2.14-1focal  
> ! all| python3-ceph-common  | 16.2.13-1focal  
>  | 16.2.14-1focal  
> - amd64  | ceph | 16.2.13-1focal  
>  | -   
> ! amd64  | ceph-base| 16.2.13-1focal  
>  | 16.2.14-1focal  
> ! amd64  | ceph-base-dbg| 16.2.13-1focal  
>  | 16.2.14-1focal  
> - amd64  | ceph-common  | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-common-dbg  | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-fuse| 16.2.13-1focal  
>  | -   
> - amd64  | ceph-fuse-dbg| 16.2.13-1focal  
>  | -   
> - amd64  | ceph-immutable-object-cache  | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-immutable-object-cache-dbg  | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-mds | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-mds-dbg | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-mgr | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-mgr-dbg | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-mon | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-mon-dbg | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-osd | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-osd-dbg | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-resource-agents | 16.2.13-1focal  
>  | -   
> - amd64  | ceph-test| 16.2.13-1focal  
>  | -   
> - amd64

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Yuri Weinstein

16.2.14 has not been released yet.

Please don't do any upgrades before we send an announcement email.

TIA

On Wed, Aug 30, 2023 at 8:45 AM Reed Dier  wrote:
>
> It looks like 16.2.14 was released, but it looks like in an incomplete way in 
> the debian repo?
>
> I first noticed it because my nightly mirror snapshot picked it up, and 
> showed that the majority of packages were removed, and only a handful had a 
> new version.
>
> focal-ceph-pacific 230829 to 230830
>   Arch   | Package  | Version in A
>  | Version in B
> - all| ceph-grafana-dashboards  | 16.2.13-1focal  
>  | -
> - all| ceph-mgr-cephadm | 16.2.13-1focal  
>  | -
> ! all| ceph-mgr-dashboard   | 16.2.13-1focal  
>  | 16.2.14-1focal
> - all| ceph-mgr-diskprediction-local| 16.2.13-1focal  
>  | -
> - all| ceph-mgr-k8sevents   | 16.2.13-1focal  
>  | -
> - all| ceph-mgr-modules-core| 16.2.13-1focal  
>  | -
> - all| ceph-mgr-rook| 16.2.13-1focal  
>  | -
> ! all| ceph-prometheus-alerts   | 16.2.13-1focal  
>  | 16.2.14-1focal
> - all| cephfs-shell | 16.2.13-1focal  
>  | -
> - all| cephfs-top   | 16.2.13-1focal  
>  | -
> - all| libcephfs-java   | 16.2.13-1focal  
>  | -
> ! all| python3-ceph-argparse| 16.2.13-1focal  
>  | 16.2.14-1focal
> ! all| python3-ceph-common  | 16.2.13-1focal  
>  | 16.2.14-1focal
> - amd64  | ceph | 16.2.13-1focal  
>  | -
> ! amd64  | ceph-base| 16.2.13-1focal  
>  | 16.2.14-1focal
> ! amd64  | ceph-base-dbg| 16.2.13-1focal  
>  | 16.2.14-1focal
> - amd64  | ceph-common  | 16.2.13-1focal  
>  | -
> - amd64  | ceph-common-dbg  | 16.2.13-1focal  
>  | -
> - amd64  | ceph-fuse| 16.2.13-1focal  
>  | -
> - amd64  | ceph-fuse-dbg| 16.2.13-1focal  
>  | -
> - amd64  | ceph-immutable-object-cache  | 16.2.13-1focal  
>  | -
> - amd64  | ceph-immutable-object-cache-dbg  | 16.2.13-1focal  
>  | -
> - amd64  | ceph-mds | 16.2.13-1focal  
>  | -
> - amd64  | ceph-mds-dbg | 16.2.13-1focal  
>  | -
> - amd64  | ceph-mgr | 16.2.13-1focal  
>  | -
> - amd64  | ceph-mgr-dbg | 16.2.13-1focal  
>  | -
> - amd64  | ceph-mon | 16.2.13-1focal  
>  | -
> - amd64  | ceph-mon-dbg | 16.2.13-1focal  
>  | -
> - amd64  | ceph-osd | 16.2.13-1focal  
>  | -
> - amd64  | ceph-osd-dbg | 16.2.13-1focal  
>  | -
> - amd64  | ceph-resource-agents | 16.2.13-1focal  
>  | -
> - amd64  | ceph-test| 16.2.13-1focal  
>  | -
> - amd64  | ceph-test-dbg| 16.2.13-1focal  
>  | -
> - amd64  | cephadm  | 16.2.13-1focal  
>  | -
> - amd64  | cephfs-mirror| 16.2.13-1focal  
>  | -
> - amd64  | cephfs-mirror-dbg| 16.2.13-1focal  
>  | -
> - amd64  | libcephfs-dev| 16.2.13-1focal  
>  | -
> - amd64  | libcephfs-jni| 16.2.13-1focal  
>  | -
> - amd64  | libcephfs2   | 16.2.13-1focal  
>  | -
> - amd64  | libcephfs2-dbg   | 16.2.13-1focal  
>  | -
> - amd64  | libjaeger| 16.2.13-1focal  
>  | -
> - amd64  | librados-dev | 16.2.13-1focal

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Burkhard Linke


Hi,

On 8/30/23 18:26, Yuri Weinstein wrote:

16.2.14 has not been released yet.

Please don't do any upgrades before we send an announcement email.



Then stop pushing packets before the announcement. This is not the first 
time this problem occurred. And given your answer I'm afraid it won't be 
the last time.


It can't be that hard to coordinate releases.


Regards,

Burkhard

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Reed Dier

This is more the sentiment that I was hoping to convey.

Sure, I have my finger on the pulse of the mailing list and the packages coming 
down the pipe, but assuming that everyone does and/or will is not a safe 
assumption.
At the minimum, publishing the versioned repos at $repourl/debian-16.2.14 but 
not cutting the symlink over for  $repourl/debian-pacific until “ready” seems 
like a very easy and useful release process improvement to prevent these 
specific issues going forward.

Likewise, $repourl/rpm-pacific is already pointing to $repourl/rpm-16.2.14 as 
well, so its not a debian specific issue, albeit it looks like there were no 
issues with missing packages on el8.
But the packages were still “pre-released” before we are supposed to use them.

Anything making it to $repourl/{deb,rpm}-$named_release should be “safe.”
Because the documentation 

 uses the named repos, as it should, and right now the documentation is 
effectively broken.

> ubuntu@mini-pacific:~$ cat /etc/apt/sources.list.d/ceph.list ; sudo apt 
> update | grep ceph ; sudo apt install ceph ceph-osd ceph-mon ceph-mgr
> deb https://download.ceph.com/debian-pacific/ focal main
> 
> WARNING: apt does not have a stable CLI interface. Use with caution in 
> scripts.
> 
> Hit:5 https://download.ceph.com/debian-pacific focal InRelease
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
> 
> The following packages have unmet dependencies:
>  ceph-mgr : Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not 
> going to be installed
>  ceph-mon : Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not 
> going to be installed
>  ceph-osd : PreDepends: ceph-common (= 15.2.17-0ubuntu0.20.04.4) but it is 
> not going to be installed
> Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not 
> going to be installed
> E: Unable to correct problems, you have held broken packages.

Reed

> On Aug 30, 2023, at 11:33 AM, Burkhard Linke 
>  wrote:
> 
> Hi,
> 
> On 8/30/23 18:26, Yuri Weinstein wrote:
>> 16.2.14 has not been released yet.
>> 
>> Please don't do any upgrades before we send an announcement email.
> 
> 
> Then stop pushing packets before the announcement. This is not the first time 
> this problem occurred. And given your answer I'm afraid it won't be the last 
> time.
> 
> It can't be that hard to coordinate releases.
> 
> 
> Regards,
> 
> Burkhard
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CLT Meeting minutes 2023-08-30

2023-08-30 Thread Nizamudeen A

Hello,

Finish v18.2.0 upgrade on LRC? It seems to be running v18.1.3
 not much of a difference in code commits

news on teuthology jobs hanging?
 cephfs issues because of network troubles
 Its resolved by Patrick

User council discussion follow-up
 Detailed info on this pad: https://pad.ceph.com/p/user_dev_relaunch
 First topic will come from David's team

16.2.14 release
 Pushing to release by this week.

Regards,
Nizam

-- 

Nizamudeen A

Software Engineer

Red Hat 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Multisite RGW setup not working when following the docs step by step

2023-08-30 Thread Petr Bena

Hello,

My goal is to setup multisite RGW with 2 separate CEPH clusters in separate 
datacenters, where RGW data are being replicated. I created a lab for this 
purpose in both locations (with latest reef ceph installed using cephadm) and 
tried to follow this guide: https://docs.ceph.com/en/reef/radosgw/multisite/

Unfortunatelly, even after multiple attempts it always failed when creating a 
secondary zone. I could succesfully pull the realm from master, but that was 
pretty much the last trully succesful step. I can notice that immediately after 
pulling the realm to secondary, radosgw-admin user list returns an empty list 
(which IMHO should contain replicated user list from master). Continuing by 
setting default real and zonegroup and creating the secondary zone in secondary 
cluster I end up having 2 zones in each cluster, both seemingly in same 
zonegroup, but with replication failing - this is what I see in sync status:

(master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin sync status
  realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
  zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
   zone 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
   current time 2023-08-30T12:58:12Z
zonegroup features enabled: resharding
   disabled: compress-encrypted
  metadata sync no sync (zone is master)
2023-08-30T12:58:13.991+ 7f583a52c780  0 ERROR: failed to fetch datalog info
  data sync source: 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
failed to retrieve sync info: (13) Permission denied


(secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin sync status
  realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
  zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
   zone 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
   current time 2023-08-30T12:58:54Z
zonegroup features enabled: resharding
   disabled: compress-encrypted
  metadata sync failed to read sync status: (2) No such file or directory
2023-08-30T12:58:55.617+ 7ff37c9db780  0 ERROR: failed to fetch datalog info
  data sync source: 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
failed to retrieve sync info: (13) Permission denied


In master there is one user created during the process (synchronization-user), 
on slave there are no users and when I try to re-create this synchronization 
user it complains I shouldn't even try and instead execute the command on 
master. I can see same realm and zonegroup IDs on both sides, zone list is 
different though:


(master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin zone list
{
"default_info": "97fb5842-713a-4995-8966-5afe1384f17f",
"zones": [
"cz-brn",
"default"
]
}


(secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin zone list
{
"default_info": "13a8c663-b241-4d8a-a424-8785fc539ec5",
"zones": [
"cz-hol",
"default"
]
}


The permission denied error is puzzling me - could it be because real pull 
didn't sync the users? I tried this multiple times with clean ceph install on 
both sides - and always ended up the same. I even tried force creating the same 
user with same secrets on the other side, but it didn't help. How can I debug 
what kind of secret is secondary trying to use when communicating with master? 
Could it be that this multisite RGW setup is not yet truly supported in reef? I 
noticed that the documentation itself seems written for older ceph versions, as 
there are no mentions about orchestrator (for example in steps where 
configuration files of RGW need to be edited, which is done differently when 
using cephadm).

I think that documentation is simply wrong at this time. Either it's missing 
some crucial steps, or it's outdated or otherwise unclear - simply by following 
all the steps as outlined there, you are likely to end up the same.


Thanks for help!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: radosgw mulsite multi zone configuration: current period realm name not same as in zonegroup

2023-08-30 Thread Adiga, Anantha

Update:  There was a networking issue between the sites, after fixing it , the 
issue reported below did not occur.

Thank you,
Anantha

From: Adiga, Anantha
Sent: Thursday, August 24, 2023 2:40 PM
To: ceph-users@ceph.io
Subject: radosgw mulsite multi zone configuration: current period realm name 
not same as in zonegroup

Hi,

I have a multi zone configuration with 4 zones.

While adding a secondary zone, getting this error:

root@cs17ca101ja0702:/# radosgw-admin realm pull --rgw-realm=global 
--url=http://10.45.128.139:8080 --default --access-key=sync_user 
--secret=sync_secret
request failed: (13) Permission denied
If the realm has been changed on the master zone, the master zone's gateway may 
need to be restarted to recognize this user.
root@cs17ca101ja0702:/#

The realm name is "global". Is the cause of the error due to the primary 
cluster having a current period listing the realm name as "default" instead of 
"global" ?  However, the realm id is of realm "global" AND the zonegroup does 
not list realm name but has the correct realm id. See below.

How to fix this issue.

root@fl31ca104ja0201:/# radosgw-admin realm get
{
"id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b",
"name": "global",
"current_period": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6",
"epoch": 2
}
root@fl31ca104ja0201:/# radosgw-admin realm get --rgw-realm=global
{
"id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b",
"name": "global",
"current_period": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6",
"epoch": 2
}

root@fl31ca104ja0201:/# radosgw-admin zonegroup list
{
"default_info": "ec8b68db-1900-464f-a21a-2f6e8c107e94",
"zonegroups": [
"alldczg"
]
}

root@fl31ca104ja0201:/# radosgw-admin zonegroup get --rgw-zonegroup=alldczg
{
"id": "ec8b68db-1900-464f-a21a-2f6e8c107e94",
"name": "alldczg",
"api_name": "alldczg",
"is_master": "true",
"endpoints": [
http://10.45.128.139:8080,
http://172.18.55.71:8080,
http://10.239.155.23:8080
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "ae267592-7cd8-4d67-8792-adc57d104cd6",
"zones": [
{
"id": "0962f0b4-beb6-4d07-a64d-07046b81529e",
"name": "CRsite",
"endpoints": [
http://172.18.55.71:8080
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 11,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
},
{
"id": "9129d118-55ac-4859-b339-b8afe0793a80",
"name": "BArga",
"endpoints": [
http://10.208.11.26:8080
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 11,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
},
{
"id": "ae267592-7cd8-4d67-8792-adc57d104cd6",
"name": "ORflex2",
"endpoints": [
http://10.45.128.139:8080
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 11,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
},
{
"id": "f5edeb4b-2a37-413b-8587-0ff40d7647ea",
"name": "SHGrasp",
"endpoints": [
http://10.239.155.23:8080
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 11,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "3da7b5ea-c44b-4d44-aced-fae2aabce97b",
"sync_policy": {
"groups": []
}
}

root@fl31ca104ja0201:/# radosgw-admin period get-current
{
"current_period": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6"
}
root@fl31ca104ja0201:/# radosgw-admin period get
{
"id": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6",
"epoch": 42,
"predecessor_uuid": "2df86f9a-d267-4b52-a13b-def8e5e612a2",
"sync_status": [],
"period_map": {
"id": "b8bc1187-2a2d-4d9e-b7be-c4f4667e3fa6",
"zonegroups": [
{
"id": "ec8b68db-1900-464f-a21a-2f6e8c107e94",
"name": "alldczg",
"api_name": "alldczg",
"is_master": "true",
"endpoints": [

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Zakhar Kirpichenko

Hi,

Please note that some packages have been pushed for Ubuntu focal as well,
but the repo is incomplete. I think it would be good if such things could
be avoided in the future.

/Z

On Wed, 30 Aug 2023 at 19:27, Yuri Weinstein  wrote:

> 16.2.14 has not been released yet.
>
> Please don't do any upgrades before we send an announcement email.
>
> TIA
>
> On Wed, Aug 30, 2023 at 8:45 AM Reed Dier  wrote:
> >
> > It looks like 16.2.14 was released, but it looks like in an incomplete
> way in the debian repo?
> >
> > I first noticed it because my nightly mirror snapshot picked it up, and
> showed that the majority of packages were removed, and only a handful had a
> new version.
> >
> > focal-ceph-pacific 230829 to 230830
> >   Arch   | Package  | Version in A
>| Version in B
> > - all| ceph-grafana-dashboards  | 16.2.13-1focal
>| -
> > - all| ceph-mgr-cephadm | 16.2.13-1focal
>| -
> > ! all| ceph-mgr-dashboard   | 16.2.13-1focal
>| 16.2.14-1focal
> > - all| ceph-mgr-diskprediction-local| 16.2.13-1focal
>| -
> > - all| ceph-mgr-k8sevents   | 16.2.13-1focal
>| -
> > - all| ceph-mgr-modules-core| 16.2.13-1focal
>| -
> > - all| ceph-mgr-rook| 16.2.13-1focal
>| -
> > ! all| ceph-prometheus-alerts   | 16.2.13-1focal
>| 16.2.14-1focal
> > - all| cephfs-shell | 16.2.13-1focal
>| -
> > - all| cephfs-top   | 16.2.13-1focal
>| -
> > - all| libcephfs-java   | 16.2.13-1focal
>| -
> > ! all| python3-ceph-argparse| 16.2.13-1focal
>| 16.2.14-1focal
> > ! all| python3-ceph-common  | 16.2.13-1focal
>| 16.2.14-1focal
> > - amd64  | ceph | 16.2.13-1focal
>| -
> > ! amd64  | ceph-base| 16.2.13-1focal
>| 16.2.14-1focal
> > ! amd64  | ceph-base-dbg| 16.2.13-1focal
>| 16.2.14-1focal
> > - amd64  | ceph-common  | 16.2.13-1focal
>| -
> > - amd64  | ceph-common-dbg  | 16.2.13-1focal
>| -
> > - amd64  | ceph-fuse| 16.2.13-1focal
>| -
> > - amd64  | ceph-fuse-dbg| 16.2.13-1focal
>| -
> > - amd64  | ceph-immutable-object-cache  | 16.2.13-1focal
>| -
> > - amd64  | ceph-immutable-object-cache-dbg  | 16.2.13-1focal
>| -
> > - amd64  | ceph-mds | 16.2.13-1focal
>| -
> > - amd64  | ceph-mds-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-mgr | 16.2.13-1focal
>| -
> > - amd64  | ceph-mgr-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-mon | 16.2.13-1focal
>| -
> > - amd64  | ceph-mon-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-osd | 16.2.13-1focal
>| -
> > - amd64  | ceph-osd-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-resource-agents | 16.2.13-1focal
>| -
> > - amd64  | ceph-test| 16.2.13-1focal
>| -
> > - amd64  | ceph-test-dbg| 16.2.13-1focal
>| -
> > - amd64  | cephadm  | 16.2.13-1focal
>| -
> > - amd64  | cephfs-mirror| 16.2.13-1focal
>| -
> > - amd64  | cephfs-mirror-dbg| 16.2.13-1focal
>| -
> > - amd64  | libcephfs-dev| 16.2.13-1focal
>| -
> > - amd64  | libcephfs-jni| 16.2.13-1focal
>| -
> > - amd64  | libcephfs2   | 16.2.13-1focal
>| -
> > - amd64  | libcephfs2-dbg   | 16.2.13-1focal
>| -

[ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung

2023-08-30 Thread Adiga, Anantha

Hi Venky,

“peer-bootstrap import” is working fine now. It was port 3300 blocked by 
firewall.
Thank you for your help.

Regards,
Anantha

From: Adiga, Anantha
Sent: Monday, August 7, 2023 1:29 PM
To: Venky Shankar ; ceph-users@ceph.io
Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung

Hi Venky,

Could this be the reason that the peer-bootstrap import is hanging?  how do I 
upgrade cephfs-mirror to Quincy?
root@fl31ca104ja0201:/# cephfs-mirror --version
ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)
root@fl31ca104ja0201:/# ceph version
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
root@fl31ca104ja0201:/#


Thank you,
Anantha
From: Adiga, Anantha
Sent: Monday, August 7, 2023 11:21 AM
To: 'Venky Shankar' mailto:vshan...@redhat.com>>; 
'ceph-users@ceph.io' mailto:ceph-users@ceph.io>>
Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung

Hi Venky,

I tried on another secondary Quincy cluster and it is the same problem. The 
peer_bootstrap mport  command hangs.



root@fl31ca104ja0201:/# ceph fs  snapshot mirror peer_bootstrap import cephfs 
eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ==

……

…….

..command does not complete..waits here
^C  to exit.
Thereafter some commands do not complete…
root@fl31ca104ja0201:/# ceph -s
  cluster:
id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
health: HEALTH_OK

  services:
mon:   3 daemons, quorum 
fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 2d)
mgr:   fl31ca104ja0201.kkoono(active, since 3d), standbys: 
fl31ca104ja0202, fl31ca104ja0203
mds:   1/1 daemons up, 2 standby
osd:   44 osds: 44 up (since 2d), 44 in (since 5w)
cephfs-mirror: 1 daemon active (1 hosts)
rgw:   3 daemons active (3 hosts, 1 zones)

  data:
volumes: 1/1 healthy
pools:   25 pools, 769 pgs
objects: 614.40k objects, 1.9 TiB
usage:   2.9 TiB used, 292 TiB / 295 TiB avail
pgs: 769 active+clean

  io:
client:   32 KiB/s rd, 0 B/s wr, 33 op/s rd, 1 op/s wr

root@fl31ca104ja0201:/#
root@fl31ca104ja0201:/# ceph fs status cephfs
This command also waits. ……

I have attached the mgr log
root@fl31ca104ja0201:/# ceph service status
{
"cephfs-mirror": {
"5306346": {
"status_stamp": "2023-08-07T17:35:56.884907+",
"last_beacon": "2023-08-07T17:45:01.903540+",
"status": {
"status_json": 
"{\"1\":{\"name\":\"cephfs\",\"directory_count\":0,\"peers\":{}}}"
}
}

Quincy secondary cluster


root@a001s008-zz14l47008:/# ceph mgr module enable mirroring

root@a001s008-zz14l47008:/# ceph fs authorize cephfs client.mirror_remote / rwps

[client.mirror_remote]

key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA==

root@a001s008-zz14l47008:/# ceph auth get client.mirror_remote

[client.mirror_remote]

key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA==

caps mds = "allow rwps fsname=cephfs"

caps mon = "allow r fsname=cephfs"

caps osd = "allow rw tag cephfs data=cephfs"

root@a001s008-zz14l47008:/#

root@a001s008-zz14l47008:/# ceph fs snapshot mirror peer_bootstrap create 
cephfs client.mirror_remote shgR-site

{"token": 
"eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ=="}

root@a001s008-zz14l47008:/#

Thank you,
Anantha

From: Adiga, Anantha
Sent: Friday, August 4, 2023 11:55 AM
To: Venky Shankar mailto:vshan...@redhat.com>>; 
ceph-users@ceph.io
Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung


Hi Venky,



Thank you so much for the guidance. Attached is the mgr log.



Note: the 4th node in the primary cluster has smaller capacity  drives, the 
other 3 nodes have the larger capacity drives.

32ssd6.98630   1.0  7.0 TiB   44 GiB   44 GiB   183 KiB  148 MiB  
6.9 TiB  0.62  0.64   40  up  osd.32

-7  76.84927 -   77 TiB  652 GiB  648 GiB20 MiB  3.0 GiB   
76 TiB  0.83  0.86-  host fl31ca104ja0203

  1ssd6.98630   1.0  7.0 TiB   73 GiB   73 GiB

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Laura Flores

Hi,

We are still in progress creating the release. Any artifacts are not yet
officially released. We will send the usual blog post and email when
everything's ready.

- Laura

On Wed, Aug 30, 2023 at 1:16 PM Zakhar Kirpichenko  wrote:

> Hi,
>
> Please note that some packages have been pushed for Ubuntu focal as well,
> but the repo is incomplete. I think it would be good if such things could
> be avoided in the future.
>
> /Z
>
> On Wed, 30 Aug 2023 at 19:27, Yuri Weinstein  wrote:
>
> > 16.2.14 has not been released yet.
> >
> > Please don't do any upgrades before we send an announcement email.
> >
> > TIA
> >
> > On Wed, Aug 30, 2023 at 8:45 AM Reed Dier  wrote:
> > >
> > > It looks like 16.2.14 was released, but it looks like in an incomplete
> > way in the debian repo?
> > >
> > > I first noticed it because my nightly mirror snapshot picked it up, and
> > showed that the majority of packages were removed, and only a handful
> had a
> > new version.
> > >
> > > focal-ceph-pacific 230829 to 230830
> > >   Arch   | Package  | Version in A
> >| Version in B
> > > - all| ceph-grafana-dashboards  | 16.2.13-1focal
> >| -
> > > - all| ceph-mgr-cephadm | 16.2.13-1focal
> >| -
> > > ! all| ceph-mgr-dashboard   | 16.2.13-1focal
> >| 16.2.14-1focal
> > > - all| ceph-mgr-diskprediction-local| 16.2.13-1focal
> >| -
> > > - all| ceph-mgr-k8sevents   | 16.2.13-1focal
> >| -
> > > - all| ceph-mgr-modules-core| 16.2.13-1focal
> >| -
> > > - all| ceph-mgr-rook| 16.2.13-1focal
> >| -
> > > ! all| ceph-prometheus-alerts   | 16.2.13-1focal
> >| 16.2.14-1focal
> > > - all| cephfs-shell | 16.2.13-1focal
> >| -
> > > - all| cephfs-top   | 16.2.13-1focal
> >| -
> > > - all| libcephfs-java   | 16.2.13-1focal
> >| -
> > > ! all| python3-ceph-argparse| 16.2.13-1focal
> >| 16.2.14-1focal
> > > ! all| python3-ceph-common  | 16.2.13-1focal
> >| 16.2.14-1focal
> > > - amd64  | ceph | 16.2.13-1focal
> >| -
> > > ! amd64  | ceph-base| 16.2.13-1focal
> >| 16.2.14-1focal
> > > ! amd64  | ceph-base-dbg| 16.2.13-1focal
> >| 16.2.14-1focal
> > > - amd64  | ceph-common  | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-common-dbg  | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-fuse| 16.2.13-1focal
> >| -
> > > - amd64  | ceph-fuse-dbg| 16.2.13-1focal
> >| -
> > > - amd64  | ceph-immutable-object-cache  | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-immutable-object-cache-dbg  | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-mds | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-mds-dbg | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-mgr | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-mgr-dbg | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-mon | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-mon-dbg | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-osd | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-osd-dbg | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-resource-agents | 16.2.13-1focal
> >| -
> > > - amd64  | ceph-test| 16.2.13-1focal
> >| -
> > > - amd64  | ceph-test-dbg| 16.2.13-1focal
> >| -
> > > - amd64  | cephadm  | 16.2.13-1focal
> >| -
> > > - amd64  | cephfs-mirror| 16.2.13-1focal
> >| -
> > > - amd64  | cephfs-mirror-dbg| 16.2.13-1focal
> >

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Paul Mezzanini

-> "At the minimum, publishing the versioned repos at $repourl/debian-16.2.14 
but not cutting the symlink over for  $repourl/debian-pacific until “ready” 
seems like a very easy and useful release process improvement to prevent these 
specific issues going forward."

This should be standard procedure for new releases.  Numerous linux 
distributions already operate this way to make sure mirrors get copes before 
the thundering herds do upgrades. 


--

Paul Mezzanini
Platform Engineer III
Research Computing

Rochester Institute of Technology

 “End users is a description, not a goal.”






From: Reed Dier 
Sent: Wednesday, August 30, 2023 1:16 PM
To: Yuri Weinstein
Cc: ceph-users
Subject: [ceph-users] Re: Pacific 16.2.14 debian Incomplete

This is more the sentiment that I was hoping to convey.

Sure, I have my finger on the pulse of the mailing list and the packages coming 
down the pipe, but assuming that everyone does and/or will is not a safe 
assumption.
At the minimum, publishing the versioned repos at $repourl/debian-16.2.14 but 
not cutting the symlink over for  $repourl/debian-pacific until “ready” seems 
like a very easy and useful release process improvement to prevent these 
specific issues going forward.

Likewise, $repourl/rpm-pacific is already pointing to $repourl/rpm-16.2.14 as 
well, so its not a debian specific issue, albeit it looks like there were no 
issues with missing packages on el8.
But the packages were still “pre-released” before we are supposed to use them.

Anything making it to $repourl/{deb,rpm}-$named_release should be “safe.”
Because the 
documentation
 uses the named repos, as it should, and right now the documentation is 
effectively broken.

ubuntu@mini-pacific:~$ cat /etc/apt/sources.list.d/ceph.list ; sudo apt update 
| grep ceph ; sudo apt install ceph ceph-osd ceph-mon ceph-mgr
deb https://download.ceph.com/debian-pacific/ focal main

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Hit:5 https://download.ceph.com/debian-pacific focal InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 ceph-mgr : Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not going 
to be installed
 ceph-mon : Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not going 
to be installed
 ceph-osd : PreDepends: ceph-common (= 15.2.17-0ubuntu0.20.04.4) but it is not 
going to be installed
Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not going 
to be installed
E: Unable to correct problems, you have held broken packages.

Reed

On Aug 30, 2023, at 11:33 AM, Burkhard Linke 
mailto:burkhard.li...@computational.bio.uni-giessen.de>>
 wrote:

Hi,

On 8/30/23 18:26, Yuri Weinstein wrote:
16.2.14 has not been released yet.

Please don't do any upgrades before we send an announcement email.


Then stop pushing packets before the announcement. This is not the first time 
this problem occurred. And given your answer I'm afraid it won't be the last 
time.

It can't be that hard to coordinate releases.


Regards,

Burkhard

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to 
ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Multisite RGW setup not working when following the docs step by step

2023-08-30 Thread Zac Dover

Petr,

My name is Zac Dover. I'm the upstream docs guy for the Ceph Foundation.

I will begin the process of correcting this part of the documentation. I will 
begin by reviewing the section "Creating a Secondary Zone".

My schedule is full until Sunday, but I will raise an issue in tracker.ceph.com 
on Sunday and I will keep you apprised of the progress.

Zac Dover
Upstream Docs
Ceph Foundation




--- Original Message ---
On Thursday, August 31st, 2023 at 12:31 AM, Petr Bena  wrote:


> 
> 
> Hello,
> 
> My goal is to setup multisite RGW with 2 separate CEPH clusters in separate 
> datacenters, where RGW data are being replicated. I created a lab for this 
> purpose in both locations (with latest reef ceph installed using cephadm) and 
> tried to follow this guide: https://docs.ceph.com/en/reef/radosgw/multisite/
> 
> Unfortunatelly, even after multiple attempts it always failed when creating a 
> secondary zone. I could succesfully pull the realm from master, but that was 
> pretty much the last trully succesful step. I can notice that immediately 
> after pulling the realm to secondary, radosgw-admin user list returns an 
> empty list (which IMHO should contain replicated user list from master). 
> Continuing by setting default real and zonegroup and creating the secondary 
> zone in secondary cluster I end up having 2 zones in each cluster, both 
> seemingly in same zonegroup, but with replication failing - this is what I 
> see in sync status:
> 
> (master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin sync status
> realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
> zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
> zone 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
> current time 2023-08-30T12:58:12Z
> zonegroup features enabled: resharding
> disabled: compress-encrypted
> metadata sync no sync (zone is master)
> 2023-08-30T12:58:13.991+ 7f583a52c780 0 ERROR: failed to fetch datalog 
> info
> data sync source: 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
> failed to retrieve sync info: (13) Permission denied
> 
> 
> (secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin sync status
> realm d2c4ebf9-e156-4c4e-9d56-3fff6a652e75 (ceph)
> zonegroup abc3c0ae-a84d-48d4-8e78-da251eb78781 (cz)
> zone 13a8c663-b241-4d8a-a424-8785fc539ec5 (cz-hol)
> current time 2023-08-30T12:58:54Z
> zonegroup features enabled: resharding
> disabled: compress-encrypted
> metadata sync failed to read sync status: (2) No such file or directory
> 2023-08-30T12:58:55.617+ 7ff37c9db780 0 ERROR: failed to fetch datalog 
> info
> data sync source: 97fb5842-713a-4995-8966-5afe1384f17f (cz-brn)
> failed to retrieve sync info: (13) Permission denied
> 
> 
> In master there is one user created during the process 
> (synchronization-user), on slave there are no users and when I try to 
> re-create this synchronization user it complains I shouldn't even try and 
> instead execute the command on master. I can see same realm and zonegroup IDs 
> on both sides, zone list is different though:
> 
> 
> (master) [ceph: root@ceph-lab-brn-01 /]# radosgw-admin zone list
> {
> "default_info": "97fb5842-713a-4995-8966-5afe1384f17f",
> "zones": [
> "cz-brn",
> "default"
> ]
> }
> 
> 
> (secondary) [ceph: root@ceph-lab-hol-01 /]# radosgw-admin zone list
> {
> "default_info": "13a8c663-b241-4d8a-a424-8785fc539ec5",
> "zones": [
> "cz-hol",
> "default"
> ]
> }
> 
> 
> The permission denied error is puzzling me - could it be because real pull 
> didn't sync the users? I tried this multiple times with clean ceph install on 
> both sides - and always ended up the same. I even tried force creating the 
> same user with same secrets on the other side, but it didn't help. How can I 
> debug what kind of secret is secondary trying to use when communicating with 
> master? Could it be that this multisite RGW setup is not yet truly supported 
> in reef? I noticed that the documentation itself seems written for older ceph 
> versions, as there are no mentions about orchestrator (for example in steps 
> where configuration files of RGW need to be edited, which is done differently 
> when using cephadm).
> 
> I think that documentation is simply wrong at this time. Either it's missing 
> some crucial steps, or it's outdated or otherwise unclear - simply by 
> following all the steps as outlined there, you are likely to end up the same.
> 
> 
> Thanks for help!
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Laura Flores

Hey users,

To follow up on my previous email, on behalf of the Ceph team, we apologize
for any confusion about pre-released packages. We are working on
streamlining the release process to avoid this next time.

- Laura

On Wed, Aug 30, 2023 at 2:14 PM Paul Mezzanini  wrote:

> -> "At the minimum, publishing the versioned repos at
> $repourl/debian-16.2.14 but not cutting the symlink over for
> $repourl/debian-pacific until “ready” seems like a very easy and useful
> release process improvement to prevent these specific issues going forward."
>
> This should be standard procedure for new releases.  Numerous linux
> distributions already operate this way to make sure mirrors get copes
> before the thundering herds do upgrades.
>
>
> --
>
> Paul Mezzanini
> Platform Engineer III
> Research Computing
>
> Rochester Institute of Technology
>
>  “End users is a description, not a goal.”
>
>
>
>
>
> 
> From: Reed Dier 
> Sent: Wednesday, August 30, 2023 1:16 PM
> To: Yuri Weinstein
> Cc: ceph-users
> Subject: [ceph-users] Re: Pacific 16.2.14 debian Incomplete
>
> This is more the sentiment that I was hoping to convey.
>
> Sure, I have my finger on the pulse of the mailing list and the packages
> coming down the pipe, but assuming that everyone does and/or will is not a
> safe assumption.
> At the minimum, publishing the versioned repos at $repourl/debian-16.2.14
> but not cutting the symlink over for  $repourl/debian-pacific until “ready”
> seems like a very easy and useful release process improvement to prevent
> these specific issues going forward.
>
> Likewise, $repourl/rpm-pacific is already pointing to $repourl/rpm-16.2.14
> as well, so its not a debian specific issue, albeit it looks like there
> were no issues with missing packages on el8.
> But the packages were still “pre-released” before we are supposed to use
> them.
>
> Anything making it to $repourl/{deb,rpm}-$named_release should be “safe.”
> Because the documentation<
> https://docs.ceph.com/en/pacific/install/get-packages/#configure-repositories-manually>
> uses the named repos, as it should, and right now the documentation is
> effectively broken.
>
> ubuntu@mini-pacific:~$ cat /etc/apt/sources.list.d/ceph.list ; sudo apt
> update | grep ceph ; sudo apt install ceph ceph-osd ceph-mon ceph-mgr
> deb https://download.ceph.com/debian-pacific/ focal main
>
> WARNING: apt does not have a stable CLI interface. Use with caution in
> scripts.
>
> Hit:5 https://download.ceph.com/debian-pacific focal InRelease
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
>
> The following packages have unmet dependencies:
>  ceph-mgr : Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not
> going to be installed
>  ceph-mon : Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not
> going to be installed
>  ceph-osd : PreDepends: ceph-common (= 15.2.17-0ubuntu0.20.04.4) but it is
> not going to be installed
> Depends: ceph-base (= 15.2.17-0ubuntu0.20.04.4) but it is not
> going to be installed
> E: Unable to correct problems, you have held broken packages.
>
> Reed
>
> On Aug 30, 2023, at 11:33 AM, Burkhard Linke <
> burkhard.li...@computational.bio.uni-giessen.de burkhard.li...@computational.bio.uni-giessen.de>> wrote:
>
> Hi,
>
> On 8/30/23 18:26, Yuri Weinstein wrote:
> 16.2.14 has not been released yet.
>
> Please don't do any upgrades before we send an announcement email.
>
>
> Then stop pushing packets before the announcement. This is not the first
> time this problem occurred. And given your answer I'm afraid it won't be
> the last time.
>
> It can't be that hard to coordinate releases.
>
>
> Regards,
>
> Burkhard
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io ceph-users-le...@ceph.io>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>

-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] v16.2.14 Pacific released

2023-08-30 Thread Yuri Weinstein

We're happy to announce the 14th backport release in the Pacific series.

https://ceph.io/en/news/blog/2023/v16-2-14-pacific-released/

Notable Changes
---

*  CEPHFS: After recovering a Ceph File System post following the
disaster recovery procedure, the recovered files under lost+found
directory can now be deleted.

*  ceph mgr dump command now displays the name of the mgr module that
registered a RADOS client in the name field added to elements of the
active_clients array. Previously, only the address of a module's RADOS
client was shown in the active_clients array.

Getting Ceph

* Git at git://github.com/ceph/ceph.git
* Tarball at https://download.ceph.com/tarballs/ceph-16.2.14.tar.gz
* Containers at https://quay.io/repository/ceph/ceph
* For packages, see https://docs.ceph.com/en/latest/install/get-packages/
* Release git sha1: 238ba602515df21ea7ffc75c88db29f9e5ef12c9
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Zakhar Kirpichenko

Now the release email comes and the repositories are still missing
packages. What a mess.

/Z

On Wed, 30 Aug 2023 at 19:27, Yuri Weinstein  wrote:

> 16.2.14 has not been released yet.
>
> Please don't do any upgrades before we send an announcement email.
>
> TIA
>
> On Wed, Aug 30, 2023 at 8:45 AM Reed Dier  wrote:
> >
> > It looks like 16.2.14 was released, but it looks like in an incomplete
> way in the debian repo?
> >
> > I first noticed it because my nightly mirror snapshot picked it up, and
> showed that the majority of packages were removed, and only a handful had a
> new version.
> >
> > focal-ceph-pacific 230829 to 230830
> >   Arch   | Package  | Version in A
>| Version in B
> > - all| ceph-grafana-dashboards  | 16.2.13-1focal
>| -
> > - all| ceph-mgr-cephadm | 16.2.13-1focal
>| -
> > ! all| ceph-mgr-dashboard   | 16.2.13-1focal
>| 16.2.14-1focal
> > - all| ceph-mgr-diskprediction-local| 16.2.13-1focal
>| -
> > - all| ceph-mgr-k8sevents   | 16.2.13-1focal
>| -
> > - all| ceph-mgr-modules-core| 16.2.13-1focal
>| -
> > - all| ceph-mgr-rook| 16.2.13-1focal
>| -
> > ! all| ceph-prometheus-alerts   | 16.2.13-1focal
>| 16.2.14-1focal
> > - all| cephfs-shell | 16.2.13-1focal
>| -
> > - all| cephfs-top   | 16.2.13-1focal
>| -
> > - all| libcephfs-java   | 16.2.13-1focal
>| -
> > ! all| python3-ceph-argparse| 16.2.13-1focal
>| 16.2.14-1focal
> > ! all| python3-ceph-common  | 16.2.13-1focal
>| 16.2.14-1focal
> > - amd64  | ceph | 16.2.13-1focal
>| -
> > ! amd64  | ceph-base| 16.2.13-1focal
>| 16.2.14-1focal
> > ! amd64  | ceph-base-dbg| 16.2.13-1focal
>| 16.2.14-1focal
> > - amd64  | ceph-common  | 16.2.13-1focal
>| -
> > - amd64  | ceph-common-dbg  | 16.2.13-1focal
>| -
> > - amd64  | ceph-fuse| 16.2.13-1focal
>| -
> > - amd64  | ceph-fuse-dbg| 16.2.13-1focal
>| -
> > - amd64  | ceph-immutable-object-cache  | 16.2.13-1focal
>| -
> > - amd64  | ceph-immutable-object-cache-dbg  | 16.2.13-1focal
>| -
> > - amd64  | ceph-mds | 16.2.13-1focal
>| -
> > - amd64  | ceph-mds-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-mgr | 16.2.13-1focal
>| -
> > - amd64  | ceph-mgr-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-mon | 16.2.13-1focal
>| -
> > - amd64  | ceph-mon-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-osd | 16.2.13-1focal
>| -
> > - amd64  | ceph-osd-dbg | 16.2.13-1focal
>| -
> > - amd64  | ceph-resource-agents | 16.2.13-1focal
>| -
> > - amd64  | ceph-test| 16.2.13-1focal
>| -
> > - amd64  | ceph-test-dbg| 16.2.13-1focal
>| -
> > - amd64  | cephadm  | 16.2.13-1focal
>| -
> > - amd64  | cephfs-mirror| 16.2.13-1focal
>| -
> > - amd64  | cephfs-mirror-dbg| 16.2.13-1focal
>| -
> > - amd64  | libcephfs-dev| 16.2.13-1focal
>| -
> > - amd64  | libcephfs-jni| 16.2.13-1focal
>| -
> > - amd64  | libcephfs2   | 16.2.13-1focal
>| -
> > - amd64  | libcephfs2-dbg   | 16.2.13-1focal
>| -
> > - amd64  | libjaeger| 16.2.13-1focal
>

[ceph-users] Re: Quincy NFS ingress failover

2023-08-30 Thread Thorne Lawler


Sorry everyone,

Is there any more detailed documentation on the high availability NFS 
functionality in current Ceph?


This is a pretty serious sticking point.

Thank you.

On 30/08/2023 9:33 am, Thorne Lawler wrote:

Fellow cephalopods,

I'm trying to get quick, seamless NFS failover happening on my 
four-node Ceph cluster.


I followed the instructions here:
https://docs.ceph.com/en/latest/cephadm/services/nfs/#high-availability-nfs 



but testing shows that failover doesn't happen. When I placed node 2 
("san2") in maintenance mode, the NFS service shut down:


Aug 24 14:19:03 san2 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq[1962479]: 
24/08/2023 04:19:03 : epoch 64b8af5a : san2 : ganesha.nfsd-8[Admin] 
do_shutdown :MAIN :EVENT :Removing all exports.
Aug 24 14:19:13 san2 bash[3235994]: time="2023-08-24T14:19:13+10:00" 
level=warning msg="StopSignal SIGTERM failed to stop container 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq 
in 10 seconds, resorting to SIGKILL"
Aug 24 14:19:13 san2 bash[3235994]: 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq
Aug 24 14:19:13 san2 
systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic 
e: 
Main process exited, code=exited, status=137/n/a
Aug 24 14:19:14 san2 
systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic 
e: 
Failed with result 'exit-code'.
Aug 24 14:19:14 san2 systemd[1]: Stopped Ceph 
nfs.xcpnfs.1.0.san2.datsvq for e2f1b934-ed43-11ec-80fa-04421a1a1d66.


And that's it. The ingress IP didn't move.

More odd, the cluster seems to have placed the ingress IP on node 1 
(san1) but seems to be using the NFS service on node 2.


Do I need to more tightly connect the NFS service to the keepalive and 
haproxy services, or do I need to expand the ingress services to refer 
to multiple NFS services?


Thank you.


--

Regards,

Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any 
attached files may be confidential information, and may also be the 
subject of legal professional privilege. _If you are not the intended 
recipient any use, disclosure or copying of this email is unauthorised. 
_If you received this email in error, please notify Discount Domain Name 
Services Pty Ltd on 03 9815 6868 to report this matter and delete all 
copies of this transmission together with any attachments. /

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Quincy NFS ingress failover

2023-08-30 Thread Thorne Lawler


If there isn't any documentation for this yet, can anyone tell me:

 * How do I inspect/change my NFS/haproxy/keepalived configuration?
 * What is it supposed to look like? Does someone have a working example?

Thank you.

On 31/08/2023 9:36 am, Thorne Lawler wrote:

Sorry everyone,

Is there any more detailed documentation on the high availability NFS 
functionality in current Ceph?


This is a pretty serious sticking point.

Thank you.

On 30/08/2023 9:33 am, Thorne Lawler wrote:

Fellow cephalopods,

I'm trying to get quick, seamless NFS failover happening on my 
four-node Ceph cluster.


I followed the instructions here:
https://docs.ceph.com/en/latest/cephadm/services/nfs/#high-availability-nfs 



but testing shows that failover doesn't happen. When I placed node 2 
("san2") in maintenance mode, the NFS service shut down:


Aug 24 14:19:03 san2 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq[1962479]: 
24/08/2023 04:19:03 : epoch 64b8af5a : san2 : ganesha.nfsd-8[Admin] 
do_shutdown :MAIN :EVENT :Removing all exports.
Aug 24 14:19:13 san2 bash[3235994]: time="2023-08-24T14:19:13+10:00" 
level=warning msg="StopSignal SIGTERM failed to stop container 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq 
in 10 seconds, resorting to SIGKILL"
Aug 24 14:19:13 san2 bash[3235994]: 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq
Aug 24 14:19:13 san2 
systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic 
e: 
Main process exited, code=exited, status=137/n/a
Aug 24 14:19:14 san2 
systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic 
e: 
Failed with result 'exit-code'.
Aug 24 14:19:14 san2 systemd[1]: Stopped Ceph 
nfs.xcpnfs.1.0.san2.datsvq for e2f1b934-ed43-11ec-80fa-04421a1a1d66.


And that's it. The ingress IP didn't move.

More odd, the cluster seems to have placed the ingress IP on node 1 
(san1) but seems to be using the NFS service on node 2.


Do I need to more tightly connect the NFS service to the keepalive 
and haproxy services, or do I need to expand the ingress services to 
refer to multiple NFS services?


Thank you.


--

Regards,

Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any 
attached files may be confidential information, and may also be the 
subject of legal professional privilege. _If you are not the intended 
recipient any use, disclosure or copying of this email is unauthorised. 
_If you received this email in error, please notify Discount Domain Name 
Services Pty Ltd on 03 9815 6868 to report this matter and delete all 
copies of this transmission together with any attachments. /

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Yuri Weinstein

We redeployed all packages again.

Please confirm that the issue is resolved.

Thank you for your help and patience!

On Wed, Aug 30, 2023 at 3:44 PM Zakhar Kirpichenko  wrote:
>
> Now the release email comes and the repositories are still missing packages. 
> What a mess.
>
> /Z
>
> On Wed, 30 Aug 2023 at 19:27, Yuri Weinstein  wrote:
>>
>> 16.2.14 has not been released yet.
>>
>> Please don't do any upgrades before we send an announcement email.
>>
>> TIA
>>
>> On Wed, Aug 30, 2023 at 8:45 AM Reed Dier  wrote:
>> >
>> > It looks like 16.2.14 was released, but it looks like in an incomplete way 
>> > in the debian repo?
>> >
>> > I first noticed it because my nightly mirror snapshot picked it up, and 
>> > showed that the majority of packages were removed, and only a handful had 
>> > a new version.
>> >
>> > focal-ceph-pacific 230829 to 230830
>> >   Arch   | Package  | Version in A 
>> > | Version in B
>> > - all| ceph-grafana-dashboards  | 16.2.13-1focal   
>> > | -
>> > - all| ceph-mgr-cephadm | 16.2.13-1focal   
>> > | -
>> > ! all| ceph-mgr-dashboard   | 16.2.13-1focal   
>> > | 16.2.14-1focal
>> > - all| ceph-mgr-diskprediction-local| 16.2.13-1focal   
>> > | -
>> > - all| ceph-mgr-k8sevents   | 16.2.13-1focal   
>> > | -
>> > - all| ceph-mgr-modules-core| 16.2.13-1focal   
>> > | -
>> > - all| ceph-mgr-rook| 16.2.13-1focal   
>> > | -
>> > ! all| ceph-prometheus-alerts   | 16.2.13-1focal   
>> > | 16.2.14-1focal
>> > - all| cephfs-shell | 16.2.13-1focal   
>> > | -
>> > - all| cephfs-top   | 16.2.13-1focal   
>> > | -
>> > - all| libcephfs-java   | 16.2.13-1focal   
>> > | -
>> > ! all| python3-ceph-argparse| 16.2.13-1focal   
>> > | 16.2.14-1focal
>> > ! all| python3-ceph-common  | 16.2.13-1focal   
>> > | 16.2.14-1focal
>> > - amd64  | ceph | 16.2.13-1focal   
>> > | -
>> > ! amd64  | ceph-base| 16.2.13-1focal   
>> > | 16.2.14-1focal
>> > ! amd64  | ceph-base-dbg| 16.2.13-1focal   
>> > | 16.2.14-1focal
>> > - amd64  | ceph-common  | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-common-dbg  | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-fuse| 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-fuse-dbg| 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-immutable-object-cache  | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-immutable-object-cache-dbg  | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-mds | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-mds-dbg | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-mgr | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-mgr-dbg | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-mon | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-mon-dbg | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-osd | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-osd-dbg | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-resource-agents | 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-test| 16.2.13-1focal   
>> > | -
>> > - amd64  | ceph-test-dbg| 16.2.13-1focal   
>> > | -
>> > - amd64  | cephadm  | 16.2.13-1focal   
>> > | -
>> > - amd64  | cephfs-mirror| 16.2.13-1focal   
>> > | -
>> > - amd64  | cephfs-mirror-dbg| 16.2.13-1focal   
>> >

[ceph-users] Re: Quincy NFS ingress failover

2023-08-30 Thread Thorne Lawler


Here are the yaml files I used to create the NFS and ingress services:

nfs-ingress.yaml

service_type: ingress
service_id: nfs.xcpnfs
placement:
  count: 2
spec:
  backend_service: nfs.xcpnfs
  frontend_port: 2049
  monitor_port: 9000
  virtual_ip: 172.16.172.199/24

nfs.yaml

service_type: nfs
service_id: xcpnfs
placement:
  hosts:
    - san1
    - san2
spec:
  port: 20490

Am I missing something here? Is there another mailing list where I 
should be asking about this?


On 31/08/2023 10:38 am, Thorne Lawler wrote:

If there isn't any documentation for this yet, can anyone tell me:

 * How do I inspect/change my NFS/haproxy/keepalived configuration?
 * What is it supposed to look like? Does someone have a working example?

Thank you.

On 31/08/2023 9:36 am, Thorne Lawler wrote:

Sorry everyone,

Is there any more detailed documentation on the high availability NFS 
functionality in current Ceph?


This is a pretty serious sticking point.

Thank you.

On 30/08/2023 9:33 am, Thorne Lawler wrote:

Fellow cephalopods,

I'm trying to get quick, seamless NFS failover happening on my 
four-node Ceph cluster.


I followed the instructions here:
https://docs.ceph.com/en/latest/cephadm/services/nfs/#high-availability-nfs 



but testing shows that failover doesn't happen. When I placed node 2 
("san2") in maintenance mode, the NFS service shut down:


Aug 24 14:19:03 san2 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq[1962479]: 
24/08/2023 04:19:03 : epoch 64b8af5a : san2 : ganesha.nfsd-8[Admin] 
do_shutdown :MAIN :EVENT :Removing all exports.
Aug 24 14:19:13 san2 bash[3235994]: time="2023-08-24T14:19:13+10:00" 
level=warning msg="StopSignal SIGTERM failed to stop container 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq 
in 10 seconds, resorting to SIGKILL"
Aug 24 14:19:13 san2 bash[3235994]: 
ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66-nfs-xcpnfs-1-0-san2-datsvq
Aug 24 14:19:13 san2 
systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic 
e: 
Main process exited, code=exited, status=137/n/a
Aug 24 14:19:14 san2 
systemd[1]:ceph-e2f1b934-ed43-11ec-80fa-04421a1a1d66@nfs.xcpnfs.1.0.san2.datsvq.servic 
e: 
Failed with result 'exit-code'.
Aug 24 14:19:14 san2 systemd[1]: Stopped Ceph 
nfs.xcpnfs.1.0.san2.datsvq for e2f1b934-ed43-11ec-80fa-04421a1a1d66.


And that's it. The ingress IP didn't move.

More odd, the cluster seems to have placed the ingress IP on node 1 
(san1) but seems to be using the NFS service on node 2.


Do I need to more tightly connect the NFS service to the keepalive 
and haproxy services, or do I need to expand the ingress services to 
refer to multiple NFS services?


Thank you.


--

Regards,

Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any 
attached files may be confidential information, and may also be the 
subject of legal professional privilege. _If you are not the intended 
recipient any use, disclosure or copying of this email is unauthorised. 
_If you received this email in error, please notify Discount Domain Name 
Services Pty Ltd on 03 9815 6868 to report this matter and delete all 
copies of this transmission together with any attachments. /

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung

2023-08-30 Thread Venky Shankar

Thanks for the follow up!

On Wed, Aug 30, 2023 at 11:49 PM Adiga, Anantha  wrote:
>
> Hi Venky,
>
>
>
> “peer-bootstrap import” is working fine now. It was port 3300 blocked by 
> firewall.
>
> Thank you for your help.
>
>
>
> Regards,
>
> Anantha
>
>
>
> From: Adiga, Anantha
> Sent: Monday, August 7, 2023 1:29 PM
> To: Venky Shankar ; ceph-users@ceph.io
> Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import 
> hung
>
>
>
> Hi Venky,
>
>
>
> Could this be the reason that the peer-bootstrap import is hanging?  how do I 
> upgrade cephfs-mirror to Quincy?
>
> root@fl31ca104ja0201:/# cephfs-mirror --version
>
> ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific 
> (stable)
>
> root@fl31ca104ja0201:/# ceph version
>
> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
>
> root@fl31ca104ja0201:/#
>
>
>
>
>
> Thank you,
>
> Anantha
>
> From: Adiga, Anantha
> Sent: Monday, August 7, 2023 11:21 AM
> To: 'Venky Shankar' ; 'ceph-users@ceph.io' 
> 
> Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import 
> hung
>
>
>
> Hi Venky,
>
>
>
> I tried on another secondary Quincy cluster and it is the same problem. The 
> peer_bootstrap mport  command hangs.
>
>
>
>
>
> root@fl31ca104ja0201:/# ceph fs  snapshot mirror peer_bootstrap import cephfs 
> eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ==
>
> ……
>
> …….
>
> ..command does not complete..waits here
>
> ^C  to exit.
>
> Thereafter some commands do not complete…
>
> root@fl31ca104ja0201:/# ceph -s
>
>   cluster:
>
> id: d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
>
> health: HEALTH_OK
>
>
>
>   services:
>
> mon:   3 daemons, quorum 
> fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 2d)
>
> mgr:   fl31ca104ja0201.kkoono(active, since 3d), standbys: 
> fl31ca104ja0202, fl31ca104ja0203
>
> mds:   1/1 daemons up, 2 standby
>
> osd:   44 osds: 44 up (since 2d), 44 in (since 5w)
>
> cephfs-mirror: 1 daemon active (1 hosts)
>
> rgw:   3 daemons active (3 hosts, 1 zones)
>
>
>
>   data:
>
> volumes: 1/1 healthy
>
> pools:   25 pools, 769 pgs
>
> objects: 614.40k objects, 1.9 TiB
>
> usage:   2.9 TiB used, 292 TiB / 295 TiB avail
>
> pgs: 769 active+clean
>
>
>
>   io:
>
> client:   32 KiB/s rd, 0 B/s wr, 33 op/s rd, 1 op/s wr
>
>
>
> root@fl31ca104ja0201:/#
>
> root@fl31ca104ja0201:/# ceph fs status cephfs
>
> This command also waits. ……
>
>
>
> I have attached the mgr log
>
> root@fl31ca104ja0201:/# ceph service status
>
> {
>
> "cephfs-mirror": {
>
> "5306346": {
>
> "status_stamp": "2023-08-07T17:35:56.884907+",
>
> "last_beacon": "2023-08-07T17:45:01.903540+",
>
> "status": {
>
> "status_json": 
> "{\"1\":{\"name\":\"cephfs\",\"directory_count\":0,\"peers\":{}}}"
>
> }
>
> }
>
>
>
> Quincy secondary cluster
>
>
>
> root@a001s008-zz14l47008:/# ceph mgr module enable mirroring
>
> root@a001s008-zz14l47008:/# ceph fs authorize cephfs client.mirror_remote / 
> rwps
>
> [client.mirror_remote]
>
> key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA==
>
> root@a001s008-zz14l47008:/# ceph auth get client.mirror_remote
>
> [client.mirror_remote]
>
> key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA==
>
> caps mds = "allow rwps fsname=cephfs"
>
> caps mon = "allow r fsname=cephfs"
>
> caps osd = "allow rw tag cephfs data=cephfs"
>
> root@a001s008-zz14l47008:/#
>
> root@a001s008-zz14l47008:/# ceph fs snapshot mirror peer_bootstrap create 
> cephfs client.mirror_remote shgR-site
>
> {"token": 
> "eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ=="}
>
> root@a001s008-zz14l47008:/#
>
>
>
> Thank you,
>
> Anantha
>
>
>
> From: Adiga, Anantha
> Sent: Friday, August 4, 2023 11:55 AM
> To: Venky Shankar ; ceph-users@ceph.io
> Subject: RE: [ceph-users] Re: cephfs snapshot mirror peer_bootstrap import 
> hung
>
>
>
> Hi Venky,
>
>
>
> Thank you so much for the guidance. Attached is the mgr log.
>
>
>
> Note: the 4th node in the primary cluster has smaller capacity  driv

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

2023-08-30 Thread Zakhar Kirpichenko

It looks much better, at least for Ubuntu focal, thanks!

/Z

On Thu, 31 Aug 2023 at 03:48, Yuri Weinstein  wrote:

> We redeployed all packages again.
>
> Please confirm that the issue is resolved.
>
> Thank you for your help and patience!
>
> On Wed, Aug 30, 2023 at 3:44 PM Zakhar Kirpichenko 
> wrote:
> >
> > Now the release email comes and the repositories are still missing
> packages. What a mess.
> >
> > /Z
> >
> > On Wed, 30 Aug 2023 at 19:27, Yuri Weinstein 
> wrote:
> >>
> >> 16.2.14 has not been released yet.
> >>
> >> Please don't do any upgrades before we send an announcement email.
> >>
> >> TIA
> >>
> >> On Wed, Aug 30, 2023 at 8:45 AM Reed Dier 
> wrote:
> >> >
> >> > It looks like 16.2.14 was released, but it looks like in an
> incomplete way in the debian repo?
> >> >
> >> > I first noticed it because my nightly mirror snapshot picked it up,
> and showed that the majority of packages were removed, and only a handful
> had a new version.
> >> >
> >> > focal-ceph-pacific 230829 to 230830
> >> >   Arch   | Package  | Version in A
>  | Version in B
> >> > - all| ceph-grafana-dashboards  | 16.2.13-1focal
>  | -
> >> > - all| ceph-mgr-cephadm | 16.2.13-1focal
>  | -
> >> > ! all| ceph-mgr-dashboard   | 16.2.13-1focal
>  | 16.2.14-1focal
> >> > - all| ceph-mgr-diskprediction-local| 16.2.13-1focal
>  | -
> >> > - all| ceph-mgr-k8sevents   | 16.2.13-1focal
>  | -
> >> > - all| ceph-mgr-modules-core| 16.2.13-1focal
>  | -
> >> > - all| ceph-mgr-rook| 16.2.13-1focal
>  | -
> >> > ! all| ceph-prometheus-alerts   | 16.2.13-1focal
>  | 16.2.14-1focal
> >> > - all| cephfs-shell | 16.2.13-1focal
>  | -
> >> > - all| cephfs-top   | 16.2.13-1focal
>  | -
> >> > - all| libcephfs-java   | 16.2.13-1focal
>  | -
> >> > ! all| python3-ceph-argparse| 16.2.13-1focal
>  | 16.2.14-1focal
> >> > ! all| python3-ceph-common  | 16.2.13-1focal
>  | 16.2.14-1focal
> >> > - amd64  | ceph | 16.2.13-1focal
>  | -
> >> > ! amd64  | ceph-base| 16.2.13-1focal
>  | 16.2.14-1focal
> >> > ! amd64  | ceph-base-dbg| 16.2.13-1focal
>  | 16.2.14-1focal
> >> > - amd64  | ceph-common  | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-common-dbg  | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-fuse| 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-fuse-dbg| 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-immutable-object-cache  | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-immutable-object-cache-dbg  | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-mds | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-mds-dbg | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-mgr | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-mgr-dbg | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-mon | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-mon-dbg | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-osd | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-osd-dbg | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-resource-agents | 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-test| 16.2.13-1focal
>  | -
> >> > - amd64  | ceph-test-dbg| 16.2.13-1focal
>  | -
> >> > - amd64  | cephadm  | 16.2.13-1focal
>  | -
> >> > - amd64  | cephfs-mirror| 16.2.13-1focal
>  | -
> >> > - amd64  | cephfs-mirror-dbg

[ceph-users] Re: Is there any way to fine tune peering/pg relocation/rebalance?

[ceph-users] Re: Reef - what happened to OSD spec?

[ceph-users] Re: lack of RGW_API_HOST in ceph dashboard, 17.2.6, causes ceph mgr dashboard problems

[ceph-users] Re: A couple OSDs not starting after host reboot

[ceph-users] Pacific 16.2.14 debian Incomplete

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] CLT Meeting minutes 2023-08-30

[ceph-users] Multisite RGW setup not working when following the docs step by step

[ceph-users] Re: radosgw mulsite multi zone configuration: current period realm name not same as in zonegroup

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] Re: Multisite RGW setup not working when following the docs step by step

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] v16.2.14 Pacific released

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] Re: Quincy NFS ingress failover

[ceph-users] Re: Quincy NFS ingress failover

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

[ceph-users] Re: Quincy NFS ingress failover

[ceph-users] Re: cephfs snapshot mirror peer_bootstrap import hung

[ceph-users] Re: Pacific 16.2.14 debian Incomplete

25 matches

Site Navigation

Mail list logo

Footer information