[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

2023-10-23 Thread 544463199 
It seems that the only way to modify the code is manually ...
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RadosGW load balancing with Kubernetes + ceph orch

2023-10-23 Thread Janek Bevendorff 

Hey all,

My Ceph cluster is managed mostly by cephadm / ceph orch to avoid 
circular dependencies between in our infrastructure deployment. Our 
RadosGW endpoints, however, are managed by Kubernetes, since it provides 
proper load balancing and service health checks.


This leaves me in the unsatisfactory situation that Ceph complains about 
'stray' RGW daemons in the cluster. The only two solutions to this that 
I found were a) to turn of the warning, which applies to all daemons and 
not just the RGWs (not pretty!), or b) to move the deployment out of 
Kubernetes. For the latter, I could define external Endpoints in 
Kubernetes, so that I still have load balancing, but then I don't have 
proper health checks any more. Meaning, if one of the RGW endpoints goes 
down, requests to our S3 endpoint will intermittently time out in 
round-robin fashion (not pretty at all!).


Can you think of a better option to solve this? I would already be 
satisfied with turning off the warning for RGW daemons only, but there 
doesn't seem to be a config option for that.


Thanks
Janek



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: quincy v17.2.7 QE Validation status

2023-10-23 Thread Yuri Weinstein 
If no one has anything else left, we have all issues resolved and
ready for the 17.2.7 release

On Mon, Oct 23, 2023 at 8:12 AM Laura Flores  wrote:
>
> Regarding the crash in quincy-p2p (tracked in
> https://tracker.ceph.com/issues/63257), @Prashant Dhange
>  and I evaluated it, and we've concluded it isn't a
> blocker for 17.2.7.
>
> So, quincy-p2p is approved.
>
> Thanks,
> Laura
>
>
>
> On Sat, Oct 21, 2023 at 12:27 AM Venky Shankar  wrote:
>
> > Hi Yuri,
> >
> > On Fri, Oct 20, 2023 at 9:44 AM Venky Shankar  wrote:
> > >
> > > Hi Yuri,
> > >
> > > On Thu, Oct 19, 2023 at 10:48 PM Venky Shankar 
> > wrote:
> > > >
> > > > Hi Yuri,
> > > >
> > > > On Thu, Oct 19, 2023 at 9:32 PM Yuri Weinstein 
> > wrote:
> > > > >
> > > > > We are still finishing off:
> > > > >
> > > > > - revert PR https://github.com/ceph/ceph/pull/54085, needs smoke
> > suite rerun
> > > > > - removed s3tests https://github.com/ceph/ceph/pull/54078 merged
> > > > >
> > > > > Venky, Casey FYI
> > > >
> > > > https://github.com/ceph/ceph/pull/53139 is causing a smoke test
> > > > failure. Details:
> > > > https://github.com/ceph/ceph/pull/53139#issuecomment-1771388202
> > > >
> > > > I've sent a revert for that change -
> > > > https://github.com/ceph/ceph/pull/54108 - will let you know when it's
> > > > ready for testing.
> > >
> > > smoke passes with this revert
> > >
> > >
> > https://pulpito.ceph.com/vshankar-2023-10-19_20:24:36-smoke-wip-vshankar-testing-quincy-20231019.172112-testing-default-smithi/
> > >
> > > fs suite running now...
> >
> > Test results are here -
> > https://tracker.ceph.com/projects/cephfs/wiki/Quincy#2023-October-19
> >
> > Yuri, please merge change - https://github.com/ceph/ceph/pull/54108
> >
> > and consider this as "fs approved".
> >
> > >
> > > >
> > > > >
> > > > > On Wed, Oct 18, 2023 at 9:07 PM Venky Shankar 
> > wrote:
> > > > > >
> > > > > > On Tue, Oct 17, 2023 at 12:23 AM Yuri Weinstein <
> > ywein...@redhat.com> wrote:
> > > > > > >
> > > > > > > Details of this release are summarized here:
> > > > > > >
> > > > > > > https://tracker.ceph.com/issues/63219#note-2
> > > > > > > Release Notes - TBD
> > > > > > >
> > > > > > > Issue https://tracker.ceph.com/issues/63192 appears to be
> > failing several runs.
> > > > > > > Should it be fixed for this release?
> > > > > > >
> > > > > > > Seeking approvals/reviews for:
> > > > > > >
> > > > > > > smoke - Laura
> > > > > >
> > > > > > There's one failure in the smoke tests
> > > > > >
> > > > > >
> > https://pulpito.ceph.com/yuriw-2023-10-18_14:58:31-smoke-quincy-release-distro-default-smithi/
> > > > > >
> > > > > > caused by
> > > > > >
> > > > > > https://github.com/ceph/ceph/pull/53647
> > > > > >
> > > > > > (which was marked DNM but got merged). However, it's a test case
> > thing
> > > > > > and we can live with it.
> > > > > >
> > > > > > Yuri mention in slack that he might do another round of
> > build/tests,
> > > > > > so, Yuri, here's the reverted change:
> > > > > >
> > > > > >https://github.com/ceph/ceph/pull/54085
> > > > > >
> > > > > > > rados - Laura, Radek, Travis, Ernesto, Adam King
> > > > > > >
> > > > > > > rgw - Casey
> > > > > > > fs - Venky
> > > > > > > orch - Adam King
> > > > > > >
> > > > > > > rbd - Ilya
> > > > > > > krbd - Ilya
> > > > > > >
> > > > > > > upgrade/quincy-p2p - Known issue IIRC, Casey pls confirm/approve
> > > > > > >
> > > > > > > client-upgrade-quincy-reef - Laura
> > > > > > >
> > > > > > > powercycle - Brad pls confirm
> > > > > > >
> > > > > > > ceph-volume - Guillaume pls take a look
> > > > > > >
> > > > > > > Please reply to this email with approval and/or trackers of known
> > > > > > > issues/PRs to address them.
> > > > > > >
> > > > > > > Josh, Neha - gibba and LRC upgrades -- N/A for quincy now after
> > reef release.
> > > > > > >
> > > > > > > Thx
> > > > > > > YuriW
> > > > > > > ___
> > > > > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Cheers,
> > > > > > Venky
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Cheers,
> > > > Venky
> > >
> > >
> > >
> > > --
> > > Cheers,
> > > Venky
> >
> >
> >
> > --
> > Cheers,
> > Venky
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: quincy v17.2.7 QE Validation status

2023-10-23 Thread Laura Flores 
Regarding the crash in quincy-p2p (tracked in
https://tracker.ceph.com/issues/63257), @Prashant Dhange
 and I evaluated it, and we've concluded it isn't a
blocker for 17.2.7.

So, quincy-p2p is approved.

Thanks,
Laura



On Sat, Oct 21, 2023 at 12:27 AM Venky Shankar  wrote:

> Hi Yuri,
>
> On Fri, Oct 20, 2023 at 9:44 AM Venky Shankar  wrote:
> >
> > Hi Yuri,
> >
> > On Thu, Oct 19, 2023 at 10:48 PM Venky Shankar 
> wrote:
> > >
> > > Hi Yuri,
> > >
> > > On Thu, Oct 19, 2023 at 9:32 PM Yuri Weinstein 
> wrote:
> > > >
> > > > We are still finishing off:
> > > >
> > > > - revert PR https://github.com/ceph/ceph/pull/54085, needs smoke
> suite rerun
> > > > - removed s3tests https://github.com/ceph/ceph/pull/54078 merged
> > > >
> > > > Venky, Casey FYI
> > >
> > > https://github.com/ceph/ceph/pull/53139 is causing a smoke test
> > > failure. Details:
> > > https://github.com/ceph/ceph/pull/53139#issuecomment-1771388202
> > >
> > > I've sent a revert for that change -
> > > https://github.com/ceph/ceph/pull/54108 - will let you know when it's
> > > ready for testing.
> >
> > smoke passes with this revert
> >
> >
> https://pulpito.ceph.com/vshankar-2023-10-19_20:24:36-smoke-wip-vshankar-testing-quincy-20231019.172112-testing-default-smithi/
> >
> > fs suite running now...
>
> Test results are here -
> https://tracker.ceph.com/projects/cephfs/wiki/Quincy#2023-October-19
>
> Yuri, please merge change - https://github.com/ceph/ceph/pull/54108
>
> and consider this as "fs approved".
>
> >
> > >
> > > >
> > > > On Wed, Oct 18, 2023 at 9:07 PM Venky Shankar 
> wrote:
> > > > >
> > > > > On Tue, Oct 17, 2023 at 12:23 AM Yuri Weinstein <
> ywein...@redhat.com> wrote:
> > > > > >
> > > > > > Details of this release are summarized here:
> > > > > >
> > > > > > https://tracker.ceph.com/issues/63219#note-2
> > > > > > Release Notes - TBD
> > > > > >
> > > > > > Issue https://tracker.ceph.com/issues/63192 appears to be
> failing several runs.
> > > > > > Should it be fixed for this release?
> > > > > >
> > > > > > Seeking approvals/reviews for:
> > > > > >
> > > > > > smoke - Laura
> > > > >
> > > > > There's one failure in the smoke tests
> > > > >
> > > > >
> https://pulpito.ceph.com/yuriw-2023-10-18_14:58:31-smoke-quincy-release-distro-default-smithi/
> > > > >
> > > > > caused by
> > > > >
> > > > > https://github.com/ceph/ceph/pull/53647
> > > > >
> > > > > (which was marked DNM but got merged). However, it's a test case
> thing
> > > > > and we can live with it.
> > > > >
> > > > > Yuri mention in slack that he might do another round of
> build/tests,
> > > > > so, Yuri, here's the reverted change:
> > > > >
> > > > >https://github.com/ceph/ceph/pull/54085
> > > > >
> > > > > > rados - Laura, Radek, Travis, Ernesto, Adam King
> > > > > >
> > > > > > rgw - Casey
> > > > > > fs - Venky
> > > > > > orch - Adam King
> > > > > >
> > > > > > rbd - Ilya
> > > > > > krbd - Ilya
> > > > > >
> > > > > > upgrade/quincy-p2p - Known issue IIRC, Casey pls confirm/approve
> > > > > >
> > > > > > client-upgrade-quincy-reef - Laura
> > > > > >
> > > > > > powercycle - Brad pls confirm
> > > > > >
> > > > > > ceph-volume - Guillaume pls take a look
> > > > > >
> > > > > > Please reply to this email with approval and/or trackers of known
> > > > > > issues/PRs to address them.
> > > > > >
> > > > > > Josh, Neha - gibba and LRC upgrades -- N/A for quincy now after
> reef release.
> > > > > >
> > > > > > Thx
> > > > > > YuriW
> > > > > > ___
> > > > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers,
> > > > > Venky
> > > > >
> > > >
> > >
> > >
> > > --
> > > Cheers,
> > > Venky
> >
> >
> >
> > --
> > Cheers,
> > Venky
>
>
>
> --
> Cheers,
> Venky
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph orch OSD redeployment after boot on stateless RAM root

2023-10-23 Thread Janek Bevendorff 

Hi,

I recently moved from a manual Ceph deployment using Saltstack to a 
hybrid of Saltstack and cephadm / ceph orch. We are provisioning our 
Ceph hosts using a stateless PXE RAM root, so I definitely need 
Saltstack to bootstrap at least the Ceph APT repository and the MON/MGR 
deployment. After that, ceph orch can take over and deploy the remaining 
daemons.


The MONs/MGRs are deployed after each reboot with

cephadm deploy --name mon.{{ ceph.node_id }} --fsid {{ 
ceph.conf.global.fsid }} --config /etc/ceph/ceph.conf
cephadm deploy --name mgr.{{ ceph.node_id }} --fsid {{ 
ceph.conf.global.fsid }} --config /etc/ceph/ceph.conf


(the MON store is provided in /var/lib/ceph/{{ ceph.conf.global.fsid 
}}/mon.{{ ceph.node_id }}).


Since cephadm ceph-volume lvm activate --all is broken (see 
https://tracker.ceph.com/issues/55395), I am activating each OSD 
individually like this:


cephadm deploy --name osd.{{ osd_id }} --fsid {{ ceph.conf.global.fsid 
}} --osd-fsid {{ osd_fsid }} --config /etc/ceph/ceph.conf


Now my question: Is there a better way to do this and can ceph orch take 
care of this in the same way it deploys my MDS?


All OSDs are listed as  in ceph orch ls (I think this is by 
design?) and I cannot find a way to activate them automatically via ceph 
orch when the host boots up. I tried


ceph cephadm osd activate HOSTNAME,

but all I get is "Created no osd(s) on host HOSTNAME; already created?"

The docs only talk about how I can create new OSDs, but not how I can 
automatically redeploy existing OSDs after a fresh boot. It seems like 
it is generally assumed that OSD deployments are persistent and next 
time the host boots, systemd simply activates the existing units.


I'd be glad about any hints!
Janek



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

2023-10-23 Thread Patrick Begou 

Le 23/10/2023 à 03:04, 544463...@qq.com a écrit :

I think you can try to roll back this part of the python code and wait for your 
good news :)



Not so easy 


[root@e9865d9a7f41 ceph]# git revert 
4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc

Auto-merging src/ceph-volume/ceph_volume/tests/util/test_device.py
CONFLICT (content): Merge conflict in 
src/ceph-volume/ceph_volume/tests/util/test_device.py

Auto-merging src/ceph-volume/ceph_volume/util/device.py
CONFLICT (content): Merge conflict in 
src/ceph-volume/ceph_volume/util/device.py

Auto-merging src/ceph-volume/ceph_volume/util/disk.py
CONFLICT (content): Merge conflict in 
src/ceph-volume/ceph_volume/util/disk.py
error: could not revert 4fc6bc394df... ceph-volume: Optionally consume 
loop devices


Patrick
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CephFS pool not releasing space after data deletion

2023-10-23 Thread Kuhring, Mathias 
Dear Ceph users,

Our CephFS is not releasing/freeing up space after deleting hundreds of 
terabytes of data.
By now, this drives us in a "nearfull" osd/pool situation and thus 
throttles IO.

We are on ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) 
quincy (stable).

Recently, we moved a bunch of data to a new pool with better EC.
This was done by adding a new EC pool to the FS.
Then assigning the FS root to the new EC pool via the directory layout xattr
(so all new data is written to the new pool).
And finally copying old data to new folders.

I swapped the data as follows to remain the old directory structures.
I also made snapshots for validation purposes.

So basically:
cp -r mymount/mydata/ mymount/new/ # this creates copy on new pool
mkdir mymount/mydata/.snap/tovalidate
mkdir mymount/new/mydata/.snap/tovalidate
mv mymount/mydata/ mymount/old/
mv mymount/new/mydata mymount/

I could see the increase of data in the new pool as expected (ceph df).
I compared the snapshots with hashdeep to make sure the new data is alright.

Then I went ahead deleting the old data, basically:
rmdir mymount/old/mydata/.snap/* # this also included a bunch of other 
older snapshots
rm -r mymount/old/mydata

At first we had a bunch of PGs with snaptrim/snaptrim_wait.
But they are done for quite some time now.
And now, already two weeks later the size of the old pool still hasn't 
really decreased.
I'm still waiting for around 500 TB to be released (and much more is 
planned).

I honestly have no clue, where to go from here.
 From my point of view (i.e. the CephFS mount), the data is gone.
I also never hard/soft-linked it anywhere.

This doesn't seem to be a regular issue.
At least I couldn't find anything related or resolved in the docs or 
user list, yet.
If anybody has an idea how to resolve this, I would highly appreciate it.

Best Wishes,
Mathias


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io