[ceph-users] Re: reef 18.2.3 QE validation status

[thomas]

On Jul 3, 2024 5:59 PM, Kaleb Keithley  wrote:

>

>

>


> Replacing the tar file is problematic too, if only because it's a potential 
> source of confusion for people who aren't paying attention.


It'd be really the worst thing to do.


> I'm not sure I believe that making this next release 18.2.4 really solves 
> anything


It solves *my* problem that the old version of the file is already in the 
Debian archive and cannot be replaced there. By all means, please find a better 
solution for long term. In the mean time, do *not* re-release an already 
released tarball.


Cheers,


Thomas Goirand (zigo)


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Kaleb Keithley]

On Wed, Jul 3, 2024 at 11:17 AM Casey Bodley  wrote:

> (cc Thomas Goirand)
>
> in April, an 18.2.3 tarball was uploaded to
> https://download.ceph.com/tarballs/ceph_18.2.3.orig.tar.gz. that's been
> picked up and packaged by the Debian project under the assumption that it
> was a supported release
>
> when we do finally release 18.2.3, we will presumably overwrite that
> existing tarball with the latest contents. Thomas has requested that we
> re-number this release to 18.2.4 to prevent further confusion. Thomas,
> could you explain the consequences of overwriting that tarball in place?
>

Debian is not alone in this regard. :-/

Putting tar files on download.ceph.com/tarballs ahead of the *real* GA is
misleading at best. Replacing the tar file is problematic too, if only
because it's a potential source of confusion for people who aren't paying
attention.

AIUI, after the tar file lands, if the testing is successful, a
corresponding src.rpm and a set of debian packaging files land in
download.ceph.com/rpm-$ver/... and download.ceph.com/debian-$ver/...
respectively.

I gather that most of the time testing goes well, and the sources
download.ceph.com/tarballs/... match whats in
download.ceph.com/{rpm,debian}-$ver and match what's tagged in github.

But IIRC the 18.2.2 testing did not go well what was ultimately tagged as
18.2.2 does not match the tar file in .../tarballs/ceph-18.2.2.tar.gz.  (Or
did the tar file get replaced?)

And the same thing is about to happen again with 18.2.3.

I'm not sure I believe that making this next release 18.2.4 really solves
anything, especially if we're not solving the underlying problem. Can we at
least fix the mistake of uploading a tar file to .../tarballs if it's not
the final, official tar file that matches what is tagged in github.

I'm sort of okay with replacing the 18.2.3 tar file. I'll just respin
Fedora and CentOS Storage SIG packages, either with the new tarball, or
with the sources extracted from the GA .../rpm-$ver or .../debian-$ver.
IOW with a higher Release than the previous spins) (Fedora and CentOS track
the source through their look-aside cache and the sources file in distgit,
so there's actually a trail of cookie crumbs.)

Otherwise I suppose I will be forced to wait for files to land in
.../rpm-$ver and .../debian-$ver if the tar file in .../tarballs can't be
relied on.


>
> On Tue, Jul 2, 2024 at 10:06 AM Yuri Weinstein 
> wrote:
>
>> After fixing the issues identified below we cherry-picked all PRs from
>> this list for 18.2.3
>> https://pad.ceph.com/p/release-cherry-pick-coordination.
>>
>> The question to the dev leads: do you think we can proceed with the
>> release without rerunning suites, as they were already approved?
>>
>> Please reply with your recommendations.
>>
>>
>> On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein 
>> wrote:
>>
>>> Please see the update from Laura below for the status of this release.
>>>
>>> Dev Leads help is appreciated to expedite fixes necessary to publish it
>>> soon.
>>>
>>> "Hi all, we have hit another blocker with this release.
>>>
>>> Due to centos 8 stream going end of life, we only have the option of
>>> releasing centos 9 stream containers.
>>>
>>> However, we did not test the efficacy of centos 9 stream containers
>>> against the orch and upgrade suites during the initial 18.2.3 release cycle.
>>>
>>> This problem is tracked here: https://tracker.ceph.com/issues/66334
>>>
>>> What needs to happen now is:
>>>
>>> 1.  The orch team needs to fix all references to centos 8 stream in the
>>> orch suite
>>> 2.  fs, rados, etc. need to fix their relative jobs the same way in the
>>> upgrade suite
>>>
>>> The easiest way to tackle that is to raise a PR against main and
>>> backport to stable releases since this problem actually affects main and
>>> all other releases.
>>>
>>> Then, we will:
>>>
>>> 1.  Rerun orch and upgrade with these fixes
>>> 2.  Re-approve orch and upgrade
>>> 3.  Re-upgrade gibba and LRC
>>>
>>> Then the release will be unblocked."
>>>
>>> On Tue, Jun 4, 2024 at 3:26 PM Laura Flores  wrote:
>>>
 Rados results were approved, and we successfully upgraded the gibba
 cluster.

 Now waiting on @Dan Mick  to upgrade the LRC.

 On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein 
 wrote:

> I reran rados on the fix
> https://github.com/ceph/ceph/pull/57794/commits
> and seeking approvals from Radek and Laure
>
> https://tracker.ceph.com/issues/65393#note-1
>
> On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein 
> wrote:
> >
> > We have discovered some issues (#1 and #2) during the final stages of
> > testing that require considering a delay in this point release until
> > all options and risks are assessed and resolved.
> >
> > We will keep you all updated on the progress.
> >
> > Thank you for your patience!
> >
> > #1 https://tracker.ceph.com/issues/66260
> > #2 

[ceph-users] Re: reef 18.2.3 QE validation status

[Venky Shankar]

Hi Yuri,

On Wed, Jul 10, 2024 at 7:28 PM Yuri Weinstein  wrote:
>
> We built a new branch with all the cherry-picks on top
> (https://pad.ceph.com/p/release-cherry-pick-coordination).
>
> I am rerunning fs:upgrade:
> https://pulpito.ceph.com/yuriw-2024-07-10_13:47:23-fs:upgrade-reef-release-distro-default-smithi/
>
> Venky, pls review it after it's done.

Thanks for retesting


https://pulpito.ceph.com/yuriw-2024-07-11_17:23:59-fs:upgrade-reef-release-distro-default-smithi/

Test run looks good!

>
> Neha, do you want to upgrade gibba/LRC to this build? (note there is
> no pre-release 18.2.4 built yet)
>
> On Tue, Jul 9, 2024 at 6:41 AM Casey Bodley  wrote:
> >
> > this was discussed in the ceph leadership team meeting yesterday, and
> > we've agreed to re-number this release to 18.2.4
> >
> > On Wed, Jul 3, 2024 at 1:08 PM  wrote:
> > >
> > >
> > > On Jul 3, 2024 5:59 PM, Kaleb Keithley  wrote:
> > > >
> > > >
> > > >
> > >
> > > > Replacing the tar file is problematic too, if only because it's a 
> > > > potential source of confusion for people who aren't paying attention.
> > >
> > > It'd be really the worst thing to do.
> > >
> > > > I'm not sure I believe that making this next release 18.2.4 really 
> > > > solves anything
> > >
> > > It solves *my* problem that the old version of the file is already in the 
> > > Debian archive and cannot be replaced there. By all means, please find a 
> > > better solution for long term. In the mean time, do *not* re-release an 
> > > already released tarball.
> > >
> > > Cheers,
> > >
> > > Thomas Goirand (zigo)
> > >
> >
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io



-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Neha Ojha]

On Wed, Jul 10, 2024 at 6:58 AM Yuri Weinstein  wrote:

> We built a new branch with all the cherry-picks on top
> (https://pad.ceph.com/p/release-cherry-pick-coordination).
>
> I am rerunning fs:upgrade:
>
> https://pulpito.ceph.com/yuriw-2024-07-10_13:47:23-fs:upgrade-reef-release-distro-default-smithi/
>
> Venky, pls review it after it's done.
>
> Neha, do you want to upgrade gibba/LRC to this build? (note there is
> no pre-release 18.2.4 built yet)
>

Upgrading LRC is part of the release process, so yes! Gibba is being used
for Squid validation so we can leave it out.

Thanks,
Neha


>
> On Tue, Jul 9, 2024 at 6:41 AM Casey Bodley  wrote:
> >
> > this was discussed in the ceph leadership team meeting yesterday, and
> > we've agreed to re-number this release to 18.2.4
> >
> > On Wed, Jul 3, 2024 at 1:08 PM  wrote:
> > >
> > >
> > > On Jul 3, 2024 5:59 PM, Kaleb Keithley  wrote:
> > > >
> > > >
> > > >
> > >
> > > > Replacing the tar file is problematic too, if only because it's a
> potential source of confusion for people who aren't paying attention.
> > >
> > > It'd be really the worst thing to do.
> > >
> > > > I'm not sure I believe that making this next release 18.2.4 really
> solves anything
> > >
> > > It solves *my* problem that the old version of the file is already in
> the Debian archive and cannot be replaced there. By all means, please find
> a better solution for long term. In the mean time, do *not* re-release an
> already released tarball.
> > >
> > > Cheers,
> > >
> > > Thomas Goirand (zigo)
> > >
> >
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

We built a new branch with all the cherry-picks on top
(https://pad.ceph.com/p/release-cherry-pick-coordination).

I am rerunning fs:upgrade:
https://pulpito.ceph.com/yuriw-2024-07-10_13:47:23-fs:upgrade-reef-release-distro-default-smithi/

Venky, pls review it after it's done.

Neha, do you want to upgrade gibba/LRC to this build? (note there is
no pre-release 18.2.4 built yet)

On Tue, Jul 9, 2024 at 6:41 AM Casey Bodley  wrote:
>
> this was discussed in the ceph leadership team meeting yesterday, and
> we've agreed to re-number this release to 18.2.4
>
> On Wed, Jul 3, 2024 at 1:08 PM  wrote:
> >
> >
> > On Jul 3, 2024 5:59 PM, Kaleb Keithley  wrote:
> > >
> > >
> > >
> >
> > > Replacing the tar file is problematic too, if only because it's a 
> > > potential source of confusion for people who aren't paying attention.
> >
> > It'd be really the worst thing to do.
> >
> > > I'm not sure I believe that making this next release 18.2.4 really solves 
> > > anything
> >
> > It solves *my* problem that the old version of the file is already in the 
> > Debian archive and cannot be replaced there. By all means, please find a 
> > better solution for long term. In the mean time, do *not* re-release an 
> > already released tarball.
> >
> > Cheers,
> >
> > Thomas Goirand (zigo)
> >
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Casey Bodley]

this was discussed in the ceph leadership team meeting yesterday, and
we've agreed to re-number this release to 18.2.4

On Wed, Jul 3, 2024 at 1:08 PM  wrote:
>
>
> On Jul 3, 2024 5:59 PM, Kaleb Keithley  wrote:
> >
> >
> >
>
> > Replacing the tar file is problematic too, if only because it's a potential 
> > source of confusion for people who aren't paying attention.
>
> It'd be really the worst thing to do.
>
> > I'm not sure I believe that making this next release 18.2.4 really solves 
> > anything
>
> It solves *my* problem that the old version of the file is already in the 
> Debian archive and cannot be replaced there. By all means, please find a 
> better solution for long term. In the mean time, do *not* re-release an 
> already released tarball.
>
> Cheers,
>
> Thomas Goirand (zigo)
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Venky Shankar]

Hi Yuri,

On Tue, Jul 2, 2024 at 7:36 PM Yuri Weinstein  wrote:

> After fixing the issues identified below we cherry-picked all PRs from
> this list for 18.2.3
> https://pad.ceph.com/p/release-cherry-pick-coordination.
>
> The question to the dev leads: do you think we can proceed with the
> release without rerunning suites, as they were already approved?
>

Yes, I think so it's safe to proceed (at least for CephFS). If time
permits, an fs:upgrade run would not hurt.


>
> Please reply with your recommendations.
>
>
> On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein  wrote:
>
>> Please see the update from Laura below for the status of this release.
>>
>> Dev Leads help is appreciated to expedite fixes necessary to publish it
>> soon.
>>
>> "Hi all, we have hit another blocker with this release.
>>
>> Due to centos 8 stream going end of life, we only have the option of
>> releasing centos 9 stream containers.
>>
>> However, we did not test the efficacy of centos 9 stream containers
>> against the orch and upgrade suites during the initial 18.2.3 release cycle.
>>
>> This problem is tracked here: https://tracker.ceph.com/issues/66334
>>
>> What needs to happen now is:
>>
>> 1.  The orch team needs to fix all references to centos 8 stream in the
>> orch suite
>> 2.  fs, rados, etc. need to fix their relative jobs the same way in the
>> upgrade suite
>>
>> The easiest way to tackle that is to raise a PR against main and backport
>> to stable releases since this problem actually affects main and all other
>> releases.
>>
>> Then, we will:
>>
>> 1.  Rerun orch and upgrade with these fixes
>> 2.  Re-approve orch and upgrade
>> 3.  Re-upgrade gibba and LRC
>>
>> Then the release will be unblocked."
>>
>> On Tue, Jun 4, 2024 at 3:26 PM Laura Flores  wrote:
>>
>>> Rados results were approved, and we successfully upgraded the gibba
>>> cluster.
>>>
>>> Now waiting on @Dan Mick  to upgrade the LRC.
>>>
>>> On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein 
>>> wrote:
>>>
 I reran rados on the fix
 https://github.com/ceph/ceph/pull/57794/commits
 and seeking approvals from Radek and Laure

 https://tracker.ceph.com/issues/65393#note-1

 On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein 
 wrote:
 >
 > We have discovered some issues (#1 and #2) during the final stages of
 > testing that require considering a delay in this point release until
 > all options and risks are assessed and resolved.
 >
 > We will keep you all updated on the progress.
 >
 > Thank you for your patience!
 >
 > #1 https://tracker.ceph.com/issues/66260
 > #2 https://tracker.ceph.com/issues/61948#note-21
 >
 > On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein 
 wrote:
 > >
 > > We've run into a problem during the last verification steps before
 > > publishing this release after upgrading the LRC to it  =>
 > > https://tracker.ceph.com/issues/65733
 > >
 > > After this issue is resolved, we will continue testing and
 publishing
 > > this point release.
 > >
 > > Thanks for your patience!
 > >
 > > On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
 > >  wrote:
 > > >
 > > > On 18.04.24 8:13 PM, Laura Flores wrote:
 > > > > Thanks for bringing this to our attention. The leads have
 decided that
 > > > > since this PR hasn't been merged to main yet and isn't
 approved, it
 > > > > will not go in v18.2.3, but it will be prioritized for v18.2.4.
 > > > > I've already added the PR to the v18.2.4 milestone so it's sure
 to be
 > > > > picked up.
 > > >
 > > > Thanks a bunch. If you miss the train, you miss the train - fair
 enough.
 > > > Nice to know there is another one going soon and that bug is
 going to be
 > > > on it !
 > > >
 > > >
 > > > Regards
 > > >
 > > > Christian
 > > > ___
 > > > ceph-users mailing list -- ceph-users@ceph.io
 > > > To unsubscribe send an email to ceph-users-le...@ceph.io
 > > >
 ___
 ceph-users mailing list -- ceph-users@ceph.io
 To unsubscribe send an email to ceph-users-le...@ceph.io

>>>
>>>
>>> --
>>>
>>> Laura Flores
>>>
>>> She/Her/Hers
>>>
>>> Software Engineer, Ceph Storage 
>>>
>>> Chicago, IL
>>>
>>> lflo...@ibm.com | lflo...@redhat.com 
>>> M: +17087388804
>>>
>>>
>>> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Casey Bodley]

(cc Thomas Goirand)

in April, an 18.2.3 tarball was uploaded to
https://download.ceph.com/tarballs/ceph_18.2.3.orig.tar.gz. that's been
picked up and packaged by the Debian project under the assumption that it
was a supported release

when we do finally release 18.2.3, we will presumably overwrite that
existing tarball with the latest contents. Thomas has requested that we
re-number this release to 18.2.4 to prevent further confusion. Thomas,
could you explain the consequences of overwriting that tarball in place?

On Tue, Jul 2, 2024 at 10:06 AM Yuri Weinstein  wrote:

> After fixing the issues identified below we cherry-picked all PRs from
> this list for 18.2.3
> https://pad.ceph.com/p/release-cherry-pick-coordination.
>
> The question to the dev leads: do you think we can proceed with the
> release without rerunning suites, as they were already approved?
>
> Please reply with your recommendations.
>
>
> On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein  wrote:
>
>> Please see the update from Laura below for the status of this release.
>>
>> Dev Leads help is appreciated to expedite fixes necessary to publish it
>> soon.
>>
>> "Hi all, we have hit another blocker with this release.
>>
>> Due to centos 8 stream going end of life, we only have the option of
>> releasing centos 9 stream containers.
>>
>> However, we did not test the efficacy of centos 9 stream containers
>> against the orch and upgrade suites during the initial 18.2.3 release cycle.
>>
>> This problem is tracked here: https://tracker.ceph.com/issues/66334
>>
>> What needs to happen now is:
>>
>> 1.  The orch team needs to fix all references to centos 8 stream in the
>> orch suite
>> 2.  fs, rados, etc. need to fix their relative jobs the same way in the
>> upgrade suite
>>
>> The easiest way to tackle that is to raise a PR against main and backport
>> to stable releases since this problem actually affects main and all other
>> releases.
>>
>> Then, we will:
>>
>> 1.  Rerun orch and upgrade with these fixes
>> 2.  Re-approve orch and upgrade
>> 3.  Re-upgrade gibba and LRC
>>
>> Then the release will be unblocked."
>>
>> On Tue, Jun 4, 2024 at 3:26 PM Laura Flores  wrote:
>>
>>> Rados results were approved, and we successfully upgraded the gibba
>>> cluster.
>>>
>>> Now waiting on @Dan Mick  to upgrade the LRC.
>>>
>>> On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein 
>>> wrote:
>>>
 I reran rados on the fix
 https://github.com/ceph/ceph/pull/57794/commits
 and seeking approvals from Radek and Laure

 https://tracker.ceph.com/issues/65393#note-1

 On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein 
 wrote:
 >
 > We have discovered some issues (#1 and #2) during the final stages of
 > testing that require considering a delay in this point release until
 > all options and risks are assessed and resolved.
 >
 > We will keep you all updated on the progress.
 >
 > Thank you for your patience!
 >
 > #1 https://tracker.ceph.com/issues/66260
 > #2 https://tracker.ceph.com/issues/61948#note-21
 >
 > On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein 
 wrote:
 > >
 > > We've run into a problem during the last verification steps before
 > > publishing this release after upgrading the LRC to it  =>
 > > https://tracker.ceph.com/issues/65733
 > >
 > > After this issue is resolved, we will continue testing and
 publishing
 > > this point release.
 > >
 > > Thanks for your patience!
 > >
 > > On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
 > >  wrote:
 > > >
 > > > On 18.04.24 8:13 PM, Laura Flores wrote:
 > > > > Thanks for bringing this to our attention. The leads have
 decided that
 > > > > since this PR hasn't been merged to main yet and isn't
 approved, it
 > > > > will not go in v18.2.3, but it will be prioritized for v18.2.4.
 > > > > I've already added the PR to the v18.2.4 milestone so it's sure
 to be
 > > > > picked up.
 > > >
 > > > Thanks a bunch. If you miss the train, you miss the train - fair
 enough.
 > > > Nice to know there is another one going soon and that bug is
 going to be
 > > > on it !
 > > >
 > > >
 > > > Regards
 > > >
 > > > Christian
 > > > ___
 > > > ceph-users mailing list -- ceph-users@ceph.io
 > > > To unsubscribe send an email to ceph-users-le...@ceph.io
 > > >
 ___
 ceph-users mailing list -- ceph-users@ceph.io
 To unsubscribe send an email to ceph-users-le...@ceph.io

>>>
>>>
>>> --
>>>
>>> Laura Flores
>>>
>>> She/Her/Hers
>>>
>>> Software Engineer, Ceph Storage 
>>>
>>> Chicago, IL
>>>
>>> lflo...@ibm.com | lflo...@redhat.com 
>>> M: +17087388804
>>>
>>>
>>> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email 

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

After fixing the issues identified below we cherry-picked all PRs from this
list for 18.2.3 https://pad.ceph.com/p/release-cherry-pick-coordination.

The question to the dev leads: do you think we can proceed with the release
without rerunning suites, as they were already approved?

Please reply with your recommendations.


On Thu, Jun 6, 2024 at 4:59 PM Yuri Weinstein  wrote:

> Please see the update from Laura below for the status of this release.
>
> Dev Leads help is appreciated to expedite fixes necessary to publish it
> soon.
>
> "Hi all, we have hit another blocker with this release.
>
> Due to centos 8 stream going end of life, we only have the option of
> releasing centos 9 stream containers.
>
> However, we did not test the efficacy of centos 9 stream containers
> against the orch and upgrade suites during the initial 18.2.3 release cycle.
>
> This problem is tracked here: https://tracker.ceph.com/issues/66334
>
> What needs to happen now is:
>
> 1.  The orch team needs to fix all references to centos 8 stream in the
> orch suite
> 2.  fs, rados, etc. need to fix their relative jobs the same way in the
> upgrade suite
>
> The easiest way to tackle that is to raise a PR against main and backport
> to stable releases since this problem actually affects main and all other
> releases.
>
> Then, we will:
>
> 1.  Rerun orch and upgrade with these fixes
> 2.  Re-approve orch and upgrade
> 3.  Re-upgrade gibba and LRC
>
> Then the release will be unblocked."
>
> On Tue, Jun 4, 2024 at 3:26 PM Laura Flores  wrote:
>
>> Rados results were approved, and we successfully upgraded the gibba
>> cluster.
>>
>> Now waiting on @Dan Mick  to upgrade the LRC.
>>
>> On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein 
>> wrote:
>>
>>> I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits
>>> and seeking approvals from Radek and Laure
>>>
>>> https://tracker.ceph.com/issues/65393#note-1
>>>
>>> On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein 
>>> wrote:
>>> >
>>> > We have discovered some issues (#1 and #2) during the final stages of
>>> > testing that require considering a delay in this point release until
>>> > all options and risks are assessed and resolved.
>>> >
>>> > We will keep you all updated on the progress.
>>> >
>>> > Thank you for your patience!
>>> >
>>> > #1 https://tracker.ceph.com/issues/66260
>>> > #2 https://tracker.ceph.com/issues/61948#note-21
>>> >
>>> > On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein 
>>> wrote:
>>> > >
>>> > > We've run into a problem during the last verification steps before
>>> > > publishing this release after upgrading the LRC to it  =>
>>> > > https://tracker.ceph.com/issues/65733
>>> > >
>>> > > After this issue is resolved, we will continue testing and publishing
>>> > > this point release.
>>> > >
>>> > > Thanks for your patience!
>>> > >
>>> > > On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
>>> > >  wrote:
>>> > > >
>>> > > > On 18.04.24 8:13 PM, Laura Flores wrote:
>>> > > > > Thanks for bringing this to our attention. The leads have
>>> decided that
>>> > > > > since this PR hasn't been merged to main yet and isn't approved,
>>> it
>>> > > > > will not go in v18.2.3, but it will be prioritized for v18.2.4.
>>> > > > > I've already added the PR to the v18.2.4 milestone so it's sure
>>> to be
>>> > > > > picked up.
>>> > > >
>>> > > > Thanks a bunch. If you miss the train, you miss the train - fair
>>> enough.
>>> > > > Nice to know there is another one going soon and that bug is going
>>> to be
>>> > > > on it !
>>> > > >
>>> > > >
>>> > > > Regards
>>> > > >
>>> > > > Christian
>>> > > > ___
>>> > > > ceph-users mailing list -- ceph-users@ceph.io
>>> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
>>> > > >
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>
>>
>>
>> --
>>
>> Laura Flores
>>
>> She/Her/Hers
>>
>> Software Engineer, Ceph Storage 
>>
>> Chicago, IL
>>
>> lflo...@ibm.com | lflo...@redhat.com 
>> M: +17087388804
>>
>>
>>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

Please see the update from Laura below for the status of this release.

Dev Leads help is appreciated to expedite fixes necessary to publish it
soon.

"Hi all, we have hit another blocker with this release.

Due to centos 8 stream going end of life, we only have the option of
releasing centos 9 stream containers.

However, we did not test the efficacy of centos 9 stream containers against
the orch and upgrade suites during the initial 18.2.3 release cycle.

This problem is tracked here: https://tracker.ceph.com/issues/66334

What needs to happen now is:

1.  The orch team needs to fix all references to centos 8 stream in the
orch suite
2.  fs, rados, etc. need to fix their relative jobs the same way in the
upgrade suite

The easiest way to tackle that is to raise a PR against main and backport
to stable releases since this problem actually affects main and all other
releases.

Then, we will:

1.  Rerun orch and upgrade with these fixes
2.  Re-approve orch and upgrade
3.  Re-upgrade gibba and LRC

Then the release will be unblocked."

On Tue, Jun 4, 2024 at 3:26 PM Laura Flores  wrote:

> Rados results were approved, and we successfully upgraded the gibba
> cluster.
>
> Now waiting on @Dan Mick  to upgrade the LRC.
>
> On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein 
> wrote:
>
>> I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits
>> and seeking approvals from Radek and Laure
>>
>> https://tracker.ceph.com/issues/65393#note-1
>>
>> On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein 
>> wrote:
>> >
>> > We have discovered some issues (#1 and #2) during the final stages of
>> > testing that require considering a delay in this point release until
>> > all options and risks are assessed and resolved.
>> >
>> > We will keep you all updated on the progress.
>> >
>> > Thank you for your patience!
>> >
>> > #1 https://tracker.ceph.com/issues/66260
>> > #2 https://tracker.ceph.com/issues/61948#note-21
>> >
>> > On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein 
>> wrote:
>> > >
>> > > We've run into a problem during the last verification steps before
>> > > publishing this release after upgrading the LRC to it  =>
>> > > https://tracker.ceph.com/issues/65733
>> > >
>> > > After this issue is resolved, we will continue testing and publishing
>> > > this point release.
>> > >
>> > > Thanks for your patience!
>> > >
>> > > On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
>> > >  wrote:
>> > > >
>> > > > On 18.04.24 8:13 PM, Laura Flores wrote:
>> > > > > Thanks for bringing this to our attention. The leads have decided
>> that
>> > > > > since this PR hasn't been merged to main yet and isn't approved,
>> it
>> > > > > will not go in v18.2.3, but it will be prioritized for v18.2.4.
>> > > > > I've already added the PR to the v18.2.4 milestone so it's sure
>> to be
>> > > > > picked up.
>> > > >
>> > > > Thanks a bunch. If you miss the train, you miss the train - fair
>> enough.
>> > > > Nice to know there is another one going soon and that bug is going
>> to be
>> > > > on it !
>> > > >
>> > > >
>> > > > Regards
>> > > >
>> > > > Christian
>> > > > ___
>> > > > ceph-users mailing list -- ceph-users@ceph.io
>> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
>> > > >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Laura Flores]

Rados results were approved, and we successfully upgraded the gibba cluster.

Now waiting on @Dan Mick  to upgrade the LRC.

On Thu, May 30, 2024 at 8:32 PM Yuri Weinstein  wrote:

> I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits
> and seeking approvals from Radek and Laure
>
> https://tracker.ceph.com/issues/65393#note-1
>
> On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein 
> wrote:
> >
> > We have discovered some issues (#1 and #2) during the final stages of
> > testing that require considering a delay in this point release until
> > all options and risks are assessed and resolved.
> >
> > We will keep you all updated on the progress.
> >
> > Thank you for your patience!
> >
> > #1 https://tracker.ceph.com/issues/66260
> > #2 https://tracker.ceph.com/issues/61948#note-21
> >
> > On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein 
> wrote:
> > >
> > > We've run into a problem during the last verification steps before
> > > publishing this release after upgrading the LRC to it  =>
> > > https://tracker.ceph.com/issues/65733
> > >
> > > After this issue is resolved, we will continue testing and publishing
> > > this point release.
> > >
> > > Thanks for your patience!
> > >
> > > On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
> > >  wrote:
> > > >
> > > > On 18.04.24 8:13 PM, Laura Flores wrote:
> > > > > Thanks for bringing this to our attention. The leads have decided
> that
> > > > > since this PR hasn't been merged to main yet and isn't approved, it
> > > > > will not go in v18.2.3, but it will be prioritized for v18.2.4.
> > > > > I've already added the PR to the v18.2.4 milestone so it's sure to
> be
> > > > > picked up.
> > > >
> > > > Thanks a bunch. If you miss the train, you miss the train - fair
> enough.
> > > > Nice to know there is another one going soon and that bug is going
> to be
> > > > on it !
> > > >
> > > >
> > > > Regards
> > > >
> > > > Christian
> > > > ___
> > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > > >
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

I reran rados on the fix https://github.com/ceph/ceph/pull/57794/commits
and seeking approvals from Radek and Laure

https://tracker.ceph.com/issues/65393#note-1

On Tue, May 28, 2024 at 2:12 PM Yuri Weinstein  wrote:
>
> We have discovered some issues (#1 and #2) during the final stages of
> testing that require considering a delay in this point release until
> all options and risks are assessed and resolved.
>
> We will keep you all updated on the progress.
>
> Thank you for your patience!
>
> #1 https://tracker.ceph.com/issues/66260
> #2 https://tracker.ceph.com/issues/61948#note-21
>
> On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein  wrote:
> >
> > We've run into a problem during the last verification steps before
> > publishing this release after upgrading the LRC to it  =>
> > https://tracker.ceph.com/issues/65733
> >
> > After this issue is resolved, we will continue testing and publishing
> > this point release.
> >
> > Thanks for your patience!
> >
> > On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
> >  wrote:
> > >
> > > On 18.04.24 8:13 PM, Laura Flores wrote:
> > > > Thanks for bringing this to our attention. The leads have decided that
> > > > since this PR hasn't been merged to main yet and isn't approved, it
> > > > will not go in v18.2.3, but it will be prioritized for v18.2.4.
> > > > I've already added the PR to the v18.2.4 milestone so it's sure to be
> > > > picked up.
> > >
> > > Thanks a bunch. If you miss the train, you miss the train - fair enough.
> > > Nice to know there is another one going soon and that bug is going to be
> > > on it !
> > >
> > >
> > > Regards
> > >
> > > Christian
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

We have discovered some issues (#1 and #2) during the final stages of
testing that require considering a delay in this point release until
all options and risks are assessed and resolved.

We will keep you all updated on the progress.

Thank you for your patience!

#1 https://tracker.ceph.com/issues/66260
#2 https://tracker.ceph.com/issues/61948#note-21

On Wed, May 1, 2024 at 3:41 PM Yuri Weinstein  wrote:
>
> We've run into a problem during the last verification steps before
> publishing this release after upgrading the LRC to it  =>
> https://tracker.ceph.com/issues/65733
>
> After this issue is resolved, we will continue testing and publishing
> this point release.
>
> Thanks for your patience!
>
> On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
>  wrote:
> >
> > On 18.04.24 8:13 PM, Laura Flores wrote:
> > > Thanks for bringing this to our attention. The leads have decided that
> > > since this PR hasn't been merged to main yet and isn't approved, it
> > > will not go in v18.2.3, but it will be prioritized for v18.2.4.
> > > I've already added the PR to the v18.2.4 milestone so it's sure to be
> > > picked up.
> >
> > Thanks a bunch. If you miss the train, you miss the train - fair enough.
> > Nice to know there is another one going soon and that bug is going to be
> > on it !
> >
> >
> > Regards
> >
> > Christian
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

We've run into a problem during the last verification steps before
publishing this release after upgrading the LRC to it  =>
https://tracker.ceph.com/issues/65733

After this issue is resolved, we will continue testing and publishing
this point release.

Thanks for your patience!

On Thu, Apr 18, 2024 at 11:29 PM Christian Rohmann
 wrote:
>
> On 18.04.24 8:13 PM, Laura Flores wrote:
> > Thanks for bringing this to our attention. The leads have decided that
> > since this PR hasn't been merged to main yet and isn't approved, it
> > will not go in v18.2.3, but it will be prioritized for v18.2.4.
> > I've already added the PR to the v18.2.4 milestone so it's sure to be
> > picked up.
>
> Thanks a bunch. If you miss the train, you miss the train - fair enough.
> Nice to know there is another one going soon and that bug is going to be
> on it !
>
>
> Regards
>
> Christian
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Christian Rohmann]

On 18.04.24 8:13 PM, Laura Flores wrote:
Thanks for bringing this to our attention. The leads have decided that 
since this PR hasn't been merged to main yet and isn't approved, it 
will not go in v18.2.3, but it will be prioritized for v18.2.4.
I've already added the PR to the v18.2.4 milestone so it's sure to be 
picked up.


Thanks a bunch. If you miss the train, you miss the train - fair enough.
Nice to know there is another one going soon and that bug is going to be 
on it !



Regards

Christian
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Laura Flores]

Hi Christian,

Thanks for bringing this to our attention. The leads have decided that
since this PR hasn't been merged to main yet and isn't approved, it will
not go in v18.2.3, but it will be prioritized for v18.2.4.
I've already added the PR to the v18.2.4 milestone so it's sure to be
picked up.

Thanks,
Laura

On Thu, Apr 18, 2024 at 4:40 AM Christian Rohmann <
christian.rohm...@inovex.de> wrote:

> Hey Laura,
>
>
> On 17.04.24 4:58 PM, Laura Flores wrote:
>
> There are two PRs that were added later to the 18.2.3 milestone concerning
> debian 
> packaging:https://github.com/ceph/ceph/pulls?q=is%3Apr+is%3Aopen+milestone%3Av18.2.3
> The user is asking if these can be included.
>
> I know everybody always wants their most anticipated PR in the next point
> release,
> but please let me kindly point you to the issue of ceph-crash not working
> due to some small glitch it's directory permissions:
>
>  * ML post to the ML
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VACLBNVXTYNSXJSNXJSRAQNZHCHABDF4/
>  * Bug Report: https://github.com/ceph/ceph/pull/55917
>  * Non-backport PR fixing this: https://tracker.ceph.com/issues/64548
>
>
> Since this is really potentially a one liner fix allowing for ceph-crash
> reports to be sent again.
> When I noticed this, I had 47 non-reported crashes queues up in one my
> clusters.
>
>
>
> Regards
>
>
> Christian
>
>
>
>
>
>

-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Christian Rohmann]

Hey Laura,


On 17.04.24 4:58 PM, Laura Flores wrote:

There are two PRs that were added later to the 18.2.3 milestone concerning
debian packaging:
https://github.com/ceph/ceph/pulls?q=is%3Apr+is%3Aopen+milestone%3Av18.2.3
The user is asking if these can be included.


I know everybody always wants their most anticipated PR in the next 
point release,
but please let me kindly point you to the issue of ceph-crash not 
working due to some small glitch it's directory permissions:


 * ML post to the ML 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VACLBNVXTYNSXJSNXJSRAQNZHCHABDF4/

 * Bug Report: https://github.com/ceph/ceph/pull/55917
 * Non-backport PR fixing this: https://tracker.ceph.com/issues/64548


Since this is really potentially a one liner fix allowing for ceph-crash 
reports to be sent again.
When I noticed this, I had 47 non-reported crashes queues up in one my 
clusters.




Regards


Christian




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Laura Flores]

Hey all,

I wanted to point out a message on the user list:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJXKJ2WR5PRQMWXIL33BKCSXJT7Q6VUA/

There are two PRs that were added later to the 18.2.3 milestone concerning
debian packaging:
https://github.com/ceph/ceph/pulls?q=is%3Apr+is%3Aopen+milestone%3Av18.2.3
The user is asking if these can be included.

Thanks,
Laura

On Wed, Apr 17, 2024 at 6:53 AM Venky Shankar  wrote:

> On Sat, Apr 13, 2024 at 12:08 AM Yuri Weinstein 
> wrote:
> >
> > Details of this release are summarized here:
> >
> > https://tracker.ceph.com/issues/65393#note-1
> > Release Notes - TBD
> > LRC upgrade - TBD
> >
> > Seeking approvals/reviews for:
> >
> > smoke - infra issues, still trying, Laura PTL
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > rgw - Casey approved?
> > fs - Venky approved?
> > orch - Adam King approved?
> >
> > krbd - Ilya approved
> > powercycle - seems fs related, Venky, Brad PTL
>
> This is a know failure: https://tracker.ceph.com/issues/64572
>
> >
> > ceph-volume - will require
> >
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > Guillaume is fixing it.
> >
> > TIA
> > ___
> > Dev mailing list -- d...@ceph.io
> > To unsubscribe send an email to dev-le...@ceph.io
> >
>
>
> --
> Cheers,
> Venky
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Venky Shankar]

On Sat, Apr 13, 2024 at 12:08 AM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/65393#note-1
> Release Notes - TBD
> LRC upgrade - TBD
>
> Seeking approvals/reviews for:
>
> smoke - infra issues, still trying, Laura PTL
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> rgw - Casey approved?
> fs - Venky approved?
> orch - Adam King approved?
>
> krbd - Ilya approved
> powercycle - seems fs related, Venky, Brad PTL

This is a know failure: https://tracker.ceph.com/issues/64572

>
> ceph-volume - will require
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> Guillaume is fixing it.
>
> TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Venky Shankar]

Hi Yuri,

On Tue, Apr 16, 2024 at 7:52 PM Yuri Weinstein  wrote:
>
> And approval is needed for:
>
> fs - Venky approved?

fs approved. failures are:
https://tracker.ceph.com/projects/cephfs/wiki/Reef#2024-04-17

> powercycle - seems fs related, Venky, Brad PTL
>
> On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein  wrote:
> >
> > Still waiting for approvals:
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
> >
> > We plan not to upgrade the LRC to 18.2.3 as we are very close to the
> > first squid RC and will be using it for this purpose.
> > Please speak up if this may present any issues.
> >
> > Thx
> >
> > On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein  wrote:
> > >
> > > Details of this release are summarized here:
> > >
> > > https://tracker.ceph.com/issues/65393#note-1
> > > Release Notes - TBD
> > > LRC upgrade - TBD
> > >
> > > Seeking approvals/reviews for:
> > >
> > > smoke - infra issues, still trying, Laura PTL
> > >
> > > rados - Radek, Laura approved? Travis?  Nizamudeen?
> > >
> > > rgw - Casey approved?
> > > fs - Venky approved?
> > > orch - Adam King approved?
> > >
> > > krbd - Ilya approved
> > > powercycle - seems fs related, Venky, Brad PTL
> > >
> > > ceph-volume - will require
> > > https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > > Guillaume is fixing it.
> > >
> > > TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io



-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Adam King]

>
> Orchestrator:
> 1. https://tracker.ceph.com/issues/64208 - test_cephadm.sh: Container
> version mismatch causes job to fail.
>

Not a blocker issue. Just a problem with the test itself that will be fixed
by https://github.com/ceph/ceph/pull/56714


On Tue, Apr 16, 2024 at 1:39 PM Laura Flores  wrote:

> On behalf of @Radoslaw Zarzynski , rados approved.
>
> Below is the summary of the rados suite failures, divided by component. @Adam
> King  @Venky Shankar  PTAL at the
> orch and cephfs failures to see if they are blockers.
>
> Failures, unrelated:
>
> RADOS:
> 1. https://tracker.ceph.com/issues/65183 - Overriding an EC pool
> needs the "--yes-i-really-mean-it" flag in addition to "force"
> 3. https://tracker.ceph.com/issues/62992 - Heartbeat crash in
> reset_timeout and clear_timeout
> 4. https://tracker.ceph.com/issues/58893 - test_map_discontinuity:
> AssertionError: wait_for_clean: failed before timeout expired
> 5. https://tracker.ceph.com/issues/61774 - centos 9 testing reveals
> rocksdb "Leak_StillReachable" memory leak in mons
> 7. https://tracker.ceph.com/issues/62776 - rados: cluster [WRN]
> overall HEALTH_WARN - do not have an application enabled
> 8. https://tracker.ceph.com/issues/59196 - ceph_test_lazy_omap_stats
> segfault while waiting for active+clean
>
> Orchestrator:
> 1. https://tracker.ceph.com/issues/64208 - test_cephadm.sh: Container
> version mismatch causes job to fail.
>
> CephFS:
> 1. https://tracker.ceph.com/issues/64946 - qa: unable to locate
> package libcephfs1
>
> Teuthology:
> 1. https://tracker.ceph.com/issues/64727 - suites/dbench.sh: Socket
> exception: No route to host (113)
>
> On Tue, Apr 16, 2024 at 9:22 AM Yuri Weinstein 
> wrote:
>
>> And approval is needed for:
>>
>> fs - Venky approved?
>> powercycle - seems fs related, Venky, Brad PTL
>>
>> On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein 
>> wrote:
>> >
>> > Still waiting for approvals:
>> >
>> > rados - Radek, Laura approved? Travis?  Nizamudeen?
>> >
>> > ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
>> >
>> > We plan not to upgrade the LRC to 18.2.3 as we are very close to the
>> > first squid RC and will be using it for this purpose.
>> > Please speak up if this may present any issues.
>> >
>> > Thx
>> >
>> > On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein 
>> wrote:
>> > >
>> > > Details of this release are summarized here:
>> > >
>> > > https://tracker.ceph.com/issues/65393#note-1
>> > > Release Notes - TBD
>> > > LRC upgrade - TBD
>> > >
>> > > Seeking approvals/reviews for:
>> > >
>> > > smoke - infra issues, still trying, Laura PTL
>> > >
>> > > rados - Radek, Laura approved? Travis?  Nizamudeen?
>> > >
>> > > rgw - Casey approved?
>> > > fs - Venky approved?
>> > > orch - Adam King approved?
>> > >
>> > > krbd - Ilya approved
>> > > powercycle - seems fs related, Venky, Brad PTL
>> > >
>> > > ceph-volume - will require
>> > >
>> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
>> > > Guillaume is fixing it.
>> > >
>> > > TIA
>> ___
>> Dev mailing list -- d...@ceph.io
>> To unsubscribe send an email to dev-le...@ceph.io
>>
>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage 
>
> Chicago, IL
>
> lflo...@ibm.com | lflo...@redhat.com 
> M: +17087388804
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Venky Shankar]

On Tue, Apr 16, 2024 at 7:52 PM Yuri Weinstein  wrote:
>
> And approval is needed for:
>
> fs - Venky approved?

Could not get to this today. Will be done tomorrow.

> powercycle - seems fs related, Venky, Brad PTL
>
> On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein  wrote:
> >
> > Still waiting for approvals:
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
> >
> > We plan not to upgrade the LRC to 18.2.3 as we are very close to the
> > first squid RC and will be using it for this purpose.
> > Please speak up if this may present any issues.
> >
> > Thx
> >
> > On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein  wrote:
> > >
> > > Details of this release are summarized here:
> > >
> > > https://tracker.ceph.com/issues/65393#note-1
> > > Release Notes - TBD
> > > LRC upgrade - TBD
> > >
> > > Seeking approvals/reviews for:
> > >
> > > smoke - infra issues, still trying, Laura PTL
> > >
> > > rados - Radek, Laura approved? Travis?  Nizamudeen?
> > >
> > > rgw - Casey approved?
> > > fs - Venky approved?
> > > orch - Adam King approved?
> > >
> > > krbd - Ilya approved
> > > powercycle - seems fs related, Venky, Brad PTL
> > >
> > > ceph-volume - will require
> > > https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > > Guillaume is fixing it.
> > >
> > > TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io



-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Laura Flores]

On behalf of @Radoslaw Zarzynski , rados approved.

Below is the summary of the rados suite failures, divided by component. @Adam
King  @Venky Shankar  PTAL at the
orch and cephfs failures to see if they are blockers.

Failures, unrelated:

RADOS:
1. https://tracker.ceph.com/issues/65183 - Overriding an EC pool needs
the "--yes-i-really-mean-it" flag in addition to "force"
3. https://tracker.ceph.com/issues/62992 - Heartbeat crash in
reset_timeout and clear_timeout
4. https://tracker.ceph.com/issues/58893 - test_map_discontinuity:
AssertionError: wait_for_clean: failed before timeout expired
5. https://tracker.ceph.com/issues/61774 - centos 9 testing reveals
rocksdb "Leak_StillReachable" memory leak in mons
7. https://tracker.ceph.com/issues/62776 - rados: cluster [WRN] overall
HEALTH_WARN - do not have an application enabled
8. https://tracker.ceph.com/issues/59196 - ceph_test_lazy_omap_stats
segfault while waiting for active+clean

Orchestrator:
1. https://tracker.ceph.com/issues/64208 - test_cephadm.sh: Container
version mismatch causes job to fail.

CephFS:
1. https://tracker.ceph.com/issues/64946 - qa: unable to locate package
libcephfs1

Teuthology:
1. https://tracker.ceph.com/issues/64727 - suites/dbench.sh: Socket
exception: No route to host (113)

On Tue, Apr 16, 2024 at 9:22 AM Yuri Weinstein  wrote:

> And approval is needed for:
>
> fs - Venky approved?
> powercycle - seems fs related, Venky, Brad PTL
>
> On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein 
> wrote:
> >
> > Still waiting for approvals:
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
> >
> > We plan not to upgrade the LRC to 18.2.3 as we are very close to the
> > first squid RC and will be using it for this purpose.
> > Please speak up if this may present any issues.
> >
> > Thx
> >
> > On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein 
> wrote:
> > >
> > > Details of this release are summarized here:
> > >
> > > https://tracker.ceph.com/issues/65393#note-1
> > > Release Notes - TBD
> > > LRC upgrade - TBD
> > >
> > > Seeking approvals/reviews for:
> > >
> > > smoke - infra issues, still trying, Laura PTL
> > >
> > > rados - Radek, Laura approved? Travis?  Nizamudeen?
> > >
> > > rgw - Casey approved?
> > > fs - Venky approved?
> > > orch - Adam King approved?
> > >
> > > krbd - Ilya approved
> > > powercycle - seems fs related, Venky, Brad PTL
> > >
> > > ceph-volume - will require
> > >
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > > Guillaume is fixing it.
> > >
> > > TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

And approval is needed for:

fs - Venky approved?
powercycle - seems fs related, Venky, Brad PTL

On Mon, Apr 15, 2024 at 5:55 PM Yuri Weinstein  wrote:
>
> Still waiting for approvals:
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857
>
> We plan not to upgrade the LRC to 18.2.3 as we are very close to the
> first squid RC and will be using it for this purpose.
> Please speak up if this may present any issues.
>
> Thx
>
> On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein  wrote:
> >
> > Details of this release are summarized here:
> >
> > https://tracker.ceph.com/issues/65393#note-1
> > Release Notes - TBD
> > LRC upgrade - TBD
> >
> > Seeking approvals/reviews for:
> >
> > smoke - infra issues, still trying, Laura PTL
> >
> > rados - Radek, Laura approved? Travis?  Nizamudeen?
> >
> > rgw - Casey approved?
> > fs - Venky approved?
> > orch - Adam King approved?
> >
> > krbd - Ilya approved
> > powercycle - seems fs related, Venky, Brad PTL
> >
> > ceph-volume - will require
> > https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> > Guillaume is fixing it.
> >
> > TIA
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Yuri Weinstein]

Still waiting for approvals:

rados - Radek, Laura approved? Travis?  Nizamudeen?

ceph-volume issue was fixed by https://github.com/ceph/ceph/pull/56857

We plan not to upgrade the LRC to 18.2.3 as we are very close to the
first squid RC and will be using it for this purpose.
Please speak up if this may present any issues.

Thx

On Fri, Apr 12, 2024 at 11:37 AM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/65393#note-1
> Release Notes - TBD
> LRC upgrade - TBD
>
> Seeking approvals/reviews for:
>
> smoke - infra issues, still trying, Laura PTL
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> rgw - Casey approved?
> fs - Venky approved?
> orch - Adam King approved?
>
> krbd - Ilya approved
> powercycle - seems fs related, Venky, Brad PTL
>
> ceph-volume - will require
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> Guillaume is fixing it.
>
> TIA
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Laura Flores]

smoke approved.

Infrastructure:
1. https://tracker.ceph.com/issues/64727 - suites/dbench.sh: Socket
exception: No route to host (113)

On Sun, Apr 14, 2024 at 2:08 PM Adam King  wrote:

> orch approved
>
> On Fri, Apr 12, 2024 at 2:38 PM Yuri Weinstein 
> wrote:
>
>> Details of this release are summarized here:
>>
>> https://tracker.ceph.com/issues/65393#note-1
>> Release Notes - TBD
>> LRC upgrade - TBD
>>
>> Seeking approvals/reviews for:
>>
>> smoke - infra issues, still trying, Laura PTL
>>
>> rados - Radek, Laura approved? Travis?  Nizamudeen?
>>
>> rgw - Casey approved?
>> fs - Venky approved?
>> orch - Adam King approved?
>>
>> krbd - Ilya approved
>> powercycle - seems fs related, Venky, Brad PTL
>>
>> ceph-volume - will require
>>
>> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
>> Guillaume is fixing it.
>>
>> TIA
>> ___
>> Dev mailing list -- d...@ceph.io
>> To unsubscribe send an email to dev-le...@ceph.io
>>
>> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Adam King]

orch approved

On Fri, Apr 12, 2024 at 2:38 PM Yuri Weinstein  wrote:

> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/65393#note-1
> Release Notes - TBD
> LRC upgrade - TBD
>
> Seeking approvals/reviews for:
>
> smoke - infra issues, still trying, Laura PTL
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> rgw - Casey approved?
> fs - Venky approved?
> orch - Adam King approved?
>
> krbd - Ilya approved
> powercycle - seems fs related, Venky, Brad PTL
>
> ceph-volume - will require
>
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> Guillaume is fixing it.
>
> TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Casey Bodley]

On Fri, Apr 12, 2024 at 2:38 PM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/65393#note-1
> Release Notes - TBD
> LRC upgrade - TBD
>
> Seeking approvals/reviews for:
>
> smoke - infra issues, still trying, Laura PTL
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> rgw - Casey approved?

rgw approved

> fs - Venky approved?
> orch - Adam King approved?
>
> krbd - Ilya approved
> powercycle - seems fs related, Venky, Brad PTL
>
> ceph-volume - will require
> https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e1a64700f8a600
> Guillaume is fixing it.
>
> TIA
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: reef 18.2.3 QE validation status

[Ilya Dryomov]

On Fri, Apr 12, 2024 at 8:38 PM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/65393#note-1
> Release Notes - TBD
> LRC upgrade - TBD
>
> Seeking approvals/reviews for:
>
> smoke - infra issues, still trying, Laura PTL
>
> rados - Radek, Laura approved? Travis?  Nizamudeen?
>
> rgw - Casey approved?
> fs - Venky approved?
> orch - Adam King approved?
>
> krbd - Ilya approved

Approved based on a rerun:

https://pulpito.ceph.com/dis-2024-04-12_20:04:30-krbd-reef-release-testing-default-smithi/

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io