Re: [ceph-users] ceph 12.2.9 release

[Matthew Vernon]

On 07/11/2018 23:28, Neha Ojha wrote:

> For those who haven't upgraded to 12.2.9 -
> 
> Please avoid this release and wait for 12.2.10.

Any idea when 12.2.10 is going to be here, please?

Regards,

Matthew


-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Graham Allan]

Thanks for the extra info, but I find the question more nuanced than that.

For example in my case I ended up with 12.2.9 on my last handful of 
newly-installed servers (and this is simply using yum rather than 
explicitly ceph-deploy).


We are replacing hardware nodes so the cluster is actively backfilling 
across nodes, which fulfills Greg's criterion of "(the bug) only 
triggers when PGs are not active+clean". Fortunately we are mostly 
backfilling from older to newer minor versions (12.2.8 to 12.2.9), so 
the problem may not manifest itself, but I'm not going to be in a 
position to have all pgs active+clean until the backfills are complete. 
Perhaps at that point it might be safe to upgrade the remaining osds to 
the same 12.2.9? The advice to also not upgrade to 12.9.10 is a bit 
disturbing since it sounds like the feature revert might make it 
incompatible with what we have.


Graham

On 11/07/2018 05:28 PM, Neha Ojha wrote:

For those on 12.2.9 -

If you have successfully upgraded to 12.2.9, there is no reason for
you to downgrade, since the bug appears while upgrading to 12.2.9 -
http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
does not have much in-store except for the revert. We are working on a
clean upgrade path for this feature and will announce it when it is
ready.

For those who haven't upgraded to 12.2.9 -

Please avoid this release and wait for 12.2.10.

More information here:
https://www.spinics.net/lists/ceph-devel/msg43509.html,
https://www.spinics.net/lists/ceph-users/msg49112.html

Again, sorry about the inconvenience and hope this helps!

Thanks,
Neha

On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
 wrote:

El Miércoles 07/11/2018 a las 10:58, Simon Ironside escribió:

On 07/11/2018 10:59, Konstantin Shalygin wrote:

I wonder if there is any release announcement for ceph 12.2.9 that I
missed. I just found the new packages on download.ceph.com, is this an
official release?


This is because 12.2.9 have a several bugs. You should avoid to use this
release and wait for 12.2.10


Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?

Simon


Is it safe to downgrade from 12.2.9 to 12.2.8?

Or should we just wait for 12.2.10?

Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Ricardo J. Barberis]

El Jueves 08/11/2018 a las 06:17, Marc Roos escribió:
> And that is why I don't like ceph-deploy. Unless you have maybe hundreds
> of disks, I don’t see why you cannot install it "manually".

We do have another cluster with 600+ disks, this one has 91 so far.

We actually started using ceph-deploy on this cluster to simplify our old 
method of installation, which would have done the same thing, actually, since 
it also uses the official repos.

We might have noticed the different version with our old method, but I can't 
be sure.

Cheers,

> -Original Message-
> From: Ricardo J. Barberis [mailto:rica...@palmtx.com.ar]
> Sent: woensdag 7 november 2018 23:23
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] ceph 12.2.9 release
>
> El Miércoles 07/11/2018 a las 11:28, Matthew Vernon escribió:
> > On 07/11/2018 14:16, Marc Roos wrote:
> > > I don't see the problem. I am installing only the ceph updates when
> > > others have done this and are running several weeks without
> > > problems. I have noticed this 12.2.9 availability also, did not see
> > > any release notes, so why install it? Especially with recent issues
> > > of other releases.
> >
> > Relevantly, if you want to upgrade to Luminous in many of the obvious
> > ways, you'll end up with 12.2.9.
> >
> > Regards,
> >
> > Matthew
>
> Also relevant: if you use ceph-deploy like I do con CentOS 7, it
> installs the latest version available, so I inadvertedly ended up with
> 12.2.9 on my last four servers.
>
> Thanks,

-- 
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Neha Ojha]

On Thu, Nov 8, 2018 at 12:16 PM, Ricardo J. Barberis
 wrote:
> Hi Neha, thank you for the info.
>
> I'd like to clarify that we didn't actually upgrade to 12.2.9, we just
> installed 4 more OSD servers and those got 12.2.9, so we have a mixture
> of 12.2.9 and 12.2.8.
>
> Should we:
> - keep as is and wait for 12.2.10+ before proceeding?
> - downgrade our newest OSDs from 12.2.9 to 12.2.8?
> - upgrade everithing to 12.2.9?

I think the best way to proceed is to upgrade the newest OSDs and
restart all OSDs with 12.2.9, or only upgrade when all PGs are
active+clean, if restart is not an option.

>
>
> Our current setup (we still have disks to add as OSDs):
>
> # ceph versions
> {
> "mon": {
> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
> luminous (stable)": 5
> },
> "mgr": {
> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
> luminous (stable)": 5
> },
> "osd": {
> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
> luminous (stable)": 75,
> "ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) 
> luminous (stable)": 16
> },
> "mds": {},
> "overall": {
> "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
> luminous (stable)": 85,
> "ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) 
> luminous (stable)": 16
> }
> }
>
>
> El Miércoles 07/11/2018 a las 20:28, Neha Ojha escribió:
>> For those on 12.2.9 -
>>
>> If you have successfully upgraded to 12.2.9, there is no reason for
>> you to downgrade, since the bug appears while upgrading to 12.2.9 -
>> http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
>> 12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
>> does not have much in-store except for the revert. We are working on a
>> clean upgrade path for this feature and will announce it when it is
>> ready.
>>
>> For those who haven't upgraded to 12.2.9 -
>>
>> Please avoid this release and wait for 12.2.10.
>>
>> More information here:
>> https://www.spinics.net/lists/ceph-devel/msg43509.html,
>> https://www.spinics.net/lists/ceph-users/msg49112.html
>>
>> Again, sorry about the inconvenience and hope this helps!
>>
>> Thanks,
>> Neha
>>
>> On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
>>
>>  wrote:
>> > El Miércoles 07/11/2018 a las 10:58, Simon Ironside escribió:
>> >> On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> >> >> I wonder if there is any release announcement for ceph 12.2.9 that I
>> >> >> missed. I just found the new packages on download.ceph.com, is this
>> >> >> an official release?
>> >> >
>> >> > This is because 12.2.9 have a several bugs. You should avoid to use
>> >> > this release and wait for 12.2.10
>> >>
>> >> Argh! What's it doing in the repos then?? I've just upgraded to it!
>> >> What are the bugs? Is there a thread about them?
>> >>
>> >> Simon
>> >
>> > Is it safe to downgrade from 12.2.9 to 12.2.8?
>> >
>> > Or should we just wait for 12.2.10?
>> >
>> > Thanks,
> --
> Ricardo J. Barberis
> Usuario Linux Nº 250625: http://counter.li.org/
> Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
> Senior SysAdmin / IT Architect - www.DonWeb.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Ricardo J. Barberis]

Hi Neha, thank you for the info.

I'd like to clarify that we didn't actually upgrade to 12.2.9, we just
installed 4 more OSD servers and those got 12.2.9, so we have a mixture
of 12.2.9 and 12.2.8.

Should we:
- keep as is and wait for 12.2.10+ before proceeding?
- downgrade our newest OSDs from 12.2.9 to 12.2.8?
- upgrade everithing to 12.2.9?


Our current setup (we still have disks to add as OSDs):

# ceph versions
{
"mon": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
luminous (stable)": 5
},
"mgr": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
luminous (stable)": 5
},
"osd": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
luminous (stable)": 75,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) 
luminous (stable)": 16
},
"mds": {},
"overall": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
luminous (stable)": 85,
"ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217) 
luminous (stable)": 16
}
}


El Miércoles 07/11/2018 a las 20:28, Neha Ojha escribió:
> For those on 12.2.9 -
>
> If you have successfully upgraded to 12.2.9, there is no reason for
> you to downgrade, since the bug appears while upgrading to 12.2.9 -
> http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
> 12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
> does not have much in-store except for the revert. We are working on a
> clean upgrade path for this feature and will announce it when it is
> ready.
>
> For those who haven't upgraded to 12.2.9 -
>
> Please avoid this release and wait for 12.2.10.
>
> More information here:
> https://www.spinics.net/lists/ceph-devel/msg43509.html,
> https://www.spinics.net/lists/ceph-users/msg49112.html
>
> Again, sorry about the inconvenience and hope this helps!
>
> Thanks,
> Neha
>
> On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
>
>  wrote:
> > El Miércoles 07/11/2018 a las 10:58, Simon Ironside escribió:
> >> On 07/11/2018 10:59, Konstantin Shalygin wrote:
> >> >> I wonder if there is any release announcement for ceph 12.2.9 that I
> >> >> missed. I just found the new packages on download.ceph.com, is this
> >> >> an official release?
> >> >
> >> > This is because 12.2.9 have a several bugs. You should avoid to use
> >> > this release and wait for 12.2.10
> >>
> >> Argh! What's it doing in the repos then?? I've just upgraded to it!
> >> What are the bugs? Is there a thread about them?
> >>
> >> Simon
> >
> > Is it safe to downgrade from 12.2.9 to 12.2.8?
> >
> > Or should we just wait for 12.2.10?
> >
> > Thanks,
-- 
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Alfredo Deza]

On Thu, Nov 8, 2018 at 3:02 AM Janne Johansson  wrote:
>
> Den ons 7 nov. 2018 kl 18:43 skrev David Turner :
> >
> > My big question is that we've had a few of these releases this year that 
> > are bugged and shouldn't be upgraded to... They don't have any release 
> > notes or announcement and the only time this comes out is when users 
> > finally ask about it weeks later.  Why is this not proactively announced to 
> > avoid a problematic release and hopefully prevent people from installing 
> > it?  It would be great if there was an actual release notes saying not to 
> > upgrade to this version or something.
>
> I think the big question is why do these packages end up publicly so
> that scripts, updates and anyone not actively trying to hold back get
> exposed to them, then we are somehow supposed to notice that the
> accompanying release notes are lacking and then from that divinate
> that we shouldn't have upgraded into this release at all. This seems
> all backwards in most possible ways.
>
> I'm not even upset about releases having bugs, stuff happens, but the
> way people are forced into it, then it's somehow your fault for
> running ceph-deploy or yum/apt upgrade against official release-repos.

It isn't your fault (or anyone in the community), we don't have a good
system in place for community repos to manage it in a way that would
help when problems like this come up.

We are in a much better place than a few years ago though, when
packages had to be placed manually when creating repos, but more work
is needed to address multi-version support in deb repos
and yanking known bad versions out of the official/latest release location.

> It's almost as if it was meant to push people into slow-moving dists
> like Blue^H^H^H^HRedhat with ceph on top.
>
> --
> May the most significant bit of your life be positive.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Martin Verges]

Hello Marc,

> - You can use this separate from the commandline?
yes, we don't take apart any feature or possible way, but we don't recommend it

> - And if I modify something from the commandline, these changes are visible 
> in the webinterface?
yes, we just ask Ceph/Linux for it's current state and show it through
the frontend (100% restful api). For example if you add a pool via CLI
you will see it after <5 seconds.

> - I can easily remove/add this webinterface? I mean sometimes you have these 
> tools that just customize the whole environment, that it is  difficult to 
> revert it.
unfortunately thats more the "hard" part. We are not only a interface,
we provide are a complete Ceph Storage solution.

We solve the problem with the different versions and the deployment
through PXE boot of the systems. You can also test this by simply
booting the normally installed system with a running Ceph from our
system. However, you should make sure that the versions of Ceph are
compatible.
Our images include all necessary programs, libraries and kernels. This
allows us to control exactly which version is running on the host.
Even months after the cluster is started, the exact same image will be
executed via PXE boot, if the admin so desires.

You can try it and use it for free in a vagrant demo
https://croit.io/croit-virtual-demo or as a real installation
https://croit.io/croit-production-guide.

As an example, just look in
https://www.youtube.com/watch?v=Jrnzlylidjs to see how we upgrade from
Ceph Luminous to Ceph Mimic.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx


2018-11-08 11:23 GMT+01:00 Marc Roos :
>
> H interesting maybe,
>
> - You can use this separate from the commandline?
> - And if I modify something from the commandline, these changes are
> visible in the webinterface?
> - I can easily remove/add this webinterface? I mean sometimes you have
> these tools that just customize the whole environment, that it is
> difficult to revert it.
>
>
>
> -Original Message-
> From: Martin Verges [mailto:martin.ver...@croit.io]
> Sent: donderdag 8 november 2018 11:07
> To: Matthew Vernon
> Cc: ceph-users
> Subject: Re: [ceph-users] ceph 12.2.9 release
>
> Sorry to say this, but that's why there's the croit management interface
> (free community edition feature).
> You don't have to worry about problems that are absolutely critical for
> reliable and stable operation. It doesn't matter if you run a cluster
> with 10 or 1000 hard disks, it just has to run!
>
> On the 12.11. on the Ceph Day in Berlin I can give you information
> directly about it.
>
> --
> Martin Verges
> Managing director
>
> Mobile: +49 174 9335695
> E-Mail: martin.ver...@croit.io
> Chat: https://t.me/MartinVerges
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht
> Munich HRB 231263
>
> Web: https://croit.io
> YouTube: https://goo.gl/PGE1Bx
>
>
> 2018-11-08 10:35 GMT+01:00 Matthew Vernon :
>> On 08/11/2018 09:17, Marc Roos wrote:
>>>
>>>   And that is why I don't like ceph-deploy. Unless you have maybe
>>> hundreds of disks, I don’t see why you cannot install it "manually".
>>
>>
>> ...as the recent ceph survey showed, plenty of people have hundreds of
>
>> disks! Ceph is meant to be operated at scale, which is why many admins
>
>> will have automation (ceph-ansible, etc.) in place.
>>
>> [our test clusters are 180 OSDs...]
>>
>> Regards,
>>
>> Matthew
>>
>>
>> --
>> The Wellcome Sanger Institute is operated by Genome Research Limited,
>> a charity registered in England with number 1021457 and a company
>> registered in England with number 2742969, whose registered office is
>> 215 Euston Road, London, NW1 2BE.
>> ___
>>
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Marc Roos]

 
H interesting maybe, 

- You can use this separate from the commandline? 
- And if I modify something from the commandline, these changes are 
visible in the webinterface? 
- I can easily remove/add this webinterface? I mean sometimes you have 
these tools that just customize the whole environment, that it is 
difficult to revert it. 



-Original Message-
From: Martin Verges [mailto:martin.ver...@croit.io] 
Sent: donderdag 8 november 2018 11:07
To: Matthew Vernon
Cc: ceph-users
Subject: Re: [ceph-users] ceph 12.2.9 release

Sorry to say this, but that's why there's the croit management interface 
(free community edition feature).
You don't have to worry about problems that are absolutely critical for 
reliable and stable operation. It doesn't matter if you run a cluster 
with 10 or 1000 hard disks, it just has to run!

On the 12.11. on the Ceph Day in Berlin I can give you information 
directly about it.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht 
Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx


2018-11-08 10:35 GMT+01:00 Matthew Vernon :
> On 08/11/2018 09:17, Marc Roos wrote:
>>
>>   And that is why I don't like ceph-deploy. Unless you have maybe 
>> hundreds of disks, I don’t see why you cannot install it "manually".
>
>
> ...as the recent ceph survey showed, plenty of people have hundreds of 

> disks! Ceph is meant to be operated at scale, which is why many admins 

> will have automation (ceph-ansible, etc.) in place.
>
> [our test clusters are 180 OSDs...]
>
> Regards,
>
> Matthew
>
>
> --
> The Wellcome Sanger Institute is operated by Genome Research Limited, 
> a charity registered in England with number 1021457 and a company 
> registered in England with number 2742969, whose registered office is 
> 215 Euston Road, London, NW1 2BE. 
> ___
>
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Martin Verges]

Sorry to say this, but that's why there's the croit management
interface (free community edition feature).
You don't have to worry about problems that are absolutely critical
for reliable and stable operation. It doesn't matter if you run a
cluster with 10 or 1000 hard disks, it just has to run!

On the 12.11. on the Ceph Day in Berlin I can give you information
directly about it.

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.ver...@croit.io
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx


2018-11-08 10:35 GMT+01:00 Matthew Vernon :
> On 08/11/2018 09:17, Marc Roos wrote:
>>
>>   And that is why I don't like ceph-deploy. Unless you have maybe hundreds
>> of disks, I don’t see why you cannot install it "manually".
>
>
> ...as the recent ceph survey showed, plenty of people have hundreds of
> disks! Ceph is meant to be operated at scale, which is why many admins will
> have automation (ceph-ansible, etc.) in place.
>
> [our test clusters are 180 OSDs...]
>
> Regards,
>
> Matthew
>
>
> --
> The Wellcome Sanger Institute is operated by Genome Research Limited, a
> charity registered in England with number 1021457 and a company registered
> in England with number 2742969, whose registered office is 215 Euston Road,
> London, NW1 2BE. ___
>
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Marc Roos]

 
I know ceph is meant to operate at scale, that’s why we are all here. 
But if you have a 180 disk cluster, you have 6-9 nodes, is nothing when 
you add a node. I would just do the manual install and especially with a 
production environment, considering all the 'little' bugs surfacing 
here. I doubt if ceph-deploy is even much faster. 




-Original Message-
From: Matthew Vernon [mailto:m...@sanger.ac.uk] 
Sent: donderdag 8 november 2018 10:36
To: ceph-users@lists.ceph.com
Cc: Marc Roos
Subject: Re: [ceph-users] ceph 12.2.9 release

On 08/11/2018 09:17, Marc Roos wrote:
>   
> And that is why I don't like ceph-deploy. Unless you have maybe 
> hundreds of disks, I don’t see why you cannot install it "manually".

...as the recent ceph survey showed, plenty of people have hundreds of 
disks! Ceph is meant to be operated at scale, which is why many admins 
will have automation (ceph-ansible, etc.) in place.

[our test clusters are 180 OSDs...]

Regards,

Matthew


--
 The Wellcome Sanger Institute is operated by Genome Research  Limited, 
a charity registered in England with number 1021457 and a  company 
registered in England with number 2742969, whose registered  office is 
215 Euston Road, London, NW1 2BE. 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Matthew Vernon]

On 08/11/2018 09:17, Marc Roos wrote:
  
And that is why I don't like ceph-deploy. Unless you have maybe hundreds

of disks, I don’t see why you cannot install it "manually".


...as the recent ceph survey showed, plenty of people have hundreds of 
disks! Ceph is meant to be operated at scale, which is why many admins 
will have automation (ceph-ansible, etc.) in place.


[our test clusters are 180 OSDs...]

Regards,

Matthew


--
The Wellcome Sanger Institute is operated by Genome Research 
Limited, a charity registered in England with number 1021457 and a 
company registered in England with number 2742969, whose registered 
office is 215 Euston Road, London, NW1 2BE. 
___

ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Simon Ironside]

On 08/11/2018 09:17, Marc Roos wrote:
  
And that is why I don't like ceph-deploy. Unless you have maybe hundreds

of disks, I don’t see why you cannot install it "manually".


On 07/11/2018 22:22, Ricardo J. Barberis wrote:
  
Also relevant: if you use ceph-deploy like I do con CentOS 7, it

installs the latest version available, so I inadvertedly ended up with
12.2.9 on my last four servers.


I use ceph-deploy (on RHEL 7) but with my own repos. I still fell into 
the 12.2.9 trap but that's because that's the package version I'd 
mirrored. If I'd downloaded 12.2.8 instead it would've been fine with 
ceph-deploy.


Simon
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Marc Roos]

 
And that is why I don't like ceph-deploy. Unless you have maybe hundreds 
of disks, I don’t see why you cannot install it "manually".




-Original Message-
From: Ricardo J. Barberis [mailto:rica...@palmtx.com.ar] 
Sent: woensdag 7 november 2018 23:23
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph 12.2.9 release

El Miércoles 07/11/2018 a las 11:28, Matthew Vernon escribió:
> On 07/11/2018 14:16, Marc Roos wrote:
> >  
> > 
> > I don't see the problem. I am installing only the ceph updates when 
> > others have done this and are running several weeks without 
> > problems. I have noticed this 12.2.9 availability also, did not see 
> > any release notes, so why install it? Especially with recent issues 
> > of other releases.
> 
> Relevantly, if you want to upgrade to Luminous in many of the obvious 
> ways, you'll end up with 12.2.9.
> 
> Regards,
> 
> Matthew

Also relevant: if you use ceph-deploy like I do con CentOS 7, it 
installs the latest version available, so I inadvertedly ended up with 
12.2.9 on my last four servers.

Thanks,
--
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/ Usuario LFS Nº 5121: 
http://www.linuxfromscratch.org/ Senior SysAdmin / IT Architect - 
www.DonWeb.com ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Valmar Kuristik]

I'll second that.

We are in progress of upgrading after just receiving new hardware, and 
it looks like right now, digging information on what to do and exactly 
how will take literally hundreds of times more time and effort than the 
upgrade itself, once you know.



On 08.11.2018 10:02, Janne Johansson wrote:

Den ons 7 nov. 2018 kl 18:43 skrev David Turner :


My big question is that we've had a few of these releases this year that are 
bugged and shouldn't be upgraded to... They don't have any release notes or 
announcement and the only time this comes out is when users finally ask about 
it weeks later.  Why is this not proactively announced to avoid a problematic 
release and hopefully prevent people from installing it?  It would be great if 
there was an actual release notes saying not to upgrade to this version or 
something.


I think the big question is why do these packages end up publicly so
that scripts, updates and anyone not actively trying to hold back get
exposed to them, then we are somehow supposed to notice that the
accompanying release notes are lacking and then from that divinate
that we shouldn't have upgraded into this release at all. This seems
all backwards in most possible ways.

I'm not even upset about releases having bugs, stuff happens, but the
way people are forced into it, then it's somehow your fault for
running ceph-deploy or yum/apt upgrade against official release-repos.
It's almost as if it was meant to push people into slow-moving dists
like Blue^H^H^H^HRedhat with ceph on top.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Janne Johansson]

Den ons 7 nov. 2018 kl 18:43 skrev David Turner :
>
> My big question is that we've had a few of these releases this year that are 
> bugged and shouldn't be upgraded to... They don't have any release notes or 
> announcement and the only time this comes out is when users finally ask about 
> it weeks later.  Why is this not proactively announced to avoid a problematic 
> release and hopefully prevent people from installing it?  It would be great 
> if there was an actual release notes saying not to upgrade to this version or 
> something.

I think the big question is why do these packages end up publicly so
that scripts, updates and anyone not actively trying to hold back get
exposed to them, then we are somehow supposed to notice that the
accompanying release notes are lacking and then from that divinate
that we shouldn't have upgraded into this release at all. This seems
all backwards in most possible ways.

I'm not even upset about releases having bugs, stuff happens, but the
way people are forced into it, then it's somehow your fault for
running ceph-deploy or yum/apt upgrade against official release-repos.
It's almost as if it was meant to push people into slow-moving dists
like Blue^H^H^H^HRedhat with ceph on top.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Christoph Adomeit]

So my question regarding the latest ceph releases still is:

Where do all these scrub errors come from and do we have to worry about ?


On Thu, Nov 08, 2018 at 12:16:05AM +0800, Ashley Merrick wrote:
> I am seeing this on the latest mimic on my test cluster aswel.
> 
> Every automatic deep-scrub comes back as inconsistent, but doing another
> manual scrub comes back as fine and clear each time.
> 
> Not sure if related or not..
> 
> On Wed, 7 Nov 2018 at 11:57 PM, Christoph Adomeit <
> christoph.adom...@gatworks.de> wrote:
> 
> > Hello together,
> >
> > we have upgraded to 12.2.9 because it was in the official repos.
> >
> > Right after the update and some scrubs we have issues.
> >
> > This morning after regular scrubs we had around 10% of all pgs inconstent:
> >
> > pgs: 4036 active+clean
> >   380  active+clean+inconsistent
> >
> > After repairung these 380 pgs we again have:
> >
> > 1/93611534 objects unfound (0.000%)
> > 28   active+clean+inconsistent
> > 1active+recovery_wait+degraded
> >
> > Now we stopped repairing because it does not seem to solve the problem and
> > more and more error messages are occuring. So far we did not see corruption
> > but we do not feel well with the cluster.
> >
> > What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
> >
> > Is ist dangerous for our Data to leave the cluster running ?
> >
> > I am sure we do not have hardware errors and that these errors came with
> > the update to 12.2.9.
> >
> > Thanks
> >   Christoph
> >
> >
> >
> > On Wed, Nov 07, 2018 at 07:39:59AM -0800, Gregory Farnum wrote:
> > > On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside 
> > > wrote:
> > >
> > > >
> > > >
> > > > On 07/11/2018 10:59, Konstantin Shalygin wrote:
> > > > >> I wonder if there is any release announcement for ceph 12.2.9 that I
> > > > missed.
> > > > >> I just found the new packages on download.ceph.com, is this an
> > official
> > > > >> release?
> > > > >
> > > > > This is because 12.2.9 have a several bugs. You should avoid to use
> > this
> > > > > release and wait for 12.2.10
> > > >
> > > > Argh! What's it doing in the repos then?? I've just upgraded to it!
> > > > What are the bugs? Is there a thread about them?
> > >
> > >
> > > If you’ve already upgraded and have no issues then you won’t have any
> > > trouble going forward — except perhaps on the next upgrade, if you do it
> > > while the cluster is unhealthy.
> > >
> > > I agree that it’s annoying when these issues make it out. We’ve had
> > ongoing
> > > discussions to try and improve the release process so it’s less drawn-out
> > > and to prevent these upgrade issues from making it through testing, but
> > > nobody has resolved it yet. If anybody has experience working with deb
> > > repositories and handling releases, the Ceph upstream could use some
> > > help... ;)
> > > -Greg
> > >
> > >
> > > >
> > > > Simon
> > > > ___
> > > > ceph-users mailing list
> > > > ceph-users@lists.ceph.com
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > >
> >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >

-- 
Was macht ein Clown im Büro ? Faxen 
Christoph Adomeit
GATWORKS GmbH
Reststrauch 191
41199 Moenchengladbach
Sitz: Moenchengladbach
Amtsgericht Moenchengladbach, HRB 6303
Geschaeftsfuehrer:
Christoph Adomeit, Hans Wilhelm Terstappen

christoph.adom...@gatworks.de Internetloesungen vom Feinsten
Fon. +49 2166 9149-32  Fax. +49 2166 9149-10
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Neha Ojha]

For those on 12.2.9 -

If you have successfully upgraded to 12.2.9, there is no reason for
you to downgrade, since the bug appears while upgrading to 12.2.9 -
http://tracker.ceph.com/issues/36686. We suggest you to not upgrade to
12.2.10, which reverts the feature that caused this bug. Also, 12.2.10
does not have much in-store except for the revert. We are working on a
clean upgrade path for this feature and will announce it when it is
ready.

For those who haven't upgraded to 12.2.9 -

Please avoid this release and wait for 12.2.10.

More information here:
https://www.spinics.net/lists/ceph-devel/msg43509.html,
https://www.spinics.net/lists/ceph-users/msg49112.html

Again, sorry about the inconvenience and hope this helps!

Thanks,
Neha

On Wed, Nov 7, 2018 at 2:38 PM, Ricardo J. Barberis
 wrote:
> El Miércoles 07/11/2018 a las 10:58, Simon Ironside escribió:
>> On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> >> I wonder if there is any release announcement for ceph 12.2.9 that I
>> >> missed. I just found the new packages on download.ceph.com, is this an
>> >> official release?
>> >
>> > This is because 12.2.9 have a several bugs. You should avoid to use this
>> > release and wait for 12.2.10
>>
>> Argh! What's it doing in the repos then?? I've just upgraded to it!
>> What are the bugs? Is there a thread about them?
>>
>> Simon
>
> Is it safe to downgrade from 12.2.9 to 12.2.8?
>
> Or should we just wait for 12.2.10?
>
> Thanks,
> --
> Ricardo J. Barberis
> Usuario Linux Nº 250625: http://counter.li.org/
> Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
> Senior SysAdmin / IT Architect - www.DonWeb.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Ricardo J. Barberis]

El Miércoles 07/11/2018 a las 10:58, Simon Ironside escribió:
> On 07/11/2018 10:59, Konstantin Shalygin wrote:
> >> I wonder if there is any release announcement for ceph 12.2.9 that I
> >> missed. I just found the new packages on download.ceph.com, is this an
> >> official release?
> >
> > This is because 12.2.9 have a several bugs. You should avoid to use this
> > release and wait for 12.2.10
>
> Argh! What's it doing in the repos then?? I've just upgraded to it!
> What are the bugs? Is there a thread about them?
>
> Simon

Is it safe to downgrade from 12.2.9 to 12.2.8?

Or should we just wait for 12.2.10?

Thanks,
-- 
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Ricardo J. Barberis]

El Miércoles 07/11/2018 a las 11:28, Matthew Vernon escribió:
> On 07/11/2018 14:16, Marc Roos wrote:
> >  
> > 
> > I don't see the problem. I am installing only the ceph updates when 
> > others have done this and are running several weeks without problems. I 
> > have noticed this 12.2.9 availability also, did not see any release 
> > notes, so why install it? Especially with recent issues of other 
> > releases.
> 
> Relevantly, if you want to upgrade to Luminous in many of the obvious
> ways, you'll end up with 12.2.9.
> 
> Regards,
> 
> Matthew

Also relevant: if you use ceph-deploy like I do con CentOS 7, it installs the 
latest version available, so I inadvertedly ended up with 12.2.9 on my last 
four servers.

Thanks,
-- 
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Ricardo J. Barberis]

El Miércoles 07/11/2018 a las 11:05, Matthew Vernon escribió:
> On 07/11/2018 10:59, Konstantin Shalygin wrote:
> >> I wonder if there is any release announcement for ceph 12.2.9 that I 
> >> missed.
> >> I just found the new packages on download.ceph.com, is this an official
> >> release?
> > 
> > This is because 12.2.9 have a several bugs. You should avoid to use this
> > release and wait for 12.2.10
> 
> It seems that maybe something isn't quite right in the release
> infrastructure, then? The 12.2.8 packages are still available, but e.g.
> debian-luminous's Packages file is pointing to the 12.2.9 (broken) packages.
> 
> Could the Debian/Ubuntu repos only have their releases updated (as
> opposed to what's in the pool) for safe/official releases? It's one
> thing letting people find pre-release things if they go looking, but
> ISTM that arranging that a mis-timed apt-get update/upgrade might
> install known-broken packages is ... unfortunate.
> 
> Regards,
> 
> Matthew

Not only Debian/Ubuntu, CentOS also:

# LANG=C yum list ceph-common
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
 * elrepo: repos.mia.lax-noc.com
 * epel: epel.gtdinternet.com
8 packages excluded due to repository priority protections
Installed Packages
ceph-common.x86_64  2:12.2.8-0.el7 @Ceph
Available Packages
ceph-common.x86_64  2:12.2.9-0.el7 Ceph


-- 
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Ken Dreyer]

On Wed, Nov 7, 2018 at 8:57 AM Kevin Olbrich  wrote:
> We solve this problem by hosting two repos. One for staging and QA and one 
> for production.
> Every release gets to staging (for example directly after building a scm tag).
>
> If QA passed, the stage repo is turned into the prod one.
> Using symlinks, it would be possible to switch back if problems occure.
> Example: https://incoming.debian.org/

With the CentOS Storage SIG's cbs.centos.org , we have the ability to
tag builds as "-candidate", "-testing", and "-released". I think that
mechanism could help here, so brave users can run "testing" early
before it goes out to the entire world in "released".

We would have to build out something like that for Ubuntu, maybe
copying around the binaries.

- Ken
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[David Turner]

My big question is that we've had a few of these releases this year that
are bugged and shouldn't be upgraded to... They don't have any release
notes or announcement and the only time this comes out is when users
finally ask about it weeks later.  Why is this not proactively announced to
avoid a problematic release and hopefully prevent people from installing
it?  It would be great if there was an actual release notes saying not to
upgrade to this version or something.

On Wed, Nov 7, 2018 at 11:16 AM Ashley Merrick 
wrote:

> I am seeing this on the latest mimic on my test cluster aswel.
>
> Every automatic deep-scrub comes back as inconsistent, but doing another
> manual scrub comes back as fine and clear each time.
>
> Not sure if related or not..
>
> On Wed, 7 Nov 2018 at 11:57 PM, Christoph Adomeit <
> christoph.adom...@gatworks.de> wrote:
>
>> Hello together,
>>
>> we have upgraded to 12.2.9 because it was in the official repos.
>>
>> Right after the update and some scrubs we have issues.
>>
>> This morning after regular scrubs we had around 10% of all pgs inconstent:
>>
>> pgs: 4036 active+clean
>>   380  active+clean+inconsistent
>>
>> After repairung these 380 pgs we again have:
>>
>> 1/93611534 objects unfound (0.000%)
>> 28   active+clean+inconsistent
>> 1active+recovery_wait+degraded
>>
>> Now we stopped repairing because it does not seem to solve the problem
>> and more and more error messages are occuring. So far we did not see
>> corruption but we do not feel well with the cluster.
>>
>> What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
>>
>> Is ist dangerous for our Data to leave the cluster running ?
>>
>> I am sure we do not have hardware errors and that these errors came with
>> the update to 12.2.9.
>>
>> Thanks
>>   Christoph
>>
>>
>>
>> On Wed, Nov 07, 2018 at 07:39:59AM -0800, Gregory Farnum wrote:
>> > On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside 
>> > wrote:
>> >
>> > >
>> > >
>> > > On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> > > >> I wonder if there is any release announcement for ceph 12.2.9 that
>> I
>> > > missed.
>> > > >> I just found the new packages on download.ceph.com, is this an
>> official
>> > > >> release?
>> > > >
>> > > > This is because 12.2.9 have a several bugs. You should avoid to use
>> this
>> > > > release and wait for 12.2.10
>> > >
>> > > Argh! What's it doing in the repos then?? I've just upgraded to it!
>> > > What are the bugs? Is there a thread about them?
>> >
>> >
>> > If you’ve already upgraded and have no issues then you won’t have any
>> > trouble going forward — except perhaps on the next upgrade, if you do it
>> > while the cluster is unhealthy.
>> >
>> > I agree that it’s annoying when these issues make it out. We’ve had
>> ongoing
>> > discussions to try and improve the release process so it’s less
>> drawn-out
>> > and to prevent these upgrade issues from making it through testing, but
>> > nobody has resolved it yet. If anybody has experience working with deb
>> > repositories and handling releases, the Ceph upstream could use some
>> > help... ;)
>> > -Greg
>> >
>> >
>> > >
>> > > Simon
>> > > ___
>> > > ceph-users mailing list
>> > > ceph-users@lists.ceph.com
>> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> > >
>>
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Ashley Merrick]

I am seeing this on the latest mimic on my test cluster aswel.

Every automatic deep-scrub comes back as inconsistent, but doing another
manual scrub comes back as fine and clear each time.

Not sure if related or not..

On Wed, 7 Nov 2018 at 11:57 PM, Christoph Adomeit <
christoph.adom...@gatworks.de> wrote:

> Hello together,
>
> we have upgraded to 12.2.9 because it was in the official repos.
>
> Right after the update and some scrubs we have issues.
>
> This morning after regular scrubs we had around 10% of all pgs inconstent:
>
> pgs: 4036 active+clean
>   380  active+clean+inconsistent
>
> After repairung these 380 pgs we again have:
>
> 1/93611534 objects unfound (0.000%)
> 28   active+clean+inconsistent
> 1active+recovery_wait+degraded
>
> Now we stopped repairing because it does not seem to solve the problem and
> more and more error messages are occuring. So far we did not see corruption
> but we do not feel well with the cluster.
>
> What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
>
> Is ist dangerous for our Data to leave the cluster running ?
>
> I am sure we do not have hardware errors and that these errors came with
> the update to 12.2.9.
>
> Thanks
>   Christoph
>
>
>
> On Wed, Nov 07, 2018 at 07:39:59AM -0800, Gregory Farnum wrote:
> > On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside 
> > wrote:
> >
> > >
> > >
> > > On 07/11/2018 10:59, Konstantin Shalygin wrote:
> > > >> I wonder if there is any release announcement for ceph 12.2.9 that I
> > > missed.
> > > >> I just found the new packages on download.ceph.com, is this an
> official
> > > >> release?
> > > >
> > > > This is because 12.2.9 have a several bugs. You should avoid to use
> this
> > > > release and wait for 12.2.10
> > >
> > > Argh! What's it doing in the repos then?? I've just upgraded to it!
> > > What are the bugs? Is there a thread about them?
> >
> >
> > If you’ve already upgraded and have no issues then you won’t have any
> > trouble going forward — except perhaps on the next upgrade, if you do it
> > while the cluster is unhealthy.
> >
> > I agree that it’s annoying when these issues make it out. We’ve had
> ongoing
> > discussions to try and improve the release process so it’s less drawn-out
> > and to prevent these upgrade issues from making it through testing, but
> > nobody has resolved it yet. If anybody has experience working with deb
> > repositories and handling releases, the Ceph upstream could use some
> > help... ;)
> > -Greg
> >
> >
> > >
> > > Simon
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
>
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Gregory Farnum]

The specific bug you are known at risk for when installing the 12.2.9
packages is http://tracker.ceph.com/issues/36686.

It only triggers when PGs are not active+clean and are running different
minor versions. (Even more specifically, it seems to only show up when
doing backfill from an OSD running new code to an OSD running old code
during the upgrade process.)

If you have encountered other issues, there are no special troubleshooting
steps I'm aware of; follow the standard advice.
-Greg

On Wed, Nov 7, 2018 at 8:08 AM Christoph Adomeit <
christoph.adom...@gatworks.de> wrote:

> Hello together,
>
> we have upgraded to 12.2.9 because it was in the official repos.
>
> Right after the update and some scrubs we have issues.
>
> This morning after regular scrubs we had around 10% of all pgs inconstent:
>
> pgs: 4036 active+clean
>   380  active+clean+inconsistent
>
> After repairung these 380 pgs we again have:
>
> 1/93611534 objects unfound (0.000%)
> 28   active+clean+inconsistent
> 1active+recovery_wait+degraded
>
> Now we stopped repairing because it does not seem to solve the problem and
> more and more error messages are occuring. So far we did not see corruption
> but we do not feel well with the cluster.
>
> What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?
>
> Is ist dangerous for our Data to leave the cluster running ?
>
> I am sure we do not have hardware errors and that these errors came with
> the update to 12.2.9.
>
> Thanks
>   Christoph
>
>
>
> On Wed, Nov 07, 2018 at 07:39:59AM -0800, Gregory Farnum wrote:
> > On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside 
> > wrote:
> >
> > >
> > >
> > > On 07/11/2018 10:59, Konstantin Shalygin wrote:
> > > >> I wonder if there is any release announcement for ceph 12.2.9 that I
> > > missed.
> > > >> I just found the new packages on download.ceph.com, is this an
> official
> > > >> release?
> > > >
> > > > This is because 12.2.9 have a several bugs. You should avoid to use
> this
> > > > release and wait for 12.2.10
> > >
> > > Argh! What's it doing in the repos then?? I've just upgraded to it!
> > > What are the bugs? Is there a thread about them?
> >
> >
> > If you’ve already upgraded and have no issues then you won’t have any
> > trouble going forward — except perhaps on the next upgrade, if you do it
> > while the cluster is unhealthy.
> >
> > I agree that it’s annoying when these issues make it out. We’ve had
> ongoing
> > discussions to try and improve the release process so it’s less drawn-out
> > and to prevent these upgrade issues from making it through testing, but
> > nobody has resolved it yet. If anybody has experience working with deb
> > repositories and handling releases, the Ceph upstream could use some
> > help... ;)
> > -Greg
> >
> >
> > >
> > > Simon
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
>
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Simon Ironside]

On 07/11/2018 15:39, Gregory Farnum wrote:
On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside > wrote:




On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> I wonder if there is any release announcement for ceph 12.2.9
that I missed.
>> I just found the new packages on download.ceph.com
, is this an official
>> release?
>
> This is because 12.2.9 have a several bugs. You should avoid to
use this
> release and wait for 12.2.10

Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?


If you’ve already upgraded and have no issues then you won’t have any 
trouble going forward — except perhaps on the next upgrade, if you do 
it while the cluster is unhealthy.


Thanks, the upgrade went fine and I've no known issues. The only warning 
I have is about too many PGs per OSD which is my fault not ceph's. I 
trust that doesn't count as a reason not to proceed to 13.2.2?


I agree that it’s annoying when these issues make it out. We’ve had 
ongoing discussions to try and improve the release process so it’s 
less drawn-out and to prevent these upgrade issues from making it 
through testing, but nobody has resolved it yet. If anybody has 
experience working with deb repositories and handling releases, the 
Ceph upstream could use some help... ;)


Totally, I get that this happens from time to time but once a bad 
release is known why not just delete the affected packages from the 
official repos? That seems to me to be a really easy step to take 
especially if release announcements haven't been sent, docs.ceph.com 
hasn't been updated yet etc. I reposync --newest-only the RPMs from the 
official repos to my own then update my ceph hosts from there which is 
how I ended up with 12.2.9.


Simon
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Christoph Adomeit]

Hello together,

we have upgraded to 12.2.9 because it was in the official repos.

Right after the update and some scrubs we have issues.

This morning after regular scrubs we had around 10% of all pgs inconstent:

pgs: 4036 active+clean
  380  active+clean+inconsistent

After repairung these 380 pgs we again have:

1/93611534 objects unfound (0.000%)
28   active+clean+inconsistent
1active+recovery_wait+degraded

Now we stopped repairing because it does not seem to solve the problem and more 
and more error messages are occuring. So far we did not see corruption but we 
do not feel well with the cluster.

What do you suggest, wait for 12.2.10 ? Roll Back to 12.2.8 ?

Is ist dangerous for our Data to leave the cluster running ?

I am sure we do not have hardware errors and that these errors came with the 
update to 12.2.9.

Thanks
  Christoph



On Wed, Nov 07, 2018 at 07:39:59AM -0800, Gregory Farnum wrote:
> On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside 
> wrote:
> 
> >
> >
> > On 07/11/2018 10:59, Konstantin Shalygin wrote:
> > >> I wonder if there is any release announcement for ceph 12.2.9 that I
> > missed.
> > >> I just found the new packages on download.ceph.com, is this an official
> > >> release?
> > >
> > > This is because 12.2.9 have a several bugs. You should avoid to use this
> > > release and wait for 12.2.10
> >
> > Argh! What's it doing in the repos then?? I've just upgraded to it!
> > What are the bugs? Is there a thread about them?
> 
> 
> If you’ve already upgraded and have no issues then you won’t have any
> trouble going forward — except perhaps on the next upgrade, if you do it
> while the cluster is unhealthy.
> 
> I agree that it’s annoying when these issues make it out. We’ve had ongoing
> discussions to try and improve the release process so it’s less drawn-out
> and to prevent these upgrade issues from making it through testing, but
> nobody has resolved it yet. If anybody has experience working with deb
> repositories and handling releases, the Ceph upstream could use some
> help... ;)
> -Greg
> 
> 
> >
> > Simon
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Kevin Olbrich]

Am Mi., 7. Nov. 2018 um 16:40 Uhr schrieb Gregory Farnum :

> On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside 
> wrote:
>
>>
>>
>> On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> >> I wonder if there is any release announcement for ceph 12.2.9 that I
>> missed.
>> >> I just found the new packages on download.ceph.com, is this an
>> official
>> >> release?
>> >
>> > This is because 12.2.9 have a several bugs. You should avoid to use
>> this
>> > release and wait for 12.2.10
>>
>> Argh! What's it doing in the repos then?? I've just upgraded to it!
>> What are the bugs? Is there a thread about them?
>
>
> If you’ve already upgraded and have no issues then you won’t have any
> trouble going forward — except perhaps on the next upgrade, if you do it
> while the cluster is unhealthy.
>
> I agree that it’s annoying when these issues make it out. We’ve had
> ongoing discussions to try and improve the release process so it’s less
> drawn-out and to prevent these upgrade issues from making it through
> testing, but nobody has resolved it yet. If anybody has experience working
> with deb repositories and handling releases, the Ceph upstream could use
> some help... ;)
> -Greg
>
>>
>>
We solve this problem by hosting two repos. One for staging and QA and one
for production.
Every release gets to staging (for example directly after building a scm
tag).

If QA passed, the stage repo is turned into the prod one.
Using symlinks, it would be possible to switch back if problems occure.
Example: https://incoming.debian.org/

Currently I would be unable to deploy new nodes if I use the official
mirrors as apt is unable to use older versions (which does work on yum/dnf).
Thats why we are implementing "mirror-sync" / rsync with a copy of the repo
and the desired packages until such solution is available.

Kevin


>> Simon
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Gregory Farnum]

On Wed, Nov 7, 2018 at 5:58 AM Simon Ironside 
wrote:

>
>
> On 07/11/2018 10:59, Konstantin Shalygin wrote:
> >> I wonder if there is any release announcement for ceph 12.2.9 that I
> missed.
> >> I just found the new packages on download.ceph.com, is this an official
> >> release?
> >
> > This is because 12.2.9 have a several bugs. You should avoid to use this
> > release and wait for 12.2.10
>
> Argh! What's it doing in the repos then?? I've just upgraded to it!
> What are the bugs? Is there a thread about them?


If you’ve already upgraded and have no issues then you won’t have any
trouble going forward — except perhaps on the next upgrade, if you do it
while the cluster is unhealthy.

I agree that it’s annoying when these issues make it out. We’ve had ongoing
discussions to try and improve the release process so it’s less drawn-out
and to prevent these upgrade issues from making it through testing, but
nobody has resolved it yet. If anybody has experience working with deb
repositories and handling releases, the Ceph upstream could use some
help... ;)
-Greg


>
> Simon
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Dietmar Rieder]

On 11/7/18 11:59 AM, Konstantin Shalygin wrote:
>> I wonder if there is any release announcement for ceph 12.2.9 that I missed.
>> I just found the new packages on download.ceph.com, is this an official
>> release?
> 
> This is because 12.2.9 have a several bugs. You should avoid to use this
> release and wait for 12.2.10

Thanks a lot!

~Dietmar


-- 
_
D i e t m a r  R i e d e r, Mag.Dr.
Innsbruck Medical University
Biocenter - Division for Bioinformatics
Email: dietmar.rie...@i-med.ac.at
Web:   http://www.icbi.at




signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Matthew Vernon]

On 07/11/2018 14:16, Marc Roos wrote:
>  
> 
> I don't see the problem. I am installing only the ceph updates when 
> others have done this and are running several weeks without problems. I 
> have noticed this 12.2.9 availability also, did not see any release 
> notes, so why install it? Especially with recent issues of other 
> releases.

Relevantly, if you want to upgrade to Luminous in many of the obvious
ways, you'll end up with 12.2.9.

Regards,

Matthew


-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Thomas White]

One of the Ceph clusters my team manages is on 12.2.9 on a Proxmox
environment seems to be running fine with simple x3 replication and RBD.
Would be interesting to know what issues have been encountered so far. All
our OSDs are simple filestore at present and our path to 12.2.9 was 10.2.7
-> 10.2.10 -> 12.2.9 in the past 2 weeks with no issues.

That said, it is disappointing these packages are making their way into
repositories without the proper announcements for an LTS release, especially
given this is enterprise orientated software.

Thomas

-Original Message-
From: ceph-users  On Behalf Of Simon
Ironside
Sent: 07 November 2018 13:58
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph 12.2.9 release



On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> I wonder if there is any release announcement for ceph 12.2.9 that I
missed.
>> I just found the new packages on download.ceph.com, is this an 
>> official release?
> 
> This is because 12.2.9 have a several bugs. You should avoid to use 
> this release and wait for 12.2.10

Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?

Simon
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Marc Roos]

 

I don't see the problem. I am installing only the ceph updates when 
others have done this and are running several weeks without problems. I 
have noticed this 12.2.9 availability also, did not see any release 
notes, so why install it? Especially with recent issues of other 
releases.

That being said, this ceph is often a major part of a production 
environment. And I am surprised how easily buggy releases are finding 
their way to the public. We had this 12.2.5 now 12.2.9 there was also 
something with upgrading to mimic. I would not expect this from a LTS 
version (or a redhat product).





-Original Message-
From: Matthew Vernon [mailto:m...@sanger.ac.uk] 
Sent: woensdag 7 november 2018 15:05
To: Konstantin Shalygin; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph 12.2.9 release

On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> I wonder if there is any release announcement for ceph 12.2.9 that I 
missed.
>> I just found the new packages on download.ceph.com, is this an 
>> official release?
> 
> This is because 12.2.9 have a several bugs. You should avoid to use 
> this release and wait for 12.2.10

It seems that maybe something isn't quite right in the release 
infrastructure, then? The 12.2.8 packages are still available, but e.g.
debian-luminous's Packages file is pointing to the 12.2.9 (broken) 
packages.

Could the Debian/Ubuntu repos only have their releases updated (as 
opposed to what's in the pool) for safe/official releases? It's one 
thing letting people find pre-release things if they go looking, but 
ISTM that arranging that a mis-timed apt-get update/upgrade might 
install known-broken packages is ... unfortunate.

Regards,

Matthew


--
 The Wellcome Sanger Institute is operated by Genome Research  Limited, 
a charity registered in England with number 1021457 and a  company 
registered in England with number 2742969, whose registered  office is 
215 Euston Road, London, NW1 2BE. 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Matthew Vernon]

On 07/11/2018 10:59, Konstantin Shalygin wrote:
>> I wonder if there is any release announcement for ceph 12.2.9 that I missed.
>> I just found the new packages on download.ceph.com, is this an official
>> release?
> 
> This is because 12.2.9 have a several bugs. You should avoid to use this
> release and wait for 12.2.10

It seems that maybe something isn't quite right in the release
infrastructure, then? The 12.2.8 packages are still available, but e.g.
debian-luminous's Packages file is pointing to the 12.2.9 (broken) packages.

Could the Debian/Ubuntu repos only have their releases updated (as
opposed to what's in the pool) for safe/official releases? It's one
thing letting people find pre-release things if they go looking, but
ISTM that arranging that a mis-timed apt-get update/upgrade might
install known-broken packages is ... unfortunate.

Regards,

Matthew


-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Simon Ironside]

On 07/11/2018 10:59, Konstantin Shalygin wrote:

I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?


This is because 12.2.9 have a several bugs. You should avoid to use this 
release and wait for 12.2.10


Argh! What's it doing in the repos then?? I've just upgraded to it!
What are the bugs? Is there a thread about them?

Simon
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph 12.2.9 release

[Konstantin Shalygin]

I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?


This is because 12.2.9 have a several bugs. You should avoid to use this 
release and wait for 12.2.10




k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph 12.2.9 release

[Dietmar Rieder]

Hi,

I wonder if there is any release announcement for ceph 12.2.9 that I missed.
I just found the new packages on download.ceph.com, is this an official
release?

~ Dietmar

-- 
_
D i e t m a r  R i e d e r, Mag.Dr.
Innsbruck Medical University
Biocenter - Division for Bioinformatics
Web:   http://www.icbi.at




signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com