[ceph-users] 14. 2.15: Question to collection_list_legacy osd bug fixed in 14.2.15

[Rainer Krienke]

Hello,

I am running a productive ceph cluster with Nautilus 14.2.13. All OSDs
are bluestore and were created with a ceph version prior to 14.2.12.

What I would like to know is how urgent I should consider the
collection_list_legacy  bug since at the moment I am not going to add a
brand new OSD to the system. However any time a disk could fail and so I
would have to destroy the OSD with the failed disk and them with a new
disk run ceph-volume to create a new bluestore OSD.

Would this scenario also lead to inconsistent pgs?

Thanks
Rainer

Am 24.11.20 um 02:35 schrieb David Galloway:
> This is the 15th backport release in the Nautilus series. This release
> fixes a ceph-volume regression introduced in v14.2.13 and includes few
> other fixes. We recommend users to update to this release.
> 
> For a detailed release notes with links & changelog please refer to the
> official blog entry at https://ceph.io/releases/v14-2-15-nautilus-released
> 
> 
> Notable Changes
> ---
> * ceph-volume: Fixes lvm batch --auto, which breaks backward
> compatibility when using non rotational devices only (SSD and/or NVMe).
> * BlueStore: Fixes a bug in collection_list_legacy which makes pgs
> inconsistent during scrub when running mixed versions of osds, prior to
> 14.2.12 with newer.
> * MGR: progress module can now be turned on/off, using the commands:
> `ceph progress on` and `ceph progress off`.
> 
> 
> Getting Ceph
> 
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://download.ceph.com/tarballs/ceph-14.2.15.tar.gz
> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
> * Release git sha1: afdd217ae5fb1ed3f60e16bd62357ca58cc650e5
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 

-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
Web: http://userpages.uni-koblenz.de/~krienke
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] osd crash: Caught signal (Aborted) thread_name:tp_osd_tp

[Milan Kupcevic]

Hello,

Three OSD daemons crash at the same time while processing the same
object located in an rbd ec4+2 pool leaving a placement group in
inactive down state. Soon after I start the osd daemons back up they
crash again choking on the same object.

8<
_dump_onode 0x5605a27ca000
4#7:8565da11:::rbd_data.6.a8a8356fd674f.003dce34:head# nid
1889617 size 0x10 (1048576) expected_object_size 0
expected_write_size 0 in 8 shards, 32768 spanning blobs
8<

Please take a look at the attached log file.


Ceph status reports:

Reduced data availability: 1 pg inactive, 1 pg down


Any hints on how to get this placement group back online would be
greatly appreciated.


Milan


-- 
Milan Kupcevic
Senior Cyberinfrastructure Engineer at Project NESE
Harvard University
FAS Research Computing
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cephfs snapshots and previous version

[Oliver Weinmann]

Today I played with a samba gateway and cephfs. I couldn’t get previous 
versions displayed on a windows client and found very little info on the net 
how to accomplish this. It seems that I need a vfs module called 
ceph_snapshots. It’s not included in the latest samba version on Centos 8. by 
this I also noticed that there is no vfs ceph module. Are these modules not 
stable and therefore not included in centos8? I can compile them but I would 
like to know why they are not included. And one more question. Are there any 
plans to add samba gateway support to cephadm?

Best regards,
Oliver

Von meinem iPhone gesendet
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] v14.2.15 Nautilus released

[David Galloway]

This is the 15th backport release in the Nautilus series. This release
fixes a ceph-volume regression introduced in v14.2.13 and includes few
other fixes. We recommend users to update to this release.

For a detailed release notes with links & changelog please refer to the
official blog entry at https://ceph.io/releases/v14-2-15-nautilus-released


Notable Changes
---
* ceph-volume: Fixes lvm batch --auto, which breaks backward
compatibility when using non rotational devices only (SSD and/or NVMe).
* BlueStore: Fixes a bug in collection_list_legacy which makes pgs
inconsistent during scrub when running mixed versions of osds, prior to
14.2.12 with newer.
* MGR: progress module can now be turned on/off, using the commands:
`ceph progress on` and `ceph progress off`.


Getting Ceph

* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-14.2.15.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: afdd217ae5fb1ed3f60e16bd62357ca58cc650e5
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Documentation of older Ceph version not accessible anymore on docs.ceph.com

[Dan Mick]

I don't know the answer to that.

On 11/23/2020 6:59 AM, Martin Palma wrote:

Hi Dan,

yes I noticed but now only "latest", "octopus" and "nautilus" are
offered to be viewed. For older versions I had to go directly to
github.

Also simply switching the URL from
"https://docs.ceph.com/en/nautilus/"; to
"https://docs.ceph.com/en/luminous/"; will not work any more.

Is it planned to make the documentation of the older version available
again through doc.ceph.com?

Best,
Martin

On Sat, Nov 21, 2020 at 2:11 AM Dan Mick  wrote:


On 11/14/2020 10:56 AM, Martin Palma wrote:

Hello,

maybe I missed the announcement but why is the documentation of the
older ceph version not accessible anymore on docs.ceph.com


It's changed UI because we're hosting them on readthedocs.com now.  See
the dropdown in the lower right corner.




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Unable to find further optimization, or distribution is already perfect

[Nathan Fish]

What does "ceph osd pool autoscale-status" report?

On Mon, Nov 23, 2020 at 12:59 PM Toby Darling  wrote:
>
> Hi
>
> We're having problems getting our erasure coded ec82pool to upmap balance.
> "ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf)
> nautilus (stable)": 554
>
> The pool consists of 20 nodes in 10 racks, each rack containing a pair
> of nodes 1@45*8TB drives and 1@10*16TB.
> https://pastebin.com/YLwu8VVi
>
> The problem is the 8TB drives are roughly 62-74% full, while the 16TB
> drives are 84-87% full.
> https://pastebin.com/j7Dx883i
>
> Neither osdmaptool or reweight-by-utilization are able to improve the
> distribution .
>
> There's an osdmap ftp://ftp.mrc-lmb.cam.ac.uk/pub/toby/osdmap.2135441.
>
> Any thoughts/pointers much appreciated.
>
> Cheers
> Toby
> --
> Toby Darling, Scientific Computing (2N249)
> MRC Laboratory of Molecular Biology
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Unable to find further optimization, or distribution is already perfect

[Toby Darling]

Hi

We're having problems getting our erasure coded ec82pool to upmap balance.
"ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) 
nautilus (stable)": 554


The pool consists of 20 nodes in 10 racks, each rack containing a pair 
of nodes 1@45*8TB drives and 1@10*16TB.

https://pastebin.com/YLwu8VVi

The problem is the 8TB drives are roughly 62-74% full, while the 16TB 
drives are 84-87% full.

https://pastebin.com/j7Dx883i

Neither osdmaptool or reweight-by-utilization are able to improve the 
distribution .


There's an osdmap ftp://ftp.mrc-lmb.cam.ac.uk/pub/toby/osdmap.2135441.

Any thoughts/pointers much appreciated.

Cheers
Toby
--
Toby Darling, Scientific Computing (2N249)
MRC Laboratory of Molecular Biology
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: PGs undersized for no reason?

[Frank Schilder]

Found it. OSDs came up in the wrong root.
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Frank Schilder 
Sent: 23 November 2020 12:46:32
To: ceph-users@ceph.io
Subject: [ceph-users] PGs undersized for no reason?

Hi all,

I'm upgrading ceph mimic 13.2.8 to 13.2.10 and make a strange observation. When 
restarting OSDs on the new version, the PGs come back as undersized. They are 
missing 1 OSD and I get a lot of objects degraded/misplaced.

I have only the noout flag set.

Can anyone help me out why the PGs don't peer until they are all complete?
Is there a flag I can set to get complete PGs before starting backfill/recovery?

Ceph is currently rebuilding objects even though all data should still be 
there. Hence, the update takes an unreasonable amount of time now and I 
remember that with the update from 13.2.2 to 13.2.8 PGs came back complete 
really fast. There was no such extended period with incomplete PGs and degraded 
redundancy.

Thanks and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] PGs undersized for no reason?

[Frank Schilder]

Hi all,

I'm upgrading ceph mimic 13.2.8 to 13.2.10 and make a strange observation. When 
restarting OSDs on the new version, the PGs come back as undersized. They are 
missing 1 OSD and I get a lot of objects degraded/misplaced.

I have only the noout flag set.

Can anyone help me out why the PGs don't peer until they are all complete?
Is there a flag I can set to get complete PGs before starting backfill/recovery?

Ceph is currently rebuilding objects even though all data should still be 
there. Hence, the update takes an unreasonable amount of time now and I 
remember that with the update from 13.2.2 to 13.2.8 PGs came back complete 
really fast. There was no such extended period with incomplete PGs and degraded 
redundancy.

Thanks and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ssd suggestion

[Anthony D'Atri]

Those are QLC, with low durability.  They may work okay for your use case if 
you keep an eye on lifetime, esp if your writes tend to sequential.   Random 
writes will eat them more quickly, as will of course EC.  

Remember that recovery and balancing contribute to writes, and ask Micron for 
the latest firmware, which can take a long time to be published on the web 
site.  




> On Nov 23, 2020, at 5:10 AM, mj  wrote:
> 
> Hi,
> 
> We are going to replace our spinning SATA 4GB filestore disks with new 4GB 
> SSD bluestore disks. Our cluster is reading far more than writing.
> 
> Comparing options, I found the interesting and cheap Micron 5210 ION 3,84TB 
> SSDs. The way we understand it, there is a performance hit, when it comes to 
> continuous writing speeds. But costwise those SSD's are very interesting. 
> (only 450 euro each)
> 
> Our cluster is only small: consisting of three servers, in a 3/2 redundancy 
> config. I was planning to replace the 8 OSDs on one server, and then take 
> some time to checkout how well (or not...) they perform.
> 
> We just wanted to ask here: anyone with suggestions on alternative SSDs we 
> should consider? Or other tips we should take into consideration..?
> 
> Thanks,
> MJ
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Documentation of older Ceph version not accessible anymore on docs.ceph.com

[Martin Palma]

Hi Dan,

yes I noticed but now only "latest", "octopus" and "nautilus" are
offered to be viewed. For older versions I had to go directly to
github.

Also simply switching the URL from
"https://docs.ceph.com/en/nautilus/"; to
"https://docs.ceph.com/en/luminous/"; will not work any more.

Is it planned to make the documentation of the older version available
again through doc.ceph.com?

Best,
Martin

On Sat, Nov 21, 2020 at 2:11 AM Dan Mick  wrote:
>
> On 11/14/2020 10:56 AM, Martin Palma wrote:
> > Hello,
> >
> > maybe I missed the announcement but why is the documentation of the
> > older ceph version not accessible anymore on docs.ceph.com
>
> It's changed UI because we're hosting them on readthedocs.com now.  See
> the dropdown in the lower right corner.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

[Denis Krienbühl]

Thanks Frédéric, we’ve done that in the meantime to work around issue #47866.

The error has been reproduced and there’s a PR associated with the issue:

https://tracker.ceph.com/issues/47866 

Cheers,

Denis

> On 23 Nov 2020, at 11:56, Frédéric Nass  
> wrote:
> 
> Hi Denis,
> 
> You might want to look at rgw_gc_obj_min_wait from [1] and try increasing the 
> default value of 7200s (2 hours) to whatever suits your need < 2^64.
> Just remind that at some point you'll have to get these objects processed by 
> the gc. Or manually through the API [2].
> 
> One thing that comes to mind regarding the "last night's missing object" is 
> maybe it was multi-part re-written and the re-write failed somehow and the 
> object was then enlisted by the gc. But that supposes this particular object 
> sometimes gets re-written which may not be the case.
> 
> Regards,
> 
> Frédéric.
> 
> [1] https://docs.ceph.com/en/latest/radosgw/config-ref/
> [2] 
> https://docs.ceph.com/en/latest/dev/radosgw/admin/adminops_nonimplemented/#manually-processes-garbage-collection-items
> 
> Le 18/11/2020 à 11:27, Denis Krienbühl a écrit :
>> By the way, since there’s some probability that this is a GC refcount issue, 
>> would it be possible and sane to somehow slow the GC down or disable it 
>> altogether? Is that something we could implement on our end as a stop-gap 
>> measure to prevent dataloss?
>> 
>>> On 18 Nov 2020, at 10:46, Denis Krienbühl  wrote:
>>> 
>>> I can now confirm that last night’s missing object was a multi-part file.
>>> 
 On 18 Nov 2020, at 10:01, Janek Bevendorff 
  wrote:
 
 Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME 
 --object=OBJECTNAME (forgot the "object" there)
 
 On 18/11/2020 09:58, Janek Bevendorff wrote:
>> The object, a Docker layer, that went missing has not been touched in 2 
>> months. It worked for a while, but then suddenly went missing.
> Was the object a multipart object? You can check by running radosgw-admin 
> stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something 
> "ns": "multipart" in the output. If it says "ns": "shadow", it's a 
> single-part object.
 ___
 ceph-users mailing list -- ceph-users@ceph.io
 To unsubscribe send an email to ceph-users-le...@ceph.io
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ssd suggestion

[mj]

Hi,

We are going to replace our spinning SATA 4GB filestore disks with new 
4GB SSD bluestore disks. Our cluster is reading far more than writing.


Comparing options, I found the interesting and cheap Micron 5210 ION 
3,84TB SSDs. The way we understand it, there is a performance hit, when 
it comes to continuous writing speeds. But costwise those SSD's are very 
interesting. (only 450 euro each)


Our cluster is only small: consisting of three servers, in a 3/2 
redundancy config. I was planning to replace the 8 OSDs on one server, 
and then take some time to checkout how well (or not...) they perform.


We just wanted to ask here: anyone with suggestions on alternative SSDs 
we should consider? Or other tips we should take into consideration..?


Thanks,
MJ
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD Memory usage

[Igor Fedotov]

Hi Seena,

just to note  - this ticket might be relevant.

https://tracker.ceph.com/issues/48276


Mind leaving a comment there?


Thanks,

Igor

On 11/23/2020 2:51 AM, Seena Fallah wrote:

Now one of my OSDs gets segfault.
Here is the full trace: https://paste.ubuntu.com/p/4KHcCG9YQx/

On Mon, Nov 23, 2020 at 2:16 AM Seena Fallah  wrote:


Hi all,

After I upgrade from 14.2.9 to 14.2.14 my OSDs are using less more memory
than before! I give each OSD 6GB memory target and before the free memory
was 20GB and now after 24h from the upgrade I have 104GB free memory of
128GB memory! Also, my OSD latency got increases!
This happens in both SSD and HDD tier.

Are there any notes from the upgrade I missed? Is it related to
bluefs_buffered_io?
If BlueFS do a direct IO shouldn't BlueFS/Bluestore use the targeted
memory for its cache and does it mean before the upgrade the memory used
was by a kernel that buffers the IO and wasn't for the ceph-osd?

Thanks.


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] HA_proxy setup

[Szabo, Istvan (Agoda)]

Hi,

I wonder is there anybody have a setup like I want to setup?

1st subnet: 10.118.170.0/24 (FE users)
2nd subnet: 10.192.150.0/24 (BE users)

The users are coming from these subnets, and I want that the FE users will come 
on the 1st interface on the loadbalancer, the BE users will come one the 2nd 
interface of the HA_Proxy loadbalancer, so somehow need to create 2 backends 
maybe in the HA_Proxy config?

Both users would go to the same rados gateways and to the same ceph cluster.

Somehow I want to create static routes on the loadbalancer, but not sure how 
can I define in the HA_Proxy config, that go to that specific interface ?

Thanks



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Sizing radosgw and monitor

[Szabo, Istvan (Agoda)]

Hi,

I haven't really find any documentation about how to size radosgw.
One redhat doc says we need to decide the ratio like 1:50 or 1:100 osd / rgw.
I had an issue earlier where I had a user who source loadbalanced so always 
went to the same radosgateway and 1 time just maxed out.

So the question is, how to monitor rgw, what kind of values or ... ?
How to size RGW?

Thank you


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk

[Frédéric Nass]

Hi Denis,

You might want to look at rgw_gc_obj_min_wait from [1] and try 
increasing the default value of 7200s (2 hours) to whatever suits your 
need < 2^64.
Just remind that at some point you'll have to get these objects 
processed by the gc. Or manually through the API [2].


One thing that comes to mind regarding the "last night's missing object" 
is maybe it was multi-part re-written and the re-write failed somehow 
and the object was then enlisted by the gc. But that supposes this 
particular object sometimes gets re-written which may not be the case.


Regards,

Frédéric.

[1] https://docs.ceph.com/en/latest/radosgw/config-ref/
[2] 
https://docs.ceph.com/en/latest/dev/radosgw/admin/adminops_nonimplemented/#manually-processes-garbage-collection-items


Le 18/11/2020 à 11:27, Denis Krienbühl a écrit :

By the way, since there’s some probability that this is a GC refcount issue, 
would it be possible and sane to somehow slow the GC down or disable it 
altogether? Is that something we could implement on our end as a stop-gap 
measure to prevent dataloss?


On 18 Nov 2020, at 10:46, Denis Krienbühl  wrote:

I can now confirm that last night’s missing object was a multi-part file.


On 18 Nov 2020, at 10:01, Janek Bevendorff  
wrote:

Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the 
"object" there)

On 18/11/2020 09:58, Janek Bevendorff wrote:

The object, a Docker layer, that went missing has not been touched in 2 months. 
It worked for a while, but then suddenly went missing.

Was the object a multipart object? You can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. 
It should say something "ns": "multipart" in the output. If it says "ns": 
"shadow", it's a single-part object.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io