date:20191031

Re: [ceph-users] changing set-require-min-compat-client will cause hiccup?

2019-10-31 Thread Philippe D'Anjou

Hi, it is NOT safe.All clients fail to mount rbds now :(
   Am Mittwoch, 30. Oktober 2019, 09:33:16 OEZ hat Konstantin Shalygin 
 Folgendes geschrieben:  
 
  
 

 Hi,I need to change set-require-min-compat-clientto use upmap mode for the PG 
balancer. Will this cause a disconnect of all clients? We're talking cephfs and 
RBD images for VMs.
Or is it save to switch that live? 
 Is safe. 

 
 

 
 
k
 
   ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] feature set mismatch CEPH_FEATURE_MON_GV kernel 5.0?

2019-10-31 Thread Philippe D'Anjou

So it seems like for some reason librados is used now instead of kernel module, 
and this produces the error. But we have all latest Nautilus repos installed on 
the clients...so why would librados throw a compatiblity issue?Client 
compatiblity level is set to Luminous.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] changing set-require-min-compat-client will cause hiccup?

2019-10-31 Thread Konstantin Shalygin


On 10/31/19 2:12 PM, Philippe D'Anjou wrote:

Hi, it is NOT safe.
All clients fail to mount rbds now :(


Your clients is upmap compatible?



k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Bluestore runs out of space and dies

2019-10-31 Thread George Shuklin


Hello.

In my lab a nautilus cluster with a bluestore suddenly went dark. As I 
found it had used 98% of the space and most of OSDs (small, 10G each) 
went offline. Any attempt to restart them failed with this message:


# /usr/bin/ceph-osd -f --cluster  ceph --id 18 --setuser ceph --setgroup 
ceph


2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors 
{default=true}
2019-10-31 09:44:37.615 7f73bff99700 -1 
bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate 
0x1 allocated 0x ffe4 min_alloc_size 0x1 available 0x 0
2019-10-31 09:44:37.615 7f73bff99700 -1 
bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed 
with (28) No space left on device
2019-10-31 09:44:37.615 7f73bff99700 -1 
bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No 
space left on device not handled on operation 10 (op 30, counting from 0)
2019-10-31 09:44:37.615 7f73bff99700 -1 
bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore, 
misconfigured cluster
/build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void 
BlueStore::_txc_add_transaction(BlueStore::TransContext*, 
ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31 
09:44:37.620694
/build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455: 
ceph_abort_msg("unexpected error")


I was able to recover cluster by adding some more space into VGs for 
some of OSDs and using this command:


ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx 
--command bluefs-bdev-expand


It worked but only because I added some space into OSD.

I'm curious, is there a way to recover such OSD without growing it? On 
the old filestore I can just remove some objects to gain space, is this 
possible for bluestore? My main concern is that OSD daemon simply 
crashes at start, so I can't just add 'more OSD' to cluster - all data 
become unavailable, because OSDs are completely dead.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore runs out of space and dies

2019-10-31 Thread Paul Emmerich

BlueStore doesn't handle running out of space gracefully because that
doesn't happen on a real disk because full_ratio (95%) and the
failsafe_full_ratio (97%? some obscure config option) kick in before
that happens.

Yeah, I've also lost some test cluster with tiny disks to this. Usual
reason is keeping it in a degraded state for weeks at a time...

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Thu, Oct 31, 2019 at 10:50 AM George Shuklin
 wrote:
>
> Hello.
>
> In my lab a nautilus cluster with a bluestore suddenly went dark. As I
> found it had used 98% of the space and most of OSDs (small, 10G each)
> went offline. Any attempt to restart them failed with this message:
>
> # /usr/bin/ceph-osd -f --cluster  ceph --id 18 --setuser ceph --setgroup
> ceph
>
> 2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors
> {default=true}
> 2019-10-31 09:44:37.615 7f73bff99700 -1
> bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate
> 0x1 allocated 0x ffe4 min_alloc_size 0x1 available 0x 0
> 2019-10-31 09:44:37.615 7f73bff99700 -1
> bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed
> with (28) No space left on device
> 2019-10-31 09:44:37.615 7f73bff99700 -1
> bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No
> space left on device not handled on operation 10 (op 30, counting from 0)
> 2019-10-31 09:44:37.615 7f73bff99700 -1
> bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore,
> misconfigured cluster
> /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void
> BlueStore::_txc_add_transaction(BlueStore::TransContext*,
> ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31
> 09:44:37.620694
> /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455:
> ceph_abort_msg("unexpected error")
>
> I was able to recover cluster by adding some more space into VGs for
> some of OSDs and using this command:
>
> ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx
> --command bluefs-bdev-expand
>
> It worked but only because I added some space into OSD.
>
> I'm curious, is there a way to recover such OSD without growing it? On
> the old filestore I can just remove some objects to gain space, is this
> possible for bluestore? My main concern is that OSD daemon simply
> crashes at start, so I can't just add 'more OSD' to cluster - all data
> become unavailable, because OSDs are completely dead.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph pg in inactive state

2019-10-31 Thread Janne Johansson

Den tors 31 okt. 2019 kl 04:22 skrev soumya tr :

> Thanks 潘东元 for the response.
>
> The creation of a new pool works, and all the PGs corresponding to that
> pool have active+clean state.
>
> When I initially set ceph 3 node cluster using juju charms (replication
> count per object was set to 3), there were issues with ceph-osd services.
> So I had to delete the units and readd them (I did all of them together,
> which must have created issues with rebalancing). I assume that the PGs in
> the inactive state points to the 3 old OSDs which were deleted.
>
> I assume I will have to create all the pools again. But my concern is
> about the default pools.
>
> ---
> pool 1 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule
> 0 object_hash rjenkins pg_num 16 pgp_num 16 last_change 15 flags hashpspool
> stripe_width 0 application rgw
> pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 19 flags hashpspool
> stripe_width 0 application rgw
> pool 3 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 23 flags hashpspool
> stripe_width 0 application rgw
> pool 4 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 27 flags hashpspool
> stripe_width 0 application rgw
> pool 5 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 31 flags hashpspool
> stripe_width 0 application rgw
> pool 6 'default.rgw.intent-log' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 35 flags hashpspool
> stripe_width 0 application rgw
> pool 7 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 39 flags hashpspool
> stripe_width 0 application rgw
> pool 8 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 43 flags hashpspool
> stripe_width 0 application rgw
> pool 9 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 47 flags hashpspool
> stripe_width 0 application rgw
> pool 10 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule
> 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 51 flags hashpspool
> stripe_width 0 application rgw
> pool 11 'default.rgw.users.swift' replicated size 3 min_size 2 crush_rule
> 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 55 flags hashpspool
> stripe_width 0 application rgw
> pool 12 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 2 pgp_num 2 last_change 59 flags hashpspool
> stripe_width 0 application rgw
> pool 13 'default.rgw.buckets.extra' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 63 flags
> hashpspool stripe_width 0 application rgw
> pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 last_change 67 flags
> hashpspool stripe_width 0 application rgw
> ---
>
> Can you please update if recreating them using rados cli will break
> anything?
>
>
Those pools belong to radosgw, and if they are missing, they will be
created on demand the next time radosgw starts up.
the "defaul" is the name of the radosgw zone, which defaults to...
"default". They are not needed by any other part of ceph.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore runs out of space and dies

2019-10-31 Thread Nathan Fish

The best way to prevent this on a testing cluster with tiny virtual
drives is probably to lower the various full_ratio's significantly.

On Thu, Oct 31, 2019 at 7:17 AM Paul Emmerich  wrote:
>
> BlueStore doesn't handle running out of space gracefully because that
> doesn't happen on a real disk because full_ratio (95%) and the
> failsafe_full_ratio (97%? some obscure config option) kick in before
> that happens.
>
> Yeah, I've also lost some test cluster with tiny disks to this. Usual
> reason is keeping it in a degraded state for weeks at a time...
>
> Paul
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
>
> On Thu, Oct 31, 2019 at 10:50 AM George Shuklin
>  wrote:
> >
> > Hello.
> >
> > In my lab a nautilus cluster with a bluestore suddenly went dark. As I
> > found it had used 98% of the space and most of OSDs (small, 10G each)
> > went offline. Any attempt to restart them failed with this message:
> >
> > # /usr/bin/ceph-osd -f --cluster  ceph --id 18 --setuser ceph --setgroup
> > ceph
> >
> > 2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors
> > {default=true}
> > 2019-10-31 09:44:37.615 7f73bff99700 -1
> > bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate
> > 0x1 allocated 0x ffe4 min_alloc_size 0x1 available 0x 0
> > 2019-10-31 09:44:37.615 7f73bff99700 -1
> > bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed
> > with (28) No space left on device
> > 2019-10-31 09:44:37.615 7f73bff99700 -1
> > bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No
> > space left on device not handled on operation 10 (op 30, counting from 0)
> > 2019-10-31 09:44:37.615 7f73bff99700 -1
> > bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore,
> > misconfigured cluster
> > /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void
> > BlueStore::_txc_add_transaction(BlueStore::TransContext*,
> > ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31
> > 09:44:37.620694
> > /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455:
> > ceph_abort_msg("unexpected error")
> >
> > I was able to recover cluster by adding some more space into VGs for
> > some of OSDs and using this command:
> >
> > ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx
> > --command bluefs-bdev-expand
> >
> > It worked but only because I added some space into OSD.
> >
> > I'm curious, is there a way to recover such OSD without growing it? On
> > the old filestore I can just remove some objects to gain space, is this
> > possible for bluestore? My main concern is that OSD daemon simply
> > crashes at start, so I can't just add 'more OSD' to cluster - all data
> > become unavailable, because OSDs are completely dead.
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore runs out of space and dies

2019-10-31 Thread George Shuklin

Thank you everyone, I got it. There is no way to fix out-of-space 
bluestore without expanding it.


Therefore, in production we would stick with 99%FREE size for LV, as it 
gives operators 'last chance' to repair the cluster in case of 
emergency. It's a bit unfortunate that we need to give up the whole per 
cent (1 % is too much for 4Tb drives).


On 31/10/2019 15:04, Nathan Fish wrote:

The best way to prevent this on a testing cluster with tiny virtual
drives is probably to lower the various full_ratio's significantly.

On Thu, Oct 31, 2019 at 7:17 AM Paul Emmerich  wrote:

BlueStore doesn't handle running out of space gracefully because that
doesn't happen on a real disk because full_ratio (95%) and the
failsafe_full_ratio (97%? some obscure config option) kick in before
that happens.

Yeah, I've also lost some test cluster with tiny disks to this. Usual
reason is keeping it in a degraded state for weeks at a time...

Paul

--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Thu, Oct 31, 2019 at 10:50 AM George Shuklin
 wrote:

Hello.

In my lab a nautilus cluster with a bluestore suddenly went dark. As I
found it had used 98% of the space and most of OSDs (small, 10G each)
went offline. Any attempt to restart them failed with this message:

# /usr/bin/ceph-osd -f --cluster  ceph --id 18 --setuser ceph --setgroup
ceph

2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors
{default=true}
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate
0x1 allocated 0x ffe4 min_alloc_size 0x1 available 0x 0
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed
with (28) No space left on device
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No
space left on device not handled on operation 10 (op 30, counting from 0)
2019-10-31 09:44:37.615 7f73bff99700 -1
bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore,
misconfigured cluster
/build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void
BlueStore::_txc_add_transaction(BlueStore::TransContext*,
ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31
09:44:37.620694
/build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455:
ceph_abort_msg("unexpected error")

I was able to recover cluster by adding some more space into VGs for
some of OSDs and using this command:

ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx
--command bluefs-bdev-expand

It worked but only because I added some space into OSD.

I'm curious, is there a way to recover such OSD without growing it? On
the old filestore I can just remove some objects to gain space, is this
possible for bluestore? My main concern is that OSD daemon simply
crashes at start, so I can't just add 'more OSD' to cluster - all data
become unavailable, because OSDs are completely dead.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore runs out of space and dies

2019-10-31 Thread Nathan Fish

You should never run into this problem on a 4TB drive in the first
place. Bluestore explodes if it can't allocate a few GB; but on a 4TB
drive, the default full_ratio of 0.95 will forbid placing any new
objects onto an OSD with less than 200GB.

On Thu, Oct 31, 2019 at 9:31 AM George Shuklin  wrote:
>
> Thank you everyone, I got it. There is no way to fix out-of-space
> bluestore without expanding it.
>
> Therefore, in production we would stick with 99%FREE size for LV, as it
> gives operators 'last chance' to repair the cluster in case of
> emergency. It's a bit unfortunate that we need to give up the whole per
> cent (1 % is too much for 4Tb drives).
>
> On 31/10/2019 15:04, Nathan Fish wrote:
> > The best way to prevent this on a testing cluster with tiny virtual
> > drives is probably to lower the various full_ratio's significantly.
> >
> > On Thu, Oct 31, 2019 at 7:17 AM Paul Emmerich  
> > wrote:
> >> BlueStore doesn't handle running out of space gracefully because that
> >> doesn't happen on a real disk because full_ratio (95%) and the
> >> failsafe_full_ratio (97%? some obscure config option) kick in before
> >> that happens.
> >>
> >> Yeah, I've also lost some test cluster with tiny disks to this. Usual
> >> reason is keeping it in a degraded state for weeks at a time...
> >>
> >> Paul
> >>
> >> --
> >> Paul Emmerich
> >>
> >> Looking for help with your Ceph cluster? Contact us at https://croit.io
> >>
> >> croit GmbH
> >> Freseniusstr. 31h
> >> 81247 München
> >> www.croit.io
> >> Tel: +49 89 1896585 90
> >>
> >> On Thu, Oct 31, 2019 at 10:50 AM George Shuklin
> >>  wrote:
> >>> Hello.
> >>>
> >>> In my lab a nautilus cluster with a bluestore suddenly went dark. As I
> >>> found it had used 98% of the space and most of OSDs (small, 10G each)
> >>> went offline. Any attempt to restart them failed with this message:
> >>>
> >>> # /usr/bin/ceph-osd -f --cluster  ceph --id 18 --setuser ceph --setgroup
> >>> ceph
> >>>
> >>> 2019-10-31 09:44:37.591 7f73d54b3f80 -1 osd.18 271 log_to_monitors
> >>> {default=true}
> >>> 2019-10-31 09:44:37.615 7f73bff99700 -1
> >>> bluestore(/var/lib/ceph/osd/ceph-18) _do_alloc_write failed to allocate
> >>> 0x1 allocated 0x ffe4 min_alloc_size 0x1 available 0x > >>> 0
> >>> 2019-10-31 09:44:37.615 7f73bff99700 -1
> >>> bluestore(/var/lib/ceph/osd/ceph-18) _do_write _do_alloc_write failed
> >>> with (28) No space left on device
> >>> 2019-10-31 09:44:37.615 7f73bff99700 -1
> >>> bluestore(/var/lib/ceph/osd/ceph-18) _txc_add_transaction error (28) No
> >>> space left on device not handled on operation 10 (op 30, counting from 0)
> >>> 2019-10-31 09:44:37.615 7f73bff99700 -1
> >>> bluestore(/var/lib/ceph/osd/ceph-18) ENOSPC from bluestore,
> >>> misconfigured cluster
> >>> /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: In function 'void
> >>> BlueStore::_txc_add_transaction(BlueStore::TransContext*,
> >>> ObjectStore::Transaction*)' thread 7f73bff99700 time 2019-10-31
> >>> 09:44:37.620694
> >>> /build/ceph-14.2.4/src/os/bluestore/BlueStore.cc: 11455:
> >>> ceph_abort_msg("unexpected error")
> >>>
> >>> I was able to recover cluster by adding some more space into VGs for
> >>> some of OSDs and using this command:
> >>>
> >>> ceph-bluestore-tool --log-level 30 --path /var/lib/ceph/osd/ceph-xx
> >>> --command bluefs-bdev-expand
> >>>
> >>> It worked but only because I added some space into OSD.
> >>>
> >>> I'm curious, is there a way to recover such OSD without growing it? On
> >>> the old filestore I can just remove some objects to gain space, is this
> >>> possible for bluestore? My main concern is that OSD daemon simply
> >>> crashes at start, so I can't just add 'more OSD' to cluster - all data
> >>> become unavailable, because OSDs are completely dead.
> >>>
> >>> ___
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore runs out of space and dies

2019-10-31 Thread Janne Johansson

Den tors 31 okt. 2019 kl 15:07 skrev George Shuklin <
george.shuk...@gmail.com>:

> Thank you everyone, I got it. There is no way to fix out-of-space
> bluestore without expanding it.
>
> Therefore, in production we would stick with 99%FREE size for LV, as it
> gives operators 'last chance' to repair the cluster in case of
> emergency. It's a bit unfortunate that we need to give up the whole per
> cent (1 % is too much for 4Tb drives).
>

In production, stuff with start giving warnings at 85%, so you would just
not get into this kinds of situations where the last percent matters or not.


-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bluestore runs out of space and dies

2019-10-31 Thread George Shuklin


On 31/10/2019 17:02, Janne Johansson wrote:

I thought about protecting everything with a proper full ratio, but I 
really afraid of the perspective of some human error (raising fullratio 
too high to help to recover some 'very important data asap') which would 
leave cluster broken for real. Those few Gb would be the next line of 
defense. It's better to have a downtime than 'unable to recover' situation.


Den tors 31 okt. 2019 kl 15:07 skrev George Shuklin 
mailto:george.shuk...@gmail.com>>:


Thank you everyone, I got it. There is no way to fix out-of-space
bluestore without expanding it.

Therefore, in production we would stick with 99%FREE size for LV,
as it
gives operators 'last chance' to repair the cluster in case of
emergency. It's a bit unfortunate that we need to give up the
whole per
cent (1 % is too much for 4Tb drives).


In production, stuff with start giving warnings at 85%, so you would 
just not get into this kinds of situations where the last percent 
matters or not.



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW/swift segments

2019-10-31 Thread Peter Eisch

I’ve verified this is still broken on Nautilus (14.2.4).  The code for deleting 
a swift object split into segments is correct.  I’m now trying to follow the 
object expirer code.  It seems it reads the metadata for the lead object 
(0-bytes in length) and expires that correctly but never understands to look 
for its associated segments in the metadata.  It could be because these are 
stored in a different bucket – I’m not sure.  Once the lead object is deleted, 
the metadata is removed and all the segments are then orphaned.

Is anyone here familiar with the RGWObjectExpirer code who I could confer?

peter


Peter Eisch
Senior Site Reliability Engineer
T1.612.659.3228
virginpulse.com
|virginpulse.com/global-challenge
Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland 
| United Kingdom | USA
Confidentiality Notice: The information contained in this e-mail, including any 
attachment(s), is intended solely for use by the designated recipient(s). 
Unauthorized use, dissemination, distribution, or reproduction of this message 
by anyone other than the intended recipient(s), or a person designated as 
responsible for delivering such messages to the intended recipient, is strictly 
prohibited and may be unlawful. This e-mail may contain proprietary, 
confidential or privileged information. Any views or opinions expressed are 
solely those of the author and do not necessarily represent those of Virgin 
Pulse, Inc. If you have received this message in error, or are not the named 
recipient(s), please immediately notify the sender and delete this e-mail 
message.
v2.64
From: ceph-users  on behalf of Peter Eisch 

Date: Monday, October 28, 2019 at 3:06 PM
To: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] RGW/swift segments

I should have noted this is with Luminous 12.2.12 and consistent with 
swiftclient versions from 3.0.0 to 3.8.1, which may not be relevant.  With a 
proper nod I can open a ticket for this – just want to make sure it’s not a 
config issue.

[client.rgw.cephrgw-s01]
  host = cephrgw-s01
  keyring = /etc/ceph/ceph.client.rgw.cephrgw-s01
  rgw_zone = 
  rgw zonegroup = us
  rgw realm = 
  rgw dns name = rgw-s00.
  rgw dynamic resharding = false
  rgw swift account in url = true
  rgw swift url = https://rgw-s00./swift/v1
  rgw keystone make new tenants = true
  rgw keystone implicit tenants = true
  rgw enable usage log = true
  rgw keystone accepted roles = _member_,admin
  rgw keystone admin domain = Default
  rgw keystone admin password = 
  rgw keystone admin project = admin
  rgw keystone admin user = admin
  rgw keystone api version = 3
  rgw keystone url = https://keystone-s00.
  rgw relaxed s3 bucket names = true
  rgw s3 auth use keystone = true
  rgw thread pool size = 4096
  rgw keystone revocation interval = 300
  rgw keystone token cache size = 1
  rgw swift versioning enabled = true
  rgw log nonexistent bucket = true

All tips accepted…

peter

From: ceph-users  on behalf of Peter Eisch 

Date: Monday, October 28, 2019 at 9:28 AM
To: "ceph-users@lists.ceph.com" 
Subject: [ceph-users] RGW/swift segments


Hi,

When uploading to RGW via swift I can set an expiration time.  The files being 
uploaded are large.  We segment them using the swift upload ‘-S’ arg.  This 
results in a 0-byte file in the bucket and all the data frags landing in a 
*_segments bucket.

When the expiration passes the 0-byte file is delete but all the segments 
remain.  Am I misconfigured or is this a bug where it won’t expire the actual 
data?  Shouldn’t RGW set the expiration on the uploaded segments too if they’re 
managed separately?

Thanks,

peter
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW/swift segments

2019-10-31 Thread Peter Eisch

Removal hints are created for the head object (of 0 bytes) correctly.  The 
expiry process is unaware of the relationship between the head object and the 
segments.  Likewise no removal hit is created for the swift segments.

I’m awaiting a blessing to file this as a bug, would any capable reader of this 
be willing to file it as a bug, please?

peter


Peter Eisch
Senior Site Reliability Engineer
T1.612.659.3228
virginpulse.com
|virginpulse.com/global-challenge
Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland 
| United Kingdom | USA
Confidentiality Notice: The information contained in this e-mail, including any 
attachment(s), is intended solely for use by the designated recipient(s). 
Unauthorized use, dissemination, distribution, or reproduction of this message 
by anyone other than the intended recipient(s), or a person designated as 
responsible for delivering such messages to the intended recipient, is strictly 
prohibited and may be unlawful. This e-mail may contain proprietary, 
confidential or privileged information. Any views or opinions expressed are 
solely those of the author and do not necessarily represent those of Virgin 
Pulse, Inc. If you have received this message in error, or are not the named 
recipient(s), please immediately notify the sender and delete this e-mail 
message.
v2.64
From: ceph-users  on behalf of Peter Eisch 

Date: Thursday, October 31, 2019 at 10:27 AM
To: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] RGW/swift segments

I’ve verified this is still broken on Nautilus (14.2.4).  The code for deleting 
a swift object split into segments is correct.  I’m now trying to follow the 
object expirer code.  It seems it reads the metadata for the lead object 
(0-bytes in length) and expires that correctly but never understands to look 
for its associated segments in the metadata.  It could be because these are 
stored in a different bucket – I’m not sure.  Once the lead object is deleted, 
the metadata is removed and all the segments are then orphaned.

Is anyone here familiar with the RGWObjectExpirer code who I could confer?

peter

From: ceph-users  on behalf of Peter Eisch 

Date: Monday, October 28, 2019 at 3:06 PM
To: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] RGW/swift segments

I should have noted this is with Luminous 12.2.12 and consistent with 
swiftclient versions from 3.0.0 to 3.8.1, which may not be relevant.  With a 
proper nod I can open a ticket for this – just want to make sure it’s not a 
config issue.

[client.rgw.cephrgw-s01]
  host = cephrgw-s01
  keyring = /etc/ceph/ceph.client.rgw.cephrgw-s01
  rgw_zone = 
  rgw zonegroup = us
  rgw realm = 
  rgw dns name = rgw-s00.
  rgw dynamic resharding = false
  rgw swift account in url = true
  rgw swift url = https://rgw-s00./swift/v1
  rgw keystone make new tenants = true
  rgw keystone implicit tenants = true
  rgw enable usage log = true
  rgw keystone accepted roles = _member_,admin
  rgw keystone admin domain = Default
  rgw keystone admin password = 
  rgw keystone admin project = admin
  rgw keystone admin user = admin
  rgw keystone api version = 3
  rgw keystone url = https://keystone-s00.
  rgw relaxed s3 bucket names = true
  rgw s3 auth use keystone = true
  rgw thread pool size = 4096
  rgw keystone revocation interval = 300
  rgw keystone token cache size = 1
  rgw swift versioning enabled = true
  rgw log nonexistent bucket = true

All tips accepted…

peter

From: ceph-users  on behalf of Peter Eisch 

Date: Monday, October 28, 2019 at 9:28 AM
To: "ceph-users@lists.ceph.com" 
Subject: [ceph-users] RGW/swift segments


Hi,

When uploading to RGW via swift I can set an expiration time.  The files being 
uploaded are large.  We segment them using the swift upload ‘-S’ arg.  This 
results in a 0-byte file in the bucket and all the data frags landing in a 
*_segments bucket.

When the expiration passes the 0-byte file is delete but all the segments 
remain.  Am I misconfigured or is this a bug where it won’t expire the actual 
data?  Shouldn’t RGW set the expiration on the uploaded segments too if they’re 
managed separately?

Thanks,

peter
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph pg in inactive state

2019-10-31 Thread soumya tr

Thanks Janne!!

I deleted all the pools.  A few default rgw pools got auto-created, and the
rest I created manually. Now Ceph looks happy.

On Thu, Oct 31, 2019 at 11:18 PM Janne Johansson 
wrote:

>
>
> Den tors 31 okt. 2019 kl 04:22 skrev soumya tr :
>
>> Thanks 潘东元 for the response.
>>
>> The creation of a new pool works, and all the PGs corresponding to that
>> pool have active+clean state.
>>
>> When I initially set ceph 3 node cluster using juju charms (replication
>> count per object was set to 3), there were issues with ceph-osd services.
>> So I had to delete the units and readd them (I did all of them together,
>> which must have created issues with rebalancing). I assume that the PGs in
>> the inactive state points to the 3 old OSDs which were deleted.
>>
>> I assume I will have to create all the pools again. But my concern is
>> about the default pools.
>>
>> ---
>> pool 1 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule
>> 0 object_hash rjenkins pg_num 16 pgp_num 16 last_change 15 flags hashpspool
>> stripe_width 0 application rgw
>> pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 19 flags hashpspool
>> stripe_width 0 application rgw
>> pool 3 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 23 flags hashpspool
>> stripe_width 0 application rgw
>> pool 4 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 27 flags hashpspool
>> stripe_width 0 application rgw
>> pool 5 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 31 flags hashpspool
>> stripe_width 0 application rgw
>> pool 6 'default.rgw.intent-log' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 35 flags hashpspool
>> stripe_width 0 application rgw
>> pool 7 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 39 flags hashpspool
>> stripe_width 0 application rgw
>> pool 8 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 43 flags hashpspool
>> stripe_width 0 application rgw
>> pool 9 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 47 flags hashpspool
>> stripe_width 0 application rgw
>> pool 10 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule
>> 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 51 flags hashpspool
>> stripe_width 0 application rgw
>> pool 11 'default.rgw.users.swift' replicated size 3 min_size 2 crush_rule
>> 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 55 flags hashpspool
>> stripe_width 0 application rgw
>> pool 12 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 2 pgp_num 2 last_change 59 flags hashpspool
>> stripe_width 0 application rgw
>> pool 13 'default.rgw.buckets.extra' replicated size 3 min_size 2
>> crush_rule 0 object_hash rjenkins pg_num 2 pgp_num 2 last_change 63 flags
>> hashpspool stripe_width 0 application rgw
>> pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2
>> crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 last_change 67 flags
>> hashpspool stripe_width 0 application rgw
>> ---
>>
>> Can you please update if recreating them using rados cli will break
>> anything?
>>
>>
> Those pools belong to radosgw, and if they are missing, they will be
> created on demand the next time radosgw starts up.
> the "defaul" is the name of the radosgw zone, which defaults to...
> "default". They are not needed by any other part of ceph.
>
> --
> May the most significant bit of your life be positive.
>


-- 
Regards,
Soumya
Linux Sytem Administrator
DirectI

"I like the dreams of the future better than the history of the past."
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] changing set-require-min-compat-client will cause hiccup?

[ceph-users] feature set mismatch CEPH_FEATURE_MON_GV kernel 5.0?

Re: [ceph-users] changing set-require-min-compat-client will cause hiccup?

[ceph-users] Bluestore runs out of space and dies

Re: [ceph-users] Bluestore runs out of space and dies

Re: [ceph-users] Ceph pg in inactive state

Re: [ceph-users] Bluestore runs out of space and dies

Re: [ceph-users] Bluestore runs out of space and dies

Re: [ceph-users] Bluestore runs out of space and dies

Re: [ceph-users] Bluestore runs out of space and dies

Re: [ceph-users] Bluestore runs out of space and dies

Re: [ceph-users] RGW/swift segments

Re: [ceph-users] RGW/swift segments

Re: [ceph-users] Ceph pg in inactive state

14 matches

Site Navigation

Mail list logo

Footer information