Re: [ceph-users] OSDs crash after deleting unfound object in Luminous 12.2.8

[Mike Lovell]

what was the object name that you marked lost? was it one of the cache tier
hit_sets?

the trace you have does seem to be failing when the OSD is trying to remove
a hit set that is no longer needed. i ran into a similar problem which
might have been why that bug you listed was created. maybe providing what i
have since discovered about hit sets might help.

the hit sets are what the cache tier uses to know which objects have been
accessed in a given period of time. these hit sets are then stored in the
object store using an object name that is generated. for the version you're
running, the code for that generation is at
https://github.com/ceph/ceph/blob/v12.2.8/src/osd/PrimaryLogPG.cc#L12667.
its bascially "hit_set__archive__" where the
times are recorded in the hit set history. that hit set history is stored
as part of the PG metadata. you can get a list of all of the hit sets the
PG has by looking at 'ceph pg  query' and looking at the
['info']['hit_set_history']['history'] array. each entry in that array has
the information on each hit set for the PG and the times are what is used
in generation of the object name. there should be one ceph object for each
hit set listed in that array.

if you told the cluster to mark one of the hit set objects as lost, its
possible the OSD cannot get that object and is hitting the assert(obc) near
the end of PrimaryLogPG::hit_set_trim in the same source file referenced
above. you can potentially verify this by a couple methods. i think if you
set debug_osd at 20 that it should log a line saying something like
"hit_set_trim removing hit_set__archive." if that name matches
one of the ones you marked lost, then is this almost certainly the cause.
you can also do a find on the OSD directory, if you're using file store,
and look for the right file name. something like 'find
/var/lib/ceph/osd/ceph-/current/_head -name
hit\*set\*\*archive\*' should work. include the \ to escape the * so
bash doesn't interpret it. if you're using bluestore, i think you can use
the ceph-objectstore-tool while the osd is stopped to get a list of
objects. you'll probably want to only look in the .ceph-internal namespace
since the hit sets are stored in that namespace.

there are a couple potential ways to get around this. what we did when we
had the problem was run a custom build of the ceph-osd where we commented
out the assert(obc); line in hit_set_trim. that build was only run for long
enough to get the cluster back online and then to flush and evict the
entire cache, remove the cache, restart using the normal ceph builds, and
then recreate the cache.

the other options are things that i don't know for sure if they'll work. if
you're using file store, you might be able to just copy another hit set to
the file name of the missing hit set object. this should be pretty benign
and its just going to remove the object in a moment anyways. also, i'm not
entirely sure how to come up with what directory to put the object in if
the osd has done any directory splitting. maybe someone on the list will
know how to do this. there might be a way with the ceph-objectstore-tool to
write in the object but i couldn't find one in my testing on hammer.

the last option i can think of, is that if you can completely stop any
traffic to the pools in question, its possible the OSDs wont crash.
hit_set_trim doesn't appear to get called if there is no client traffic
reaching the osds and the hit sets aren't being updated. if you can stop
anything from using the pools in question and guarantee nothing will come
in, then it might be possible to keep the OSDs long enough to flush
everything from the cache tier, remove it, and recreate it. this option
seems like a long shot and i don't know for sure it'll work. it just seemed
to me like the OSDs would stay up in a similar scenario on my hammer test
cluster. its possible things have changed in luminous and hit_set_trim gets
called more often. i also didn't test whether the process of flushing and
evicting the objects in the cache caused hit_set_trim to get called.

hopefully that gives you some more info on what might be going on and ways
around it. i'm not entirely sure why there is still the assert(obj); in
hit_set_trim still. there was a little bit of discussion about removing it
since it means the object its trying to remove is gone anyways. i think
that just happened for a little bit in irc. i guess it didn't happen cause
no one followed up on it.

good luck and hopefully you don't blame me if things get worse. :)
mike

On Fri, Oct 12, 2018 at 7:34 AM Lawrence Smith <
lawrence.sm...@uni-muenster.de> wrote:

> Hi all,
>
> we are running a luminous 12.2.8 cluster with a 3 fold replicated cache
> pool with a min_size of 2. We recently encountered an "object unfound"
> error in one of our pgs in this pool. After marking this object lost,
> two of the acting osds crashed and were unable to start up again, with
> only the primary osd staying up. Hoping the cluster might rem

Re: [ceph-users] OSDs crash after deleting unfound object in Luminous 12.2.8

[Paul Emmerich]

It would be helpful to have a full crash log with debug osd = 0/20 and
the information in which pool and pg you marked the object as lost.
You might be able to use ceph-objectstore-tool to remove the bad
object from the OSD if it still exists in either the cache pool or
underlying pool.

Ugly fix if that doesn't work: patch the code to just ignore that
instead of asserting. Only do that after verifying that it's acutally
crashing on the object you deleted, ideally you'd also check the
object name and only do that on the known bad object. Also, no
guarantee that this is safe to do, but I think it should be here
(after a very short look at the crashing code)

I also had to hardcode an object name of a badly corrupted object into
an OSD to ignore it to prevent a crash an OSD with the last surving
copy of a PG, fun times... (ceph-objectstore-tool wouldn't even
recognize that the object with that name exists in my case)


Paul
Am Fr., 12. Okt. 2018 um 15:34 Uhr schrieb Lawrence Smith
:
>
> Hi all,
>
> we are running a luminous 12.2.8 cluster with a 3 fold replicated cache
> pool with a min_size of 2. We recently encountered an "object unfound"
> error in one of our pgs in this pool. After marking this object lost,
> two of the acting osds crashed and were unable to start up again, with
> only the primary osd staying up. Hoping the cluster might remap the
> copies of this pg, we marked the two crashed osds as out. Now the
> primary osd of this pg has also gone down leaving again only one active
> osd with the cluster reporting a degraded filesystem. All the affected
> osds are running filestore, while about half the cluster has already
> been upgraded to run bluestore osds.
>
> All three of the crashed osds fail to restart, reporting the following
> error during startup:
>
> Oct 12 13:19:12 kaa-109 ceph-osd[166266]:  0> 2018-10-12
> 13:19:12.782652 7f1f2d79b700 -1
> /var/tmp/portage/sys-cluster/ceph-12.2.8/work/ceph-12.2.8/src/osd/PrimaryLogPG.cc:
> In function '
> void PrimaryLogPG::hit_set_trim(PrimaryLogPG::OpContextUPtr&, unsigned
> int)' thread 7f1f2d79b700 time 2018-10-12 13:19:12.779813
> /var/tmp/portage/sys-cluster/ceph-12.2.8/work/ceph-12.2.8/src/osd/PrimaryLogPG.cc:
> 12985: FAILED assert(obc)
>
>   ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
> luminous (stable)
>   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x109) [0x562265bfda9c]
>   2:
> (PrimaryLogPG::hit_set_trim(std::unique_ptr std::default_delete >&, unsigned int)+0x942)
> [0x5622657d6cea]
>   3: (PrimaryLogPG::hit_set_persist()+0xa4b) [0x5622657e5fab]
>   4: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x426a)
> [0x562265800c64]
>   5: (PrimaryLogPG::do_request(boost::intrusive_ptr&,
> ThreadPool::TPHandle&)+0xc1f) [0x5622657b94ed]
>   6: (OSD::dequeue_op(boost::intrusive_ptr,
> boost::intrusive_ptr, ThreadPool::TPHandle&)+0x396)
> [0x562265655cf8]
>   7: (PGQueueable::RunVis::operator()(boost::intrusive_ptr
> const&)+0x5a) [0x5622658c09a6]
>   8: (OSD::ShardedOpWQ::_process(unsigned int,
> ceph::heartbeat_handle_d*)+0x1ab6) [0x562265657918]
>   9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5cd)
> [0x562265c026f5]
>   10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x562265c05e72]
>   11: (()+0x751e) [0x7f1f4fd7f51e]
>   12: (clone()+0x3f) [0x7f1f4ed7ef0f]
>
> A search in the bug tracker revealed that a similar error has been
> resolved for jewel http://tracker.ceph.com/issues/19223, yet I don't
> know if this is in any way relevant.
>
> We are currently at a loss how to get these osds back up. Any
> suggestions how to approach this would be very welcome. If there is any
> further information that is needed or additional context please let me know.
>
> Thanks,
>
> Lawrence
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshooting hanging storage backend whenever there is any cluster change

[Paul Emmerich]

PGs switching to the peering state after a failure is normal and
expected. The important thing is how long they stay in that state; it
shouldn't be longer than a few seconds. It looks like less than 5
seconds from your log.

What might help here is the ceph -w log (or mon cluster log file)
during an outage.

Also, get rid of that min_size = 1 setting, that will bite you in the long run.

Paul
Am Fr., 12. Okt. 2018 um 23:27 Uhr schrieb Stefan Priebe - Profihost
AG :
>
> Hi David,
>
> Am 12.10.2018 um 15:59 schrieb David Turner:
> > The PGs per OSD does not change unless the OSDs are marked out.  You
> > have noout set, so that doesn't change at all during this test.  All of
> > your PGs peered quickly at the beginning and then were active+undersized
> > the rest of the time, you never had any blocked requests, and you always
> > had 100MB/s+ client IO.  I didn't see anything wrong with your cluster
> > to indicate that your clients had any problems whatsoever accessing data.
> >
> > Can you confirm that you saw the same problems while you were running
> > those commands?  The next thing would seem that possibly a client isn't
> > getting an updated OSD map to indicate that the host and its OSDs are
> > down and it's stuck trying to communicate with host7.  That would
> > indicate a potential problem with the client being unable to communicate
> > with the Mons maybe?
> May be but what about this status
> 'PG_AVAILABILITY Reduced data availability: pgs peering'
>
> See the log here: https://pastebin.com/wxUKzhgB
>
> PG_AVAILABILITY is noted at timestamps [2018-10-12 12:16:15.403394] and
> [2018-10-12 12:17:40.072655].
>
> And why does Ceph docs say:
>
> Data availability is reduced, meaning that the cluster is unable to
> service potential read or write requests for some data in the cluster.
> Specifically, one or more PGs is in a state that does not allow IO
> requests to be serviced. Problematic PG states include peering, stale,
> incomplete, and the lack of active (if those conditions do not clear
> quickly).
>
>
> Greets,
> Stefan
> >
> > On Fri, Oct 12, 2018 at 8:35 AM Nils Fahldieck - Profihost AG
> > mailto:n.fahldi...@profihost.ag>> wrote:
> >
> > Hi, in our `ceph.conf` we have:
> >
> >   mon_max_pg_per_osd = 300
> >
> > While the host is offline (9 OSDs down):
> >
> >   4352 PGs * 3 / 62 OSDs ~ 210 PGs per OSD
> >
> > If all OSDs are online:
> >
> >   4352 PGs * 3 / 71 OSDs ~ 183 PGs per OSD
> >
> > ... so this doesn't seem to be the issue.
> >
> > If I understood you right, that's what you've meant. If I got you wrong,
> > would you mind to point to one of those threads you mentioned?
> >
> > Thanks :)
> >
> > Am 12.10.2018 um 14:03 schrieb Burkhard Linke:
> > > Hi,
> > >
> > >
> > > On 10/12/2018 01:55 PM, Nils Fahldieck - Profihost AG wrote:
> > >> I rebooted a Ceph host and logged `ceph status` & `ceph health
> > detail`
> > >> every 5 seconds. During this I encountered 'PG_AVAILABILITY
> > Reduced data
> > >> availability: pgs peering'. At the same time some VMs hung as
> > described
> > >> before.
> > >
> > > Just a wild guess... you have 71 OSDs and about 4500 PG with size=3.
> > > 13500 PG instance overall, resulting in ~190 PGs per OSD under normal
> > > circumstances.
> > >
> > > If one host is down and the PGs have to re-peer, you might reach the
> > > limit of 200 PG/OSDs on some of the OSDs, resulting in stuck peering.
> > >
> > > You can try to raise this limit. There are several threads on the
> > > mailing list about this.
> > >
> > > Regards,
> > > Burkhard
> > >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Troubleshooting hanging storage backend whenever there is any cluster change

[Stefan Priebe - Profihost AG]

Hi David,

Am 12.10.2018 um 15:59 schrieb David Turner:
> The PGs per OSD does not change unless the OSDs are marked out.  You
> have noout set, so that doesn't change at all during this test.  All of
> your PGs peered quickly at the beginning and then were active+undersized
> the rest of the time, you never had any blocked requests, and you always
> had 100MB/s+ client IO.  I didn't see anything wrong with your cluster
> to indicate that your clients had any problems whatsoever accessing data.
> 
> Can you confirm that you saw the same problems while you were running
> those commands?  The next thing would seem that possibly a client isn't
> getting an updated OSD map to indicate that the host and its OSDs are
> down and it's stuck trying to communicate with host7.  That would
> indicate a potential problem with the client being unable to communicate
> with the Mons maybe?
May be but what about this status
'PG_AVAILABILITY Reduced data availability: pgs peering'

See the log here: https://pastebin.com/wxUKzhgB

PG_AVAILABILITY is noted at timestamps [2018-10-12 12:16:15.403394] and
[2018-10-12 12:17:40.072655].

And why does Ceph docs say:

Data availability is reduced, meaning that the cluster is unable to
service potential read or write requests for some data in the cluster.
Specifically, one or more PGs is in a state that does not allow IO
requests to be serviced. Problematic PG states include peering, stale,
incomplete, and the lack of active (if those conditions do not clear
quickly).


Greets,
Stefan
> 
> On Fri, Oct 12, 2018 at 8:35 AM Nils Fahldieck - Profihost AG
> mailto:n.fahldi...@profihost.ag>> wrote:
> 
> Hi, in our `ceph.conf` we have:
> 
>   mon_max_pg_per_osd = 300
> 
> While the host is offline (9 OSDs down):
> 
>   4352 PGs * 3 / 62 OSDs ~ 210 PGs per OSD
> 
> If all OSDs are online:
> 
>   4352 PGs * 3 / 71 OSDs ~ 183 PGs per OSD
> 
> ... so this doesn't seem to be the issue.
> 
> If I understood you right, that's what you've meant. If I got you wrong,
> would you mind to point to one of those threads you mentioned?
> 
> Thanks :)
> 
> Am 12.10.2018 um 14:03 schrieb Burkhard Linke:
> > Hi,
> >
> >
> > On 10/12/2018 01:55 PM, Nils Fahldieck - Profihost AG wrote:
> >> I rebooted a Ceph host and logged `ceph status` & `ceph health
> detail`
> >> every 5 seconds. During this I encountered 'PG_AVAILABILITY
> Reduced data
> >> availability: pgs peering'. At the same time some VMs hung as
> described
> >> before.
> >
> > Just a wild guess... you have 71 OSDs and about 4500 PG with size=3.
> > 13500 PG instance overall, resulting in ~190 PGs per OSD under normal
> > circumstances.
> >
> > If one host is down and the PGs have to re-peer, you might reach the
> > limit of 200 PG/OSDs on some of the OSDs, resulting in stuck peering.
> >
> > You can try to raise this limit. There are several threads on the
> > mailing list about this.
> >
> > Regards,
> > Burkhard
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] list admin issues

[shubjero]

Happens to me too, on gmail. I'm on half a dozen other mailman lists with
no issues at all. I've escalate this problem to the ceph mailing list
maintainer and they said its an issue with their provider, but this was
probably a year ago.

On Tue, Oct 9, 2018 at 7:04 AM Elias Abacioglu <
elias.abacio...@deltaprojects.com> wrote:

> Maybe there are some advice here that can help remedy the situation a bit?
> https://support.google.com/mail/answer/81126?hl=en
> https://support.google.com/mail/answer/6227174?hl=en
>
> /Elias
>
> On Tue, Oct 9, 2018 at 2:24 AM Alex Gorbachev 
> wrote:
>
>> On Mon, Oct 8, 2018 at 7:48 AM Elias Abacioglu
>>  wrote:
>> >
>> > If it's attachments causing this, perhaps forbid attachments? Force
>> people to use pastebin / imgur type of services?
>> >
>> > /E
>> >
>> > On Mon, Oct 8, 2018 at 1:33 PM Martin Palma  wrote:
>> >>
>> >> Same here also on Gmail with G Suite.
>> >> On Mon, Oct 8, 2018 at 12:31 AM Paul Emmerich 
>> wrote:
>> >> >
>> >> > I'm also seeing this once every few months or so on Gmail with G
>> Suite.
>> >> >
>> >> > Paul
>> >> > Am So., 7. Okt. 2018 um 08:18 Uhr schrieb Joshua Chen
>> >> > :
>> >> > >
>> >> > > I also got removed once, got another warning once (need to
>> re-enable).
>> >> > >
>> >> > > Cheers
>> >> > > Joshua
>> >> > >
>> >> > >
>> >> > > On Sun, Oct 7, 2018 at 5:38 AM Svante Karlsson <
>> svante.karls...@csi.se> wrote:
>> >> > >>
>> >> > >> I'm also getting removed but not only from ceph. I subscribe
>> d...@kafka.apache.org list and the same thing happens there.
>> >> > >>
>> >> > >> Den lör 6 okt. 2018 kl 23:24 skrev Jeff Smith <
>> j...@unbiasedgeek.com>:
>> >> > >>>
>> >> > >>> I have been removed twice.
>> >> > >>> On Sat, Oct 6, 2018 at 7:07 AM Elias Abacioglu
>> >> > >>>  wrote:
>> >> > >>> >
>> >> > >>> > Hi,
>> >> > >>> >
>> >> > >>> > I'm bumping this old thread cause it's getting annoying. My
>> membership get disabled twice a month.
>> >> > >>> > Between my two Gmail accounts I'm in more than 25 mailing
>> lists and I see this behavior only here. Why is only ceph-users only
>> affected? Maybe Christian was on to something, is this intentional?
>> >> > >>> > Reality is that there is a lot of ceph-users with Gmail
>> accounts, perhaps it wouldn't be so bad to actually trying to figure this
>> one out?
>> >> > >>> >
>> >> > >>> > So can the maintainers of this list please investigate what
>> actually gets bounced? Look at my address if you want.
>> >> > >>> > I got disabled 20181006, 20180927, 20180916, 20180725,
>> 20180718 most recently.
>> >> > >>> > Please help!
>> >> > >>> >
>> >> > >>> > Thanks,
>> >> > >>> > Elias
>> >> > >>> >
>> >> > >>> > On Mon, Oct 16, 2017 at 5:41 AM Christian Balzer <
>> ch...@gol.com> wrote:
>> >> > >>> >>
>> >> > >>> >>
>> >> > >>> >> Most mails to this ML score low or negatively with
>> SpamAssassin, however
>> >> > >>> >> once in a while (this is a recent one) we get relatively high
>> scores.
>> >> > >>> >> Note that the forged bits are false positives, but the SA is
>> up to date and
>> >> > >>> >> google will have similar checks:
>> >> > >>> >> ---
>> >> > >>> >> X-Spam-Status: No, score=3.9 required=10.0
>> tests=BAYES_00,DCC_CHECK,
>> >> > >>> >>
>> FORGED_MUA_MOZILLA,FORGED_YAHOO_RCVD,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,
>> >> > >>> >>
>> HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MIME_HTML_MOSTLY,RCVD_IN_MSPIKE_H4,
>> >> > >>> >>  RCVD_IN_MSPIKE_WL,RDNS_NONE,T_DKIM_INVALID shortcircuit=no
>> autolearn=no
>> >> > >>> >> ---
>> >> > >>> >>
>> >> > >>> >> Between attachment mails and some of these and you're well on
>> your way out.
>> >> > >>> >>
>> >> > >>> >> The default mailman settings and logic require 5 bounces to
>> trigger
>> >> > >>> >> unsubscription and 7 days of NO bounces to reset the counter.
>> >> > >>> >>
>> >> > >>> >> Christian
>> >> > >>> >>
>> >> > >>> >> On Mon, 16 Oct 2017 12:23:25 +0900 Christian Balzer wrote:
>> >> > >>> >>
>> >> > >>> >> > On Mon, 16 Oct 2017 14:15:22 +1100 Blair Bethwaite wrote:
>> >> > >>> >> >
>> >> > >>> >> > > Thanks Christian,
>> >> > >>> >> > >
>> >> > >>> >> > > You're no doubt on the right track, but I'd really like
>> to figure out
>> >> > >>> >> > > what it is at my end - I'm unlikely to be the only person
>> subscribed
>> >> > >>> >> > > to ceph-users via a gmail account.
>> >> > >>> >> > >
>> >> > >>> >> > > Re. attachments, I'm surprised mailman would be allowing
>> them in the
>> >> > >>> >> > > first place, and even so gmail's attachment requirements
>> are less
>> >> > >>> >> > > strict than most corporate email setups (those that don't
>> already use
>> >> > >>> >> > > a cloud provider).
>> >> > >>> >> > >
>> >> > >>> >> > Mailman doesn't do anything with this by default AFAIK, but
>> see below.
>> >> > >>> >> > Strict is fine if you're in control, corporate mail can be
>> hell, doubly so
>> >> > >>> >> > if on M$ cloud.
>> >> > >>> >> >
>> >> > >>> >> > > This started happening earlier in the year after I turned
>> off digest
>> >> >

Re: [ceph-users] add existing rbd to new tcmu iscsi gateways

[Jason Dillaman]

On Wed, Oct 10, 2018 at 5:42 PM Brady Deetz  wrote:
>
> Looks like that may have recently been broken.
>
> Unfortunately no real logs of use in rbd-target-api.log or rbd-target-gw.log. 
> Is there an increased log level I can enable for whatever web-service is 
> handling this?
>
> [root@dc1srviscsi01 ~]# rbd -p vmware_ssd_metadata --data-pool vmware_ssd 
> --size 2T create ssd_test_0
>
> [root@dc1srviscsi01 ~]# rbd -p vmware_ssd_metadata info ssd_test_0
> rbd image 'ssd_test_0':
> size 2 TiB in 524288 objects
> order 22 (4 MiB objects)
> id: b4343f6b8b4567
> data_pool: vmware_ssd
> block_name_prefix: rbd_data.56.b4343f6b8b4567
> format: 2
> features: layering, exclusive-lock, object-map, fast-diff, 
> deep-flatten, data-pool
> op_features:
> flags:
> create_timestamp: Wed Oct 10 16:36:18 2018
>
> [root@dc1srviscsi01 ~]# gwcli
> /> cd disks/
> /disks> create pool=vmware_ssd_metadata image=vmware_ssd_metadata.ssd_test_0 
> size=2T
> Failed : 500 INTERNAL SERVER ERROR

You have the pool name specified twice -- you shouldn't pass the pool
name to the "image=" parameter. Also, I think you might need to rename
the image since the ceph-iscsi-config package will restrict the image
name to alpha-numeric + hyphens (no underscores apparently). I tried
running it on the latest master build of gwcli and it worked fine.

> /disks>
>
>
> [root@dc1srviscsi02 ~]# rpm -qa |egrep "ceph|iscsi|tcmu|rst|kernel-ml"
> ceph-mds-13.2.2-0.el7.x86_64
> ceph-mgr-13.2.2-0.el7.x86_64
> ceph-release-1-1.el7.noarch
> tcmu-runner-1.4.0-1.el7.x86_64
> ceph-iscsi-cli-2.7-54.g9b18a3b.el7.noarch
> ceph-common-13.2.2-0.el7.x86_64
> ceph-mon-13.2.2-0.el7.x86_64
> ceph-13.2.2-0.el7.x86_64
> tcmu-runner-debuginfo-1.4.0-1.el7.x86_64
> ceph-iscsi-config-2.6-42.gccca57d.el7.noarch
> libcephfs2-13.2.2-0.el7.x86_64
> ceph-base-13.2.2-0.el7.x86_64
> ceph-osd-13.2.2-0.el7.x86_64
> ceph-radosgw-13.2.2-0.el7.x86_64
> kernel-ml-4.18.12-1.el7.elrepo.x86_64
> python-cephfs-13.2.2-0.el7.x86_64
> ceph-selinux-13.2.2-0.el7.x86_64
>
>
>
> On Tue, Oct 9, 2018 at 3:51 PM Jason Dillaman  wrote:
>>
>> On Tue, Oct 9, 2018 at 3:14 PM Brady Deetz  wrote:
>> >
>> > I am attempting to migrate to the new tcmu iscsi gateway. Is there a way 
>> > to configure gwcli to export an rbd that was created outside gwcli?
>>
>> You should be able to just run "/disks create .
>> " from within "gwcli" to have it add an existing image.
>>
>> > This is necessary for me because I have a lun exported from an old LIO 
>> > gateway to a Windows host that I need to transition to the new tcmu based 
>> > cluster.
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> Jason



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph dashboard ac-* commands not working (Mimic)

[Hayashida, Mami]

I set up a new Mimic cluster recently and have just enabled the Dashboard.
I first tried to add a (Dashboard) user with the "ac-user-create" command
following this version of documentation (
http://docs.ceph.com/docs/master/mgr/dashboard/), but the command did not
work.  Following the   /mimic/mgr/dashboard/ version, I used the
"set-login-credentials" command, I was able to create a user with a
password, which was successful.  But with none of the ac-* command working,
how can we manage the dashboard user accounts?  At this point, I cannot
figure out what level of permissions have been given to the (test)
dashboard user I have just created.  Neither have I figured out how to
delete a user or obtain a list of dashboard users created so far.

I am using Ceph version 13.2.2 and  all the ac-* commands I have tried
returns exactly the same message.

mon0:~$ ceph dashboard ac-user-show  test-user
no valid command found; 10 closest matches:
dashboard get-rgw-api-user-id
dashboard get-rest-requests-timeout
dashboard set-rgw-api-host 
dashboard set-rgw-api-secret-key 
dashboard get-rgw-api-access-key
dashboard set-rest-requests-timeout 
dashboard get-rgw-api-scheme
dashboard get-rgw-api-host
dashboard set-login-credentials  
dashboard set-session-expire 
Error EINVAL: invalid command


-- 
-

*Mami Hayashida*

*Research Computing Associate*
Research Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayash...@uky.edu
(859)323-7521
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] bluestore compression enabled but no data compressed

[Frank Schilder]

Hi David,

thanks, now I see what you mean. If you are right, that would mean that the 
documentation is wrong. Under 
"http://docs.ceph.com/docs/master/rados/operations/pools/#set-pool-values"; is 
stated that "Sets inline compression algorithm to use for underlying BlueStore. 
This setting overrides the global setting of bluestore compression algorithm". 
In other words, the global setting should be irrelevant if compression is 
enabled on a pool.

Well, I will try how setting both to "aggressive" or "force" works out and let 
you know.

Thanks and have a nice weekend,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: David Turner 
Sent: 12 October 2018 16:50:31
To: Frank Schilder
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] bluestore compression enabled but no data compressed

If you go down just a little farther you'll see the settings that you put into 
your ceph.conf under the osd section (although I'd probably do global).  That's 
where the OSDs get the settings from.  As a note, once these are set, future 
writes will be compressed (if they match the compression settings which you can 
see there about minimum ratios, blob sizes, etc).  To compress current data, 
you need to re-write it.

On Fri, Oct 12, 2018 at 10:41 AM Frank Schilder 
mailto:fr...@dtu.dk>> wrote:
Hi David,

thanks for your quick answer. When I look at both references, I see exactly the 
same commands:

ceph osd pool set {pool-name} {key} {value}

where on one page only keys specific for compression are described. This is the 
command I found and used. However, I can't see any compression happening. If 
you know about something else than "ceph osd pool set" - commands, please let 
me know.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: David Turner mailto:drakonst...@gmail.com>>
Sent: 12 October 2018 15:47:20
To: Frank Schilder
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] bluestore compression enabled but no data compressed

It's all of the settings that you found in your first email when you dumped the 
configurations and such.  
http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression

On Fri, Oct 12, 2018 at 7:36 AM Frank Schilder 
mailto:fr...@dtu.dk>>> 
wrote:
Hi David,

thanks for your answer. I did enable compression on the pools as described in 
the link you sent below (ceph osd pool set sr-fs-data-test compression_mode 
aggressive, I also tried force to no avail). However, I could not find anything 
on enabling compression per OSD. Could you possibly provide a source or sample 
commands?

Thanks and best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: David Turner 
mailto:drakonst...@gmail.com>>>
Sent: 09 October 2018 17:42
To: Frank Schilder
Cc: 
ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] bluestore compression enabled but no data compressed

When I've tested compression before there are 2 places you need to configure 
compression.  On the OSDs in the configuration settings that you mentioned, but 
also on the [1] pools themselves.  If you have the compression mode on the 
pools set to none, then it doesn't matter what the OSDs configuration is and 
vice versa unless you are using the setting of force.  If you want to default 
compress everything, set pools to passive and osds to aggressive.  If you want 
to only compress specific pools, set the osds to passive and the specific pools 
to aggressive.  Good luck.


[1] http://docs.ceph.com/docs/mimic/rados/operations/pools/#set-pool-values

On Tue, Sep 18, 2018 at 7:11 AM Frank Schilder 
mailto:fr...@dtu.dk>>

Re: [ceph-users] Anyone tested Samsung 860 DCT SSDs?

[Kenneth Van Alstyne]

Thanks for the feedback everyone.  Based on the TBW figures, it sounds like 
these drives are terrible for us as the idea is to NOT use them simply for 
archive.  This will be a high read/write workload, so totally a show stopper.  
I’m interested in the Seagate Nytro myself.

Thanks,

--
Kenneth Van Alstyne
Systems Architect
Knight Point Systems, LLC
Service-Disabled Veteran-Owned Business
1775 Wiehle Avenue Suite 101 | Reston, VA 20190
c: 228-547-8045 f: 571-266-3106
www.knightpoint.com 
DHS EAGLE II Prime Contractor: FC1 SDVOSB Track
GSA Schedule 70 SDVOSB: GS-35F-0646S
GSA MOBIS Schedule: GS-10F-0404Y
ISO 2 / ISO 27001 / CMMI Level 3

Notice: This e-mail message, including any attachments, is for the sole use of 
the intended recipient(s) and may contain confidential and privileged 
information. Any unauthorized review, copy, use, disclosure, or distribution is 
STRICTLY prohibited. If you are not the intended recipient, please contact the 
sender by reply e-mail and destroy all copies of the original message.

> On Oct 12, 2018, at 9:31 AM, Corin Langosch  
> wrote:
> 
> Hi
> 
> It has only TBW of 349 TB, so might die quite soon. But what about the
> "Seagate Nytro 1551 DuraWrite 3DWPD Mainstream Endurance 960GB, SATA"?
> Seems really cheap too and has TBW 5.25PB. Anybody tested that? What
> about (RBD) performance? 
> 
> Cheers
> Corin
> 
> On Fri, 2018-10-12 at 13:53 +, Kenneth Van Alstyne wrote:
>> Cephers:
>>  As the subject suggests, has anyone tested Samsung 860 DCT
>> SSDs?  They are really inexpensive and we are considering buying some
>> to test.
>> 
>> Thanks,
>> 
>> --
>> Kenneth Van Alstyne
>> Systems Architect
>> Knight Point Systems, LLC
>> Service-Disabled Veteran-Owned Business
>> 1775 Wiehle Avenue Suite 101 | Reston, VA 20190
>> c: 228-547-8045 f: 571-266-3106
>> www.knightpoint.com 
>> DHS EAGLE II Prime Contractor: FC1 SDVOSB Track
>> GSA Schedule 70 SDVOSB: GS-35F-0646S
>> GSA MOBIS Schedule: GS-10F-0404Y
>> ISO 2 / ISO 27001 / CMMI Level 3
>> 
>> Notice: This e-mail message, including any attachments, is for the
>> sole use of the intended recipient(s) and may contain confidential
>> and privileged information. Any unauthorized review, copy, use,
>> disclosure, or distribution is STRICTLY prohibited. If you are not
>> the intended recipient, please contact the sender by reply e-mail and
>> destroy all copies of the original message.
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] bluestore compression enabled but no data compressed

[David Turner]

If you go down just a little farther you'll see the settings that you put
into your ceph.conf under the osd section (although I'd probably do
global).  That's where the OSDs get the settings from.  As a note, once
these are set, future writes will be compressed (if they match the
compression settings which you can see there about minimum ratios, blob
sizes, etc).  To compress current data, you need to re-write it.

On Fri, Oct 12, 2018 at 10:41 AM Frank Schilder  wrote:

> Hi David,
>
> thanks for your quick answer. When I look at both references, I see
> exactly the same commands:
>
> ceph osd pool set {pool-name} {key} {value}
>
> where on one page only keys specific for compression are described. This
> is the command I found and used. However, I can't see any compression
> happening. If you know about something else than "ceph osd pool set" -
> commands, please let me know.
>
> Best regards,
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: David Turner 
> Sent: 12 October 2018 15:47:20
> To: Frank Schilder
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] bluestore compression enabled but no data
> compressed
>
> It's all of the settings that you found in your first email when you
> dumped the configurations and such.
> http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression
>
> On Fri, Oct 12, 2018 at 7:36 AM Frank Schilder  fr...@dtu.dk>> wrote:
> Hi David,
>
> thanks for your answer. I did enable compression on the pools as described
> in the link you sent below (ceph osd pool set sr-fs-data-test
> compression_mode aggressive, I also tried force to no avail). However, I
> could not find anything on enabling compression per OSD. Could you possibly
> provide a source or sample commands?
>
> Thanks and best regards,
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: David Turner mailto:drakonst...@gmail.com>>
> Sent: 09 October 2018 17:42
> To: Frank Schilder
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] bluestore compression enabled but no data
> compressed
>
> When I've tested compression before there are 2 places you need to
> configure compression.  On the OSDs in the configuration settings that you
> mentioned, but also on the [1] pools themselves.  If you have the
> compression mode on the pools set to none, then it doesn't matter what the
> OSDs configuration is and vice versa unless you are using the setting of
> force.  If you want to default compress everything, set pools to passive
> and osds to aggressive.  If you want to only compress specific pools, set
> the osds to passive and the specific pools to aggressive.  Good luck.
>
>
> [1]
> http://docs.ceph.com/docs/mimic/rados/operations/pools/#set-pool-values
>
> On Tue, Sep 18, 2018 at 7:11 AM Frank Schilder  fr...@dtu.dk>>> wrote:
> I seem to have a problem getting bluestore compression to do anything. I
> followed the documentation and enabled bluestore compression on various
> pools by executing "ceph osd pool set  compression_mode
> aggressive". Unfortunately, it seems like no data is compressed at all. As
> an example, below is some diagnostic output for a data pool used by a
> cephfs:
>
> [root@ceph-01 ~]# ceph --version
> ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous
> (stable)
>
> All defaults are OK:
>
> [root@ceph-01 ~]# ceph --show-config | grep compression
> [...]
> bluestore_compression_algorithm = snappy
> bluestore_compression_max_blob_size = 0
> bluestore_compression_max_blob_size_hdd = 524288
> bluestore_compression_max_blob_size_ssd = 65536
> bluestore_compression_min_blob_size = 0
> bluestore_compression_min_blob_size_hdd = 131072
> bluestore_compression_min_blob_size_ssd = 8192
> bluestore_compression_mode = none
> bluestore_compression_required_ratio = 0.875000
> [...]
>
> Compression is reported as enabled:
>
> [root@ceph-01 ~]# ceph osd pool ls detail
> [...]
> pool 24 'sr-fs-data-test' erasure size 8 min_size 7 crush_rule 10
> object_hash rjenkins pg_num 50 pgp_num 50 last_change 7726 flags
> hashpspool,ec_overwrites stripe_width 24576 compression_algorithm snappy
> compression_mode aggressive application cephfs
> [...]
>
> [root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_mode
> compression_mode: aggressive
> [root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_algorithm
> compression_algorithm: snappy
>
> We dumped a 4Gib file with dd from /dev/zero. Should be easy to compress
> with excellent ratio. Search for a PG:
>
> [root@ceph-01 ~]# ceph pg ls-by-pool sr-fs-data-test
> PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
>  LOG DISK_LOG STATESTATE_STAMPVERSION  REPORTED UP
>  UP_PRIMARY ACTING   

Re: [ceph-users] bluestore compression enabled but no data compressed

[Frank Schilder]

Hi David,

thanks for your quick answer. When I look at both references, I see exactly the 
same commands:

ceph osd pool set {pool-name} {key} {value}

where on one page only keys specific for compression are described. This is the 
command I found and used. However, I can't see any compression happening. If 
you know about something else than "ceph osd pool set" - commands, please let 
me know.

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: David Turner 
Sent: 12 October 2018 15:47:20
To: Frank Schilder
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] bluestore compression enabled but no data compressed

It's all of the settings that you found in your first email when you dumped the 
configurations and such.  
http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression

On Fri, Oct 12, 2018 at 7:36 AM Frank Schilder 
mailto:fr...@dtu.dk>> wrote:
Hi David,

thanks for your answer. I did enable compression on the pools as described in 
the link you sent below (ceph osd pool set sr-fs-data-test compression_mode 
aggressive, I also tried force to no avail). However, I could not find anything 
on enabling compression per OSD. Could you possibly provide a source or sample 
commands?

Thanks and best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: David Turner mailto:drakonst...@gmail.com>>
Sent: 09 October 2018 17:42
To: Frank Schilder
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] bluestore compression enabled but no data compressed

When I've tested compression before there are 2 places you need to configure 
compression.  On the OSDs in the configuration settings that you mentioned, but 
also on the [1] pools themselves.  If you have the compression mode on the 
pools set to none, then it doesn't matter what the OSDs configuration is and 
vice versa unless you are using the setting of force.  If you want to default 
compress everything, set pools to passive and osds to aggressive.  If you want 
to only compress specific pools, set the osds to passive and the specific pools 
to aggressive.  Good luck.


[1] http://docs.ceph.com/docs/mimic/rados/operations/pools/#set-pool-values

On Tue, Sep 18, 2018 at 7:11 AM Frank Schilder 
mailto:fr...@dtu.dk>>> 
wrote:
I seem to have a problem getting bluestore compression to do anything. I 
followed the documentation and enabled bluestore compression on various pools 
by executing "ceph osd pool set  compression_mode aggressive". 
Unfortunately, it seems like no data is compressed at all. As an example, below 
is some diagnostic output for a data pool used by a cephfs:

[root@ceph-01 ~]# ceph --version
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)

All defaults are OK:

[root@ceph-01 ~]# ceph --show-config | grep compression
[...]
bluestore_compression_algorithm = snappy
bluestore_compression_max_blob_size = 0
bluestore_compression_max_blob_size_hdd = 524288
bluestore_compression_max_blob_size_ssd = 65536
bluestore_compression_min_blob_size = 0
bluestore_compression_min_blob_size_hdd = 131072
bluestore_compression_min_blob_size_ssd = 8192
bluestore_compression_mode = none
bluestore_compression_required_ratio = 0.875000
[...]

Compression is reported as enabled:

[root@ceph-01 ~]# ceph osd pool ls detail
[...]
pool 24 'sr-fs-data-test' erasure size 8 min_size 7 crush_rule 10 object_hash 
rjenkins pg_num 50 pgp_num 50 last_change 7726 flags hashpspool,ec_overwrites 
stripe_width 24576 compression_algorithm snappy compression_mode aggressive 
application cephfs
[...]

[root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_mode
compression_mode: aggressive
[root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_algorithm
compression_algorithm: snappy

We dumped a 4Gib file with dd from /dev/zero. Should be easy to compress with 
excellent ratio. Search for a PG:

[root@ceph-01 ~]# ceph pg ls-by-pool sr-fs-data-test
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG 
DISK_LOG STATESTATE_STAMPVERSION  REPORTED UP   
UP_PRIMARY ACTING   ACTING_PRIMARY LAST_SCRUB 
SCRUB_STAMPLAST_DEEP_SCRUB DEEP_SCRUB_STAMP
24.0 15  00 0   0  62914560  77 
  77 active+clean 2018-09-14 01:07:14.593007  7698'77 7735:142 
[53,47,36,30,14,55,57,5] 53 [53,47,36,30,14,55,57,5] 53
7698'77 2018-09-14 01:07:14.592966 0'0 2018-09-11 08:06:29.309010

There is about 250MB data on the primary OSD, but noting seems to be compressed:

[root@ceph-07 ~]# ceph daemon osd.53 perf dump | grep blue
[...]
"bluestore_allocated": 313917440,
"bluestore_stored": 264362803,
"blu

Re: [ceph-users] Anyone tested Samsung 860 DCT SSDs?

[Corin Langosch]

Hi

It has only TBW of 349 TB, so might die quite soon. But what about the
"Seagate Nytro 1551 DuraWrite 3DWPD Mainstream Endurance 960GB, SATA"?
Seems really cheap too and has TBW 5.25PB. Anybody tested that? What
about (RBD) performance? 

Cheers
Corin

On Fri, 2018-10-12 at 13:53 +, Kenneth Van Alstyne wrote:
> Cephers:
>   As the subject suggests, has anyone tested Samsung 860 DCT
> SSDs?  They are really inexpensive and we are considering buying some
> to test.
> 
> Thanks,
> 
> --
> Kenneth Van Alstyne
> Systems Architect
> Knight Point Systems, LLC
> Service-Disabled Veteran-Owned Business
> 1775 Wiehle Avenue Suite 101 | Reston, VA 20190
> c: 228-547-8045 f: 571-266-3106
> www.knightpoint.com 
> DHS EAGLE II Prime Contractor: FC1 SDVOSB Track
> GSA Schedule 70 SDVOSB: GS-35F-0646S
> GSA MOBIS Schedule: GS-10F-0404Y
> ISO 2 / ISO 27001 / CMMI Level 3
> 
> Notice: This e-mail message, including any attachments, is for the
> sole use of the intended recipient(s) and may contain confidential
> and privileged information. Any unauthorized review, copy, use,
> disclosure, or distribution is STRICTLY prohibited. If you are not
> the intended recipient, please contact the sender by reply e-mail and
> destroy all copies of the original message.
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Anyone tested Samsung 860 DCT SSDs?

[Alexandre DERUMIER]

don't have tested them, but be careful of dwpd

0.2 DWPD

:/

- Mail original -
De: "Kenneth Van Alstyne" 
À: "ceph-users" 
Envoyé: Vendredi 12 Octobre 2018 15:53:43
Objet: [ceph-users] Anyone tested Samsung 860 DCT SSDs?

Cephers: 
As the subject suggests, has anyone tested Samsung 860 DCT SSDs? They are 
really inexpensive and we are considering buying some to test. 

Thanks, 

-- 
Kenneth Van Alstyne 
Systems Architect 
Knight Point Systems, LLC 
Service-Disabled Veteran-Owned Business 
1775 Wiehle Avenue Suite 101 | Reston, VA 20190 
c: 228-547-8045 f: 571-266-3106 
www.knightpoint.com 
DHS EAGLE II Prime Contractor: FC1 SDVOSB Track 
GSA Schedule 70 SDVOSB: GS-35F-0646S 
GSA MOBIS Schedule: GS-10F-0404Y 
ISO 2 / ISO 27001 / CMMI Level 3 

Notice: This e-mail message, including any attachments, is for the sole use of 
the intended recipient(s) and may contain confidential and privileged 
information. Any unauthorized review, copy, use, disclosure, or distribution is 
STRICTLY prohibited. If you are not the intended recipient, please contact the 
sender by reply e-mail and destroy all copies of the original message. 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Anyone tested Samsung 860 DCT SSDs?

[David Turner]

What do you want to use these for?  "5 Year or 0.2 DWPD" is the durability
of this drive which is absolutely awful for most every use in Ceph.
Possibly if you're using these for data disks (not DB or WAL) and you plan
to have a more durable media to host the DB+WAL on... this could work.  Or
if you're just doing archival storage... but then you should be using much
cheaper spinners.  Back in the days of Filestore and SSD Journals I had
some disks that had 0.3 DWPD and I had to replace all of the disks in under
a year because they ran out of writes.

On Fri, Oct 12, 2018 at 9:55 AM Kenneth Van Alstyne <
kvanalst...@knightpoint.com> wrote:

> Cephers:
> As the subject suggests, has anyone tested Samsung 860 DCT SSDs?
> They are really inexpensive and we are considering buying some to test.
>
> Thanks,
>
> --
> Kenneth Van Alstyne
> Systems Architect
> Knight Point Systems, LLC
> Service-Disabled Veteran-Owned Business
> 1775 Wiehle Avenue Suite 101 | Reston, VA 20190
> 
> c: 228-547-8045 <(228)%20547-8045> f: 571-266-3106 <(571)%20266-3106>
> www.knightpoint.com
> DHS EAGLE II Prime Contractor: FC1 SDVOSB Track
> GSA Schedule 70 SDVOSB: GS-35F-0646S
> GSA MOBIS Schedule: GS-10F-0404Y
> ISO 2 / ISO 27001 / CMMI Level 3
>
> Notice: This e-mail message, including any attachments, is for the sole
> use of the intended recipient(s) and may contain confidential and
> privileged information. Any unauthorized review, copy, use, disclosure, or
> distribution is STRICTLY prohibited. If you are not the intended recipient,
> please contact the sender by reply e-mail and destroy all copies of the
> original message.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshooting hanging storage backend whenever there is any cluster change

[David Turner]

The PGs per OSD does not change unless the OSDs are marked out.  You have
noout set, so that doesn't change at all during this test.  All of your PGs
peered quickly at the beginning and then were active+undersized the rest of
the time, you never had any blocked requests, and you always had 100MB/s+
client IO.  I didn't see anything wrong with your cluster to indicate that
your clients had any problems whatsoever accessing data.

Can you confirm that you saw the same problems while you were running those
commands?  The next thing would seem that possibly a client isn't getting
an updated OSD map to indicate that the host and its OSDs are down and it's
stuck trying to communicate with host7.  That would indicate a potential
problem with the client being unable to communicate with the Mons maybe?
Have you completely ruled out any network problems between all nodes and
all of the IPs in the cluster.  What does your client log show during these
times?

On Fri, Oct 12, 2018 at 8:35 AM Nils Fahldieck - Profihost AG <
n.fahldi...@profihost.ag> wrote:

> Hi, in our `ceph.conf` we have:
>
>   mon_max_pg_per_osd = 300
>
> While the host is offline (9 OSDs down):
>
>   4352 PGs * 3 / 62 OSDs ~ 210 PGs per OSD
>
> If all OSDs are online:
>
>   4352 PGs * 3 / 71 OSDs ~ 183 PGs per OSD
>
> ... so this doesn't seem to be the issue.
>
> If I understood you right, that's what you've meant. If I got you wrong,
> would you mind to point to one of those threads you mentioned?
>
> Thanks :)
>
> Am 12.10.2018 um 14:03 schrieb Burkhard Linke:
> > Hi,
> >
> >
> > On 10/12/2018 01:55 PM, Nils Fahldieck - Profihost AG wrote:
> >> I rebooted a Ceph host and logged `ceph status` & `ceph health detail`
> >> every 5 seconds. During this I encountered 'PG_AVAILABILITY Reduced data
> >> availability: pgs peering'. At the same time some VMs hung as described
> >> before.
> >
> > Just a wild guess... you have 71 OSDs and about 4500 PG with size=3.
> > 13500 PG instance overall, resulting in ~190 PGs per OSD under normal
> > circumstances.
> >
> > If one host is down and the PGs have to re-peer, you might reach the
> > limit of 200 PG/OSDs on some of the OSDs, resulting in stuck peering.
> >
> > You can try to raise this limit. There are several threads on the
> > mailing list about this.
> >
> > Regards,
> > Burkhard
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Anyone tested Samsung 860 DCT SSDs?

[Kenneth Van Alstyne]

Cephers:
As the subject suggests, has anyone tested Samsung 860 DCT SSDs?  They 
are really inexpensive and we are considering buying some to test.

Thanks,

--
Kenneth Van Alstyne
Systems Architect
Knight Point Systems, LLC
Service-Disabled Veteran-Owned Business
1775 Wiehle Avenue Suite 101 | Reston, VA 20190
c: 228-547-8045 f: 571-266-3106
www.knightpoint.com 
DHS EAGLE II Prime Contractor: FC1 SDVOSB Track
GSA Schedule 70 SDVOSB: GS-35F-0646S
GSA MOBIS Schedule: GS-10F-0404Y
ISO 2 / ISO 27001 / CMMI Level 3

Notice: This e-mail message, including any attachments, is for the sole use of 
the intended recipient(s) and may contain confidential and privileged 
information. Any unauthorized review, copy, use, disclosure, or distribution is 
STRICTLY prohibited. If you are not the intended recipient, please contact the 
sender by reply e-mail and destroy all copies of the original message.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] bluestore compression enabled but no data compressed

[David Turner]

It's all of the settings that you found in your first email when you dumped
the configurations and such.
http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#inline-compression

On Fri, Oct 12, 2018 at 7:36 AM Frank Schilder  wrote:

> Hi David,
>
> thanks for your answer. I did enable compression on the pools as described
> in the link you sent below (ceph osd pool set sr-fs-data-test
> compression_mode aggressive, I also tried force to no avail). However, I
> could not find anything on enabling compression per OSD. Could you possibly
> provide a source or sample commands?
>
> Thanks and best regards,
>
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: David Turner 
> Sent: 09 October 2018 17:42
> To: Frank Schilder
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] bluestore compression enabled but no data
> compressed
>
> When I've tested compression before there are 2 places you need to
> configure compression.  On the OSDs in the configuration settings that you
> mentioned, but also on the [1] pools themselves.  If you have the
> compression mode on the pools set to none, then it doesn't matter what the
> OSDs configuration is and vice versa unless you are using the setting of
> force.  If you want to default compress everything, set pools to passive
> and osds to aggressive.  If you want to only compress specific pools, set
> the osds to passive and the specific pools to aggressive.  Good luck.
>
>
> [1]
> http://docs.ceph.com/docs/mimic/rados/operations/pools/#set-pool-values
>
> On Tue, Sep 18, 2018 at 7:11 AM Frank Schilder  fr...@dtu.dk>> wrote:
> I seem to have a problem getting bluestore compression to do anything. I
> followed the documentation and enabled bluestore compression on various
> pools by executing "ceph osd pool set  compression_mode
> aggressive". Unfortunately, it seems like no data is compressed at all. As
> an example, below is some diagnostic output for a data pool used by a
> cephfs:
>
> [root@ceph-01 ~]# ceph --version
> ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous
> (stable)
>
> All defaults are OK:
>
> [root@ceph-01 ~]# ceph --show-config | grep compression
> [...]
> bluestore_compression_algorithm = snappy
> bluestore_compression_max_blob_size = 0
> bluestore_compression_max_blob_size_hdd = 524288
> bluestore_compression_max_blob_size_ssd = 65536
> bluestore_compression_min_blob_size = 0
> bluestore_compression_min_blob_size_hdd = 131072
> bluestore_compression_min_blob_size_ssd = 8192
> bluestore_compression_mode = none
> bluestore_compression_required_ratio = 0.875000
> [...]
>
> Compression is reported as enabled:
>
> [root@ceph-01 ~]# ceph osd pool ls detail
> [...]
> pool 24 'sr-fs-data-test' erasure size 8 min_size 7 crush_rule 10
> object_hash rjenkins pg_num 50 pgp_num 50 last_change 7726 flags
> hashpspool,ec_overwrites stripe_width 24576 compression_algorithm snappy
> compression_mode aggressive application cephfs
> [...]
>
> [root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_mode
> compression_mode: aggressive
> [root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_algorithm
> compression_algorithm: snappy
>
> We dumped a 4Gib file with dd from /dev/zero. Should be easy to compress
> with excellent ratio. Search for a PG:
>
> [root@ceph-01 ~]# ceph pg ls-by-pool sr-fs-data-test
> PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES
>  LOG DISK_LOG STATESTATE_STAMPVERSION  REPORTED UP
>  UP_PRIMARY ACTING   ACTING_PRIMARY
> LAST_SCRUB SCRUB_STAMPLAST_DEEP_SCRUB DEEP_SCRUB_STAMP
> 24.0 15  00 0   0  62914560
> 77   77 active+clean 2018-09-14 01:07:14.593007  7698'77 7735:142
> [53,47,36,30,14,55,57,5] 53 [53,47,36,30,14,55,57,5]
>  537698'77 2018-09-14 01:07:14.592966 0'0 2018-09-11
> 08:06:29.309010
>
> There is about 250MB data on the primary OSD, but noting seems to be
> compressed:
>
> [root@ceph-07 ~]# ceph daemon osd.53 perf dump | grep blue
> [...]
> "bluestore_allocated": 313917440,
> "bluestore_stored": 264362803,
> "bluestore_compressed": 0,
> "bluestore_compressed_allocated": 0,
> "bluestore_compressed_original": 0,
> [...]
>
> Just to make sure, I checked one of the objects' contents:
>
> [root@ceph-01 ~]# rados ls -p sr-fs-data-test
> 104.039c
> [...]
> 104.039f
>
> It is 4M chunks ...
> [root@ceph-01 ~]# rados -p sr-fs-data-test stat 104.039f
> sr-fs-data-test/104.039f mtime 2018-09-11 14:39:38.00,
> size 4194304
>
> ... with all zeros:
>
> [root@ceph-01 ~]# rados -p sr-fs-data-test get 104.039f obj
>
> [root@ceph-01 ~]# hexdump -C obj
>   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> ||
> *
> 004000

[ceph-users] OSDs crash after deleting unfound object in Luminous 12.2.8

[Lawrence Smith]

Hi all,

we are running a luminous 12.2.8 cluster with a 3 fold replicated cache 
pool with a min_size of 2. We recently encountered an "object unfound" 
error in one of our pgs in this pool. After marking this object lost, 
two of the acting osds crashed and were unable to start up again, with 
only the primary osd staying up. Hoping the cluster might remap the 
copies of this pg, we marked the two crashed osds as out. Now the 
primary osd of this pg has also gone down leaving again only one active 
osd with the cluster reporting a degraded filesystem. All the affected 
osds are running filestore, while about half the cluster has already 
been upgraded to run bluestore osds.


All three of the crashed osds fail to restart, reporting the following 
error during startup:


Oct 12 13:19:12 kaa-109 ceph-osd[166266]:  0> 2018-10-12 
13:19:12.782652 7f1f2d79b700 -1 
/var/tmp/portage/sys-cluster/ceph-12.2.8/work/ceph-12.2.8/src/osd/PrimaryLogPG.cc: 
In function '
void PrimaryLogPG::hit_set_trim(PrimaryLogPG::OpContextUPtr&, unsigned 
int)' thread 7f1f2d79b700 time 2018-10-12 13:19:12.779813
/var/tmp/portage/sys-cluster/ceph-12.2.8/work/ceph-12.2.8/src/osd/PrimaryLogPG.cc: 
12985: FAILED assert(obc)


 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) 
luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x109) [0x562265bfda9c]
 2: 
(PrimaryLogPG::hit_set_trim(std::unique_ptrstd::default_delete >&, unsigned int)+0x942) 
[0x5622657d6cea]

 3: (PrimaryLogPG::hit_set_persist()+0xa4b) [0x5622657e5fab]
 4: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x426a) 
[0x562265800c64]
 5: (PrimaryLogPG::do_request(boost::intrusive_ptr&, 
ThreadPool::TPHandle&)+0xc1f) [0x5622657b94ed]
 6: (OSD::dequeue_op(boost::intrusive_ptr, 
boost::intrusive_ptr, ThreadPool::TPHandle&)+0x396) 
[0x562265655cf8]
 7: (PGQueueable::RunVis::operator()(boost::intrusive_ptr 
const&)+0x5a) [0x5622658c09a6]
 8: (OSD::ShardedOpWQ::_process(unsigned int, 
ceph::heartbeat_handle_d*)+0x1ab6) [0x562265657918]
 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5cd) 
[0x562265c026f5]

 10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x562265c05e72]
 11: (()+0x751e) [0x7f1f4fd7f51e]
 12: (clone()+0x3f) [0x7f1f4ed7ef0f]

A search in the bug tracker revealed that a similar error has been 
resolved for jewel http://tracker.ceph.com/issues/19223, yet I don't 
know if this is in any way relevant.


We are currently at a loss how to get these osds back up. Any 
suggestions how to approach this would be very welcome. If there is any 
further information that is needed or additional context please let me know.


Thanks,

Lawrence


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshooting hanging storage backend whenever there is any cluster change

[Nils Fahldieck - Profihost AG]

Hi, in our `ceph.conf` we have:

  mon_max_pg_per_osd = 300

While the host is offline (9 OSDs down):

  4352 PGs * 3 / 62 OSDs ~ 210 PGs per OSD

If all OSDs are online:

  4352 PGs * 3 / 71 OSDs ~ 183 PGs per OSD

... so this doesn't seem to be the issue.

If I understood you right, that's what you've meant. If I got you wrong,
would you mind to point to one of those threads you mentioned?

Thanks :)

Am 12.10.2018 um 14:03 schrieb Burkhard Linke:
> Hi,
> 
> 
> On 10/12/2018 01:55 PM, Nils Fahldieck - Profihost AG wrote:
>> I rebooted a Ceph host and logged `ceph status` & `ceph health detail`
>> every 5 seconds. During this I encountered 'PG_AVAILABILITY Reduced data
>> availability: pgs peering'. At the same time some VMs hung as described
>> before.
> 
> Just a wild guess... you have 71 OSDs and about 4500 PG with size=3.
> 13500 PG instance overall, resulting in ~190 PGs per OSD under normal
> circumstances.
> 
> If one host is down and the PGs have to re-peer, you might reach the
> limit of 200 PG/OSDs on some of the OSDs, resulting in stuck peering.
> 
> You can try to raise this limit. There are several threads on the
> mailing list about this.
> 
> Regards,
> Burkhard
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshooting hanging storage backend whenever there is any cluster change

[Burkhard Linke]

Hi,


On 10/12/2018 01:55 PM, Nils Fahldieck - Profihost AG wrote:

I rebooted a Ceph host and logged `ceph status` & `ceph health detail`
every 5 seconds. During this I encountered 'PG_AVAILABILITY Reduced data
availability: pgs peering'. At the same time some VMs hung as described
before.


Just a wild guess... you have 71 OSDs and about 4500 PG with size=3. 
13500 PG instance overall, resulting in ~190 PGs per OSD under normal 
circumstances.


If one host is down and the PGs have to re-peer, you might reach the 
limit of 200 PG/OSDs on some of the OSDs, resulting in stuck peering.


You can try to raise this limit. There are several threads on the 
mailing list about this.


Regards,
Burkhard

--
Dr. rer. nat. Burkhard Linke
Bioinformatics and Systems Biology
Justus-Liebig-University Giessen
35392 Giessen, Germany
Phone: (+49) (0)641 9935810

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Troubleshooting hanging storage backend whenever there is any cluster change

[Nils Fahldieck - Profihost AG]

I rebooted a Ceph host and logged `ceph status` & `ceph health detail`
every 5 seconds. During this I encountered 'PG_AVAILABILITY Reduced data
availability: pgs peering'. At the same time some VMs hung as described
before.

See the log here: https://pastebin.com/wxUKzhgB

PG_AVAILABILITY is noted at timestamps [2018-10-12 12:16:15.403394] and
[2018-10-12 12:17:40.072655].

Ceph docs say regarding PG_AVAILABILITY:

Data availability is reduced, meaning that the cluster is unable to
service potential read or write requests for some data in the cluster.
Specifically, one or more PGs is in a state that does not allow IO
requests to be serviced. Problematic PG states include peering, stale,
incomplete, and the lack of active (if those conditions do not clear
quickly).

Do you know why those PGs are stuck peering and how I might troubleshoot
this any further?
Am 11.10.2018 um 22:27 schrieb David Turner:
> You should definitely stop using `size 3 min_size 1` on your pools.  Go
> back to the default `min_size 2`.  I'm a little confused why you have 3
> different CRUSH rules.  They're all identical.  You only need different
> CRUSH rules if you're using Erasure Coding or targeting a different set
> of OSDs like SSD vs HDD OSDs for different pools.
> 
> All of that said, I don't see anything in those rules that would
> indicate why you're having problems with accessing your data when a node
> is being restarted.  The `ceph status` and `ceph health detail` outputs
> will be helpful while it's happening.
> 
> On Thu, Oct 11, 2018 at 3:02 PM Nils Fahldieck - Profihost AG
> mailto:n.fahldi...@profihost.ag>> wrote:
> 
> Thanks for your reply. I'll capture a `ceph status` the next time I
> encounter a not working RBD. Here's the other output you asked for:
> 
> $ ceph osd crush rule dump
> [
>     {
>         "rule_id": 0,
>         "rule_name": "data",
>         "ruleset": 0,
>         "type": 1,
>         "min_size": 1,
>         "max_size": 10,
>         "steps": [
>             {
>                 "op": "take",
>                 "item": -1,
>                 "item_name": "root"
>             },
>             {
>                 "op": "chooseleaf_firstn",
>                 "num": 0,
>                 "type": "host"
>             },
>             {
>                 "op": "emit"
>             }
>         ]
>     },
>     {
>         "rule_id": 1,
>         "rule_name": "metadata",
>         "ruleset": 1,
>         "type": 1,
>         "min_size": 1,
>         "max_size": 10,
>         "steps": [
>             {
>                 "op": "take",
>                 "item": -1,
>                 "item_name": "root"
>             },
>             {
>                 "op": "chooseleaf_firstn",
>                 "num": 0,
>                 "type": "host"
>             },
>             {
>                 "op": "emit"
>             }
>         ]
>     },
>     {
>         "rule_id": 2,
>         "rule_name": "rbd",
>         "ruleset": 2,
>         "type": 1,
>         "min_size": 1,
>         "max_size": 10,
>         "steps": [
>             {
>                 "op": "take",
>                 "item": -1,
>                 "item_name": "root"
>             },
>             {
>                 "op": "chooseleaf_firstn",
>                 "num": 0,
>                 "type": "host"
>             },
>             {
>                 "op": "emit"
>             }
>         ]
>     }
> ]
> 
> $ ceph osd pool ls detail
> pool 5 'cephstor1' replicated size 3 min_size 1 crush_rule 0 object_hash
> rjenkins pg_num 4096 pgp_num 4096 last_change 1217074 flags hashpspool
> min_read_recency_for_promote 1 min_write_recency_for_promote 1
> stripe_width 0 application rbd
>         removed_snaps
> 
> [1~9,b~1,d~7d1e8,7d1f6~3d05f,ba256~4bd9,bee30~357,bf188~5531,c46ba~85b3,ccc6e~b599,d820b~1,d820d~1,d820f~1,d8211~1,d8214~1,d8216~1,d8219~2,d821d~1,d821f~1,d8221~1,d8223~1,d8226~2,d8229~1,d822b~2,d822e~2,d8231~3,d8236~1,d8238~2,d823b~1,d823d~3,d8241~1,d8243~1,d8245~1,d8247~3,d824d~1,d824f~1,d8251~1,d8253~1,d8255~2,d8258~1,d825c~1,d825e~2,d8262~1,d8264~1,d8266~1,d8268~2,d826e~2,d8272~1,d8274~1,d8276~8,d8280~1,d8282~1,d8284~1,d8286~1,d8288~1,d828a~1,d828c~1,d828e~1,d8290~1,d8292~1,d8294~3,d8298~1,d829a~2,d829d~1,d82a0~4,d82a6~1,d82a8~2,d82ac~1,d82ae~1,d82b0~1,d82b2~1,d82b5~1,d82b7~1,d82b9~1,d82bb~1,d82bd~1,d82bf~1,d82c1~1,d82c3~2,d82c6~2,d82c9~1,d82cb~1,d82ce~1,d82d0~2,d82d3~1,d82d6~4,d82db~1,d82de~1,d82e0~1,d82e2~1,d82e4~1,d82e6~1,d82e8~1,d82ea~1,d82ed~1,d82ef~1,d82f1~1,d82f3~2,d82f7~2,d82fb~2,d82ff~1,d8301~1,d8303~1,d8305~1,d8307~1,d8309~1,d830b~1,d830e~1,d8311~2,d8314~3,d8318~1,d831a~1,d831c~1,d831f~3,d8323~2,d8329~1,d832

Re: [ceph-users] bluestore compression enabled but no data compressed

[Frank Schilder]

Hi David,

thanks for your answer. I did enable compression on the pools as described in 
the link you sent below (ceph osd pool set sr-fs-data-test compression_mode 
aggressive, I also tried force to no avail). However, I could not find anything 
on enabling compression per OSD. Could you possibly provide a source or sample 
commands?

Thanks and best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: David Turner 
Sent: 09 October 2018 17:42
To: Frank Schilder
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] bluestore compression enabled but no data compressed

When I've tested compression before there are 2 places you need to configure 
compression.  On the OSDs in the configuration settings that you mentioned, but 
also on the [1] pools themselves.  If you have the compression mode on the 
pools set to none, then it doesn't matter what the OSDs configuration is and 
vice versa unless you are using the setting of force.  If you want to default 
compress everything, set pools to passive and osds to aggressive.  If you want 
to only compress specific pools, set the osds to passive and the specific pools 
to aggressive.  Good luck.


[1] http://docs.ceph.com/docs/mimic/rados/operations/pools/#set-pool-values

On Tue, Sep 18, 2018 at 7:11 AM Frank Schilder 
mailto:fr...@dtu.dk>> wrote:
I seem to have a problem getting bluestore compression to do anything. I 
followed the documentation and enabled bluestore compression on various pools 
by executing "ceph osd pool set  compression_mode aggressive". 
Unfortunately, it seems like no data is compressed at all. As an example, below 
is some diagnostic output for a data pool used by a cephfs:

[root@ceph-01 ~]# ceph --version
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)

All defaults are OK:

[root@ceph-01 ~]# ceph --show-config | grep compression
[...]
bluestore_compression_algorithm = snappy
bluestore_compression_max_blob_size = 0
bluestore_compression_max_blob_size_hdd = 524288
bluestore_compression_max_blob_size_ssd = 65536
bluestore_compression_min_blob_size = 0
bluestore_compression_min_blob_size_hdd = 131072
bluestore_compression_min_blob_size_ssd = 8192
bluestore_compression_mode = none
bluestore_compression_required_ratio = 0.875000
[...]

Compression is reported as enabled:

[root@ceph-01 ~]# ceph osd pool ls detail
[...]
pool 24 'sr-fs-data-test' erasure size 8 min_size 7 crush_rule 10 object_hash 
rjenkins pg_num 50 pgp_num 50 last_change 7726 flags hashpspool,ec_overwrites 
stripe_width 24576 compression_algorithm snappy compression_mode aggressive 
application cephfs
[...]

[root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_mode
compression_mode: aggressive
[root@ceph-01 ~]# ceph osd pool get sr-fs-data-test compression_algorithm
compression_algorithm: snappy

We dumped a 4Gib file with dd from /dev/zero. Should be easy to compress with 
excellent ratio. Search for a PG:

[root@ceph-01 ~]# ceph pg ls-by-pool sr-fs-data-test
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG 
DISK_LOG STATESTATE_STAMPVERSION  REPORTED UP   
UP_PRIMARY ACTING   ACTING_PRIMARY LAST_SCRUB 
SCRUB_STAMPLAST_DEEP_SCRUB DEEP_SCRUB_STAMP
24.0 15  00 0   0  62914560  77 
  77 active+clean 2018-09-14 01:07:14.593007  7698'77 7735:142 
[53,47,36,30,14,55,57,5] 53 [53,47,36,30,14,55,57,5] 53
7698'77 2018-09-14 01:07:14.592966 0'0 2018-09-11 08:06:29.309010

There is about 250MB data on the primary OSD, but noting seems to be compressed:

[root@ceph-07 ~]# ceph daemon osd.53 perf dump | grep blue
[...]
"bluestore_allocated": 313917440,
"bluestore_stored": 264362803,
"bluestore_compressed": 0,
"bluestore_compressed_allocated": 0,
"bluestore_compressed_original": 0,
[...]

Just to make sure, I checked one of the objects' contents:

[root@ceph-01 ~]# rados ls -p sr-fs-data-test
104.039c
[...]
104.039f

It is 4M chunks ...
[root@ceph-01 ~]# rados -p sr-fs-data-test stat 104.039f
sr-fs-data-test/104.039f mtime 2018-09-11 14:39:38.00, size 
4194304

... with all zeros:

[root@ceph-01 ~]# rados -p sr-fs-data-test get 104.039f obj

[root@ceph-01 ~]# hexdump -C obj
  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ||
*
0040

All as it should be, except for compression. Am I overlooking something?

Best regards,

=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users