Re: [ceph-users] dealing with the full osd / help reweight

2016-03-25 Thread lin zhou 
Yeah,I think the main reason is the setting of pg_num and pgp_num of some key 
pool.
This site will tell you the correct value:http://ceph.com/pgcalc/


Before you adjust pg_num and pgp_num,if this is a product environment,you 
should set as Christian Balzer said:
--- 
osd_max_backfills = 1 
osd_backfill_scan_min = 4 
osd_backfill_scan_max = 32 
osd_recovery_max_active = 1 
osd recovery threads = 1 
osd recovery op priority = 1 
—
you can use 'ceph tell osd.* injectargs “--odd-max-backfills 1” to change this 
setting.

and recommended process is 8->16->32->64->128->256->…  the more you increase 
the more data will rebalance,and more time it will take.so be careful.


--  
hnuzhoul...@gmail.com

在 2016年3月24日 20:57:17, Jacek Jarosiewicz (jjarosiew...@supermedia.pl) 写到:
> [root@cf01 ceph]# ceph osd pool ls detail
> pool 0 'vms' replicated size 2 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 67 flags hashpspool stripe_width 0
> pool 1 '.rgw.root' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 117 flags hashpspool
> stripe_width 0
> pool 2 '.rgw.control' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 118 flags hashpspool
> stripe_width 0
> pool 3 '.rgw.gc' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 119 flags hashpspool
> stripe_width 0
> pool 4 '.rgw.buckets_cache' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 121 flags hashpspool
> stripe_width 0
> pool 5 '.rgw.buckets.index' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 122 flags hashpspool
> stripe_width 0
> pool 6 '.rgw.buckets.extra' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 123 flags hashpspool
> stripe_width 0
> pool 7 '.log' replicated size 2 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 124 flags hashpspool stripe_width 0
> pool 8 '.intent-log' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 125 flags hashpspool
> stripe_width 0
> pool 9 '.usage' replicated size 2 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 126 flags hashpspool stripe_width 0
> pool 10 '.users' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 127 flags hashpspool
> stripe_width 0
> pool 11 '.users.email' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 128 flags hashpspool
> stripe_width 0
> pool 12 '.users.swift' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 129 flags hashpspool
> stripe_width 0
> pool 13 '.users.uid' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 130 flags hashpspool
> stripe_width 0
> pool 14 '.rgw.buckets' replicated size 2 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 64 pgp_num 64 last_change 5717 flags
> hashpspool stripe_width 0
> pool 15 '.rgw' replicated size 2 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 135 owner 18446744073709551615
> flags hashpspool stripe_width 0
> pool 17 'one' replicated size 3 min_size 2 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 611 flags hashpspool
> stripe_width 0
> removed_snaps [1~e,13~1]
>  
> I've tried reweight-by-utilization, but after some data shifting the
> cluster came up with a near full osd again..
>  
> Do I assume correctly that a lower weight of an osd means 'use that osd
> less'?
>  
> J
>  
> On 03/24/2016 01:43 PM, koukou73gr wrote:
> > What is your pool size? 304 pgs sound awfuly small for 20 OSDs.
> > More pgs will help distribute full pgs better.
> >
> > But with a full or near full OSD in hand, increasing pgs is a no-no
> > operation. If you search in the list archive, I believe there was a
> > thread last month or so which provided a walkthrough-sort of for dealing
> > with uneven distribution and a full OSD.
> >
> > -K.
> >
> >
> > On 03/24/2016 01:54 PM, Jacek Jarosiewicz wrote:
> >> disk usage on the full osd is as below. What are the *_TEMP directories
> >> for? How can I make sure which pg directories are safe to remove?
> >>
> >> [root@cf04 current]# du -hs *
> >> 156G 0.14_head
> >> 156G 0.21_head
> >> 155G 0.32_head
> >> 157G 0.3a_head
> >> 155G 0.e_head
> >> 156G 0.f_head
> >> 40K 10.2_head
> >> 4.0K 11.3_head
> >> 218G 14.13_head
> >> 218G 14.15_head
> >> 218G 14.1b_head
> >> 219G 14.1f_head
> >> 9.1G 14.29_head
> >> 219G 14.2a_head
> >> 75G 14.2d_head
> >> 125G 14.2e_head
> >> 113G 14.32_head
> >> 163G 14.33_head
> >> 218G 14.35_head
> >> 151G 14.39_head
> >> 218G 14.3b_head
> >> 103G 14.3d_head
> >> 217G 14.3f_head
> >> 219G 14.a_head
> >> 773M

Re: [ceph-users] Ceph-fuse huge performance gap between different block sizes

2016-03-25 Thread Christian Balzer 
On Fri, 25 Mar 2016 09:17:08 + Zhang Qiang wrote:

> Hi Christian, Thanks for your reply, here're the test specs:
> >>>
> [global]
> ioengine=libaio
> runtime=90
> direct=1
There it is.

You do understand what that flag does and what latencies are, right?

You're basically telling the I/O stack to not acknowledge the write until
it has reached the actual storage.

So your 4KB block has to traverse all the way from your client to the
storage node holding the primary PG, being written to the journal there,
same with any replicas, THEN that primary PG needs to send back an ACK to
the client that it has been done.
That amounts to about 3300 IOPS in your case, not actually bad, 0.3ms RTT.

As I said before, that's the sum of all your network latencies and Ceph
code overhead. 

If you have RBD caching (look it up) enabled on your fuse client machine
AND set direct=0, those writes can be coalesced ideally into 4MB perfect
Ceph block sized operations.

Christian

> group_reporting
> iodepth=16
> ramp_time=5
> size=1G
> 
> [seq_w_4k_20]
> bs=4k
> filename=seq_w_4k_20
> rw=write
> numjobs=20
> 
> [seq_w_1m_20]
> bs=1m
> filename=seq_w_1m_20
> rw=write
> numjobs=20
> 
> 
> Test results: 4k -  aggrb=13245KB/s, 1m - aggrb=1102.6MB/s
> 
> Mount options:  ceph-fuse /ceph -m 10.3.138.36:6789
> 
> Ceph configurations:
> 
> filestore_xattr_use_omap = true
> auth cluster required = cephx
> auth service required = cephx
> auth client required = cephx
> osd journal size = 128
> osd pool default size = 2
> osd pool default min size = 1
> osd pool default pg num = 512
> osd pool default pgp num = 512
> osd crush chooseleaf type = 1
> 
> 
> Other configurations are all default.
> 
> Status:
>  health HEALTH_OK
>  monmap e5: 5 mons at {1=
> 10.3.138.37:6789/0,2=10.3.138.39:6789/0,3=10.3.138.40:6789/0,4=10.3.138.59:6789/0,GGZ-YG-S0311-PLATFORM-138=10.3.138.36:6789/0
> }
> election epoch 28, quorum 0,1,2,3,4
> GGZ-YG-S0311-PLATFORM-138,1,2,3,4
>  mdsmap e55: 1/1/1 up {0=1=up:active}
>  osdmap e1290: 20 osds: 20 up, 20 in
>   pgmap v7180: 1000 pgs, 2 pools, 14925 MB data, 3851 objects
> 37827 MB used, 20837 GB / 21991 GB avail
> 1000 active+clean
> 
> On Fri, 25 Mar 2016 at 16:44 Christian Balzer  wrote:
> 
> >
> > Hello,
> >
> > On Fri, 25 Mar 2016 08:11:27 + Zhang Qiang wrote:
> >
> > > Hi all,
> > >
> > > According to fio,
> > Exact fio command please.
> >
> > >with 4k block size, the sequence write performance of
> > > my ceph-fuse mount
> >
> > Exact mount options, ceph config (RBD cache) please.
> >
> > >is just about 20+ M/s, only 200 Mb of 1 Gb full
> > > duplex NIC outgoing bandwidth was used for maximum. But for 1M block
> > > size the performance could achieve as high as 1000 M/s, approaching
> > > the limit of the NIC bandwidth. Why the performance stats differs so
> > > mush for different block sizes?
> > That's exactly why.
> > You can see that with local attached storage as well, many small
> > requests are slower than large (essential sequential) writes.
> > Network attached storage in general (latency) and thus Ceph as well
> > (plus code overhead) amplify that.
> >
> > >Can I configure ceph-fuse mount's block size
> > > for maximum performance?
> > >
> > Very little to do with that if you're using sync writes (thus the fio
> > command line pleasE), if not RBD cache could/should help.
> >
> > Christian
> >
> > > Basic information about the cluster: 20 OSDs on separate PCIe hard
> > > disks distributed across 2 servers, each with write performance
> > > about 300 M/s; 5 MONs; 1 MDS. Ceph version 0.94.6
> > > (e832001feaf8c176593e0325c8298e3f16dfb403).
> > >
> > > Thanks :)
> >
> >
> > --
> > Christian BalzerNetwork/Systems Engineer
> > ch...@gol.com   Global OnLine Japan/Rakuten Communications
> > http://www.gol.com/
> >


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Question about cache tier and backfill/recover

2016-03-25 Thread Christian Balzer 
On Fri, 25 Mar 2016 14:14:37 -0700 Bob R wrote:

> Mike,
> 
> Recovery would be based on placement groups and those degraded groups
> would only exist on the storage pool(s) rather than the cache tier in
> this scenario.
> 
Precisely.

They are entirely different entities.
There may be partially identical data (clean objects) in them, but that
will with near certainty never be enough for recovery/backfill operation.
Never mind that it would be akin to crossing the streams.

Christian

> Bob
> 
> On Fri, Mar 25, 2016 at 8:30 AM, Mike Miller 
> wrote:
> 
> > Hi,
> >
> > in case of a failure in the storage tier, say single OSD disk failure
> > or complete system failure with several OSD disks, will the remaining
> > cache tier (on other nodes) be used for rapid backfilling/recovering
> > first until it is full? Or is backfill/recovery done directly to the
> > storage tier?
> >
> > Thanks and regards,
> >
> > Mike
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Losing data in healthy cluster

2016-03-25 Thread Christian Balzer 

Hello,

this was of course discussed here in the very recent thread
"data corruption with hammer" 

Read it, it contains fixes and a workaround as well.
Also from that thread: http://tracker.ceph.com/issues/12814

You don't need to remove the cache tier to fix things.

And as also discussed here, getting the last (header) object out of the
cache requires stopping VMs in the case of RBD images and I would suspect
something similar with cephfs.

Christian


On Fri, 25 Mar 2016 16:47:11 -0700 Blade Doyle wrote:

> Help, my Ceph cluster is losing data slowly over time.  I keep finding
> files that are the same length as they should be, but all the content
> has been lost & replaced by nulls.
> 
> Here is an example:
> 
> (from a backup I have the original file)
> 
> [root@blotter docker]# ls -lart
> /backup/space/docker/ceph-monitor/ceph-w-monitor.py
> /space/docker/ceph-monitor/ceph-w-monitor.py
> -rwxrwxrwx 1 root root 7237 Mar 12 07:34
> /backup/space/docker/ceph-monitor/ceph-w-monitor.py
> -rwxrwxrwx 1 root root 7237 Mar 12 07:34
> /space/docker/ceph-monitor/ceph-w-monitor.py
> 
> [root@blotter docker]# sum
> /backup/space/docker/ceph-monitor/ceph-w-monitor.py 19803 8
> 
> [root@blotter docker]# sum /space/docker/ceph-monitor/ceph-w-monitor.py
> 0 8
> 
> 
> 
> If I had to _guess_ I would blame a recent change to the writeback cache
> tier layer.  I turned it off and flushed it last weekendabout the
> same time I started to notice this data loss.
> 
> I disabled it using instructions from here:
> http://docs.ceph.com/docs/master/rados/operations/cache-tiering/
> 
> Basically, I just set it to "forward" and then "flushed it".
> ceph osd tier cache-mode ssd_cache forward
> and
> rados -p ssd_cache cache-flush-evict-all
> 
> After that I removed the overlay.  But that failed (and still fails)
> with:
> 
> 
> Finally, tried to remove the cache t from the backing pool, but that
> failed (still fails) with:
> 
> $ ceph osd tier remove-overlay cephfs_data
> Error EBUSY: pool 'cephfs_data' is in use by CephFS via its tier
> 
> At that point I thought, because I had set the cache-mode to "forward",
> it would be safe to just leave it as is until I had time to debug
> further.
> 
> I should mention that after the cluster settled down and did some
> scrubbing, there was one inconsistent page.  I ran a "ceph fix page xxx"
> command to resolve that and the health was good again.
> 
> 
> I can do some experimenting this weekend if somebody wants to help me
> through it.  Otherwise I'll probably try to put the cache-tier back into
> "writeback" to see if that helps.  If not, I'll recreate the entire ceph
> cluster.
> 
> Thanks,
> Blade.
> 
> 
> P.S. My cluster is made of mixed ARM and x86_64..
> $ ceph version
> ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)
> 
> # ceph version
> ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
> 
> etc...
> 
> 
> PPS:
> 
> $ ceph df
> GLOBAL:
> SIZE  AVAIL RAW USED %RAW USED
> 2456G 1492G 839G 34.16
> POOLS:
> NAMEID USED   %USED MAX AVAIL OBJECTS
> rbd 0139G  5.66  185G   36499
> cephfs_data 1235G  9.59  185G  102883
> cephfs_metadata 2  33642k 0  185G5530
> ssd_cache   4   0 0  370G   0


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Losing data in healthy cluster

2016-03-25 Thread Blade Doyle 
Help, my Ceph cluster is losing data slowly over time.  I keep finding files
that are the same length as they should be, but all the content has been
lost & replaced by nulls.

Here is an example:

(from a backup I have the original file)

[root@blotter docker]# ls -lart
/backup/space/docker/ceph-monitor/ceph-w-monitor.py
/space/docker/ceph-monitor/ceph-w-monitor.py
-rwxrwxrwx 1 root root 7237 Mar 12 07:34
/backup/space/docker/ceph-monitor/ceph-w-monitor.py
-rwxrwxrwx 1 root root 7237 Mar 12 07:34
/space/docker/ceph-monitor/ceph-w-monitor.py

[root@blotter docker]# sum
/backup/space/docker/ceph-monitor/ceph-w-monitor.py 19803 8

[root@blotter docker]# sum /space/docker/ceph-monitor/ceph-w-monitor.py
0 8



If I had to _guess_ I would blame a recent change to the writeback cache
tier layer.  I turned it off and flushed it last weekendabout the same
time I started to notice this data loss.

I disabled it using instructions from here:
http://docs.ceph.com/docs/master/rados/operations/cache-tiering/

Basically, I just set it to "forward" and then "flushed it".
ceph osd tier cache-mode ssd_cache forward
and
rados -p ssd_cache cache-flush-evict-all

After that I removed the overlay.  But that failed (and still fails) with:


Finally, tried to remove the cache t from the backing pool, but that failed
(still fails) with:

$ ceph osd tier remove-overlay cephfs_data
Error EBUSY: pool 'cephfs_data' is in use by CephFS via its tier

At that point I thought, because I had set the cache-mode to "forward", it
would be safe to just leave it as is until I had time to debug further.

I should mention that after the cluster settled down and did some scrubbing,
there was one inconsistent page.  I ran a "ceph fix page xxx" command to
resolve that and the health was good again.


I can do some experimenting this weekend if somebody wants to help me
through it.  Otherwise I'll probably try to put the cache-tier back into
"writeback" to see if that helps.  If not, I'll recreate the entire ceph
cluster.

Thanks,
Blade.


P.S. My cluster is made of mixed ARM and x86_64..
$ ceph version
ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)

# ceph version
ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)

etc...


PPS:

$ ceph df
GLOBAL:
SIZE  AVAIL RAW USED %RAW USED
2456G 1492G 839G 34.16
POOLS:
NAMEID USED   %USED MAX AVAIL OBJECTS
rbd 0139G  5.66  185G   36499
cephfs_data 1235G  9.59  185G  102883
cephfs_metadata 2  33642k 0  185G5530
ssd_cache   4   0 0  370G   0
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pg incomplete second osd in acting set still available

2016-03-25 Thread John-Paul Robinson 
So I think I know what might have gone wrong.

When I took might osd's out of the cluster and shut them down, the first
set of osds likely came back up and in the cluster before 300 seconds
expired.  This would have prevented cluster triggering recovery of the
pg from the replica osd.

So the question is, can I force this to happen?  Can I take the supposed
primary osd down for 300+ seconds to allow the cluster to start
recovering the pgs (this will of course affect all other pgs on the
osds).  Or is there a better way?

Note that all my secondary osds in these pgs have the expected amount of
data for the pg, remained up during the primary's downtime and should
have the state to become the primary for the acting set.

Thanks for listening.

~jpr


On 03/25/2016 11:57 AM, John-Paul Robinson wrote:
> Hi Folks,
>
> One last dip into my old bobtail cluster.  (new hardware is on order)
>
> I have three pg in an incomplete state.  The cluster was previously
> stable but with a health warn state due to a few near full osds.  I
> started resizing drives on one host to expand space after taking the
> osds that served them out and down.  My failure domain is two levels
> osds and hosts and have two copies per placement group.
>
> I have three of my pgs flagging incomplete.
>
> root@d90-b1-1c-3a-c4-8f:~# date; sudo ceph --id nova health detail |
> grep incomplete
> Fri Mar 25 11:28:47 CDT 2016
> HEALTH_WARN 168 pgs backfill; 107 pgs backfilling; 241 pgs degraded; 3
> pgs incomplete; 3 pgs stuck inactive; 287 pgs stuck unclean; recovery
> 4913393/39589336 degraded (12.411%);  recovering 120 o/s, 481MB/s; 4
> near full osd(s)
> pg 3.5 is stuck inactive since forever, current state incomplete, last
> acting [53,22]
> pg 3.150 is stuck inactive since forever, current state incomplete, last
> acting [50,74]
> pg 3.38c is stuck inactive since forever, current state incomplete, last
> acting [14,70]
> pg 3.5 is stuck unclean since forever, current state incomplete, last
> acting [53,22]
> pg 3.150 is stuck unclean since forever, current state incomplete, last
> acting [50,74]
> pg 3.38c is stuck unclean since forever, current state incomplete, last
> acting [14,70]
> pg 3.38c is incomplete, acting [14,70]
> pg 3.150 is incomplete, acting [50,74]
> pg 3.5 is incomplete, acting [53,22]
>
> Given that incomplete means:
>
> "Ceph detects that a placement group is missing information about writes
> that may have occurred, or does not have any healthy copies. If you see
> this state, try to start any failed OSDs that may contain the needed
> information or temporarily adjust min_size to allow recovery."
>
> I have restarted all osds in these acting sets and they log normally,
> opening their respective journals and such. However, the incomplete
> state remains.
>
> All three of the primary osds 53,50,14 have were reformatted to expand
> size, so I know there's no "spare" journal if its referring to what was
> there before.  Btw, I did take all osds to out and down before resizing
> their drives, so I'm not sure how these pg would actually be expecting
> old journal.
>
> I suspect I need to forgo the journal and let the secondaries become
> primary for these pg.
>
> I sure hope that's possible.
>
> As always, thanks for any pointers.
>
> ~jpr
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Question about cache tier and backfill/recover

2016-03-25 Thread Bob R 
Mike,

Recovery would be based on placement groups and those degraded groups would
only exist on the storage pool(s) rather than the cache tier in this
scenario.

Bob

On Fri, Mar 25, 2016 at 8:30 AM, Mike Miller 
wrote:

> Hi,
>
> in case of a failure in the storage tier, say single OSD disk failure or
> complete system failure with several OSD disks, will the remaining cache
> tier (on other nodes) be used for rapid backfilling/recovering first until
> it is full? Or is backfill/recovery done directly to the storage tier?
>
> Thanks and regards,
>
> Mike
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] pg incomplete second osd in acting set still available

2016-03-25 Thread John-Paul Robinson 
Hi Folks,

One last dip into my old bobtail cluster.  (new hardware is on order)

I have three pg in an incomplete state.  The cluster was previously
stable but with a health warn state due to a few near full osds.  I
started resizing drives on one host to expand space after taking the
osds that served them out and down.  My failure domain is two levels
osds and hosts and have two copies per placement group.

I have three of my pgs flagging incomplete.

root@d90-b1-1c-3a-c4-8f:~# date; sudo ceph --id nova health detail |
grep incomplete
Fri Mar 25 11:28:47 CDT 2016
HEALTH_WARN 168 pgs backfill; 107 pgs backfilling; 241 pgs degraded; 3
pgs incomplete; 3 pgs stuck inactive; 287 pgs stuck unclean; recovery
4913393/39589336 degraded (12.411%);  recovering 120 o/s, 481MB/s; 4
near full osd(s)
pg 3.5 is stuck inactive since forever, current state incomplete, last
acting [53,22]
pg 3.150 is stuck inactive since forever, current state incomplete, last
acting [50,74]
pg 3.38c is stuck inactive since forever, current state incomplete, last
acting [14,70]
pg 3.5 is stuck unclean since forever, current state incomplete, last
acting [53,22]
pg 3.150 is stuck unclean since forever, current state incomplete, last
acting [50,74]
pg 3.38c is stuck unclean since forever, current state incomplete, last
acting [14,70]
pg 3.38c is incomplete, acting [14,70]
pg 3.150 is incomplete, acting [50,74]
pg 3.5 is incomplete, acting [53,22]

Given that incomplete means:

"Ceph detects that a placement group is missing information about writes
that may have occurred, or does not have any healthy copies. If you see
this state, try to start any failed OSDs that may contain the needed
information or temporarily adjust min_size to allow recovery."

I have restarted all osds in these acting sets and they log normally,
opening their respective journals and such. However, the incomplete
state remains.

All three of the primary osds 53,50,14 have were reformatted to expand
size, so I know there's no "spare" journal if its referring to what was
there before.  Btw, I did take all osds to out and down before resizing
their drives, so I'm not sure how these pg would actually be expecting
old journal.

I suspect I need to forgo the journal and let the secondaries become
primary for these pg.

I sure hope that's possible.

As always, thanks for any pointers.

~jpr

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Question about cache tier and backfill/recover

2016-03-25 Thread Mike Miller 

Hi,

in case of a failure in the storage tier, say single OSD disk failure or 
complete system failure with several OSD disks, will the remaining cache 
tier (on other nodes) be used for rapid backfilling/recovering first 
until it is full? Or is backfill/recovery done directly to the storage tier?


Thanks and regards,

Mike
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-fuse huge performance gap between different block sizes

2016-03-25 Thread Jan Schermer 
FYI when I performed testing on our cluster I saw the same thing.
fio randwrite 4k test over a large volume was a lot faster with larger RBD 
object size (8mb was marginally better than the default 4mb). It makes no sense 
to me unless there is a huge overhead with increasing number of objects. Or 
maybe there is some sort of alignment problem that causes small objects overlap 
with the actual workload. (In my cluster some objects are mysteriously sized as 
4MiB-4KiB).

Jan

> On 25. 3. 2016, at 10:17, Zhang Qiang  wrote:
> 
> Hi Christian, Thanks for your reply, here're the test specs:
> >>>
> [global]
> ioengine=libaio
> runtime=90
> direct=1
> group_reporting
> iodepth=16
> ramp_time=5
> size=1G
> 
> [seq_w_4k_20]
> bs=4k
> filename=seq_w_4k_20
> rw=write
> numjobs=20
> 
> [seq_w_1m_20]
> bs=1m
> filename=seq_w_1m_20
> rw=write
> numjobs=20
> 
> 
> Test results: 4k -  aggrb=13245KB/s, 1m - aggrb=1102.6MB/s
> 
> Mount options:  ceph-fuse /ceph -m 10.3.138.36:6789
> 
> Ceph configurations:
> 
> filestore_xattr_use_omap = true
> auth cluster required = cephx
> auth service required = cephx
> auth client required = cephx
> osd journal size = 128
> osd pool default size = 2
> osd pool default min size = 1
> osd pool default pg num = 512
> osd pool default pgp num = 512
> osd crush chooseleaf type = 1
> 
> 
> Other configurations are all default.
> 
> Status:
>  health HEALTH_OK
>  monmap e5: 5 mons at 
> {1=10.3.138.37:6789/0,2=10.3.138.39:6789/0,3=10.3.138.40:6789/0,4=10.3.138.59:6789/0,GGZ-YG-S0311-PLATFORM-138=10.3.138.36:6789/0}
> election epoch 28, quorum 0,1,2,3,4 
> GGZ-YG-S0311-PLATFORM-138,1,2,3,4
>  mdsmap e55: 1/1/1 up {0=1=up:active}
>  osdmap e1290: 20 osds: 20 up, 20 in
>   pgmap v7180: 1000 pgs, 2 pools, 14925 MB data, 3851 objects
> 37827 MB used, 20837 GB / 21991 GB avail
> 1000 active+clean
> 
>> On Fri, 25 Mar 2016 at 16:44 Christian Balzer  wrote:
>> 
>> Hello,
>> 
>> On Fri, 25 Mar 2016 08:11:27 + Zhang Qiang wrote:
>> 
>> > Hi all,
>> >
>> > According to fio,
>> Exact fio command please.
>> 
>> >with 4k block size, the sequence write performance of
>> > my ceph-fuse mount
>> 
>> Exact mount options, ceph config (RBD cache) please.
>> 
>> >is just about 20+ M/s, only 200 Mb of 1 Gb full
>> > duplex NIC outgoing bandwidth was used for maximum. But for 1M block
>> > size the performance could achieve as high as 1000 M/s, approaching the
>> > limit of the NIC bandwidth. Why the performance stats differs so mush
>> > for different block sizes?
>> That's exactly why.
>> You can see that with local attached storage as well, many small requests
>> are slower than large (essential sequential) writes.
>> Network attached storage in general (latency) and thus Ceph as well (plus
>> code overhead) amplify that.
>> 
>> >Can I configure ceph-fuse mount's block size
>> > for maximum performance?
>> >
>> Very little to do with that if you're using sync writes (thus the fio
>> command line pleasE), if not RBD cache could/should help.
>> 
>> Christian
>> 
>> > Basic information about the cluster: 20 OSDs on separate PCIe hard disks
>> > distributed across 2 servers, each with write performance about 300 M/s;
>> > 5 MONs; 1 MDS. Ceph version 0.94.6
>> > (e832001feaf8c176593e0325c8298e3f16dfb403).
>> >
>> > Thanks :)
>> 
>> 
>> --
>> Christian BalzerNetwork/Systems Engineer
>> ch...@gol.com   Global OnLine Japan/Rakuten Communications
>> http://www.gol.com/
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] xfs: v4 or v5?

2016-03-25 Thread Jan Schermer 
V5 is supposedly stable, but that only means it will be just as bad as any 
other XFS.

I recommend avoiding XFS whenever possible. Ext4 works perfectly and I never 
lost any data with it, even when it got corrupted, while XFS still likes to eat 
the data when something goes wrong (and it will, like when you hit bit rot or a 
data cable fails).

Jan

> On 25. 3. 2016, at 11:44, Dzianis Kahanovich  wrote:
> 
> Before adding/replacing new OSDs:
> 
> What version of xfs is preferred by ceph developers/testers now?
> Time ago I move all to v5 (crc=1,finobt=1), it works, exclude
> "logbsize=256k,logbufs=8" in 4.4. Now I see, v5 is default mode (xfsprogs &
> kernel 4.5 at least).
> 
> I in doubts: make new OSDs old-style v4 + logbsize=256k,logbufs=8 (and remove 
> v5
> crc worloads) - increase linear performance (exclude rm/ls operations), or 
> make
> current default v5 to other benefits.
> 
> Are xfs v4 still mainstream for ceph?
> 
> PS I use too fresh "unstable" Gentoo ~amd64, so don't know some normal distros
> reality...
> 
> -- 
> WBR, Dzianis Kahanovich AKA Denis Kaganovich, http://mahatma.bspu.unibel.by/
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] xfs: v4 or v5?

2016-03-25 Thread Dzianis Kahanovich 
Before adding/replacing new OSDs:

What version of xfs is preferred by ceph developers/testers now?
Time ago I move all to v5 (crc=1,finobt=1), it works, exclude
"logbsize=256k,logbufs=8" in 4.4. Now I see, v5 is default mode (xfsprogs &
kernel 4.5 at least).

I in doubts: make new OSDs old-style v4 + logbsize=256k,logbufs=8 (and remove v5
crc worloads) - increase linear performance (exclude rm/ls operations), or make
current default v5 to other benefits.

Are xfs v4 still mainstream for ceph?

PS I use too fresh "unstable" Gentoo ~amd64, so don't know some normal distros
reality...

-- 
WBR, Dzianis Kahanovich AKA Denis Kaganovich, http://mahatma.bspu.unibel.by/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] after upgrade from 0.80.11 to 0.94.6, rbd cmd core dump

2016-03-25 Thread archer.wudong 
sorry, it's my fault. I have found the problem: because on that host, there is 
wrong version librados.so, this version is built long time ago and I forgot 
this, so it mislead me. I have removed it and link to the right one.


2016-03-25



archer.wudong



发件人：Dong Wu 
发送时间：2016-03-25 16:02
主题：after upgrade from 0.80.11 to 0.94.6, rbd cmd core dump
收件人："ceph-users"
抄送：

Hi all, I upgraded my cluster from 0.80.11 to 0.94.6, everything is ok 
except that rbd cmd cord dump on one host and success on others. 
I have disabled auth in  ceph.conf: 
auth_cluster_required = none 
auth_service_required = none 
auth_client_required = none 

here is the core message. 

$ sudo rbd ls 
2016-03-25 16:00:43.043000 7f3ae6c13780  1 -- :/0 messenger.start 
2016-03-25 16:00:43.043329 7f3ae6c13780  1 -- :/1008171 --> 
10.180.0.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 
0x434a330 con 0x4349fc0 
2016-03-25 16:00:43.043377 7f3ae6c13780  0 -- :/1008171 submit_message 
auth(proto 0 30 bytes epoch 0) v1 
 : 00 00 00 00 00 00 00 00 ff ff 00 00 00 00 00 00 :  
0010 : 00 00 00 00 00 00 1e 00 00 00 01 01 00 00 00 01 :  
0020 : 00 00 00 08 00 00 00 05 00 00 00 61 64 6d 69 6e : ...admin 
0030 : 00 00 00 00 00 00 00 00 00 00 00 00 :  

2016-03-25 16:00:43.043450 7f3adb7fe700  1 monclient(hunting): continuing hunt 
2016-03-25 16:00:43.043489 7f3adb7fe700  1 -- :/1008171 mark_down 
0x4349fc0 -- 0x4349d30 
2016-03-25 16:00:43.043614 7f3adb7fe700  1 -- :/1008171 --> 
10.180.0.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 
0x7f39cc001060 con 0x7f39cc000cf0 
2016-03-25 16:00:43.043648 7f3adb7fe700  0 -- :/1008171 submit_message 
auth(proto 0 30 bytes epoch 0) v1 
 : 00 00 00 00 00 00 00 00 ff ff 00 00 00 00 00 00 :  
0010 : 00 00 00 00 00 00 1e 00 00 00 01 01 00 00 00 01 :  
0020 : 00 00 00 08 00 00 00 05 00 00 00 61 64 6d 69 6e : ...admin 
0030 : 00 00 00 00 00 00 00 00 00 00 00 00 :  

2016-03-25 16:00:43.043694 7f3ae6c13780  0 monclient(hunting): 
authenticate timed out after 2.47033e-321 
*** Caught signal (Segmentation fault) ** 
 in thread 7f3adbfff700 
2016-03-25 16:00:43.043756 7f3adb7fe700  1 monclient(hunting): continuing hunt 
2016-03-25 16:00:43.043749 7f3ae6c13780  0 librados: client.admin 
authentication error (110) Connection timed out 
 ceph version 0.94.6-2-gbb98b8f (bb98b8fcb0bb0bd3688310f6a1688736ef422b25) 
 1: rbd() [0x60408c] 
 2: (()+0xf8d0) [0x7f3ae4ea88d0] 
 3: rbd() [0x52b841] 
 4: (Mutex::~Mutex()+0x9b) [0x562a6b] 
 5: (Connection::~Connection()+0x6e) [0x7f3ae5550fce] 
 6: (Connection::~Connection()+0x9) [0x7f3ae5551049] 
 7: (Pipe::~Pipe()+0x90) [0x7f3ae553f330] 
 8: (Pipe::~Pipe()+0x9) [0x7f3ae553f4e9] 
 9: (SimpleMessenger::reaper()+0x8a9) [0x7f3aebf9] 
 10: (SimpleMessenger::reaper_entry()+0x88) [0x7f3ae5556b38] 
 11: (SimpleMessenger::ReaperThread::entry()+0xd) [0x7f3ae555ba8d] 
 12: (()+0x80a4) [0x7f3ae4ea10a4] 
 13: (clone()+0x6d) [0x7f3ae3a2d04d] 
2016-03-25 16:00:43.045278 7f3adbfff700 -1 *** Caught signal 
(Segmentation fault) ** 
 in thread 7f3adbfff700 

 ceph version 0.94.6-2-gbb98b8f (bb98b8fcb0bb0bd3688310f6a1688736ef422b25) 
 1: rbd() [0x60408c] 
 2: (()+0xf8d0) [0x7f3ae4ea88d0] 
 3: rbd() [0x52b841] 
 4: (Mutex::~Mutex()+0x9b) [0x562a6b] 
 5: (Connection::~Connection()+0x6e) [0x7f3ae5550fce] 
 6: (Connection::~Connection()+0x9) [0x7f3ae5551049] 
 7: (Pipe::~Pipe()+0x90) [0x7f3ae553f330] 
 8: (Pipe::~Pipe()+0x9) [0x7f3ae553f4e9] 
 9: (SimpleMessenger::reaper()+0x8a9) [0x7f3aebf9] 
 10: (SimpleMessenger::reaper_entry()+0x88) [0x7f3ae5556b38] 
 11: (SimpleMessenger::ReaperThread::entry()+0xd) [0x7f3ae555ba8d] 
 12: (()+0x80a4) [0x7f3ae4ea10a4] 
 13: (clone()+0x6d) [0x7f3ae3a2d04d] 
 NOTE: a copy of the executable, or `objdump -rdS ` is 
needed to interpret this. 

--- begin dump of recent events --- 
   -39> 2016-03-25 16:00:43.036565 7f3ae6c13780  5 asok(0x42f1830) 
register_command perfcounters_dump hook 0x42f5000 
   -38> 2016-03-25 16:00:43.036596 7f3ae6c13780  5 asok(0x42f1830) 
register_command 1 hook 0x42f5000 
   -37> 2016-03-25 16:00:43.036608 7f3ae6c13780  5 asok(0x42f1830) 
register_command perf dump hook 0x42f5000 
   -36> 2016-03-25 16:00:43.036621 7f3ae6c13780  5 asok(0x42f1830) 
register_command perfcounters_schema hook 0x42f5000 
   -35> 2016-03-25 16:00:43.036630 7f3ae6c13780  5 asok(0x42f1830) 
register_command 2 hook 0x42f5000 
   -34> 2016-03-25 16:00:43.036634 7f3ae6c13780  5 asok(0x42f1830) 
register_command perf schema hook 0x42f5000 
   -33> 2016-03-25 16:00:43.036639 7f3ae6c13780  5 asok(0x42f1830) 
register_command perf reset hook 0x42f5000 
   -32> 2016-03-25 16:00:43.036643 7f3ae6c13780  5 asok(0x42f1830) 
register_command config show hook 0x42f5000 
   -31> 2016-03-25 16:00:43.036651 7f3ae6c13780  5 asok(0x42f1830) 
register_command config set hook 0x42f5000

Re: [ceph-users] Ceph-fuse huge performance gap between different block sizes

2016-03-25 Thread Zhang Qiang 
Hi Christian, Thanks for your reply, here're the test specs:
>>>
[global]
ioengine=libaio
runtime=90
direct=1
group_reporting
iodepth=16
ramp_time=5
size=1G

[seq_w_4k_20]
bs=4k
filename=seq_w_4k_20
rw=write
numjobs=20

[seq_w_1m_20]
bs=1m
filename=seq_w_1m_20
rw=write
numjobs=20


Test results: 4k -  aggrb=13245KB/s, 1m - aggrb=1102.6MB/s

Mount options:  ceph-fuse /ceph -m 10.3.138.36:6789

Ceph configurations:

filestore_xattr_use_omap = true
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 128
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 512
osd pool default pgp num = 512
osd crush chooseleaf type = 1


Other configurations are all default.

Status:
 health HEALTH_OK
 monmap e5: 5 mons at {1=
10.3.138.37:6789/0,2=10.3.138.39:6789/0,3=10.3.138.40:6789/0,4=10.3.138.59:6789/0,GGZ-YG-S0311-PLATFORM-138=10.3.138.36:6789/0
}
election epoch 28, quorum 0,1,2,3,4
GGZ-YG-S0311-PLATFORM-138,1,2,3,4
 mdsmap e55: 1/1/1 up {0=1=up:active}
 osdmap e1290: 20 osds: 20 up, 20 in
  pgmap v7180: 1000 pgs, 2 pools, 14925 MB data, 3851 objects
37827 MB used, 20837 GB / 21991 GB avail
1000 active+clean

On Fri, 25 Mar 2016 at 16:44 Christian Balzer  wrote:

>
> Hello,
>
> On Fri, 25 Mar 2016 08:11:27 + Zhang Qiang wrote:
>
> > Hi all,
> >
> > According to fio,
> Exact fio command please.
>
> >with 4k block size, the sequence write performance of
> > my ceph-fuse mount
>
> Exact mount options, ceph config (RBD cache) please.
>
> >is just about 20+ M/s, only 200 Mb of 1 Gb full
> > duplex NIC outgoing bandwidth was used for maximum. But for 1M block
> > size the performance could achieve as high as 1000 M/s, approaching the
> > limit of the NIC bandwidth. Why the performance stats differs so mush
> > for different block sizes?
> That's exactly why.
> You can see that with local attached storage as well, many small requests
> are slower than large (essential sequential) writes.
> Network attached storage in general (latency) and thus Ceph as well (plus
> code overhead) amplify that.
>
> >Can I configure ceph-fuse mount's block size
> > for maximum performance?
> >
> Very little to do with that if you're using sync writes (thus the fio
> command line pleasE), if not RBD cache could/should help.
>
> Christian
>
> > Basic information about the cluster: 20 OSDs on separate PCIe hard disks
> > distributed across 2 servers, each with write performance about 300 M/s;
> > 5 MONs; 1 MDS. Ceph version 0.94.6
> > (e832001feaf8c176593e0325c8298e3f16dfb403).
> >
> > Thanks :)
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Global OnLine Japan/Rakuten Communications
> http://www.gol.com/
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-fuse huge performance gap between different block sizes

2016-03-25 Thread Christian Balzer 

Hello,

On Fri, 25 Mar 2016 08:11:27 + Zhang Qiang wrote:

> Hi all,
> 
> According to fio, 
Exact fio command please.

>with 4k block size, the sequence write performance of
> my ceph-fuse mount 

Exact mount options, ceph config (RBD cache) please.

>is just about 20+ M/s, only 200 Mb of 1 Gb full
> duplex NIC outgoing bandwidth was used for maximum. But for 1M block
> size the performance could achieve as high as 1000 M/s, approaching the
> limit of the NIC bandwidth. Why the performance stats differs so mush
> for different block sizes? 
That's exactly why.
You can see that with local attached storage as well, many small requests
are slower than large (essential sequential) writes.
Network attached storage in general (latency) and thus Ceph as well (plus
code overhead) amplify that.

>Can I configure ceph-fuse mount's block size
> for maximum performance?
> 
Very little to do with that if you're using sync writes (thus the fio
command line pleasE), if not RBD cache could/should help.

Christian

> Basic information about the cluster: 20 OSDs on separate PCIe hard disks
> distributed across 2 servers, each with write performance about 300 M/s;
> 5 MONs; 1 MDS. Ceph version 0.94.6
> (e832001feaf8c176593e0325c8298e3f16dfb403).
> 
> Thanks :)


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph-fuse huge performance gap between different block sizes

2016-03-25 Thread Zhang Qiang 
Hi all,

According to fio, with 4k block size, the sequence write performance of my
ceph-fuse mount is just about 20+ M/s, only 200 Mb of 1 Gb full duplex NIC
outgoing bandwidth was used for maximum. But for 1M block size the
performance could achieve as high as 1000 M/s, approaching the limit of the
NIC bandwidth. Why the performance stats differs so mush for different
block sizes? Can I configure ceph-fuse mount's block size for maximum
performance?

Basic information about the cluster: 20 OSDs on separate PCIe hard disks
distributed across 2 servers, each with write performance about 300 M/s; 5
MONs; 1 MDS. Ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403).

Thanks :)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] after upgrade from 0.80.11 to 0.94.6, rbd cmd core dump

2016-03-25 Thread Dong Wu 
Hi all, I upgraded my cluster from 0.80.11 to 0.94.6, everything is ok
except that rbd cmd cord dump on one host and success on others.
I have disabled auth in  ceph.conf:
auth_cluster_required = none
auth_service_required = none
auth_client_required = none

here is the core message.

$ sudo rbd ls
2016-03-25 16:00:43.043000 7f3ae6c13780  1 -- :/0 messenger.start
2016-03-25 16:00:43.043329 7f3ae6c13780  1 -- :/1008171 -->
10.180.0.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0
0x434a330 con 0x4349fc0
2016-03-25 16:00:43.043377 7f3ae6c13780  0 -- :/1008171 submit_message
auth(proto 0 30 bytes epoch 0) v1
 : 00 00 00 00 00 00 00 00 ff ff 00 00 00 00 00 00 : 
0010 : 00 00 00 00 00 00 1e 00 00 00 01 01 00 00 00 01 : 
0020 : 00 00 00 08 00 00 00 05 00 00 00 61 64 6d 69 6e : ...admin
0030 : 00 00 00 00 00 00 00 00 00 00 00 00 : 

2016-03-25 16:00:43.043450 7f3adb7fe700  1 monclient(hunting): continuing hunt
2016-03-25 16:00:43.043489 7f3adb7fe700  1 -- :/1008171 mark_down
0x4349fc0 -- 0x4349d30
2016-03-25 16:00:43.043614 7f3adb7fe700  1 -- :/1008171 -->
10.180.0.31:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0
0x7f39cc001060 con 0x7f39cc000cf0
2016-03-25 16:00:43.043648 7f3adb7fe700  0 -- :/1008171 submit_message
auth(proto 0 30 bytes epoch 0) v1
 : 00 00 00 00 00 00 00 00 ff ff 00 00 00 00 00 00 : 
0010 : 00 00 00 00 00 00 1e 00 00 00 01 01 00 00 00 01 : 
0020 : 00 00 00 08 00 00 00 05 00 00 00 61 64 6d 69 6e : ...admin
0030 : 00 00 00 00 00 00 00 00 00 00 00 00 : 

2016-03-25 16:00:43.043694 7f3ae6c13780  0 monclient(hunting):
authenticate timed out after 2.47033e-321
*** Caught signal (Segmentation fault) **
 in thread 7f3adbfff700
2016-03-25 16:00:43.043756 7f3adb7fe700  1 monclient(hunting): continuing hunt
2016-03-25 16:00:43.043749 7f3ae6c13780  0 librados: client.admin
authentication error (110) Connection timed out
 ceph version 0.94.6-2-gbb98b8f (bb98b8fcb0bb0bd3688310f6a1688736ef422b25)
 1: rbd() [0x60408c]
 2: (()+0xf8d0) [0x7f3ae4ea88d0]
 3: rbd() [0x52b841]
 4: (Mutex::~Mutex()+0x9b) [0x562a6b]
 5: (Connection::~Connection()+0x6e) [0x7f3ae5550fce]
 6: (Connection::~Connection()+0x9) [0x7f3ae5551049]
 7: (Pipe::~Pipe()+0x90) [0x7f3ae553f330]
 8: (Pipe::~Pipe()+0x9) [0x7f3ae553f4e9]
 9: (SimpleMessenger::reaper()+0x8a9) [0x7f3aebf9]
 10: (SimpleMessenger::reaper_entry()+0x88) [0x7f3ae5556b38]
 11: (SimpleMessenger::ReaperThread::entry()+0xd) [0x7f3ae555ba8d]
 12: (()+0x80a4) [0x7f3ae4ea10a4]
 13: (clone()+0x6d) [0x7f3ae3a2d04d]
2016-03-25 16:00:43.045278 7f3adbfff700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7f3adbfff700

 ceph version 0.94.6-2-gbb98b8f (bb98b8fcb0bb0bd3688310f6a1688736ef422b25)
 1: rbd() [0x60408c]
 2: (()+0xf8d0) [0x7f3ae4ea88d0]
 3: rbd() [0x52b841]
 4: (Mutex::~Mutex()+0x9b) [0x562a6b]
 5: (Connection::~Connection()+0x6e) [0x7f3ae5550fce]
 6: (Connection::~Connection()+0x9) [0x7f3ae5551049]
 7: (Pipe::~Pipe()+0x90) [0x7f3ae553f330]
 8: (Pipe::~Pipe()+0x9) [0x7f3ae553f4e9]
 9: (SimpleMessenger::reaper()+0x8a9) [0x7f3aebf9]
 10: (SimpleMessenger::reaper_entry()+0x88) [0x7f3ae5556b38]
 11: (SimpleMessenger::ReaperThread::entry()+0xd) [0x7f3ae555ba8d]
 12: (()+0x80a4) [0x7f3ae4ea10a4]
 13: (clone()+0x6d) [0x7f3ae3a2d04d]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.

--- begin dump of recent events ---
   -39> 2016-03-25 16:00:43.036565 7f3ae6c13780  5 asok(0x42f1830)
register_command perfcounters_dump hook 0x42f5000
   -38> 2016-03-25 16:00:43.036596 7f3ae6c13780  5 asok(0x42f1830)
register_command 1 hook 0x42f5000
   -37> 2016-03-25 16:00:43.036608 7f3ae6c13780  5 asok(0x42f1830)
register_command perf dump hook 0x42f5000
   -36> 2016-03-25 16:00:43.036621 7f3ae6c13780  5 asok(0x42f1830)
register_command perfcounters_schema hook 0x42f5000
   -35> 2016-03-25 16:00:43.036630 7f3ae6c13780  5 asok(0x42f1830)
register_command 2 hook 0x42f5000
   -34> 2016-03-25 16:00:43.036634 7f3ae6c13780  5 asok(0x42f1830)
register_command perf schema hook 0x42f5000
   -33> 2016-03-25 16:00:43.036639 7f3ae6c13780  5 asok(0x42f1830)
register_command perf reset hook 0x42f5000
   -32> 2016-03-25 16:00:43.036643 7f3ae6c13780  5 asok(0x42f1830)
register_command config show hook 0x42f5000
   -31> 2016-03-25 16:00:43.036651 7f3ae6c13780  5 asok(0x42f1830)
register_command config set hook 0x42f5000
   -30> 2016-03-25 16:00:43.036654 7f3ae6c13780  5 asok(0x42f1830)
register_command config get hook 0x42f5000
   -29> 2016-03-25 16:00:43.036659 7f3ae6c13780  5 asok(0x42f1830)
register_command config diff hook 0x42f5000
   -28> 2016-03-25 16:00:43.036662 7f3ae6c13780  5 asok(0x42f1830)
register_command log flush hook 0x42f5000
   -27> 2016-03-25 16:00:43.036667 7f3ae6c13780  5 asok(0x42f1830)
register_command log dump hook 0x42f5000
   -26> 2016-03-25 16:00:43.036670 7f3ae6c13780  5 asok(0x42f1830)