Re: [ceph-users] v9.1.0 Infernalis release candidate released

[Alfredo Deza]

Trusty has just been pushed out and should be ready to be used right
away. We didn't realize this until today, sorry!

-Alfredo

On Wed, Oct 14, 2015 at 9:24 PM, Sage Weil  wrote:
> On Thu, 15 Oct 2015, Francois Lafont wrote:
>
>> Sorry, another remark.
>>
>> On 13/10/2015 23:01, Sage Weil wrote:
>>
>> > The v9.1.0 packages are pushed to the development release repositories::
>> >
>> >   http://download.ceph.com/rpm-testing
>> >   http://download.ceph.com/debian-testing
>>
>> I don't see the 9.1.0 available for Ubuntu Trusty :
>>
>> 
>> http://download.ceph.com/debian-testing/dists/trusty/main/binary-amd64/Packages
>> (the string "9.1" is not present in this page currently)
>>
>> The 9.0.3 is available but, after a quick test, this version of
>> the package doesn't create the ceph unix account.
>
> You're right.. I see jessie but not trusty in the archive.  Alfredo, can
> you verify it synced properly?
>
> Thanks!
> sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mon crash after update to Hammer 0.94.3 from Firefly 0.80.10

[Richard Bade]

Ok, debugging increased
ceph tell mon.[abc] injectargs --debug-mon 20
ceph tell mon.[abc] injectargs --debug-ms 1

Regards,
Richard

On 17 October 2015 at 01:38, Sage Weil  wrote:

> This doesn't look familiar.  Are you able to enable a higher log level so
> that if it happens again we'll have more info?
>
> debug mon = 20
> debug ms = 1
>
> Thanks!
> sage
>
> On Fri, 16 Oct 2015, Dan van der Ster wrote:
>
> > Hmm, that's strange. I didn't see anything in the tracker that looks
> > related. Hopefully an expert can chime in...
> >
> > Cheers, Dan
> >
> > On Fri, Oct 16, 2015 at 1:38 PM, Richard Bade  wrote:
> > > Thanks for your quick response Dan, but no. All the ceph-mon.*.log
> files are
> > > empty.
> > > I did track this down in syslog though, in case it helps:
> > > ceph-mon: 2015-10-16 21:25:00.117115 7f4c9f458700 -1 *** Caught signal
> > > (Segmentation fault) **#012 in thread 7f4c9f458700#012#012 ceph version
> > > 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)#012 1:
> /usr/bin/ceph-mon()
> > > [0x928b05]#012 2: (()+0xfcb0) [0x7f4ca50e0cb0]#012 3:
> > > (get_str_map_key(std::map std::less,
> > > std::allocator > const&,
> > > std::string const&, std::string const*)+0x37) [0x87d8e7]#012 4:
> > > (LogMonitor::update_from_paxos(bool*)+0x801) [0x6846e1]#012 5:
> > > (PaxosService::refresh(bool*)+0x3c6) [0x5dc326]#012 6:
> > > (Monitor::refresh_from_paxos(bool*)+0x36b) [0x588aab]#012 7:
> > > (Paxos::do_refresh()+0x4c) [0x5c465c]#012 8:
> > > (Paxos::handle_commit(MMonPaxos*)+0x243) [0x5cb2d3]#012 9:
> > > (Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5d3fbb]#012 10:
> > > (Monitor::dispatch(MonSession*, Message*, bool)+0x864) [0x5ab0d4]#012
> 11:
> > > (Monitor::_ms_dispatch(Message*)+0x2c9) [0x5a8a19]#012 12:
> > > (Monitor::ms_dispatch(Message*)+0x32) [0x5c3952]#012 13:
> > > (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x8ac987]#012 14:
> > > (DispatchQueue::entry()+0x44a) [0x8a9b2a]#012 15:
> > > (DispatchQueue::DispatchThread::entry()+0xd) [0x79e4ad]#012 16:
> (()+0x7e9a)
> > > [0x7f4ca50d8e9a]#012 17: (clone()+0x6d) [0x7f4ca3dca38d]#012 NOTE: a
> copy of
> > > the executable, or `objdump -rdS ` is needed to interpret
> this.
> > >
> > > Regards,
> > > Richard
> > >
> > > On 17 October 2015 at 00:33, Dan van der Ster 
> wrote:
> > >>
> > >> Hi,
> > >> Is there a backtrace in /var/log/ceph/ceph-mon.*.log ?
> > >> Cheers, Dan
> > >>
> > >> On Fri, Oct 16, 2015 at 12:46 PM, Richard Bade 
> wrote:
> > >> > Hi Everyone,
> > >> > I upgraded our cluster to Hammer 0.94.3 a couple of days ago and
> today
> > >> > we've
> > >> > had one monitor crash twice and another one once. We have 3 monitors
> > >> > total
> > >> > and have been running Firefly 0.80.10 for quite some time without
> any
> > >> > monitor issues.
> > >> > When the monitor crashes it leaves a core file and a crash file in
> > >> > /var/crash
> > >> > I can't see anything obviously the same goolging about.
> > >> > Has anyone seen anything like this?
> > >> > Any suggestions? What other info would be useful to help track down
> the
> > >> > issue.
> > >> >
> > >> > Regards,
> > >> > Richard
> > >> >
> > >> > ___
> > >> > ceph-users mailing list
> > >> > ceph-users@lists.ceph.com
> > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >> >
> > >
> > >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CephFS and page cache

[Burkhard Linke]

Hi,

I've noticed that CephFS (both ceph-fuse and kernel client in version 
4.2.3) remove files from page cache as soon as they are not in use by a 
process anymore.


Is this intended behaviour? We use CephFS as a replacement for NFS in 
our HPC cluster. It should serve large files which are read by multiple 
jobs on multiple hosts, so keeping them in the page cache over the 
duration of several job invocations is crucial.


Mount options are defaults,noatime,_netdev (+ extra options for the 
kernel client). Is there an option to keep data in page cache just like 
any other filesystem?


Regards,
Burkhard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

[Björn Lässig]

On 10/15/2015 09:40 PM, Ken Dreyer wrote:

I wonder if there are routing issues with IPv6 in the DreamHost cloud?

http://ipv6-test.com/validate.php prints the right IP, but then "web
server is unreachable : Connection timed out"


Getting the same error here.
With sixxs 4 out of 5 wgets are failing. ping6 is dropped.

We tried from different sites in .at .uk and .de. Tries from uk and de 
are failing mostly. at is working.


does ''ping6 download.ceph.com'' works for you?

regards
  Björn Lässig

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] troubleshooting ceph

[Vickie ch]

One more thing, did you check the setting of firewall？



Best wishes,
Mika


2015-10-16 14:54 GMT+08:00 Vickie ch :

> Hi Artie,
> Did you check your mon ？ How many monitors in this cluster？
>
>
>
> Best wishes,
> Mika
>
>
> 2015-10-16 9:23 GMT+08:00 Artie Ziff :
>
>> Hello Ceph-users!
>>
>> This is my first attempt at getting ceph running.
>>
>> Does the following, in isolation, indicate any potential troubleshooting
>> directions
>>
>> # ceph -s
>> 2015-10-15 18:12:45.586529 7fc86041b700  0 -- :/1006343 >>
>> 10.10.20.60:6789/0 pipe(0x7fc85c00d4c0 sd=3 :0 s=1 pgs=0 cs=0 l=1
>> c=0x7fc85c011760).fault
>> 2015-10-15 18:12:48.586607 7fc86031a700  0 -- :/1006343 >>
>> 10.10.20.60:6789/0 pipe(0x7fc858c0 sd=4 :0 s=1 pgs=0 cs=0 l=1
>> c=0x7fc850004b60).fault
>> ^CError connecting to cluster: InterruptedOrTimeoutError
>>
>>
>> The password-less keys are working amongst root@everyhost.
>> ceph was installed with # make install  (the defaults)
>>
>> I am reading the troubleshooting page
>> , of course.
>> Does the error above occur if the monitor is not running?
>>
>> I believe a log directory (/usr/local/var/log/ceph) was created by `make
>> install`. Is there some additional step to enable logs? Does no log file
>> indicate the monitor is not running?
>>
>> Thank you in advance for some pointers.
>>
>> -az
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] troubleshooting ceph

[Vickie ch]

Hi Artie,
Did you check your mon ？ How many monitors in this cluster？



Best wishes,
Mika


2015-10-16 9:23 GMT+08:00 Artie Ziff :

> Hello Ceph-users!
>
> This is my first attempt at getting ceph running.
>
> Does the following, in isolation, indicate any potential troubleshooting
> directions
>
> # ceph -s
> 2015-10-15 18:12:45.586529 7fc86041b700  0 -- :/1006343 >>
> 10.10.20.60:6789/0 pipe(0x7fc85c00d4c0 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fc85c011760).fault
> 2015-10-15 18:12:48.586607 7fc86031a700  0 -- :/1006343 >>
> 10.10.20.60:6789/0 pipe(0x7fc858c0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fc850004b60).fault
> ^CError connecting to cluster: InterruptedOrTimeoutError
>
>
> The password-less keys are working amongst root@everyhost.
> ceph was installed with # make install  (the defaults)
>
> I am reading the troubleshooting page
> , of course.
> Does the error above occur if the monitor is not running?
>
> I believe a log directory (/usr/local/var/log/ceph) was created by `make
> install`. Is there some additional step to enable logs? Does no log file
> indicate the monitor is not running?
>
> Thank you in advance for some pointers.
>
> -az
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Potential OSD deadlock?

[Max A. Krasilnikov]

Hello!

On Fri, Oct 09, 2015 at 01:45:42PM +0200, jan wrote:

> Have you tried running iperf between the nodes? Capturing a pcap of the 
> (failing) Ceph comms from both sides could help narrow it down.
> Is there any SDN layer involved that could add overhead/padding to the frames?

> What about some intermediate MTU like 8000 - does that work?
> Oh and if there's any bonding/trunking involved, beware that you need to set 
> the same MTU and offloads on all interfaces on certains kernels - flags like 
> MTU/offloads should propagate between the master/slave interfaces but in 
> reality it's not the case and they get reset even if you unplug/replug the 
> ethernet cable.

I'm sorry for long time to answer, but I have fixed problem with Jumbo frames
with sysctl:
#
net.ipv4.tcp_moderate_rcvbuf = 0
#
net.ipv4.tcp_rmem= 1024000 8738000 1677721600
net.ipv4.tcp_wmem= 1024000 8738000 1677721600
net.ipv4.tcp_mem= 1024000 8738000 1677721600
net.core.rmem_max=1677721600
net.core.rmem_default=167772160
net.core.wmem_max=1677721600
net.core.wmem_default=167772160

And now i can load my cluster without any slow requests. The essential setting
is net.ipv4.tcp_moderate_rcvbuf = 0. All other are just tunings.

> Jan

>> On 09 Oct 2015, at 13:21, Max A. Krasilnikov  wrote:
>> 
>> Hello!
>> 
>> On Fri, Oct 09, 2015 at 11:05:59AM +0200, jan wrote:
>> 
>>> Are there any errors on the NICs? (ethtool -s ethX)
>> 
>> No errors. Neither on nodes, nor on switches.
>> 
>>> Also take a look at the switch and look for flow control statistics - do 
>>> you have flow control enabled or disabled?
>> 
>> flow control disabled everywhere.
>> 
>>> We had to disable flow control as it would pause all IO on the port 
>>> whenever any path got congested which you don't want to happen with a 
>>> cluster like Ceph. It's better to let the frame drop/retransmit in this 
>>> case (and you should size it so it doesn't happen in any case).
>>> And how about NIC offloads? Do they play nice with jumbo frames? I wouldn't 
>>> put my money on that...
>> 
>> I tried to completely disable all offloads and setting mtu back to 9000 
>> after.
>> No luck.
>> I am speaking with my NOC about MTU in 10G network. If I have update, I will
>> write here. I can hardly beleave that it is ceph side, but nothing is
>> impossible.
>> 
>>> Jan
>> 
>> 
 On 09 Oct 2015, at 10:48, Max A. Krasilnikov  wrote:
 
 Hello!
 
 On Thu, Oct 08, 2015 at 11:44:09PM -0600, robert wrote:
 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
 
> Sage,
 
> After trying to bisect this issue (all test moved the bisect towards
> Infernalis) and eventually testing the Infernalis branch again, it
> looks like the problem still exists although it is handled a tad
> better in Infernalis. I'm going to test against Firefly/Giant next
> week and then try and dive into the code to see if I can expose any
> thing.
 
> If I can do anything to provide you with information, please let me know.
 
 I have fixed my troubles by setting MTU back to 1500 from 9000 in 2x10G 
 network
 between nodes (2x Cisco Nexus 5020, one link per switch, LACP, linux 
 bounding
 driver: bonding mode=4 lacp_rate=1 xmit_hash_policy=1 miimon=100, Intel 
 82599ES
 Adapter, non-intel sfp+). When setting it to 9000 on nodes and 9216 on 
 Nexus 5020
 switch with Jumbo frames enabled i have performance drop and slow 
 requests. When
 setting 1500 on nodes and not touching Nexus all problems are fixed.
 
 I have rebooted all my ceph services when changing MTU and changing things 
 to
 9000 and 1500 several times in order to be sure. It is reproducable in my
 environment.
 
> Thanks,
> -BEGIN PGP SIGNATURE-
> Version: Mailvelope v1.2.0
> Comment: https://www.mailvelope.com
 
> wsFcBAEBCAAQBQJWF1QlCRDmVDuy+mK58QAAWLgP/2l+TkcpeKihDxF8h/kw
> YFffNWODNfOMq8FVDQkQceo2mFCFc29JnBYiAeqW+XPelwuU5S86LG998aUB
> BvIU4EHaJNJ31X1NCIA7nwi8rXlFYfSG2qQn58+IzqZoWCQM5vD/THISV1rP
> qQKtoOAEuRxz+vOAJGI1A1xJSOiFwTRjs4LjE1zYjSP26LdEF61D/lb+AVzV
> ufxi/ci6mAla/4VTAH4VqEviDgC8AbAZnWFGfUPcTUxJQS99kFrfjJnWvgyF
> V9EmWtQCvhRO74hQLBqspOwdAxEJesPfGcJT1LjR0eEAMWvbGPtaqbSFAEWa
> jjyy5wP9+4NnGLdhba6UBtLphjqTcl0e2vVwRj0zLhI14moAOlbhIKmZ1Dt+
> 1P6vfgOUGvO76xgDMwrVKRoQgWJO/0Tup9+oqInnNYgf4W+ZWsLgLgo7ETAF
> VcI7LP1wkwAI3lz5YphY/TnKNGs6i+wVjKBamOt3R1yz9WeylaG0T6xgGHrs
> VugrRSUuO+ND9+mE5EsUgITCZoaavXJESJMb30XkK6hYGB+T/q+hBafc6Wle
> Jgs+aT2m1erdSyZn0ZC9a6CjWmwJXY6FCSGhE53BbefBxmCFxn+8tVav+Q8W
> 7s14TntP6ex4ca7eTwGuSXC9FU5fAVa+3+3aXDAC1QPAkeVkXyB716W1XG6b
> BCFo
> =GJL4
> -END PGP SIGNATURE-
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
 
 
> On Wed, Oct 7, 2015 at 1:25 PM, Robert LeBlanc 

Re: [ceph-users] download.ceph.com unreachable IPv6 [was: v9.1.0 Infernalis release candidate released]

[Corin Langosch]

download.ceph.com resolves to 2607:f298:6050:51f3:f816:3eff:fe50:5ec here. Ping 
seems to be blocked. Connect to port 80
works every few requests, probably 50%. So I assume there's some load-balancer 
there with a dead backend, which the
load-balancer didn't detect/ kick...just guessing. Best Corin

Am 16.10.2015 um 08:27 schrieb Björn Lässig:
> Getting the same error here.
> With sixxs 4 out of 5 wgets are failing. ping6 is dropped.
> 
> We tried from different sites in .at .uk and .de. Tries from uk and de are 
> failing mostly. at is working.
> 
> does ''ping6 download.ceph.com'' works for you?
> 
> regards
>   Björn Lässig
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-mon crash after update to Hammer 0.94.3 from Firefly 0.80.10

[Richard Bade]

Hi Everyone,
I upgraded our cluster to Hammer 0.94.3 a couple of days ago and today
we've had one monitor crash twice and another one once. We have 3 monitors
total and have been running Firefly 0.80.10 for quite some time without any
monitor issues.
When the monitor crashes it leaves a core file and a crash file in
/var/crash
I can't see anything obviously the same goolging about.
Has anyone seen anything like this?
Any suggestions? What other info would be useful to help track down the
issue.

Regards,
Richard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mon crash after update to Hammer 0.94.3 from Firefly 0.80.10

[Sage Weil]

This doesn't look familiar.  Are you able to enable a higher log level so 
that if it happens again we'll have more info?

debug mon = 20
debug ms = 1

Thanks!
sage

On Fri, 16 Oct 2015, Dan van der Ster wrote:

> Hmm, that's strange. I didn't see anything in the tracker that looks
> related. Hopefully an expert can chime in...
> 
> Cheers, Dan
> 
> On Fri, Oct 16, 2015 at 1:38 PM, Richard Bade  wrote:
> > Thanks for your quick response Dan, but no. All the ceph-mon.*.log files are
> > empty.
> > I did track this down in syslog though, in case it helps:
> > ceph-mon: 2015-10-16 21:25:00.117115 7f4c9f458700 -1 *** Caught signal
> > (Segmentation fault) **#012 in thread 7f4c9f458700#012#012 ceph version
> > 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)#012 1: /usr/bin/ceph-mon()
> > [0x928b05]#012 2: (()+0xfcb0) [0x7f4ca50e0cb0]#012 3:
> > (get_str_map_key(std::map > std::allocator > const&,
> > std::string const&, std::string const*)+0x37) [0x87d8e7]#012 4:
> > (LogMonitor::update_from_paxos(bool*)+0x801) [0x6846e1]#012 5:
> > (PaxosService::refresh(bool*)+0x3c6) [0x5dc326]#012 6:
> > (Monitor::refresh_from_paxos(bool*)+0x36b) [0x588aab]#012 7:
> > (Paxos::do_refresh()+0x4c) [0x5c465c]#012 8:
> > (Paxos::handle_commit(MMonPaxos*)+0x243) [0x5cb2d3]#012 9:
> > (Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5d3fbb]#012 10:
> > (Monitor::dispatch(MonSession*, Message*, bool)+0x864) [0x5ab0d4]#012 11:
> > (Monitor::_ms_dispatch(Message*)+0x2c9) [0x5a8a19]#012 12:
> > (Monitor::ms_dispatch(Message*)+0x32) [0x5c3952]#012 13:
> > (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x8ac987]#012 14:
> > (DispatchQueue::entry()+0x44a) [0x8a9b2a]#012 15:
> > (DispatchQueue::DispatchThread::entry()+0xd) [0x79e4ad]#012 16: (()+0x7e9a)
> > [0x7f4ca50d8e9a]#012 17: (clone()+0x6d) [0x7f4ca3dca38d]#012 NOTE: a copy of
> > the executable, or `objdump -rdS ` is needed to interpret this.
> >
> > Regards,
> > Richard
> >
> > On 17 October 2015 at 00:33, Dan van der Ster  wrote:
> >>
> >> Hi,
> >> Is there a backtrace in /var/log/ceph/ceph-mon.*.log ?
> >> Cheers, Dan
> >>
> >> On Fri, Oct 16, 2015 at 12:46 PM, Richard Bade  wrote:
> >> > Hi Everyone,
> >> > I upgraded our cluster to Hammer 0.94.3 a couple of days ago and today
> >> > we've
> >> > had one monitor crash twice and another one once. We have 3 monitors
> >> > total
> >> > and have been running Firefly 0.80.10 for quite some time without any
> >> > monitor issues.
> >> > When the monitor crashes it leaves a core file and a crash file in
> >> > /var/crash
> >> > I can't see anything obviously the same goolging about.
> >> > Has anyone seen anything like this?
> >> > Any suggestions? What other info would be useful to help track down the
> >> > issue.
> >> >
> >> > Regards,
> >> > Richard
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
> >
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mon crash after update to Hammer 0.94.3 from Firefly 0.80.10

[Dan van der Ster]

Hmm, that's strange. I didn't see anything in the tracker that looks
related. Hopefully an expert can chime in...

Cheers, Dan

On Fri, Oct 16, 2015 at 1:38 PM, Richard Bade  wrote:
> Thanks for your quick response Dan, but no. All the ceph-mon.*.log files are
> empty.
> I did track this down in syslog though, in case it helps:
> ceph-mon: 2015-10-16 21:25:00.117115 7f4c9f458700 -1 *** Caught signal
> (Segmentation fault) **#012 in thread 7f4c9f458700#012#012 ceph version
> 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)#012 1: /usr/bin/ceph-mon()
> [0x928b05]#012 2: (()+0xfcb0) [0x7f4ca50e0cb0]#012 3:
> (get_str_map_key(std::map std::allocator > const&,
> std::string const&, std::string const*)+0x37) [0x87d8e7]#012 4:
> (LogMonitor::update_from_paxos(bool*)+0x801) [0x6846e1]#012 5:
> (PaxosService::refresh(bool*)+0x3c6) [0x5dc326]#012 6:
> (Monitor::refresh_from_paxos(bool*)+0x36b) [0x588aab]#012 7:
> (Paxos::do_refresh()+0x4c) [0x5c465c]#012 8:
> (Paxos::handle_commit(MMonPaxos*)+0x243) [0x5cb2d3]#012 9:
> (Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5d3fbb]#012 10:
> (Monitor::dispatch(MonSession*, Message*, bool)+0x864) [0x5ab0d4]#012 11:
> (Monitor::_ms_dispatch(Message*)+0x2c9) [0x5a8a19]#012 12:
> (Monitor::ms_dispatch(Message*)+0x32) [0x5c3952]#012 13:
> (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x8ac987]#012 14:
> (DispatchQueue::entry()+0x44a) [0x8a9b2a]#012 15:
> (DispatchQueue::DispatchThread::entry()+0xd) [0x79e4ad]#012 16: (()+0x7e9a)
> [0x7f4ca50d8e9a]#012 17: (clone()+0x6d) [0x7f4ca3dca38d]#012 NOTE: a copy of
> the executable, or `objdump -rdS ` is needed to interpret this.
>
> Regards,
> Richard
>
> On 17 October 2015 at 00:33, Dan van der Ster  wrote:
>>
>> Hi,
>> Is there a backtrace in /var/log/ceph/ceph-mon.*.log ?
>> Cheers, Dan
>>
>> On Fri, Oct 16, 2015 at 12:46 PM, Richard Bade  wrote:
>> > Hi Everyone,
>> > I upgraded our cluster to Hammer 0.94.3 a couple of days ago and today
>> > we've
>> > had one monitor crash twice and another one once. We have 3 monitors
>> > total
>> > and have been running Firefly 0.80.10 for quite some time without any
>> > monitor issues.
>> > When the monitor crashes it leaves a core file and a crash file in
>> > /var/crash
>> > I can't see anything obviously the same goolging about.
>> > Has anyone seen anything like this?
>> > Any suggestions? What other info would be useful to help track down the
>> > issue.
>> >
>> > Regards,
>> > Richard
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mon crash after update to Hammer 0.94.3 from Firefly 0.80.10

[Richard Bade]

Thanks for your quick response Dan, but no. All the ceph-mon.*.log files
are empty.
I did track this down in syslog though, in case it helps:
ceph-mon: 2015-10-16 21:25:00.117115 7f4c9f458700 -1 *** Caught signal
(Segmentation fault) **#012 in thread 7f4c9f458700#012#012 ceph version
0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)#012 1:
/usr/bin/ceph-mon() [0x928b05]#012 2: (()+0xfcb0) [0x7f4ca50e0cb0]#012 3:
(get_str_map_key(std::map > const&,
std::string const&, std::string const*)+0x37) [0x87d8e7]#012 4:
(LogMonitor::update_from_paxos(bool*)+0x801) [0x6846e1]#012 5:
(PaxosService::refresh(bool*)+0x3c6) [0x5dc326]#012 6:
(Monitor::refresh_from_paxos(bool*)+0x36b) [0x588aab]#012 7:
(Paxos::do_refresh()+0x4c) [0x5c465c]#012 8:
(Paxos::handle_commit(MMonPaxos*)+0x243) [0x5cb2d3]#012 9:
(Paxos::dispatch(PaxosServiceMessage*)+0x22b) [0x5d3fbb]#012 10:
(Monitor::dispatch(MonSession*, Message*, bool)+0x864) [0x5ab0d4]#012 11:
(Monitor::_ms_dispatch(Message*)+0x2c9) [0x5a8a19]#012 12:
(Monitor::ms_dispatch(Message*)+0x32) [0x5c3952]#012 13:
(Messenger::ms_deliver_dispatch(Message*)+0x77) [0x8ac987]#012 14:
(DispatchQueue::entry()+0x44a) [0x8a9b2a]#012 15:
(DispatchQueue::DispatchThread::entry()+0xd) [0x79e4ad]#012 16: (()+0x7e9a)
[0x7f4ca50d8e9a]#012 17: (clone()+0x6d) [0x7f4ca3dca38d]#012 NOTE: a copy
of the executable, or `objdump -rdS ` is needed to interpret
this.

Regards,
Richard

On 17 October 2015 at 00:33, Dan van der Ster  wrote:

> Hi,
> Is there a backtrace in /var/log/ceph/ceph-mon.*.log ?
> Cheers, Dan
>
> On Fri, Oct 16, 2015 at 12:46 PM, Richard Bade  wrote:
> > Hi Everyone,
> > I upgraded our cluster to Hammer 0.94.3 a couple of days ago and today
> we've
> > had one monitor crash twice and another one once. We have 3 monitors
> total
> > and have been running Firefly 0.80.10 for quite some time without any
> > monitor issues.
> > When the monitor crashes it leaves a core file and a crash file in
> > /var/crash
> > I can't see anything obviously the same goolging about.
> > Has anyone seen anything like this?
> > Any suggestions? What other info would be useful to help track down the
> > issue.
> >
> > Regards,
> > Richard
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mon crash after update to Hammer 0.94.3 from Firefly 0.80.10

[Dan van der Ster]

Hi,
Is there a backtrace in /var/log/ceph/ceph-mon.*.log ?
Cheers, Dan

On Fri, Oct 16, 2015 at 12:46 PM, Richard Bade  wrote:
> Hi Everyone,
> I upgraded our cluster to Hammer 0.94.3 a couple of days ago and today we've
> had one monitor crash twice and another one once. We have 3 monitors total
> and have been running Firefly 0.80.10 for quite some time without any
> monitor issues.
> When the monitor crashes it leaves a core file and a crash file in
> /var/crash
> I can't see anything obviously the same goolging about.
> Has anyone seen anything like this?
> Any suggestions? What other info would be useful to help track down the
> issue.
>
> Regards,
> Richard
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Error after upgrading to Infernalis

[German Anders]

Hi all,

I'm trying to upgrade a ceph cluster (prev hammer release 0.94.3) to the
last release of *infernalis* (9.1.0-61-gf2b9f89). So far so good while
upgrading the mon servers, all work fine. But then when trying to upgrade
the OSD servers I got an error while trying to start the osd services again:

What I did is first to upgrade the packages, then stop the osd daemons, run
the chown -R ceph:ceph /var/lib/ceph command, and then try to start again
all the daemons. Well, they are not coming back and the error on one of the
OSD is the following:

(...)
5 10:21:05.910850
os/FileStore.cc: 1698: FAILED assert(r == 0)

 ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x8b) [0x7fec7b74489b]
 2: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
 3: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
 4: (OSD::init()+0x269) [0x7fec7b1bc2f9]
 5: (main()+0x2817) [0x7fec7b142bb7]
 6: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
 7: (()+0x30a9e7) [0x7fec7b1729e7]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed
to interpret this.

any ideas? find the complete log output:

http://pastebin.com/raw.php?i=zVABvJX3


Cheers,

*German* 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error after upgrading to Infernalis

[Robert LeBlanc]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

You need to make sure that you go through the 0.94.4 (not yet released
version) before the OSD will boot in the latest Infernalis. You can
get the packages from gitbuilder.ceph.com in the Hammer branch.
Install the packages (downgrade), and start up the OSDs, even if the
monitors are down it will still make the necessary changes, then
upgrade to Infernalis and chown the files again (some will get created
as root). To speed up the second chown, only look for files owned by
root (find /var/lib/ceph -user root -exec chown ceph. {} +)
- 
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Oct 16, 2015 at 7:41 AM, German Anders  wrote:
> Hi all,
>
> I'm trying to upgrade a ceph cluster (prev hammer release 0.94.3) to the
> last release of infernalis (9.1.0-61-gf2b9f89). So far so good while
> upgrading the mon servers, all work fine. But then when trying to upgrade
> the OSD servers I got an error while trying to start the osd services again:
>
> What I did is first to upgrade the packages, then stop the osd daemons, run
> the chown -R ceph:ceph /var/lib/ceph command, and then try to start again
> all the daemons. Well, they are not coming back and the error on one of the
> OSD is the following:
>
> (...)
> 5 10:21:05.910850
> os/FileStore.cc: 1698: FAILED assert(r == 0)
>
>  ceph version 9.1.0-61-gf2b9f89 (f2b9f898074db6473d993436e6aa566a945e3b40)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x8b) [0x7fec7b74489b]
>  2: (FileStore::init_temp_collections()+0xb2d) [0x7fec7b40ea9d]
>  3: (FileStore::mount()+0x33bb) [0x7fec7b41206b]
>  4: (OSD::init()+0x269) [0x7fec7b1bc2f9]
>  5: (main()+0x2817) [0x7fec7b142bb7]
>  6: (__libc_start_main()+0xf5) [0x7fec77d68ec5]
>  7: (()+0x30a9e7) [0x7fec7b1729e7]
>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.
>
> any ideas? find the complete log output:
>
> http://pastebin.com/raw.php?i=zVABvJX3
>
>
> Cheers,
>
> German
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-BEGIN PGP SIGNATURE-
Version: Mailvelope v1.2.0
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJWIRnLCRDmVDuy+mK58QAA/CMQAJOvA7sK2HY+JDYmUDVk
2k+wtj2K0gYOYvdQrhUWNZauBasXc5s44iAltOfaS5eRyTIklhfjOqGtwjVR
aVbBJKorFz5JKyp3x9fFLEre76AmPBT9BrCka+UEVrtavh9KZKfzK5xjrud1
WXRWoqE/QkbcLl96nP2kHg1XiEtg7lDRoPKWN+UKq5OU7xHhKO5jUv11vRfX
6FIsSmgcozirI88onbScvHRKYOigQNRZkvnloHgh6LzsBm5la4HTpwrmoHZu
YNnkvH6igC6lFcQ2ZMUoV82v8jIwi1lGJei0mDzT90jVECrpVayftEuwivnl
oRFiKy+7BDAvudOfaaqSq72Cd5poDXuuExNIiZQiyXBjSm2oDZmJfYZEceJf
USzeL+04hu2UatiGGqJXGu2O+d91SYJd24s4tsbj0B6NueD5p50LA371170y
a1uA90nde2wNz3dQHQ/JBwWbDNkaxcT7X1ghBCUWK5skTfOzrCP34xfODDet
l/vmhzuaGQ9w9oVyfNuP2VrUS5tFxL2oQ7rTaBvnyID4C//0ErP8wFJ4nUu/
BE5Te8mFIz4+yGVtd61h19R00YeYUlvFGnZMvmPWNUKYlWl5FkuoABvdSGen
WFaf8Z1GEmSShtL87n5nMHQRSSq4RoFOMOk/+9o+OVfROrWWyRNHmGUuszxB
lkb+
=XD35
-END PGP SIGNATURE-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache Tiering Question

[Robert LeBlanc]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

OK, I've set this up and now all I/O is locked up. I've reduced
target_max_bytes because one OSD was reporting 97% usage, there was
some I/O for a few seconds as things flushed, but client I/O is still
blocked. Anyone have some thoughts?

ceph osd crush rule create-simple ssd-tier ssd host firstn
ceph osd pool create ssd-pool 128 replicated ssd-tier
ceph osd tier add rbd ssd-pool
ceph osd tier cache-mode ssd-pool writeback
ceph osd tier set-overlay rbd ssd-pool
ceph osd pool set ssd-pool hit_set_type bloom
ceph osd pool set ssd-pool hit_set_count 6
ceph osd pool set ssd-pool hit_set_period 600
ceph osd pool set ssd-pool min_read_recency_for_promote 6
ceph osd pool set ssd-pool cache_target_dirty_ratio 0.4
ceph osd pool set ssd-pool cache_target_full_ratio 0.8
ceph osd pool set ssd-pool target_max_bytes 795642691584

ceph version 0.94.3-252-g629b631 (629b631488f044150422371ac77dfc005f3de1bc)

# ceph status
cluster 050309fd-723e-42aa-9624-3b3e033ab359
 health HEALTH_OK
 monmap e1: 1 mons at {nodez=192.168.55.15:6789/0}
election epoch 2, quorum 0 nodez
 osdmap e1333: 18 osds: 18 up, 18 in
  pgmap v87157: 384 pgs, 2 pools, 3326 GB data, 1368 kobjects
3010 GB used, 20262 GB / 24518 GB avail
 384 active+clean

# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
18 0.2  1.0   208G   135G 64764M 64.63 4.77
 5 0.20999  1.0   210G   181G 18392M 86.36 6.38
19 0.21999  1.0   208G   161G 37941M 77.17 5.70
10 0.18999  1.0   210G   167G 32712M 79.70 5.89
 7 0.20999  1.0   210G   181G 18405M 86.35 6.38
20 0.2  1.0   208G   119G 80247M 57.39 4.24
22 0.2  1.0   208G 87596M   112G 40.95 3.02
 8 0.20999  1.0   210G   170G 29422M 81.23 6.00
23 0.20999  1.0   208G   151G 47404M 72.75 5.37
 1 0.20999  1.0   210G   105G 96245M 50.17 3.71
 6 0.20999  1.0   210G   131G 69937M 62.40 4.61
21 0.2  1.0   208G   192G  5667M 92.26 6.81
 0 3.64000  1.0  3667G   231G  3249G  6.32 0.47
 9 3.57999  1.0  3667G   262G  3219G  7.15 0.53
 2 3.64000  1.0  3667G   273G  3207G  7.47 0.55
 3 3.64000  1.0  3667G   256G  3224G  6.99 0.52
 4 3.64000  1.0  3667G   239G  3241G  6.54 0.48
24 3.57999  1.0  3667G   272G  3208G  7.42 0.55
  TOTAL 24518G  3320G 19952G 13.54
MIN/MAX VAR: 0.47/6.81  STDDEV: 48.64

After dropping target_max_bytes to 644470580183:
# ceph df
GLOBAL:
SIZE   AVAIL  RAW USED %RAW USED
24518G 20241G3031G 12.36
POOLS:
NAME ID USED  %USED MAX AVAIL OBJECTS
rbd  0  2856G 11.65 6379G 1158862
ssd-pool 3   470G  1.92  162G  242140
# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USE AVAIL   %USE  VAR
18 0.2  1.0   208G116G  83987M 55.64 4.50
 5 0.20999  1.0   210G151G  49392M 71.95 5.82
19 0.21999  1.0   208G134G  65792M 64.15 5.19
10 0.18999  1.0   210G138G  61961M 66.11 5.35
 7 0.20999  1.0   210G149G  50672M 71.36 5.77
20 0.2  1.0   208G 101842M 101167M 47.61 3.85
22 0.2  1.0   208G  72511M127G 33.90 2.74
 8 0.20999  1.0   210G145G  55381M 69.17 5.59
23 0.20999  1.0   208G127G  72305M 61.11 4.94
 1 0.20999  1.0   210G  95656M105G 44.46 3.60
 6 0.20999  1.0   210G109G  92154M 52.07 4.21
21 0.2  1.0   208G158G  40521M 75.97 6.14
 0 3.64000  1.0  3667G231G   3249G  6.32 0.51
 9 3.57999  1.0  3667G262G   3219G  7.15 0.58
 2 3.64000  1.0  3667G273G   3207G  7.47 0.60
 3 3.64000  1.0  3667G256G   3224G  6.99 0.57
 4 3.64000  1.0  3667G239G   3241G  6.54 0.53
24 3.57999  1.0  3667G272G   3208G  7.42 0.60
  TOTAL 24518G   3031G  20241G 12.36
MIN/MAX VAR: 0.51/6.14  STDDEV: 39.87

# ceph osd tree
ID  WEIGHT   TYPE NAME   UP/DOWN REWEIGHT PRIMARY-AFFINITY
 -9  2.46991 root ssd
 -8  0.40999 host nodew-ssd
 18  0.2 osd.18   up  1.0  1.0
  5  0.20999 osd.5up  1.0  1.0
- -10  0.40997 host nodev-ssd
 19  0.21999 osd.19   up  1.0  1.0
 10  0.18999 osd.10   up  1.0  1.0
- -11  0.40999 host nodezz-ssd
  7  0.20999 osd.7up  1.0  1.0
 20  0.2 osd.20   up  1.0  1.0
- -12  0.40999 host nodey-ssd
 22  0.2 osd.22   up  1.0  1.0
  8  0.20999 osd.8up  1.0  1.0
- -13  0.41998 host nodex-ssd
 23  0.20999 osd.23   up  1.0  1.0
  1  0.20999 osd.1up  1.0  1.0
- -14  0.40999 host nodez-ssd
  6  0.20999 osd.6up  1.0  1.0
 21  0.2 osd.21  

Re: [ceph-users] Cache Tiering Question

[Robert LeBlanc]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Is the only option to restart the librbd client in this case? Anything
I can do to help resolve it?

Thanks,
- 
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Oct 16, 2015 at 10:17 AM, Sage Weil  wrote:
> On Fri, 16 Oct 2015, Robert LeBlanc wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>> I started another fio test to one of the same RBDs (leaving the hung
>> ones still hung) and it is working OK, but the hungs ones are still
>> just hung.
>
> There is a full-disk failsafe that is still somewhat buggy that could
> explain the hung requests (if they were writes and submitted while the
> osd(s) were near full).
>
> sage
>
>
>> - 
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>
>>
>> On Fri, Oct 16, 2015 at 10:00 AM, Robert LeBlanc  wrote:
>> > -BEGIN PGP SIGNED MESSAGE-
>> > Hash: SHA256
>> >
>> > OK, I've set this up and now all I/O is locked up. I've reduced
>> > target_max_bytes because one OSD was reporting 97% usage, there was
>> > some I/O for a few seconds as things flushed, but client I/O is still
>> > blocked. Anyone have some thoughts?
>> >
>> > ceph osd crush rule create-simple ssd-tier ssd host firstn
>> > ceph osd pool create ssd-pool 128 replicated ssd-tier
>> > ceph osd tier add rbd ssd-pool
>> > ceph osd tier cache-mode ssd-pool writeback
>> > ceph osd tier set-overlay rbd ssd-pool
>> > ceph osd pool set ssd-pool hit_set_type bloom
>> > ceph osd pool set ssd-pool hit_set_count 6
>> > ceph osd pool set ssd-pool hit_set_period 600
>> > ceph osd pool set ssd-pool min_read_recency_for_promote 6
>> > ceph osd pool set ssd-pool cache_target_dirty_ratio 0.4
>> > ceph osd pool set ssd-pool cache_target_full_ratio 0.8
>> > ceph osd pool set ssd-pool target_max_bytes 795642691584
>> >
>> > ceph version 0.94.3-252-g629b631 (629b631488f044150422371ac77dfc005f3de1bc)
>> >
>> > # ceph status
>> > cluster 050309fd-723e-42aa-9624-3b3e033ab359
>> >  health HEALTH_OK
>> >  monmap e1: 1 mons at {nodez=192.168.55.15:6789/0}
>> > election epoch 2, quorum 0 nodez
>> >  osdmap e1333: 18 osds: 18 up, 18 in
>> >   pgmap v87157: 384 pgs, 2 pools, 3326 GB data, 1368 kobjects
>> > 3010 GB used, 20262 GB / 24518 GB avail
>> >  384 active+clean
>> >
>> > # ceph osd df
>> > ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>> > 18 0.2  1.0   208G   135G 64764M 64.63 4.77
>> >  5 0.20999  1.0   210G   181G 18392M 86.36 6.38
>> > 19 0.21999  1.0   208G   161G 37941M 77.17 5.70
>> > 10 0.18999  1.0   210G   167G 32712M 79.70 5.89
>> >  7 0.20999  1.0   210G   181G 18405M 86.35 6.38
>> > 20 0.2  1.0   208G   119G 80247M 57.39 4.24
>> > 22 0.2  1.0   208G 87596M   112G 40.95 3.02
>> >  8 0.20999  1.0   210G   170G 29422M 81.23 6.00
>> > 23 0.20999  1.0   208G   151G 47404M 72.75 5.37
>> >  1 0.20999  1.0   210G   105G 96245M 50.17 3.71
>> >  6 0.20999  1.0   210G   131G 69937M 62.40 4.61
>> > 21 0.2  1.0   208G   192G  5667M 92.26 6.81
>> >  0 3.64000  1.0  3667G   231G  3249G  6.32 0.47
>> >  9 3.57999  1.0  3667G   262G  3219G  7.15 0.53
>> >  2 3.64000  1.0  3667G   273G  3207G  7.47 0.55
>> >  3 3.64000  1.0  3667G   256G  3224G  6.99 0.52
>> >  4 3.64000  1.0  3667G   239G  3241G  6.54 0.48
>> > 24 3.57999  1.0  3667G   272G  3208G  7.42 0.55
>> >   TOTAL 24518G  3320G 19952G 13.54
>> > MIN/MAX VAR: 0.47/6.81  STDDEV: 48.64
>> >
>> > After dropping target_max_bytes to 644470580183:
>> > # ceph df
>> > GLOBAL:
>> > SIZE   AVAIL  RAW USED %RAW USED
>> > 24518G 20241G3031G 12.36
>> > POOLS:
>> > NAME ID USED  %USED MAX AVAIL OBJECTS
>> > rbd  0  2856G 11.65 6379G 1158862
>> > ssd-pool 3   470G  1.92  162G  242140
>> > # ceph osd df
>> > ID WEIGHT  REWEIGHT SIZE   USE AVAIL   %USE  VAR
>> > 18 0.2  1.0   208G116G  83987M 55.64 4.50
>> >  5 0.20999  1.0   210G151G  49392M 71.95 5.82
>> > 19 0.21999  1.0   208G134G  65792M 64.15 5.19
>> > 10 0.18999  1.0   210G138G  61961M 66.11 5.35
>> >  7 0.20999  1.0   210G149G  50672M 71.36 5.77
>> > 20 0.2  1.0   208G 101842M 101167M 47.61 3.85
>> > 22 0.2  1.0   208G  72511M127G 33.90 2.74
>> >  8 0.20999  1.0   210G145G  55381M 69.17 5.59
>> > 23 0.20999  1.0   208G127G  72305M 61.11 4.94
>> >  1 0.20999  1.0   210G  95656M105G 44.46 3.60
>> >  6 0.20999  1.0   210G109G  92154M 52.07 4.21
>> > 21 0.2  1.0   208G158G  40521M 75.97 6.14
>> >  0 3.64000  1.0  3667G231G   3249G  6.32 0.51
>> >  9 3.57999  1.0  3667G262G   3219G  7.15 0.58
>> >  2 3.64000  1.0  3667G273G   3207G  

Re: [ceph-users] Cache Tiering Question

[Robert LeBlanc]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

I started another fio test to one of the same RBDs (leaving the hung
ones still hung) and it is working OK, but the hungs ones are still
just hung.
- 
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Fri, Oct 16, 2015 at 10:00 AM, Robert LeBlanc  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> OK, I've set this up and now all I/O is locked up. I've reduced
> target_max_bytes because one OSD was reporting 97% usage, there was
> some I/O for a few seconds as things flushed, but client I/O is still
> blocked. Anyone have some thoughts?
>
> ceph osd crush rule create-simple ssd-tier ssd host firstn
> ceph osd pool create ssd-pool 128 replicated ssd-tier
> ceph osd tier add rbd ssd-pool
> ceph osd tier cache-mode ssd-pool writeback
> ceph osd tier set-overlay rbd ssd-pool
> ceph osd pool set ssd-pool hit_set_type bloom
> ceph osd pool set ssd-pool hit_set_count 6
> ceph osd pool set ssd-pool hit_set_period 600
> ceph osd pool set ssd-pool min_read_recency_for_promote 6
> ceph osd pool set ssd-pool cache_target_dirty_ratio 0.4
> ceph osd pool set ssd-pool cache_target_full_ratio 0.8
> ceph osd pool set ssd-pool target_max_bytes 795642691584
>
> ceph version 0.94.3-252-g629b631 (629b631488f044150422371ac77dfc005f3de1bc)
>
> # ceph status
> cluster 050309fd-723e-42aa-9624-3b3e033ab359
>  health HEALTH_OK
>  monmap e1: 1 mons at {nodez=192.168.55.15:6789/0}
> election epoch 2, quorum 0 nodez
>  osdmap e1333: 18 osds: 18 up, 18 in
>   pgmap v87157: 384 pgs, 2 pools, 3326 GB data, 1368 kobjects
> 3010 GB used, 20262 GB / 24518 GB avail
>  384 active+clean
>
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
> 18 0.2  1.0   208G   135G 64764M 64.63 4.77
>  5 0.20999  1.0   210G   181G 18392M 86.36 6.38
> 19 0.21999  1.0   208G   161G 37941M 77.17 5.70
> 10 0.18999  1.0   210G   167G 32712M 79.70 5.89
>  7 0.20999  1.0   210G   181G 18405M 86.35 6.38
> 20 0.2  1.0   208G   119G 80247M 57.39 4.24
> 22 0.2  1.0   208G 87596M   112G 40.95 3.02
>  8 0.20999  1.0   210G   170G 29422M 81.23 6.00
> 23 0.20999  1.0   208G   151G 47404M 72.75 5.37
>  1 0.20999  1.0   210G   105G 96245M 50.17 3.71
>  6 0.20999  1.0   210G   131G 69937M 62.40 4.61
> 21 0.2  1.0   208G   192G  5667M 92.26 6.81
>  0 3.64000  1.0  3667G   231G  3249G  6.32 0.47
>  9 3.57999  1.0  3667G   262G  3219G  7.15 0.53
>  2 3.64000  1.0  3667G   273G  3207G  7.47 0.55
>  3 3.64000  1.0  3667G   256G  3224G  6.99 0.52
>  4 3.64000  1.0  3667G   239G  3241G  6.54 0.48
> 24 3.57999  1.0  3667G   272G  3208G  7.42 0.55
>   TOTAL 24518G  3320G 19952G 13.54
> MIN/MAX VAR: 0.47/6.81  STDDEV: 48.64
>
> After dropping target_max_bytes to 644470580183:
> # ceph df
> GLOBAL:
> SIZE   AVAIL  RAW USED %RAW USED
> 24518G 20241G3031G 12.36
> POOLS:
> NAME ID USED  %USED MAX AVAIL OBJECTS
> rbd  0  2856G 11.65 6379G 1158862
> ssd-pool 3   470G  1.92  162G  242140
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USE AVAIL   %USE  VAR
> 18 0.2  1.0   208G116G  83987M 55.64 4.50
>  5 0.20999  1.0   210G151G  49392M 71.95 5.82
> 19 0.21999  1.0   208G134G  65792M 64.15 5.19
> 10 0.18999  1.0   210G138G  61961M 66.11 5.35
>  7 0.20999  1.0   210G149G  50672M 71.36 5.77
> 20 0.2  1.0   208G 101842M 101167M 47.61 3.85
> 22 0.2  1.0   208G  72511M127G 33.90 2.74
>  8 0.20999  1.0   210G145G  55381M 69.17 5.59
> 23 0.20999  1.0   208G127G  72305M 61.11 4.94
>  1 0.20999  1.0   210G  95656M105G 44.46 3.60
>  6 0.20999  1.0   210G109G  92154M 52.07 4.21
> 21 0.2  1.0   208G158G  40521M 75.97 6.14
>  0 3.64000  1.0  3667G231G   3249G  6.32 0.51
>  9 3.57999  1.0  3667G262G   3219G  7.15 0.58
>  2 3.64000  1.0  3667G273G   3207G  7.47 0.60
>  3 3.64000  1.0  3667G256G   3224G  6.99 0.57
>  4 3.64000  1.0  3667G239G   3241G  6.54 0.53
> 24 3.57999  1.0  3667G272G   3208G  7.42 0.60
>   TOTAL 24518G   3031G  20241G 12.36
> MIN/MAX VAR: 0.51/6.14  STDDEV: 39.87
>
> # ceph osd tree
> ID  WEIGHT   TYPE NAME   UP/DOWN REWEIGHT PRIMARY-AFFINITY
>  -9  2.46991 root ssd
>  -8  0.40999 host nodew-ssd
>  18  0.2 osd.18   up  1.0  1.0
>   5  0.20999 osd.5up  1.0  1.0
> - -10  0.40997 host nodev-ssd
>  19  0.21999 osd.19   up  1.0  1.0
>  10  0.18999 osd.10   up  1.0  1.0
> - -11  0.40999 host nodezz-ssd
>   7  0.20999 osd.7up  1.0  

Re: [ceph-users] Cache Tiering Question

[Sage Weil]

On Fri, 16 Oct 2015, Robert LeBlanc wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> I started another fio test to one of the same RBDs (leaving the hung
> ones still hung) and it is working OK, but the hungs ones are still
> just hung.

There is a full-disk failsafe that is still somewhat buggy that could 
explain the hung requests (if they were writes and submitted while the 
osd(s) were near full).

sage


> - 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> 
> 
> On Fri, Oct 16, 2015 at 10:00 AM, Robert LeBlanc  wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > OK, I've set this up and now all I/O is locked up. I've reduced
> > target_max_bytes because one OSD was reporting 97% usage, there was
> > some I/O for a few seconds as things flushed, but client I/O is still
> > blocked. Anyone have some thoughts?
> >
> > ceph osd crush rule create-simple ssd-tier ssd host firstn
> > ceph osd pool create ssd-pool 128 replicated ssd-tier
> > ceph osd tier add rbd ssd-pool
> > ceph osd tier cache-mode ssd-pool writeback
> > ceph osd tier set-overlay rbd ssd-pool
> > ceph osd pool set ssd-pool hit_set_type bloom
> > ceph osd pool set ssd-pool hit_set_count 6
> > ceph osd pool set ssd-pool hit_set_period 600
> > ceph osd pool set ssd-pool min_read_recency_for_promote 6
> > ceph osd pool set ssd-pool cache_target_dirty_ratio 0.4
> > ceph osd pool set ssd-pool cache_target_full_ratio 0.8
> > ceph osd pool set ssd-pool target_max_bytes 795642691584
> >
> > ceph version 0.94.3-252-g629b631 (629b631488f044150422371ac77dfc005f3de1bc)
> >
> > # ceph status
> > cluster 050309fd-723e-42aa-9624-3b3e033ab359
> >  health HEALTH_OK
> >  monmap e1: 1 mons at {nodez=192.168.55.15:6789/0}
> > election epoch 2, quorum 0 nodez
> >  osdmap e1333: 18 osds: 18 up, 18 in
> >   pgmap v87157: 384 pgs, 2 pools, 3326 GB data, 1368 kobjects
> > 3010 GB used, 20262 GB / 24518 GB avail
> >  384 active+clean
> >
> > # ceph osd df
> > ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
> > 18 0.2  1.0   208G   135G 64764M 64.63 4.77
> >  5 0.20999  1.0   210G   181G 18392M 86.36 6.38
> > 19 0.21999  1.0   208G   161G 37941M 77.17 5.70
> > 10 0.18999  1.0   210G   167G 32712M 79.70 5.89
> >  7 0.20999  1.0   210G   181G 18405M 86.35 6.38
> > 20 0.2  1.0   208G   119G 80247M 57.39 4.24
> > 22 0.2  1.0   208G 87596M   112G 40.95 3.02
> >  8 0.20999  1.0   210G   170G 29422M 81.23 6.00
> > 23 0.20999  1.0   208G   151G 47404M 72.75 5.37
> >  1 0.20999  1.0   210G   105G 96245M 50.17 3.71
> >  6 0.20999  1.0   210G   131G 69937M 62.40 4.61
> > 21 0.2  1.0   208G   192G  5667M 92.26 6.81
> >  0 3.64000  1.0  3667G   231G  3249G  6.32 0.47
> >  9 3.57999  1.0  3667G   262G  3219G  7.15 0.53
> >  2 3.64000  1.0  3667G   273G  3207G  7.47 0.55
> >  3 3.64000  1.0  3667G   256G  3224G  6.99 0.52
> >  4 3.64000  1.0  3667G   239G  3241G  6.54 0.48
> > 24 3.57999  1.0  3667G   272G  3208G  7.42 0.55
> >   TOTAL 24518G  3320G 19952G 13.54
> > MIN/MAX VAR: 0.47/6.81  STDDEV: 48.64
> >
> > After dropping target_max_bytes to 644470580183:
> > # ceph df
> > GLOBAL:
> > SIZE   AVAIL  RAW USED %RAW USED
> > 24518G 20241G3031G 12.36
> > POOLS:
> > NAME ID USED  %USED MAX AVAIL OBJECTS
> > rbd  0  2856G 11.65 6379G 1158862
> > ssd-pool 3   470G  1.92  162G  242140
> > # ceph osd df
> > ID WEIGHT  REWEIGHT SIZE   USE AVAIL   %USE  VAR
> > 18 0.2  1.0   208G116G  83987M 55.64 4.50
> >  5 0.20999  1.0   210G151G  49392M 71.95 5.82
> > 19 0.21999  1.0   208G134G  65792M 64.15 5.19
> > 10 0.18999  1.0   210G138G  61961M 66.11 5.35
> >  7 0.20999  1.0   210G149G  50672M 71.36 5.77
> > 20 0.2  1.0   208G 101842M 101167M 47.61 3.85
> > 22 0.2  1.0   208G  72511M127G 33.90 2.74
> >  8 0.20999  1.0   210G145G  55381M 69.17 5.59
> > 23 0.20999  1.0   208G127G  72305M 61.11 4.94
> >  1 0.20999  1.0   210G  95656M105G 44.46 3.60
> >  6 0.20999  1.0   210G109G  92154M 52.07 4.21
> > 21 0.2  1.0   208G158G  40521M 75.97 6.14
> >  0 3.64000  1.0  3667G231G   3249G  6.32 0.51
> >  9 3.57999  1.0  3667G262G   3219G  7.15 0.58
> >  2 3.64000  1.0  3667G273G   3207G  7.47 0.60
> >  3 3.64000  1.0  3667G256G   3224G  6.99 0.57
> >  4 3.64000  1.0  3667G239G   3241G  6.54 0.53
> > 24 3.57999  1.0  3667G272G   3208G  7.42 0.60
> >   TOTAL 24518G   3031G  20241G 12.36
> > MIN/MAX VAR: 0.51/6.14  STDDEV: 39.87
> >
> > # ceph osd tree
> > ID  WEIGHT   TYPE NAME   UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >  -9  2.46991 root ssd
> >  -8  0.40999

Re: [ceph-users] Cache Tiering Question

[Sage Weil]

On Fri, 16 Oct 2015, Robert LeBlanc wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> Is the only option to restart the librbd client in this case? Anything
> I can do to help resolve it?

If you know which OSD the request is outstanding against (ceph daemon 
 objecter_requests) you can mark that osd down.. that should 
make the client resend.

sage


> 
> Thanks,
> - 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> 
> 
> On Fri, Oct 16, 2015 at 10:17 AM, Sage Weil  wrote:
> > On Fri, 16 Oct 2015, Robert LeBlanc wrote:
> >> -BEGIN PGP SIGNED MESSAGE-
> >> Hash: SHA256
> >>
> >> I started another fio test to one of the same RBDs (leaving the hung
> >> ones still hung) and it is working OK, but the hungs ones are still
> >> just hung.
> >
> > There is a full-disk failsafe that is still somewhat buggy that could
> > explain the hung requests (if they were writes and submitted while the
> > osd(s) were near full).
> >
> > sage
> >
> >
> >> - 
> >> Robert LeBlanc
> >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> >>
> >>
> >> On Fri, Oct 16, 2015 at 10:00 AM, Robert LeBlanc  wrote:
> >> > -BEGIN PGP SIGNED MESSAGE-
> >> > Hash: SHA256
> >> >
> >> > OK, I've set this up and now all I/O is locked up. I've reduced
> >> > target_max_bytes because one OSD was reporting 97% usage, there was
> >> > some I/O for a few seconds as things flushed, but client I/O is still
> >> > blocked. Anyone have some thoughts?
> >> >
> >> > ceph osd crush rule create-simple ssd-tier ssd host firstn
> >> > ceph osd pool create ssd-pool 128 replicated ssd-tier
> >> > ceph osd tier add rbd ssd-pool
> >> > ceph osd tier cache-mode ssd-pool writeback
> >> > ceph osd tier set-overlay rbd ssd-pool
> >> > ceph osd pool set ssd-pool hit_set_type bloom
> >> > ceph osd pool set ssd-pool hit_set_count 6
> >> > ceph osd pool set ssd-pool hit_set_period 600
> >> > ceph osd pool set ssd-pool min_read_recency_for_promote 6
> >> > ceph osd pool set ssd-pool cache_target_dirty_ratio 0.4
> >> > ceph osd pool set ssd-pool cache_target_full_ratio 0.8
> >> > ceph osd pool set ssd-pool target_max_bytes 795642691584
> >> >
> >> > ceph version 0.94.3-252-g629b631 
> >> > (629b631488f044150422371ac77dfc005f3de1bc)
> >> >
> >> > # ceph status
> >> > cluster 050309fd-723e-42aa-9624-3b3e033ab359
> >> >  health HEALTH_OK
> >> >  monmap e1: 1 mons at {nodez=192.168.55.15:6789/0}
> >> > election epoch 2, quorum 0 nodez
> >> >  osdmap e1333: 18 osds: 18 up, 18 in
> >> >   pgmap v87157: 384 pgs, 2 pools, 3326 GB data, 1368 kobjects
> >> > 3010 GB used, 20262 GB / 24518 GB avail
> >> >  384 active+clean
> >> >
> >> > # ceph osd df
> >> > ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
> >> > 18 0.2  1.0   208G   135G 64764M 64.63 4.77
> >> >  5 0.20999  1.0   210G   181G 18392M 86.36 6.38
> >> > 19 0.21999  1.0   208G   161G 37941M 77.17 5.70
> >> > 10 0.18999  1.0   210G   167G 32712M 79.70 5.89
> >> >  7 0.20999  1.0   210G   181G 18405M 86.35 6.38
> >> > 20 0.2  1.0   208G   119G 80247M 57.39 4.24
> >> > 22 0.2  1.0   208G 87596M   112G 40.95 3.02
> >> >  8 0.20999  1.0   210G   170G 29422M 81.23 6.00
> >> > 23 0.20999  1.0   208G   151G 47404M 72.75 5.37
> >> >  1 0.20999  1.0   210G   105G 96245M 50.17 3.71
> >> >  6 0.20999  1.0   210G   131G 69937M 62.40 4.61
> >> > 21 0.2  1.0   208G   192G  5667M 92.26 6.81
> >> >  0 3.64000  1.0  3667G   231G  3249G  6.32 0.47
> >> >  9 3.57999  1.0  3667G   262G  3219G  7.15 0.53
> >> >  2 3.64000  1.0  3667G   273G  3207G  7.47 0.55
> >> >  3 3.64000  1.0  3667G   256G  3224G  6.99 0.52
> >> >  4 3.64000  1.0  3667G   239G  3241G  6.54 0.48
> >> > 24 3.57999  1.0  3667G   272G  3208G  7.42 0.55
> >> >   TOTAL 24518G  3320G 19952G 13.54
> >> > MIN/MAX VAR: 0.47/6.81  STDDEV: 48.64
> >> >
> >> > After dropping target_max_bytes to 644470580183:
> >> > # ceph df
> >> > GLOBAL:
> >> > SIZE   AVAIL  RAW USED %RAW USED
> >> > 24518G 20241G3031G 12.36
> >> > POOLS:
> >> > NAME ID USED  %USED MAX AVAIL OBJECTS
> >> > rbd  0  2856G 11.65 6379G 1158862
> >> > ssd-pool 3   470G  1.92  162G  242140
> >> > # ceph osd df
> >> > ID WEIGHT  REWEIGHT SIZE   USE AVAIL   %USE  VAR
> >> > 18 0.2  1.0   208G116G  83987M 55.64 4.50
> >> >  5 0.20999  1.0   210G151G  49392M 71.95 5.82
> >> > 19 0.21999  1.0   208G134G  65792M 64.15 5.19
> >> > 10 0.18999  1.0   210G138G  61961M 66.11 5.35
> >> >  7 0.20999  1.0   210G149G  50672M 71.36 5.77
> >> > 20 0.2  1.0   208G 101842M 101167M 47.61 3.85
> >> > 22 0.2  1.0   208G  72511M127G 33.90 2.74
> >> >  8 0.20999  1.0