Re: [ceph-users] CephFS is not maintianing conistency

2016-02-01 Thread Yan, Zheng 
On Tue, Feb 2, 2016 at 2:27 AM, Mykola Dvornik  wrote:
> What version are you running on your servers and clients?
>

Are you using 4.1 or 4.2 kernel?
https://bugzilla.kernel.org/show_bug.cgi?id=104911. Upgrade to 4.3+
kernel or 4.1.17 kernel or 4.2.8 kernel can resolve this issue.

>
> On the clients:
>
> ceph-fuse --version
> ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
>
> MDS/OSD/MON:
>
> ceph --version
> ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
>
>  Exactly what changes are you making that aren't visible?
>
>
> I am creating some new files in non-root folders.
>
> What's the output of "ceph -s"?
>
>
> ceph -s
>
> cluster 98d72518-6619-4b5c-b148-9a781ef13bcb
>  health HEALTH_OK
>  monmap e1: 1 mons at {000-s-ragnarok=XXX.XXX.XXX.XXX:6789/0}
> election epoch 1, quorum 0 000-s-ragnarok
>  mdsmap e576: 1/1/1 up {0=000-s-ragnarok=up:active}
>  osdmap e233: 16 osds: 16 up, 16 in
> flags sortbitwise
>   pgmap v1927636: 1088 pgs, 2 pools, 1907 GB data, 2428 kobjects
> 3844 GB used, 25949 GB / 29793 GB avail
> 1088 active+clean
>   client io 4381 B/s wr, 2 op
>
> In addition on the clients' side I have
>
> cat /etc/fuse.conf
>
> user_allow_other
> auto_cache
> large_read
> max_write = 16777216
> max_read = 16777216
>
>
> -Mykola
>
>
> On Mon, Feb 1, 2016 at 5:06 PM, Gregory Farnum  wrote:
>
> On Monday, February 1, 2016, Mykola Dvornik 
> wrote:
>>
>> Hi guys,
>>
>> This is sort of rebuttal.
>>
>> I have a CephFS deployed and mounted on a couple of clients via ceph-fuse
>> (due to quota support and possibility to kill the ceph-fuse process to avoid
>> stale mounts).
>>
>> So the problems is that some times the changes made on one client are not
>> visible on the others. It appears to me as rather random process. The only
>> solution is to touch a new file in any particular folder that apparently
>> triggers synchronization.
>>
>> I've been using a kernel-side client before with no such kind of problems.
>> So the questions is it expected behavior of ceph-fuse?
>
>
> What version are you running on your servers and clients? Exactly what
> changes are you making that aren't visible? What's the output of "ceph -s"?
> We see bugs like this occasionally but I can't think of any recent ones in
> ceph-fuse -- they're actually seen a lot more often in the kernel client.
> -Greg
>
>
>>
>>
>> Regards,
>>
>> Mykola
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrading Ceph

2016-02-01 Thread david 
Hi Vlad,
I just upgraded my ceph cluster from firefly to hammer and all right. 
Please do it according to the manuals in www.ceph.com  
and restart monitors and then osds. I restarted osds one by one, which means 
restart one OSD and waits for it runs normal and then restart the other OSD and 
so on.


> On Feb 2, 2016, at 09:10, Vlad Blando  wrote:
> 
> What if the upgrade fails, what is the rollback scenario?
> 
> 
> 
> On Wed, Jan 27, 2016 at 10:10 PM,  > wrote:
> I just upgraded from firefly to infernalis (firefly to hammer to
> infernalis) my cluster
> All came like a charm
> I upgraded mon first, then the osd, one by one, restarting the daemon
> after each upgrade
> Remember to change uid of your file, as the osd daemon now run under the
> user ceph (and not root): can be pretty long operation .. do the chown,
> then upgrade, then shut the daemon, rechown the remaining files, start
> the daemon (use chown --from=0:0 ceph:ceph -R /var/lib/ceph/osd/ceph-2
> to speed up the last chown)
> 
> To conclude, I upgraded the whole cluster without a single downtime
> (pretty surprised, didn't expect the process to be "that" robust)
> 
> On 27/01/2016 15:00, Vlad Blando wrote:
> > Hi,
> >
> > I have a production Ceph Cluster
> > - 3 nodes
> > - 3 mons on each nodes
> > - 9 OSD @ 4TB per node
> > - using ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
> >
> > ​Now I want to upgrade it to Hammer, I saw the documentation on upgrading,
> > it looks straight forward, but I want to know to those who have tried
> > upgrading a production environment, any precautions, caveats, preparation
> > that I need to do before doing it?
> >
> > - Vlad
> > ᐧ
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> > 
> >
> 
> 
> ᐧ
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPHFS: standby-replay mds crash

2016-02-01 Thread Yan, Zheng 
On Tue, Feb 2, 2016 at 12:52 PM, Goncalo Borges
 wrote:
> Hi...
>
> Seems very similar to
>
> http://tracker.ceph.com/issues/14144
>
> Can you confirm it is the same issue?

should be the same. it's fixed by https://github.com/ceph/ceph/pull/7132

Regards
Yan, Zheng

>
> Cheers
> G.
>
>
>
> 
> From: Goncalo Borges
> Sent: 02 February 2016 15:30
> To: ceph-us...@ceph.com
> Cc: rct...@coepp.org.au
> Subject: CEPHFS: standby-replay mds crash
>
> Hi CephFS experts.
>
> 1./ We are using Ceph and CephFS 9.2.0 with an active mds and a
> standby-replay mds (standard config)
>
> # ceph -s
> cluster 
>  health HEALTH_OK
>  monmap e1: 3 mons at
> {mon1=:6789/0,mon2=:6789/0,mon3=:6789/0}
> election epoch 98, quorum 0,1,2 mon1,mon3,mon2
>  mdsmap e102: 1/1/1 up {0=mds2=up:active}, 1 up:standby-replay
>  osdmap e689: 64 osds: 64 up, 64 in
> flags sortbitwise
>   pgmap v2006627: 3072 pgs, 3 pools, 106 GB data, 85605 objects
> 323 GB used, 174 TB / 174 TB avail
> 3072 active+clean
>   client io 1191 B/s rd, 2 op/s
>
>
>
> 2./ Today, the standby-replay mds crashed but the active mds continued ok.
> The logs (following this email) show a problem creating a thread.
>
> 3./ Our ganglia monitoring shows:
>
> - a tremendous increase of load in the system
> - a tremendous peak of network connectivity for inbound traffic
> - No excessive memory usage nor excessive number of processes running.
>
> 4./ For now, we just restarted the standby-replay mds, which seems to be
> happy again.
>
>
> Have any of you hit this issue before?
>
> TIA
> Goncalo
> .
>
> # cat /var/log/ceph/ceph-mds.mds.log
>
> (... snap...)
>
> 2016-02-02 02:53:28.608130 7f047679d700  1 mds.0.0 standby_replay_restart
> (as standby)
> 2016-02-02 02:53:28.614498 7f0474799700  1 mds.0.0 replay_done (as standby)
> 2016-02-02 02:53:29.614593 7f047679d700  1 mds.0.0 standby_replay_restart
> (as standby)
> 2016-02-02 02:53:29.620953 7f0474799700  1 mds.0.0 replay_done (as standby)
> 2016-02-02 02:53:30.621036 7f047679d700  1 mds.0.0 standby_replay_restart
> (as standby)
> 2016-02-02 02:53:30.640483 7f0474799700 -1 common/Thread.cc: In function
> 'void Thread::create(size_t)' thread 7f0474799700 time 2016-02-02
> 02:53:30.626626
> common/Thread.cc: 154: FAILED assert(ret == 0)
>
>  ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x85) [0x7f047ee2e105]
>  2: (Thread::create(unsigned long)+0x8a) [0x7f047ee19d1a]
>  3: (MDLog::replay(MDSInternalContextBase*)+0xe8) [0x7f047ecacd78]
>  4: (MDSRank::boot_start(MDSRank::BootStep, int)+0xe2e) [0x7f047ea88d3e]
>  5: (MDSRank::_standby_replay_restart_finish(int, unsigned long)+0x96)
> [0x7f047ea898b6]
>  6: (MDSIOContextBase::complete(int)+0xa4) [0x7f047eca3cc4]
>  7: (Finisher::finisher_thread_entry()+0x168) [0x7f047ed63fb8]
>  8: (()+0x7dc5) [0x7f047dc7adc5]
>  9: (clone()+0x6d) [0x7f047cb6521d]
>  NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.
>
> --- begin dump of recent events ---
>
> (... snap...)
>
>-10> 2016-02-02 02:53:30.621036 7f047679d700  1 mds.0.0
> standby_replay_restart (as standby)
> -9> 2016-02-02 02:53:30.621091 7f047679d700  1 --
> :6801/31961 --> :6808/23967 --
> osd_op(mds.245029.0:7272223 200. [read 0~0] 5.844f3494
> ack+read+known_if_redirected+full_force e689) v6 -- ?+0 0x7f0492e67600 con
> 0x7f04931c5b80
> -8> 2016-02-02 02:53:30.624919 7f0466c66700  1 --
> :6801/31961 <== osd.62 :6808/23967 1556070
>  osd_op_reply(7272223 200. [read 0~90] v0'0 uv8348 ondisk = 0)
> v6  179+0+90 (526420493 0 2009462618) 0x7f04b53fc000 con 0x7f04931c5b80
> -7> 2016-02-02 02:53:30.625029 7f0474799700  1 mds.245029.journaler(ro)
> probing for end of the log
> -6> 2016-02-02 02:53:30.625094 7f0474799700  1 --
> :6801/31961 --> :6814/11760 --
> osd_op(mds.245029.0:7272224 200.0537 [stat] 5.a003dca
> ack+read+rwordered+known_if_redirected+full_force e689) v6 -- ?+0
> 0x7f04bd8c6680 con 0x7f04af654aa0
> -5> 2016-02-02 02:53:30.625168 7f0474799700  1 --
> :6801/31961 --> :6814/11663 --
> osd_op(mds.245029.0:7272225 200.0538 [stat] 5.aa28907c
> ack+read+rwordered+known_if_redirected+full_force e689) v6 -- ?+0
> 0x7f04a72d1440 con 0x7f04b4b08d60
> -4> 2016-02-02 02:53:30.626365 7f00e3e1c700  1 --
> :6801/31961 <== osd.1 :6814/11663 92601 
> osd_op_reply(7272225 200.0538 [stat] v0'0 uv0 ack = -2 ((2) No such file
> or directory)) v6  179+0+0 (3023689365 0 0) 0x7f04913899c0 con
> 0x7f04b4b08d60
> -3> 2016-02-02 02:53:30.626433 7f0374d69700  1 --
> :6801/31961 <== osd.24 :6814/11760 93044 
> osd_op_reply(7272224 200.0537 [stat] v0'0 uv1909 ondisk = 0) v6
>  179+0+16 (736135707 0 822294467) 0x7f0482b3cc00 con 0x7f04af654aa0
> -2> 2016-02-02 02:53:30.626500 7f0474799700  1 mds.245029.journaler(ro)
> _finish_reprobe ne

Re: [ceph-users] CEPHFS: standby-replay mds crash

2016-02-01 Thread Goncalo Borges 
Hi...

Seems very similar to

http://tracker.ceph.com/issues/14144

Can you confirm it is the same issue?

Cheers
G.




From: Goncalo Borges
Sent: 02 February 2016 15:30
To: ceph-us...@ceph.com
Cc: rct...@coepp.org.au
Subject: CEPHFS: standby-replay mds crash

Hi CephFS experts.

1./ We are using Ceph and CephFS 9.2.0 with an active mds and a standby-replay 
mds (standard config)

# ceph -s
cluster 
 health HEALTH_OK
 monmap e1: 3 mons at 
{mon1=:6789/0,mon2=:6789/0,mon3=:6789/0}
election epoch 98, quorum 0,1,2 mon1,mon3,mon2
 mdsmap e102: 1/1/1 up {0=mds2=up:active}, 1 up:standby-replay
 osdmap e689: 64 osds: 64 up, 64 in
flags sortbitwise
  pgmap v2006627: 3072 pgs, 3 pools, 106 GB data, 85605 objects
323 GB used, 174 TB / 174 TB avail
3072 active+clean
  client io 1191 B/s rd, 2 op/s


2./ Today, the standby-replay mds crashed but the active mds continued ok. The 
logs (following this email) show a problem creating a thread.

3./ Our ganglia monitoring shows:
- a tremendous increase of load in the system
- a tremendous peak of network connectivity for inbound traffic
- No excessive memory usage nor excessive number of processes running.

4./ For now, we just restarted the standby-replay mds, which seems to be happy 
again.


Have any of you hit this issue before?

TIA
Goncalo
.

# cat /var/log/ceph/ceph-mds.mds.log

(... snap...)

2016-02-02 02:53:28.608130 7f047679d700  1 mds.0.0 standby_replay_restart (as 
standby)
2016-02-02 02:53:28.614498 7f0474799700  1 mds.0.0 replay_done (as standby)
2016-02-02 02:53:29.614593 7f047679d700  1 mds.0.0 standby_replay_restart (as 
standby)
2016-02-02 02:53:29.620953 7f0474799700  1 mds.0.0 replay_done (as standby)
2016-02-02 02:53:30.621036 7f047679d700  1 mds.0.0 standby_replay_restart (as 
standby)
2016-02-02 02:53:30.640483 7f0474799700 -1 common/Thread.cc: In function 'void 
Thread::create(size_t)' thread 7f0474799700 time 2016-02-02 02:53:30.626626
common/Thread.cc: 154: FAILED assert(ret == 0)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) 
[0x7f047ee2e105]
 2: (Thread::create(unsigned long)+0x8a) [0x7f047ee19d1a]
 3: (MDLog::replay(MDSInternalContextBase*)+0xe8) [0x7f047ecacd78]
 4: (MDSRank::boot_start(MDSRank::BootStep, int)+0xe2e) [0x7f047ea88d3e]
 5: (MDSRank::_standby_replay_restart_finish(int, unsigned long)+0x96) 
[0x7f047ea898b6]
 6: (MDSIOContextBase::complete(int)+0xa4) [0x7f047eca3cc4]
 7: (Finisher::finisher_thread_entry()+0x168) [0x7f047ed63fb8]
 8: (()+0x7dc5) [0x7f047dc7adc5]
 9: (clone()+0x6d) [0x7f047cb6521d]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

--- begin dump of recent events ---

(... snap...)

   -10> 2016-02-02 02:53:30.621036 7f047679d700  1 mds.0.0 
standby_replay_restart (as standby)
-9> 2016-02-02 02:53:30.621091 7f047679d700  1 -- 
:6801/31961 --> :6808/23967 -- 
osd_op(mds.245029.0:7272223 200. [read 0~0] 5.844f3494 
ack+read+known_if_redirected+full_force e689) v6 -- ?+0 0x7f0492e67600 con 
0x7f04931c5b80
-8> 2016-02-02 02:53:30.624919 7f0466c66700  1 -- 
:6801/31961 <== osd.62 :6808/23967 1556070  
osd_op_reply(7272223 200. [read 0~90] v0'0 uv8348 ondisk = 0) v6  
179+0+90 (526420493 0 2009462618) 0x7f04b53fc000 con 0x7f04931c5b80
-7> 2016-02-02 02:53:30.625029 7f0474799700  1 mds.245029.journaler(ro) 
probing for end of the log
-6> 2016-02-02 02:53:30.625094 7f0474799700  1 -- 
:6801/31961 --> :6814/11760 -- 
osd_op(mds.245029.0:7272224 200.0537 [stat] 5.a003dca 
ack+read+rwordered+known_if_redirected+full_force e689) v6 -- ?+0 
0x7f04bd8c6680 con 0x7f04af654aa0
-5> 2016-02-02 02:53:30.625168 7f0474799700  1 -- 
:6801/31961 --> :6814/11663 -- 
osd_op(mds.245029.0:7272225 200.0538 [stat] 5.aa28907c 
ack+read+rwordered+known_if_redirected+full_force e689) v6 -- ?+0 
0x7f04a72d1440 con 0x7f04b4b08d60
-4> 2016-02-02 02:53:30.626365 7f00e3e1c700  1 -- 
:6801/31961 <== osd.1 :6814/11663 92601  
osd_op_reply(7272225 200.0538 [stat] v0'0 uv0 ack = -2 ((2) No such file or 
directory)) v6  179+0+0 (3023689365 0 0) 0x7f04913899c0 con 0x7f04b4b08d60
-3> 2016-02-02 02:53:30.626433 7f0374d69700  1 -- 
:6801/31961 <== osd.24 :6814/11760 93044  
osd_op_reply(7272224 200.0537 [stat] v0'0 uv1909 ondisk = 0) v6
 179+0+16 (736135707 0 822294467) 0x7f0482b3cc00 con 0x7f04af654aa0
-2> 2016-02-02 02:53:30.626500 7f0474799700  1 mds.245029.journaler(ro) 
_finish_reprobe new_end = 5600997154 (header had 5600996718).
-1> 2016-02-02 02:53:30.626525 7f0474799700  2 mds.0.0 boot_start 2: 
replaying mds log
 0> 2016-02-02 02:53:30.640483 7f0474799700 -1 common/Thread.cc: In 
function 'void Thread::create(size_t)' thread 7f0474799700 time 2016-02-02 
02:53:30.626626
common/Thread.cc: 154: FAILED assert(ret == 0)

---

[ceph-users] CEPHFS: standby-replay mds crash

2016-02-01 Thread Goncalo Borges 
Hi CephFS experts.

1./ We are using Ceph and CephFS 9.2.0 with an active mds and a standby-replay 
mds (standard config)

# ceph -s
cluster 
 health HEALTH_OK
 monmap e1: 3 mons at 
{mon1=:6789/0,mon2=:6789/0,mon3=:6789/0}
election epoch 98, quorum 0,1,2 mon1,mon3,mon2
 mdsmap e102: 1/1/1 up {0=mds2=up:active}, 1 up:standby-replay
 osdmap e689: 64 osds: 64 up, 64 in
flags sortbitwise
  pgmap v2006627: 3072 pgs, 3 pools, 106 GB data, 85605 objects
323 GB used, 174 TB / 174 TB avail
3072 active+clean
  client io 1191 B/s rd, 2 op/s


2./ Today, the standby-replay mds crashed but the active mds continued ok. The 
logs (following this email) show a problem creating a thread.

3./ Our ganglia monitoring shows:
- a tremendous increase of load in the system
- a tremendous peak of network connectivity for inbound traffic
- No excessive memory usage nor excessive number of processes running.

4./ For now, we just restarted the standby-replay mds, which seems to be happy 
again.


Have any of you hit this issue before?

TIA
Goncalo
.

# cat /var/log/ceph/ceph-mds.mds.log

(... snap...)

2016-02-02 02:53:28.608130 7f047679d700  1 mds.0.0 standby_replay_restart (as 
standby)
2016-02-02 02:53:28.614498 7f0474799700  1 mds.0.0 replay_done (as standby)
2016-02-02 02:53:29.614593 7f047679d700  1 mds.0.0 standby_replay_restart (as 
standby)
2016-02-02 02:53:29.620953 7f0474799700  1 mds.0.0 replay_done (as standby)
2016-02-02 02:53:30.621036 7f047679d700  1 mds.0.0 standby_replay_restart (as 
standby)
2016-02-02 02:53:30.640483 7f0474799700 -1 common/Thread.cc: In function 'void 
Thread::create(size_t)' thread 7f0474799700 time 2016-02-02 02:53:30.626626
common/Thread.cc: 154: FAILED assert(ret == 0)

 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) 
[0x7f047ee2e105]
 2: (Thread::create(unsigned long)+0x8a) [0x7f047ee19d1a]
 3: (MDLog::replay(MDSInternalContextBase*)+0xe8) [0x7f047ecacd78]
 4: (MDSRank::boot_start(MDSRank::BootStep, int)+0xe2e) [0x7f047ea88d3e]
 5: (MDSRank::_standby_replay_restart_finish(int, unsigned long)+0x96) 
[0x7f047ea898b6]
 6: (MDSIOContextBase::complete(int)+0xa4) [0x7f047eca3cc4]
 7: (Finisher::finisher_thread_entry()+0x168) [0x7f047ed63fb8]
 8: (()+0x7dc5) [0x7f047dc7adc5]
 9: (clone()+0x6d) [0x7f047cb6521d]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

--- begin dump of recent events ---

(... snap...)

   -10> 2016-02-02 02:53:30.621036 7f047679d700  1 mds.0.0 
standby_replay_restart (as standby)
-9> 2016-02-02 02:53:30.621091 7f047679d700  1 -- 
:6801/31961 --> :6808/23967 -- 
osd_op(mds.245029.0:7272223 200. [read 0~0] 5.844f3494 
ack+read+known_if_redirected+full_force e689) v6 -- ?+0 0x7f0492e67600 con 
0x7f04931c5b80
-8> 2016-02-02 02:53:30.624919 7f0466c66700  1 -- 
:6801/31961 <== osd.62 :6808/23967 1556070  
osd_op_reply(7272223 200. [read 0~90] v0'0 uv8348 ondisk = 0) v6  
179+0+90 (526420493 0 2009462618) 0x7f04b53fc000 con 0x7f04931c5b80
-7> 2016-02-02 02:53:30.625029 7f0474799700  1 mds.245029.journaler(ro) 
probing for end of the log
-6> 2016-02-02 02:53:30.625094 7f0474799700  1 -- 
:6801/31961 --> :6814/11760 -- 
osd_op(mds.245029.0:7272224 200.0537 [stat] 5.a003dca 
ack+read+rwordered+known_if_redirected+full_force e689) v6 -- ?+0 
0x7f04bd8c6680 con 0x7f04af654aa0
-5> 2016-02-02 02:53:30.625168 7f0474799700  1 -- 
:6801/31961 --> :6814/11663 -- 
osd_op(mds.245029.0:7272225 200.0538 [stat] 5.aa28907c 
ack+read+rwordered+known_if_redirected+full_force e689) v6 -- ?+0 
0x7f04a72d1440 con 0x7f04b4b08d60
-4> 2016-02-02 02:53:30.626365 7f00e3e1c700  1 -- 
:6801/31961 <== osd.1 :6814/11663 92601  
osd_op_reply(7272225 200.0538 [stat] v0'0 uv0 ack = -2 ((2) No such file or 
directory)) v6  179+0+0 (3023689365 0 0) 0x7f04913899c0 con 0x7f04b4b08d60
-3> 2016-02-02 02:53:30.626433 7f0374d69700  1 -- 
:6801/31961 <== osd.24 :6814/11760 93044  
osd_op_reply(7272224 200.0537 [stat] v0'0 uv1909 ondisk = 0) v6
 179+0+16 (736135707 0 822294467) 0x7f0482b3cc00 con 0x7f04af654aa0
-2> 2016-02-02 02:53:30.626500 7f0474799700  1 mds.245029.journaler(ro) 
_finish_reprobe new_end = 5600997154 (header had 5600996718).
-1> 2016-02-02 02:53:30.626525 7f0474799700  2 mds.0.0 boot_start 2: 
replaying mds log
 0> 2016-02-02 02:53:30.640483 7f0474799700 -1 common/Thread.cc: In 
function 'void Thread::create(size_t)' thread 7f0474799700 time 2016-02-02 
02:53:30.626626
common/Thread.cc: 154: FAILED assert(ret == 0)

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd

Re: [ceph-users] Upgrading Ceph

2016-02-01 Thread Ben Hines 
Upgrades have been easy for me, following the steps.

I would say to be careful not to 'miss' one OSD, or forget to restart it
after updating, since having an OSD on a different version than the rest of
the cluster for too long during an upgrade started to cause issues when i
missed one once.

-Ben

On Wed, Jan 27, 2016 at 6:00 AM, Vlad Blando  wrote:

> Hi,
>
> I have a production Ceph Cluster
> - 3 nodes
> - 3 mons on each nodes
> - 9 OSD @ 4TB per node
> - using ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
>
> ​Now I want to upgrade it to Hammer, I saw the documentation on upgrading,
> it looks straight forward, but I want to know to those who have tried
> upgrading a production environment, any precautions, caveats, preparation
> that I need to do before doing it?
>
> - Vlad
> ᐧ
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS - Trying to understand direct OSD connection to ceph-fuse cephfs clients

2016-02-01 Thread Goncalo Borges 
Hi Greg.


We are using Ceph and CephFS 9.2.0. CephFS clients are being mounted via 
ceph-fuse.

We recently noticed the firewall from certain CephFS clients dropping 
connections with OSDs as SRC. This is something which is not systematic but we 
noticed happening at least once. Here is an example of the firewall log.

IPTABLES-IN IN=eth0 OUT= MAC= SRC= 
DST= LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42315 DF PROTO=TCP 
SPT=6802 DPT=57950 WINDOW=243 RES=0x00 ACK URGP=0

and of the netstat output executed in the OSD at the same time

tcp0  0 :6800:37893 
ESTABLISHED 23987/ceph-osd
tcp0  0 :6810:50875 
ESTABLISHED 23971/ceph-osd
tcp1  1 :6802:57950 CLOSING
 -   <-
(...)

In normal situations, we always have seen that it is the ceph-fuse client that 
starts the connections to OSDs, and the connection back from OSDs is accepted 
by the client firewall because it is a related,established connection. The fact 
that we saw drops in our client firewalls tell us that OSDs are trying to 
connect to the clients as a new connection.

Up to now I haven't seen the situation again but the infrastructure is not 
being used at the moment while when it happened it was under (some) load. My 
basic dd bare tests do not show anything abnormal and I have not been able to 
replicate the event.

We do not understand our observation and we wonder if this is
1) something expected (and in this case we will open our clients firewalls)
2) an unexpected behaviour (such as ceph-fuse breaking the established 
connections and the OSD is not aware of it)

> Yeah, it doesn't make much sense that the OSD would be opening a connection 
> -- I can't think of any way this could happen. Is it possible your firewall 
> software is referring to packets
> rather than streams when it identifies source and destination? (I don't do 
> much network watching so that dump output means very little to me.)


 If this is not something expected, we may be seeing a bug. We will keep 
tracking this down and let you guys know what else do we find.

Cheers
G.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrading Ceph

2016-02-01 Thread Vlad Blando 
What if the upgrade fails, what is the rollback scenario?



On Wed, Jan 27, 2016 at 10:10 PM,  wrote:

> I just upgraded from firefly to infernalis (firefly to hammer to
> infernalis) my cluster
> All came like a charm
> I upgraded mon first, then the osd, one by one, restarting the daemon
> after each upgrade
> Remember to change uid of your file, as the osd daemon now run under the
> user ceph (and not root): can be pretty long operation .. do the chown,
> then upgrade, then shut the daemon, rechown the remaining files, start
> the daemon (use chown --from=0:0 ceph:ceph -R /var/lib/ceph/osd/ceph-2
> to speed up the last chown)
>
> To conclude, I upgraded the whole cluster without a single downtime
> (pretty surprised, didn't expect the process to be "that" robust)
>
> On 27/01/2016 15:00, Vlad Blando wrote:
> > Hi,
> >
> > I have a production Ceph Cluster
> > - 3 nodes
> > - 3 mons on each nodes
> > - 9 OSD @ 4TB per node
> > - using ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
> >
> > ​Now I want to upgrade it to Hammer, I saw the documentation on
> upgrading,
> > it looks straight forward, but I want to know to those who have tried
> > upgrading a production environment, any precautions, caveats, preparation
> > that I need to do before doing it?
> >
> > - Vlad
> > ᐧ
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
ᐧ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] attempt to access beyond end of device on osd prepare

2016-02-01 Thread Wade Holler 
327.3.1 as I recall. Thank for the additional information / confirmation.
On Mon, Feb 1, 2016 at 6:05 PM Simon Ironside 
wrote:

> On 01/02/16 17:47, Wade Holler wrote:
> > I can at least say that I've seen this. (a lot)
> >
> > Running Infernalis with Btrfs on Cent 7.2.  I haven't seen any other
> > issue in the cluster that I would say is related.
> >
> > Take it for what you will.
>
> Thanks Wade,
>
> That's reassuring at least that it hasn't caused you problems longer term.
>
> I've not been able to reproduce the message myself by mounting and
> dismounting the file systems manually and I note that if I reboot the
> server without starting ceph services (i.e. chkconfig ceph off; reboot)
> the messages still appear for each OSD (xfs) disk even though they were
> never mounted by ceph or fstab:
>
> Feb  1 22:24:34 san1-osd2 kernel: XFS (sdb1): Mounting V4 Filesystem
> Feb  1 22:24:34 san1-osd2 kernel: XFS (sdb1): Ending clean mount
> Feb  1 22:24:34 san1-osd2 kernel: attempt to access beyond end of device
> Feb  1 22:24:34 san1-osd2 kernel: sdb1: rw=0, want=7814035088,
> limit=7814035087
>
> FWIW, the messages seem to only appear once per disk and have not yet
> re-appeared during normal use of the cluster on either OSD host.
>
> Maybe this is a kernel thing rather than a ceph thing. I'm probably
> running the similar if not same version as you: 3.10.0-327
>
> Regards,
> Simon.
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] attempt to access beyond end of device on osd prepare

2016-02-01 Thread Simon Ironside 

On 01/02/16 17:47, Wade Holler wrote:

I can at least say that I've seen this. (a lot)

Running Infernalis with Btrfs on Cent 7.2.  I haven't seen any other
issue in the cluster that I would say is related.

Take it for what you will.


Thanks Wade,

That's reassuring at least that it hasn't caused you problems longer term.

I've not been able to reproduce the message myself by mounting and 
dismounting the file systems manually and I note that if I reboot the 
server without starting ceph services (i.e. chkconfig ceph off; reboot) 
the messages still appear for each OSD (xfs) disk even though they were 
never mounted by ceph or fstab:


Feb  1 22:24:34 san1-osd2 kernel: XFS (sdb1): Mounting V4 Filesystem
Feb  1 22:24:34 san1-osd2 kernel: XFS (sdb1): Ending clean mount
Feb  1 22:24:34 san1-osd2 kernel: attempt to access beyond end of device
Feb  1 22:24:34 san1-osd2 kernel: sdb1: rw=0, want=7814035088, 
limit=7814035087


FWIW, the messages seem to only appear once per disk and have not yet 
re-appeared during normal use of the cluster on either OSD host.


Maybe this is a kernel thing rather than a ceph thing. I'm probably 
running the similar if not same version as you: 3.10.0-327


Regards,
Simon.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Developer Monthly (CDM)

2016-02-01 Thread Patrick McGarry 
Hey cephers,

Just as a reminder, the new format for our periodic developer meetings
is starting this Wed @ 12:30p EST.

http://tracker.ceph.com/projects/ceph/wiki/Planning

This meeting will be replacing the once quarterly (ish) Ceph Developer
Summits. Our hope is that a less formal, and more frequent, meeting
will allow for tighter coordination on Ceph work between the many
stakeholders. We’re hoping to keep this pretty short and on-track, so
if you have a topic to discuss, please add it to the wiki:

http://tracker.ceph.com/projects/ceph/wiki/CDM_03-FEB-2016

Thanks!

-- 

Best Regards,

Patrick McGarry
Director Ceph Community || Red Hat
http://ceph.com  ||  http://community.redhat.com
@scuttlemonkey || @ceph
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] virsh, virt-filesystems, guestmount, virt-install not working well with ceph rbd yet?

2016-02-01 Thread Jelle de Jong 
Hello everybody,

This is a cross post to libvirt-users, libguestfs and ceph-users.

I came back from FOSDEM 2016 and this was my 7th year or so and seen the
awesome development around visualization going on and want to thank
everybody for there contributions.

I seen presentations from oVirt, OpenStack and quite a few great Redhat
people, just like the last previous years.

I personally been supporting Linux-HA (pacemaker) DRBD/iSCSI/KVM
platform for years now and last year I started supporting Ceph RBD
clusters with KVM hosts.

But I keep hitting some “pains” and I been wondering why they have not
been solved and how I can best request help around this?

Tools that don't seem to fully work with Ceph RBD yet:
- virsh snapshot-create --quiesce --disk-only $quest
- virt-filesystems ${array_volume[@]}
- guestmount --ro ${array_volume[@]} -m $mount /mnt/$name-$disk
- virt-install also doesn't have Ceph RBD and I currently use virsh edit
to add the RBD storage disks.

My request is to get these tools working as well with Ceph RBD and maybe
not only integrate oVirt or OpenStack alike systems. I use several types
of back-up strategies depending on the size on the data storage use. I
am not sure how oVirt and alike create secure encrypted incremental
off-site back-ups of large data sets but I use(d) combinations of
rdiff-backup, duplicity and full guest dumps with dd, rbd export, xz.
Are these issues known and important enough, roadmap, bug reports?
Should I start working for Redhat as I don't have the resources myself,
but I can help with some money towards a crowed fund or bounty.

Maybe my software is outdated?
virsh 1.2.9
ceph version 0.80.10
libguestfs-tools 1:1.28.1-1
3.16.7-ckt20-1+deb8u2

My current workarounds for full storage exports:
virsh domfsfreeze $quest
sleep 2
virsh domblklist $quest
rbd snap create --snap snapshot $blkdevice
virsh domfsthaw $quest
rbd export $blkdevice@snapshot - | xz -1 | ssh -p 222 $user@$server "dd
of=/$location/$blkdevice$snapshot-$daystamp.dd.disk.gz"
rbd snap rm $blkdevice@snapshot

Kind regards,

Jelle de Jong
irc: #tuxcrafter
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Switching cache from writeback to forward causes I/O error in Firefly

2016-02-01 Thread Christian Balzer 

Hello,

Latest Firefly (and no, I can't safely upgrade before the cache is in).

While running some high-intensity I/O (for the crappy test cluster that
is) with "rados -p rbd  bench 60 write -t 32 -b 4096" turning a cache-tier
on (from forward to writeback mode) works as expected and without any
noteworthy interruptions.

However switching from writeback to forward gives me repeatedly this:
---
error during benchmark: -5
error 5: (5) Input/output error
---

Nothing in the ceph logs.

So my questions are:

a) known problem (or expected behavior, hopefully not)
b) any ideas on how to further investigate this

Regards,

Christian

-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Remove MDS

2016-02-01 Thread John Spray 
You can get rid of a daemon by simply stopping it and removing its
directory in /var/lib/ceph/mds -- no important state is stored locally
on the servers running MDS daemons.

Hopefully your new server is already up and running, so you can see
both the new and the old servers' daemons in "ceph mds dump".  Once
you stop the old one, it should disappear from the mds status within a
minute, or you can pre-empt that by using "ceph mds fail".

Cheers,
John

On Mon, Feb 1, 2016 at 6:32 PM, Don Laursen  wrote:
> My cluster is 0.94.5
>
> At first I installed the MDS role on an OSD node. I now have a dedicated
> server for MDS. How do I remove from the OSD node?
>
>
>
> TIA,
>
> Don
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Remove MDS

2016-02-01 Thread Don Laursen 
My cluster is 0.94.5
At first I installed the MDS role on an OSD node. I now have a dedicated server 
for MDS. How do I remove from the OSD node?

TIA,
Don

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS is not maintianing conistency

2016-02-01 Thread Mykola Dvornik 

What version are you running on your servers and clients?


On the clients:

ceph-fuse --version
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)

MDS/OSD/MON:

ceph --version
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)


 Exactly what changes are you making that aren't visible?


I am creating some new files in non-root folders.


What's the output of "ceph -s"?


ceph -s

   cluster 98d72518-6619-4b5c-b148-9a781ef13bcb
health HEALTH_OK
monmap e1: 1 mons at {000-s-ragnarok=XXX.XXX.XXX.XXX:6789/0}
   election epoch 1, quorum 0 000-s-ragnarok
mdsmap e576: 1/1/1 up {0=000-s-ragnarok=up:active}
osdmap e233: 16 osds: 16 up, 16 in
   flags sortbitwise
 pgmap v1927636: 1088 pgs, 2 pools, 1907 GB data, 2428 kobjects
   3844 GB used, 25949 GB / 29793 GB avail
   1088 active+clean
 client io 4381 B/s wr, 2 op

In addition on the clients' side I have

cat /etc/fuse.conf

user_allow_other
auto_cache
large_read
max_write = 16777216
max_read = 16777216


-Mykola


On Mon, Feb 1, 2016 at 5:06 PM, Gregory Farnum  
wrote:
On Monday, February 1, 2016, Mykola Dvornik 
 wrote:

Hi guys,

This is sort of rebuttal.

I have a CephFS deployed and mounted on a couple of clients via 
ceph-fuse (due to quota support and possibility to kill the 
ceph-fuse process to avoid stale mounts).


So the problems is that some times the changes made on one client 
are not visible on the others. It appears to me as rather random 
process. The only solution is to touch a new file in any particular 
folder that apparently triggers synchronization.


I've been using a kernel-side client before with no such kind of 
problems. So the questions is it expected behavior of ceph-fuse?


What version are you running on your servers and clients? Exactly 
what changes are you making that aren't visible? What's the output of 
"ceph -s"?
We see bugs like this occasionally but I can't think of any recent 
ones in ceph-fuse -- they're actually seen a lot more often in the 
kernel client.

-Greg




Regards,

Mykola










___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] attempt to access beyond end of device on osd prepare

2016-02-01 Thread Wade Holler 
I can at least say that I've seen this. (a lot)

Running Infernalis with Btrfs on Cent 7.2.  I haven't seen any other issue
in the cluster that I would say is related.

Take it for what you will.

Best Regards,
Wade


On Mon, Feb 1, 2016 at 12:39 PM Simon Ironside 
wrote:

> Has nobody else encountered this (or can explain it away) then?
>
> That's a bit of a worry, I wish I'd never spotted it now. I've just
> built the second OSD server and it produces exactly the same "attempt to
> access beyond end of device" message for each osd prepared.
>
> Regards,
> Simon.
>
> On 27/01/16 12:26, Simon Ironside wrote:
> > Hi All,
> >
> > I'm setting up a new cluster and owing to an unrelated mistake I made
> > during setup I've been paying particular attention to the system log on
> > the OSD server.
> >
> > When I run the ceph-deploy osd prepare step, an 'access beyond end of
> > device' error is logged just after the file system is mounted where it
> > seems to be attempting to reach one block beyond the end of the disk:
> >
> > Jan 26 16:13:47 ceph-osd1 kernel: attempt to access beyond end of device
> > Jan 26 16:13:47 ceph-osd1 kernel: sde1: rw=0, want=7814035088,
> > limit=7814035087
> >
> > This happens with every disk in the OSD host, whether configured for
> > internal or external journals, and also appears when the OSDs are
> > remounted if the host is rebooted. The OSDs start up and seem to
> > function just fine.
> >
> > I've checked the XFS file system is without the bounds of the partition
> > with xfs_info, and it is. I've also checked the OSD hosts in my other
> > live clusters but they don't show this error so I'm reluctant to ignore
> it.
> >
> > As this is a new cluster it doesn't yet contain anything important so
> > it's no problem to flatten it and start again. I'm using hammer 0.94.5
> > on RHEL 7.2.
> >
> > Thanks,
> > Simon.
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] attempt to access beyond end of device on osd prepare

2016-02-01 Thread Simon Ironside 

Has nobody else encountered this (or can explain it away) then?

That's a bit of a worry, I wish I'd never spotted it now. I've just 
built the second OSD server and it produces exactly the same "attempt to 
access beyond end of device" message for each osd prepared.


Regards,
Simon.

On 27/01/16 12:26, Simon Ironside wrote:

Hi All,

I'm setting up a new cluster and owing to an unrelated mistake I made
during setup I've been paying particular attention to the system log on
the OSD server.

When I run the ceph-deploy osd prepare step, an 'access beyond end of
device' error is logged just after the file system is mounted where it
seems to be attempting to reach one block beyond the end of the disk:

Jan 26 16:13:47 ceph-osd1 kernel: attempt to access beyond end of device
Jan 26 16:13:47 ceph-osd1 kernel: sde1: rw=0, want=7814035088,
limit=7814035087

This happens with every disk in the OSD host, whether configured for
internal or external journals, and also appears when the OSDs are
remounted if the host is rebooted. The OSDs start up and seem to
function just fine.

I've checked the XFS file system is without the bounds of the partition
with xfs_info, and it is. I've also checked the OSD hosts in my other
live clusters but they don't show this error so I'm reluctant to ignore it.

As this is a new cluster it doesn't yet contain anything important so
it's no problem to flatten it and start again. I'm using hammer 0.94.5
on RHEL 7.2.

Thanks,
Simon.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Install ceph with infiniband

2016-02-01 Thread Robert LeBlanc 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

RDMA for Ceph is not quite production ready yet. For now, you will
have to go with IPoIB and when RDMA is available, you hopefully can
migrate pretty easy. Matt Benjamin should be able to give you a better
idea about timeline. I haven't heard a lot about the progress lately,
but that doesn't mean nothing is going on.
- 
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Mon, Feb 1, 2016 at 6:37 AM, 1 <10...@candesoft.com> wrote:
> Hello,
> I am going to install ceph with infiniband and RDMA support. However, it
> seems there is no documents for doing that. I can only find 'RDMA support'
> in ceph's release notes, and it seems using  Accelio (libxio) library. Most
> of the documents are based on Ethernet configuration. So are there any
> documents about deploy ceph with Infiniband and RDMA? Does ceph already
> intergrated the Accelio (libxio) library or I have to download libxio,
> install and configure it before doing ceph installation?
> Thanks.
>
> Yingdi Guo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

-BEGIN PGP SIGNATURE-
Version: Mailvelope v1.3.4
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJWr4fdCRDmVDuy+mK58QAASHkP/jQEpVv1LQXNO2vNK6iN
XUmL0OhPfb/AFQeSHg3sVYpYfRu/kTRvLUype4Nh5OMrTtS9dBYYxKh3mqZk
e2M4aVuKRq6kvMh43dGIjThd1pEWrAktMEUxWxMUfKMVNJIHWVmFDz6keDAX
e+kMVDqBgQIg8WorWi75DMOClEXV3bEVQlxjgBlpiYNB4J+CfdjiQtyZaFkX
8kVov1YVkz+raCNkikIR9A5vVGj1tyMzHk0I0Wc1ql8/RCN+7Y6HAqGZlvP5
ol+cugA/L9BVLyC13NKN0NRbh+Eh2L5es0PlZ+4ElScW0IkQCJIe5p0yCETb
a+mNtKQhfPjiIVGQNSNHqsSwIFNaCVVpF1W2cgZmutU7hLPcG+2BOtCy8Hfw
ZgHOMEtUUdJIAqBIgau44XzAHt762zjg0RXqAk1z8RfNLtsfFh+FaYoxNRXM
VBUBrobETcaXV+vk59AEnnKYBqMCUGJ2X7X79t6gUYelBpRDI0Fu98A0B6hS
R3fSLQu3XzxidoKiaiYhnsowxAFfb4+J0KkWrDDkTbQdwYMjYzHpKJn0/q/w
7QxIYlLl0sY9Gqqu61FgZO0Ygrod1lA+GHmelUVp75hpf4txiJhgldoIG4v5
SXMjH8/7dSWwwbGBVCfbsPWQkPqRFA9Qu0PoS6UKEcVRM4r45B9ij3nkPIlB
UZEq
=56mC
-END PGP SIGNATURE-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS - Trying to understand direct OSD connection to ceph-fuse cephfs clients

2016-02-01 Thread Gregory Farnum 
On Sunday, January 31, 2016, Goncalo Borges 
wrote:

> Dear CephFS experts...
>
> We are using Ceph and CephFS 9.2.0. CephFS clients are being mounted via
> ceph-fuse.
>
> We recently noticed the firewall from certain CephFS clients dropping
> connections with OSDs as SRC. This is something which is not systematic but
> we noticed happening at least once. Here is an example of the firewall log.
>
> IPTABLES-IN IN=eth0 OUT= MAC= SRC=
> DST= LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42315 DF PROTO=TCP
> SPT=6802 DPT=57950 WINDOW=243 RES=0x00 ACK URGP=0
>
> and of the netstat output executed in the OSD at the same time
>
> tcp0  0 :6800:37893
>  ESTABLISHED 23987/ceph-osd
> tcp0  0 :6810:50875
>  ESTABLISHED 23971/ceph-osd
> tcp1  1 :6802:57950
>  CLOSING -   <-
> (...)
>
> In normal situations, we always have seen that it is the ceph-fuse client
> that starts the connections to OSDs, and the connection back from OSDs is
> accepted by the client firewall because it is a related,established
> connection. The fact that we saw drops in our client firewalls tell us that
> OSDs are trying to connect to the clients as a new connection.
>
> Up to now I haven't seen the situation again but the infrastructure is not
> being used at the moment while when it happened it was under (some) load.
> My basic dd bare tests do not show anything abnormal and I have not been
> able to replicate the event.
>
> We do not understand our observation and we wonder if this is
> 1) something expected (and in this case we will open our clients firewalls)
> 2) an unexpected behaviour (such as ceph-fuse breaking the established
> connections and the OSD is not aware of it)


Yeah, it doesn't make much sense that the OSD would be opening a connection
-- I can't think of any way this could happen. Is it possible your firewall
software is referring to packets rather than streams when it identifies
source and destination? (I don't do much network watching so that dump
output means very little to me.)
-Greg


>
> Any comment is welcome.
>
> TIA
> G.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS is not maintianing conistency

2016-02-01 Thread Gregory Farnum 
On Monday, February 1, 2016, Mykola Dvornik 
wrote:

> Hi guys,
>
> This is sort of rebuttal.
>
> I have a CephFS deployed and mounted on a couple of clients via ceph-fuse
> (due to quota support and possibility to kill the ceph-fuse process to
> avoid stale mounts).
>
> So the problems is that some times the changes made on one client are not
> visible on the others. It appears to me as rather random process. The only
> solution is to touch a new file in any particular folder that apparently
> triggers synchronization.
>
> I've been using a kernel-side client before with no such kind of problems.
> So the questions is it expected behavior of ceph-fuse?
>

What version are you running on your servers and clients? Exactly what
changes are you making that aren't visible? What's the output of "ceph -s"?
We see bugs like this occasionally but I can't think of any recent ones in
ceph-fuse -- they're actually seen a lot more often in the kernel client.
-Greg



>
> Regards,
>
> Mykola
>
>
>
>
>
>
>
>
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Need help to develop CEPH EC Plugin for array type of Erasure Code

2016-02-01 Thread Syed Hussain 
Hi,

I've been working to develop a CEPH EC plgin for array type erasure code,
for example
RAID-DP (i.e RDP). Later I realized that I can't continue with (k+m) format
in CEPH as
like normal RS code if I want to save disk IO (or Read data).  Since I need
to ready the individual symbols from each device rather that k symbols. One
"chunk" from OSD will contain k-symbols
in contiguous form. The same thing will be an issue for MSR(Minimum Storage
Regeneration) Erasure code plugin.

As a solution I thought to consider k*k data chunk and k*m coding chunk and
use LRC kind of
layering to pass the info (in plugin) which is data chunk and coding chunk.
However, I've doubt if the decoding API will supply me the correct info on
lost chunks to get the info on lost erasure coding nodes.

Does anybody have good solution to this issue using CRUSH map?
Could Pl. suggest any pointer or link that explain the above issue?

Thanks,
Syed Abid Hussain
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] download.ceph.com not reachable over IPv6

2016-02-01 Thread Wido den Hollander 
Hi,

My monitoring just showed me that download.ceph.com isn't reachable over IPv6.

I tried from multipe locations, but on all I get a connection refused:

root@soekris:~# telnet -6 download.ceph.com 80
Trying 2607:f298:6050:51f3:f816:3eff:fe50:5ec...
telnet: Unable to connect to remote host: Connection refused
root@soekris:~#

wido@crew:~$ telnet -6 download.ceph.com 80
Trying 2607:f298:6050:51f3:f816:3eff:fe50:5ec...
telnet: Unable to connect to remote host: Connection refused
wido@crew:~$

Could somebody take a look at this?

Thanks!

Wido
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CephFS is not maintianing conistency

2016-02-01 Thread Mykola Dvornik 

Hi guys,

This is sort of rebuttal.

I have a CephFS deployed and mounted on a couple of clients via 
ceph-fuse (due to quota support and possibility to kill the ceph-fuse 
process to avoid stale mounts).


So the problems is that some times the changes made on one client are 
not visible on the others. It appears to me as rather random process. 
The only solution is to touch a new file in any particular folder that 
apparently triggers synchronization.


I've been using a kernel-side client before with no such kind of 
problems. So the questions is it expected behavior of ceph-fuse?


Regards,

Mykola











___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Install ceph with infiniband

2016-02-01 Thread 10000 
Hello,
I am going to install ceph with infiniband and RDMA support. However, it 
seems there is no documents for doing that. I can only find 'RDMA support' in 
ceph's release notes, and it seems using  Accelio (libxio) library. Most of the 
documents are based on Ethernet configuration. So are there any documents about 
deploy ceph with Infiniband and RDMA? Does ceph already intergrated the Accelio 
(libxio) library or I have to download libxio, install and configure it before 
doing ceph installation?
Thanks.

Yingdi Guo___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Stats back to Calamari

2016-02-01 Thread Daniel Rolfe 
hmmm we are running infernalis...

root@ceph1:~# ceph -v
ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
root@ceph1:~#


So you think everything should work with hammer ?

As this is all just a proof of concept on some vm's so we know how to build
into prod

Regards, Daniel

On Mon, Feb 1, 2016 at 11:22 PM, John Spray  wrote:

> The "assert path[-1] == 'type'" is the error you get when using the
> calamari diamond branch with a >= infernalis version of Ceph (where
> new fields were added to the perf schema output).  No idea if anyone
> has worked on updating Calamari+Diamond for latest ceph.
>
> John
>
> On Mon, Feb 1, 2016 at 12:09 PM, Daniel Rolfe 
> wrote:
> > I can see the is ok files are there
> >
> > root@ceph1:/var/run/ceph# ls -la
> > total 0
> > drwxrwx---  2 ceph ceph  80 Feb  1 10:51 .
> > drwxr-xr-x 18 root root 640 Feb  1 10:52 ..
> > srwxr-xr-x  1 ceph ceph   0 Feb  1 10:51 ceph-mon.ceph1.asok
> > srwxr-xr-x  1 root root   0 Jan 27 15:08 ceph-osd.0.asok
> > root@ceph1:/var/run/ceph#
> > root@ceph1:/var/run/ceph#
> > root@ceph1:/var/run/ceph#
> >
> >
> > Running diamond in debug show the below
> >
> > [2016-02-01 10:55:23,774] [Thread-1] Collecting data from:
> NetworkCollector
> > [2016-02-01 10:56:23,484] [Thread-1] Collecting data from: CPUCollector
> > [2016-02-01 10:56:23,487] [Thread-6] Collecting data from:
> MemoryCollector
> > [2016-02-01 10:56:23,489] [Thread-7] Collecting data from:
> SockstatCollector
> > [2016-02-01 10:56:23,768] [Thread-1] Collecting data from: CephCollector
> > [2016-02-01 10:56:23,768] [Thread-1] gathering service stats for
> > /var/run/ceph/ceph-mon.ceph1.asok
> > [2016-02-01 10:56:24,094] [Thread-1] Traceback (most recent call last):
> >   File "/usr/lib/pymodules/python2.7/diamond/collector.py", line 412, in
> > _run
> > self.collect()
> >   File "/usr/share/diamond/collectors/ceph/ceph.py", line 464, in collect
> > self._collect_service_stats(path)
> >   File "/usr/share/diamond/collectors/ceph/ceph.py", line 450, in
> > _collect_service_stats
> > self._publish_stats(counter_prefix, stats, schema, GlobalName)
> >   File "/usr/share/diamond/collectors/ceph/ceph.py", line 305, in
> > _publish_stats
> > assert path[-1] == 'type'
> > AssertionError
> >
> > [2016-02-01 10:56:24,096] [Thread-8] Collecting data from:
> > LoadAverageCollector
> > [2016-02-01 10:56:24,098] [Thread-1] Collecting data from:
> VMStatCollector
> > [2016-02-01 10:56:24,099] [Thread-1] Collecting data from:
> > DiskUsageCollector
> > [2016-02-01 10:56:24,104] [Thread-9] Collecting data from:
> > DiskSpaceCollector
> >
> >
> >
> > Check the md5 on the file returns the below:
> >
> > root@ceph1:/var/run/ceph# md5sum
> /usr/share/diamond/collectors/ceph/ceph.py
> > aeb3915f8ac7fdea61495805d2c99f33
> /usr/share/diamond/collectors/ceph/ceph.py
> > root@ceph1:/var/run/ceph#
> >
> >
> >
> > I've found that replacing the ceph.py file with the below stops the
> diamond
> > error
> >
> >
> >
> https://raw.githubusercontent.com/BrightcoveOS/Diamond/master/src/collectors/ceph/ceph.py
> >
> > root@ceph1:/usr/share/diamond/collectors/ceph# md5sum ceph.py
> > 13ac74ce0df39a5def879cb5fc530015  ceph.py
> >
> >
> > [2016-02-01 11:14:33,116] [Thread-42] Collecting data from:
> MemoryCollector
> > [2016-02-01 11:14:33,117] [Thread-1] Collecting data from: CPUCollector
> > [2016-02-01 11:14:33,123] [Thread-43] Collecting data from:
> > SockstatCollector
> > [2016-02-01 11:14:35,453] [Thread-1] Collecting data from: CephCollector
> > [2016-02-01 11:14:35,454] [Thread-1] checking
> > /var/run/ceph/ceph-mon.ceph1.asok
> > [2016-02-01 11:14:35,552] [Thread-1] checking
> /var/run/ceph/ceph-osd.0.asok
> > [2016-02-01 11:14:35,685] [Thread-44] Collecting data from:
> > LoadAverageCollector
> > [2016-02-01 11:14:35,686] [Thread-1] Collecting data from:
> VMStatCollector
> > [2016-02-01 11:14:35,687] [Thread-1] Collecting data from:
> > DiskUsageCollector
> > [2016-02-01 11:14:35,692] [Thread-45] Collecting data from:
> > DiskSpaceCollector
> >
> >
> > But after all that it's still NOT working
> >
> > What diamond version are you running ?
> >
> > I'm running Diamond version 3.4.67
> >
> > On Mon, Feb 1, 2016 at 11:01 PM, Daniel Rolfe  >
> > wrote:
> >>
> >> I can see the is ok files are there
> >>
> >> root@ceph1:/var/run/ceph# ls -la
> >> total 0
> >> drwxrwx---  2 ceph ceph  80 Feb  1 10:51 .
> >> drwxr-xr-x 18 root root 640 Feb  1 10:52 ..
> >> srwxr-xr-x  1 ceph ceph   0 Feb  1 10:51 ceph-mon.ceph1.asok
> >> srwxr-xr-x  1 root root   0 Jan 27 15:08 ceph-osd.0.asok
> >> root@ceph1:/var/run/ceph#
> >> root@ceph1:/var/run/ceph#
> >> root@ceph1:/var/run/ceph#
> >>
> >>
> >> Running diamond in debug show the below
> >>
> >> [2016-02-01 10:55:23,774] [Thread-1] Collecting data from:
> >> NetworkCollector
> >> [2016-02-01 10:56:23,484] [Thread-1] Collecting data from: CPUCollector
> >> [2016-02-01 10:56:23,487] [Thread-6] Collecting data from:
> MemoryCollector
> >> [2016-02-01 10:56:23,489

Re: [ceph-users] remove rbd volume with watchers

2016-02-01 Thread Mihai Gheorghe 
I managed to delete it after running "rados -p cache-ssd listwatchers
rbd_header.". It was one of the monitors that was watching and keeping
the image busy. Any light on why did the monitor kept keeping the image
busy? I would like to know!

Thanks!

2016-02-01 14:32 GMT+02:00 mcapsali :

> Hi,
>
> I accidentally terminated an instance in openstack that was running a
> system upgrade. It's boot volume was a cinder volume backed by ceph cache
> tier pool.
>
> Now i can not delete the volume neither from cinder of directly from ceph.
> If i try to delete it from cinder i get "rbd volume busy. Try again in 30
> sec". If i try to delete it with rbd rm i get "Unable to delete. Volume
> still has watchers".
>
> I can delete the cinder volume manually from db, but i get stuck with the
> rbd volume that is present on the cache tier pool and cold-storage pool.
>
> Is there a way to remove the watchers from the image or force delete it
> with watchers active?
>
> Thank you!
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] remove rbd volume with watchers

2016-02-01 Thread mcapsali 
Hi,

I accidentally terminated an instance in openstack that was running a
system upgrade. It's boot volume was a cinder volume backed by ceph cache
tier pool.

Now i can not delete the volume neither from cinder of directly from ceph.
If i try to delete it from cinder i get "rbd volume busy. Try again in 30
sec". If i try to delete it with rbd rm i get "Unable to delete. Volume
still has watchers".

I can delete the cinder volume manually from db, but i get stuck with the
rbd volume that is present on the cache tier pool and cold-storage pool.

Is there a way to remove the watchers from the image or force delete it
with watchers active?

Thank you!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Stats back to Calamari

2016-02-01 Thread John Spray 
The "assert path[-1] == 'type'" is the error you get when using the
calamari diamond branch with a >= infernalis version of Ceph (where
new fields were added to the perf schema output).  No idea if anyone
has worked on updating Calamari+Diamond for latest ceph.

John

On Mon, Feb 1, 2016 at 12:09 PM, Daniel Rolfe  wrote:
> I can see the is ok files are there
>
> root@ceph1:/var/run/ceph# ls -la
> total 0
> drwxrwx---  2 ceph ceph  80 Feb  1 10:51 .
> drwxr-xr-x 18 root root 640 Feb  1 10:52 ..
> srwxr-xr-x  1 ceph ceph   0 Feb  1 10:51 ceph-mon.ceph1.asok
> srwxr-xr-x  1 root root   0 Jan 27 15:08 ceph-osd.0.asok
> root@ceph1:/var/run/ceph#
> root@ceph1:/var/run/ceph#
> root@ceph1:/var/run/ceph#
>
>
> Running diamond in debug show the below
>
> [2016-02-01 10:55:23,774] [Thread-1] Collecting data from: NetworkCollector
> [2016-02-01 10:56:23,484] [Thread-1] Collecting data from: CPUCollector
> [2016-02-01 10:56:23,487] [Thread-6] Collecting data from: MemoryCollector
> [2016-02-01 10:56:23,489] [Thread-7] Collecting data from: SockstatCollector
> [2016-02-01 10:56:23,768] [Thread-1] Collecting data from: CephCollector
> [2016-02-01 10:56:23,768] [Thread-1] gathering service stats for
> /var/run/ceph/ceph-mon.ceph1.asok
> [2016-02-01 10:56:24,094] [Thread-1] Traceback (most recent call last):
>   File "/usr/lib/pymodules/python2.7/diamond/collector.py", line 412, in
> _run
> self.collect()
>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 464, in collect
> self._collect_service_stats(path)
>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 450, in
> _collect_service_stats
> self._publish_stats(counter_prefix, stats, schema, GlobalName)
>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 305, in
> _publish_stats
> assert path[-1] == 'type'
> AssertionError
>
> [2016-02-01 10:56:24,096] [Thread-8] Collecting data from:
> LoadAverageCollector
> [2016-02-01 10:56:24,098] [Thread-1] Collecting data from: VMStatCollector
> [2016-02-01 10:56:24,099] [Thread-1] Collecting data from:
> DiskUsageCollector
> [2016-02-01 10:56:24,104] [Thread-9] Collecting data from:
> DiskSpaceCollector
>
>
>
> Check the md5 on the file returns the below:
>
> root@ceph1:/var/run/ceph# md5sum /usr/share/diamond/collectors/ceph/ceph.py
> aeb3915f8ac7fdea61495805d2c99f33  /usr/share/diamond/collectors/ceph/ceph.py
> root@ceph1:/var/run/ceph#
>
>
>
> I've found that replacing the ceph.py file with the below stops the diamond
> error
>
>
> https://raw.githubusercontent.com/BrightcoveOS/Diamond/master/src/collectors/ceph/ceph.py
>
> root@ceph1:/usr/share/diamond/collectors/ceph# md5sum ceph.py
> 13ac74ce0df39a5def879cb5fc530015  ceph.py
>
>
> [2016-02-01 11:14:33,116] [Thread-42] Collecting data from: MemoryCollector
> [2016-02-01 11:14:33,117] [Thread-1] Collecting data from: CPUCollector
> [2016-02-01 11:14:33,123] [Thread-43] Collecting data from:
> SockstatCollector
> [2016-02-01 11:14:35,453] [Thread-1] Collecting data from: CephCollector
> [2016-02-01 11:14:35,454] [Thread-1] checking
> /var/run/ceph/ceph-mon.ceph1.asok
> [2016-02-01 11:14:35,552] [Thread-1] checking /var/run/ceph/ceph-osd.0.asok
> [2016-02-01 11:14:35,685] [Thread-44] Collecting data from:
> LoadAverageCollector
> [2016-02-01 11:14:35,686] [Thread-1] Collecting data from: VMStatCollector
> [2016-02-01 11:14:35,687] [Thread-1] Collecting data from:
> DiskUsageCollector
> [2016-02-01 11:14:35,692] [Thread-45] Collecting data from:
> DiskSpaceCollector
>
>
> But after all that it's still NOT working
>
> What diamond version are you running ?
>
> I'm running Diamond version 3.4.67
>
> On Mon, Feb 1, 2016 at 11:01 PM, Daniel Rolfe 
> wrote:
>>
>> I can see the is ok files are there
>>
>> root@ceph1:/var/run/ceph# ls -la
>> total 0
>> drwxrwx---  2 ceph ceph  80 Feb  1 10:51 .
>> drwxr-xr-x 18 root root 640 Feb  1 10:52 ..
>> srwxr-xr-x  1 ceph ceph   0 Feb  1 10:51 ceph-mon.ceph1.asok
>> srwxr-xr-x  1 root root   0 Jan 27 15:08 ceph-osd.0.asok
>> root@ceph1:/var/run/ceph#
>> root@ceph1:/var/run/ceph#
>> root@ceph1:/var/run/ceph#
>>
>>
>> Running diamond in debug show the below
>>
>> [2016-02-01 10:55:23,774] [Thread-1] Collecting data from:
>> NetworkCollector
>> [2016-02-01 10:56:23,484] [Thread-1] Collecting data from: CPUCollector
>> [2016-02-01 10:56:23,487] [Thread-6] Collecting data from: MemoryCollector
>> [2016-02-01 10:56:23,489] [Thread-7] Collecting data from:
>> SockstatCollector
>> [2016-02-01 10:56:23,768] [Thread-1] Collecting data from: CephCollector
>> [2016-02-01 10:56:23,768] [Thread-1] gathering service stats for
>> /var/run/ceph/ceph-mon.ceph1.asok
>> [2016-02-01 10:56:24,094] [Thread-1] Traceback (most recent call last):
>>   File "/usr/lib/pymodules/python2.7/diamond/collector.py", line 412, in
>> _run
>> self.collect()
>>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 464, in collect
>> self._collect_service_stats(path)
>>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 450, in

Re: [ceph-users] Ceph Stats back to Calamari

2016-02-01 Thread Daniel Rolfe 
I can see the is ok files are there

*root@ceph1:/var/run/ceph# ls -la*
*total 0*
*drwxrwx---  2 ceph ceph  80 Feb  1 10:51 .*
*drwxr-xr-x 18 root root 640 Feb  1 10:52 ..*
*srwxr-xr-x  1 ceph ceph   0 Feb  1 10:51 ceph-mon.ceph1.asok*
*srwxr-xr-x  1 root root   0 Jan 27 15:08 ceph-osd.0.asok*
*root@ceph1:/var/run/ceph#*
*root@ceph1:/var/run/ceph#*
*root@ceph1:/var/run/ceph#*


Running diamond in debug show the below

[2016-02-01 10:55:23,774] [Thread-1] Collecting data from: NetworkCollector
[2016-02-01 10:56:23,484] [Thread-1] Collecting data from: CPUCollector
[2016-02-01 10:56:23,487] [Thread-6] Collecting data from: MemoryCollector
[2016-02-01 10:56:23,489] [Thread-7] Collecting data from: SockstatCollector
[2016-02-01 10:56:23,768] [Thread-1] Collecting data from: CephCollector
[2016-02-01 10:56:23,768] [Thread-1] gathering service stats for
/var/run/ceph/ceph-mon.ceph1.asok
[2016-02-01 10:56:24,094] [Thread-1] Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/diamond/collector.py", line 412, in
_run
self.collect()
  File "/usr/share/diamond/collectors/ceph/ceph.py", line 464, in collect
self._collect_service_stats(path)
  File "/usr/share/diamond/collectors/ceph/ceph.py", line 450, in
_collect_service_stats
self._publish_stats(counter_prefix, stats, schema, GlobalName)
  File "/usr/share/diamond/collectors/ceph/ceph.py", line 305, in
_publish_stats
assert path[-1] == 'type'
AssertionError

[2016-02-01 10:56:24,096] [Thread-8] Collecting data from:
LoadAverageCollector
[2016-02-01 10:56:24,098] [Thread-1] Collecting data from: VMStatCollector
[2016-02-01 10:56:24,099] [Thread-1] Collecting data from:
DiskUsageCollector
[2016-02-01 10:56:24,104] [Thread-9] Collecting data from:
DiskSpaceCollector



Check the md5 on the file returns the below:

*root@ceph1:/var/run/ceph# md5sum
/usr/share/diamond/collectors/ceph/ceph.py*
*aeb3915f8ac7fdea61495805d2c99f33  /usr/share/diamond/collectors/ceph/ceph.py*
*root@ceph1:/var/run/ceph#*



I've found that replacing the ceph.py file with the below stops the diamond
error


https://raw.githubusercontent.com/BrightcoveOS/Diamond/master/src/collectors/ceph/ceph.py

*root@ceph1:/usr/share/diamond/collectors/ceph# md5sum ceph.py*
*13ac74ce0df39a5def879cb5fc530015  ceph.py*


[2016-02-01 11:14:33,116] [Thread-42] Collecting data from: MemoryCollector
[2016-02-01 11:14:33,117] [Thread-1] Collecting data from: CPUCollector
[2016-02-01 11:14:33,123] [Thread-43] Collecting data from:
SockstatCollector
*[2016-02-01 11:14:35,453] [Thread-1] Collecting data from: CephCollector*
*[2016-02-01 11:14:35,454] [Thread-1] checking
/var/run/ceph/ceph-mon.ceph1.asok*
*[2016-02-01 11:14:35,552] [Thread-1] checking
/var/run/ceph/ceph-osd.0.asok*
[2016-02-01 11:14:35,685] [Thread-44] Collecting data from:
LoadAverageCollector
[2016-02-01 11:14:35,686] [Thread-1] Collecting data from: VMStatCollector
[2016-02-01 11:14:35,687] [Thread-1] Collecting data from:
DiskUsageCollector
[2016-02-01 11:14:35,692] [Thread-45] Collecting data from:
DiskSpaceCollector


But after all that it's still NOT working

What diamond version are you running ?

I'm running Diamond version 3.4.67

On Mon, Feb 1, 2016 at 11:01 PM, Daniel Rolfe 
wrote:

> I can see the is ok files are there
>
> *root@ceph1:/var/run/ceph# ls -la*
> *total 0*
> *drwxrwx---  2 ceph ceph  80 Feb  1 10:51 .*
> *drwxr-xr-x 18 root root 640 Feb  1 10:52 ..*
> *srwxr-xr-x  1 ceph ceph   0 Feb  1 10:51 ceph-mon.ceph1.asok*
> *srwxr-xr-x  1 root root   0 Jan 27 15:08 ceph-osd.0.asok*
> *root@ceph1:/var/run/ceph#*
> *root@ceph1:/var/run/ceph#*
> *root@ceph1:/var/run/ceph#*
>
>
> Running diamond in debug show the below
>
> [2016-02-01 10:55:23,774] [Thread-1] Collecting data from: NetworkCollector
> [2016-02-01 10:56:23,484] [Thread-1] Collecting data from: CPUCollector
> [2016-02-01 10:56:23,487] [Thread-6] Collecting data from: MemoryCollector
> [2016-02-01 10:56:23,489] [Thread-7] Collecting data from:
> SockstatCollector
> [2016-02-01 10:56:23,768] [Thread-1] Collecting data from: CephCollector
> [2016-02-01 10:56:23,768] [Thread-1] gathering service stats for
> /var/run/ceph/ceph-mon.ceph1.asok
> [2016-02-01 10:56:24,094] [Thread-1] Traceback (most recent call last):
>   File "/usr/lib/pymodules/python2.7/diamond/collector.py", line 412, in
> _run
> self.collect()
>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 464, in collect
> self._collect_service_stats(path)
>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 450, in
> _collect_service_stats
> self._publish_stats(counter_prefix, stats, schema, GlobalName)
>   File "/usr/share/diamond/collectors/ceph/ceph.py", line 305, in
> _publish_stats
> assert path[-1] == 'type'
> AssertionError
>
> [2016-02-01 10:56:24,096] [Thread-8] Collecting data from:
> LoadAverageCollector
> [2016-02-01 10:56:24,098] [Thread-1] Collecting data from: VMStatCollector
> [2016-02-01 10:56:24,099] [Thread-1] Collecting data from:
> DiskUsa

Re: [ceph-users] SSD Journal

2016-02-01 Thread Jan Schermer 
Hi,
unfortunately I'm not a dev, so it's gonna be someone else ripping the journal 
out and trying.
But I came to understand that getting rid of journal is not that easy of a task.

To me, more important would be if the devs understood what I'm trying to say :-)
because without that any new development will still try and accomodate whatever 
the
original design provided, and in this way it will be inherently "flawed".

I never used RADOS directly (the closest I came was trying RadosGW to replace a 
Swift cluster),
so it's possible someone build a custom app on top of that and it's using some 
more powerful
features that RBD/S3 don't need. Is there such a project? Is someone actually 
building on top of
librados, or is it in reality only tied to RBD/S3 as we know it?

If journal is really only for crash consitency in case of abrupt OSD failure, 
then I can tell you right now
RBD doesn't need it. The tricky part comes when you have to compare different 
data on OSDs after
a crash - from filesystem perspective anything goes, but we need to stick to 
one "version" of the data
(or leave marked it as unused in a bitmap if one is to be used, who cares what 
data was actually in there).
But no need to reiterate I guess, there are more scenarios I haven't thought of.

I'd like to see Ceph competetive to vSAN, ScaleIO or even some concoction that 
I can brew using
NBD/DM/ZFS/whatever, and it's pretty obvious to me something isn't right in the 
design - at least
for the "most important" workload which is RBD.

Jan


> On 29 Jan 2016, at 18:05, Robert LeBlanc  wrote:
> 
> Signed PGP part
> Jan,
> 
> I know that Sage has worked through a lot of this and spent a lot of
> time on it, so I'm somewhat inclined to say that if he says it needs
> to be there, then it needs to be there. I, however, have been known to
> stare at the tress so much that I miss the forest and I understand
> some of the points that you bring up about the data consistency and
> recovery from the client prospective. One thing that might be helpful
> is for you (or someone else) to get in the code and disable the
> journal pieces (not sure how difficult this would be) and test it
> against your theories. It seems like you have some deep and sincere
> interests in seeing Ceph be successful. If you theory holds up, then
> presenting the data and results will help others understand and be
> more interested in it. It took me a few months of this kind of work
> with the WeightedPriorityQueue, and I think the developers and
> understanding the limitations of the PrioritizedQueue and how
> WeightedPriorityQueue can overcome them with the battery of tests I've
> done with a proof of concept. Theory and actual results can be
> different, but results are generally more difficult to argue.
> 
> Some of the decision about the journal may be based on RADOS and not
> RBD. For instance, the decision may have been made that if a RADOS
> write has been given to the cluster, it is to be assumed that the
> write is durable without waiting for an ACK. I can't see why an
> S3/RADOS client can't wait for an ACK from the web server/OSD, but I
> haven't gotten into that area yet. That is something else to keep in
> mind.
> 
> Lionel,
> 
> I don't think the journal is used for anything more than crash
> consistency of the OSD. I don't believe the journal is used a playback
> instrument for bringing other OSDs into sync. An osd that is out of
> sync will write it's updates to it's journal to speed up the process,
> but that is the extent. The OSD providing the update has to read the
> updates to send from disk/page cache. My understanding that the
> journal is "never" read from, only when the OSD process crashes.
> 
> I'm happy to be corrected if I've misstated anything.
> 
> Robert LeBlanc
> 
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
> 
> 
> On Fri, Jan 29, 2016 at 9:27 AM, Lionel Bouton
>  wrote:
> > Le 29/01/2016 16:25, Jan Schermer a écrit :
> >
> > [...]
> >
> > But if I understand correctly, there is indeed a log of the recent
> > modifications in the filestore which is used when a PG is recovering
> > because another OSD is lagging behind (not when Ceph reports a full
> > backfill where I suppose all objects' versions of a PG are compared).
> >
> > That list of transactions becomes useful only when OSD crashes and comes
> > back up - it needs to catch up somehow and this is one of the options. But
> > do you really need the "content" of those transactions which is what the
> > journal does?
> > If you have no such list then you need to either rely on things like mtime
> > of the object, or simply compare the hash of the objects (scrub).
> >
> >
> > This didn't seem robust enough to me but I think I had forgotten about the
> > monitors' role in maintaining coherency.
> >
> > Let's say you use a pool with size=3 and min_size=2. You begin with a PG
> > with 3 active OSDs then you lose a first OSD for this PG an

[ceph-users] High IOWAIT On OpenStack Instance

2016-02-01 Thread Ferhat Ozkasgarli 
Hi All,

We are testing Ceph with OpenStack and installed 3 Mon (This three monitor
nodes are also OpenStack controller and network node), 6 OSD (3 of the OSDs
are also Nova Computer Node).

There are total 24 OSDs (21 SAS, 3 SSD and all journals are in SSD).

There is no cache tiering for now.

Before power problem, I had great test and achieved following results:

Running VM: 106
Software: iometer
Hardware: HP DL 360e Gen8
Network: 10G Network for Storage
IOPS: 40K IOPS (30/70 write,read)

Now after the incident, I have installed the cluster from scratch and
having 90 to 100 % iowait on all of the vm I have created.

I know this might be from hardware failure or network but I need to
pinpoint who the culprit is.

Does any one has good procedure to pinpoint this kind of problems?

Thx
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com