Re: [ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

[Benedikt Fraunhofer]

Hi Jan,

> Doesn't look near the limit currently (but I suppose you rebooted it in the 
> meantime?).

the box this numbers came from has an uptime of 13 days
so it's one of the boxes that did survive yesterdays half-cluster-wide-reboot.

> Did iostat say anything about the drives? (btw dm-1 and dm-6 are what? Is 
> that your data drives?) - were they overloaded really?

no they didn't have any load and or iops.
Basically the whole box had nothing to do.

If I understand the load correctly, this just reports threads
that are ready and willing to work but - in this case -
don't get any data to work with.

Thx

 Benedikt


2015-12-08 8:44 GMT+01:00 Jan Schermer :
>
> Jan
>
>
>> On 08 Dec 2015, at 08:41, Benedikt Fraunhofer  wrote:
>>
>> Hi Jan,
>>
>> we had 65k for pid_max, which made
>> kernel.threads-max = 1030520.
>> or
>> kernel.threads-max = 256832
>> (looks like it depends on the number of cpus?)
>>
>> currently we've
>>
>> root@ceph1-store209:~# sysctl -a | grep -e thread -e pid
>> kernel.cad_pid = 1
>> kernel.core_uses_pid = 0
>> kernel.ns_last_pid = 60298
>> kernel.pid_max = 65535
>> kernel.threads-max = 256832
>> vm.nr_pdflush_threads = 0
>> root@ceph1-store209:~# ps axH |wc -l
>> 17548
>>
>> we'll see how it behaves once puppet has come by and adjusted it.
>>
>> Thx!
>>
>> Benedikt
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

[Jan Schermer]

Doesn't look near the limit currently (but I suppose you rebooted it in the 
meantime?).
Did iostat say anything about the drives? (btw dm-1 and dm-6 are what? Is that 
your data drives?) - were they overloaded really?

Jan


> On 08 Dec 2015, at 08:41, Benedikt Fraunhofer  wrote:
> 
> Hi Jan,
> 
> we had 65k for pid_max, which made
> kernel.threads-max = 1030520.
> or
> kernel.threads-max = 256832
> (looks like it depends on the number of cpus?)
> 
> currently we've
> 
> root@ceph1-store209:~# sysctl -a | grep -e thread -e pid
> kernel.cad_pid = 1
> kernel.core_uses_pid = 0
> kernel.ns_last_pid = 60298
> kernel.pid_max = 65535
> kernel.threads-max = 256832
> vm.nr_pdflush_threads = 0
> root@ceph1-store209:~# ps axH |wc -l
> 17548
> 
> we'll see how it behaves once puppet has come by and adjusted it.
> 
> Thx!
> 
> Benedikt

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

[Benedikt Fraunhofer]

Hi Jan,

we had 65k for pid_max, which made
kernel.threads-max = 1030520.
or
kernel.threads-max = 256832
(looks like it depends on the number of cpus?)

currently we've

root@ceph1-store209:~# sysctl -a | grep -e thread -e pid
kernel.cad_pid = 1
kernel.core_uses_pid = 0
kernel.ns_last_pid = 60298
kernel.pid_max = 65535
kernel.threads-max = 256832
vm.nr_pdflush_threads = 0
root@ceph1-store209:~# ps axH |wc -l
17548

we'll see how it behaves once puppet has come by and adjusted it.

Thx!

 Benedikt
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

[Jan Schermer]

And how many pids do you have currently?
This should do it I think
# ps axH |wc -l

Jan

> On 08 Dec 2015, at 08:26, Benedikt Fraunhofer  wrote:
> 
> Hi Jan,
> 
> we initially had to bump it once we had more than 12 osds
> per box. But it'll change that to the values you provided.
> 
> Thx!
> 
> Benedikt
> 
> 2015-12-08 8:15 GMT+01:00 Jan Schermer :
>> What is the setting of sysctl kernel.pid_max?
>> You relly need to have this:
>> kernel.pid_max = 4194304
>> (I think it also sets this as well: kernel.threads-max = 4194304)
>> 
>> I think you are running out of processs IDs.
>> 
>> Jan
>> 
>>> On 08 Dec 2015, at 08:10, Benedikt Fraunhofer  wrote:
>>> 
>>> Hello Cephers,
>>> 
>>> lately, our ceph-cluster started to show some weird behavior:
>>> 
>>> the osd boxes show a load of 5000-15000 before the osds get marked down.
>>> Usually the box is fully usable, even "apt-get dist-upgrade" runs smoothly,
>>> you can read and write to any disk, only things you can't do are strace the 
>>> osd
>>> processes, sync or reboot.
>>> 
>>> we only find some logs about the "xfsaild = XFS Access Item List Daemon"
>>> as hung_task warnings.
>>> 
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016108]
>>> [] ? kthread_create_on_node+0x1c0/0x1c0
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016112] INFO: task
>>> xfsaild/dm-1:1445 blocked for more than 120 seconds.
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016329]   Tainted:
>>> G C 3.19.0-39-generic #44~14.04.1-Ubuntu
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016558] "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016802] xfsaild/dm-1
>>> D 8807faa03af8 0  1445  2 0x
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016805]
>>> 8807faa03af8 8808098989d0 00013e80 8807faa03fd8
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016808]
>>> 00013e80 88080bb775c0 8808098989d0 88011381b2a8
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016812]
>>> 8807faa03c50 7fff 8807faa03c48 8808098989d0
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016815] Call Trace:
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016819]
>>> [] schedule+0x29/0x70
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016823]
>>> [] schedule_timeout+0x20c/0x280
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016826]
>>> [] ? sched_clock_cpu+0x85/0xc0
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016830]
>>> [] ? try_to_wake_up+0x1f1/0x340
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016834]
>>> [] wait_for_completion+0xa4/0x170
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016836]
>>> [] ? wake_up_state+0x20/0x20
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016840]
>>> [] flush_work+0xed/0x1c0
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016846]
>>> [] ? destroy_worker+0x90/0x90
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016870]
>>> [] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016873]
>>> [] ? lock_timer_base.isra.36+0x2b/0x50
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016878]
>>> [] ? try_to_del_timer_sync+0x4f/0x70
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016901]
>>> [] _xfs_log_force+0x60/0x270 [xfs]
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016904]
>>> [] ? internal_add_timer+0x80/0x80
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016926]
>>> [] xfs_log_force+0x2a/0x90 [xfs]
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016948]
>>> [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016970]
>>> [] xfsaild+0x140/0x5a0 [xfs]
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016992]
>>> [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016996]
>>> [] kthread+0xd2/0xf0
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017000]
>>> [] ? kthread_create_on_node+0x1c0/0x1c0
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017005]
>>> [] ret_from_fork+0x58/0x90
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017009]
>>> [] ? kthread_create_on_node+0x1c0/0x1c0
>>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017013] INFO: task
>>> xfsaild/dm-6:1616 blocked for more than 120 seconds.
>>> 
>>> kswapd is also reported as hung, but we don't have swap on the osds.
>>> 
>>> It looks like either all ceph-osd-threads are reporting in as willing to 
>>> work,
>>> or it's the xfs-maintenance-process itself like described in [1,2]
>>> 
>>> Usually if we aint fast enough setting no{out,scrub,deep-scrub} this
>>> has an avalanche
>>> effect where we usually end up ipmi-power-cycling half of the cluster
>>> because all the osd-nodes
>>> are busy doing nothing (according to iostat or top, exept the load).
>>> 
>>> Is this a known bug for kernel 3.19.0-39 (ubuntu 14.04 with the vivid 
>>> kernel)?

Re: [ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

[Benedikt Fraunhofer]

Hi Jan,

we initially had to bump it once we had more than 12 osds
per box. But it'll change that to the values you provided.

Thx!

 Benedikt

2015-12-08 8:15 GMT+01:00 Jan Schermer :
> What is the setting of sysctl kernel.pid_max?
> You relly need to have this:
> kernel.pid_max = 4194304
> (I think it also sets this as well: kernel.threads-max = 4194304)
>
> I think you are running out of processs IDs.
>
> Jan
>
>> On 08 Dec 2015, at 08:10, Benedikt Fraunhofer  wrote:
>>
>> Hello Cephers,
>>
>> lately, our ceph-cluster started to show some weird behavior:
>>
>> the osd boxes show a load of 5000-15000 before the osds get marked down.
>> Usually the box is fully usable, even "apt-get dist-upgrade" runs smoothly,
>> you can read and write to any disk, only things you can't do are strace the 
>> osd
>> processes, sync or reboot.
>>
>> we only find some logs about the "xfsaild = XFS Access Item List Daemon"
>> as hung_task warnings.
>>
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016108]
>> [] ? kthread_create_on_node+0x1c0/0x1c0
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016112] INFO: task
>> xfsaild/dm-1:1445 blocked for more than 120 seconds.
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016329]   Tainted:
>> G C 3.19.0-39-generic #44~14.04.1-Ubuntu
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016558] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016802] xfsaild/dm-1
>> D 8807faa03af8 0  1445  2 0x
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016805]
>> 8807faa03af8 8808098989d0 00013e80 8807faa03fd8
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016808]
>> 00013e80 88080bb775c0 8808098989d0 88011381b2a8
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016812]
>> 8807faa03c50 7fff 8807faa03c48 8808098989d0
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016815] Call Trace:
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016819]
>> [] schedule+0x29/0x70
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016823]
>> [] schedule_timeout+0x20c/0x280
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016826]
>> [] ? sched_clock_cpu+0x85/0xc0
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016830]
>> [] ? try_to_wake_up+0x1f1/0x340
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016834]
>> [] wait_for_completion+0xa4/0x170
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016836]
>> [] ? wake_up_state+0x20/0x20
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016840]
>> [] flush_work+0xed/0x1c0
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016846]
>> [] ? destroy_worker+0x90/0x90
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016870]
>> [] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016873]
>> [] ? lock_timer_base.isra.36+0x2b/0x50
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016878]
>> [] ? try_to_del_timer_sync+0x4f/0x70
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016901]
>> [] _xfs_log_force+0x60/0x270 [xfs]
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016904]
>> [] ? internal_add_timer+0x80/0x80
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016926]
>> [] xfs_log_force+0x2a/0x90 [xfs]
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016948]
>> [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016970]
>> [] xfsaild+0x140/0x5a0 [xfs]
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016992]
>> [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016996]
>> [] kthread+0xd2/0xf0
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017000]
>> [] ? kthread_create_on_node+0x1c0/0x1c0
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017005]
>> [] ret_from_fork+0x58/0x90
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017009]
>> [] ? kthread_create_on_node+0x1c0/0x1c0
>> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017013] INFO: task
>> xfsaild/dm-6:1616 blocked for more than 120 seconds.
>>
>> kswapd is also reported as hung, but we don't have swap on the osds.
>>
>> It looks like either all ceph-osd-threads are reporting in as willing to 
>> work,
>> or it's the xfs-maintenance-process itself like described in [1,2]
>>
>> Usually if we aint fast enough setting no{out,scrub,deep-scrub} this
>> has an avalanche
>> effect where we usually end up ipmi-power-cycling half of the cluster
>> because all the osd-nodes
>> are busy doing nothing (according to iostat or top, exept the load).
>>
>> Is this a known bug for kernel 3.19.0-39 (ubuntu 14.04 with the vivid 
>> kernel)?
>> Do the xfs-tweaks described here
>> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg25295.html
>> (i know this is for a pull request modifying the write-paths)
>> look decent or worth a try?
>>
>> Currently we're running with "back to defaults" and less load
>> (des

Re: [ceph-users] after loss of journal, osd fails to start with failed assert OSDMapRef OSDService::get_map(epoch_t) ret != null

[Benedikt Fraunhofer]

Hi Jan,

2015-12-08 8:12 GMT+01:00 Jan Schermer :

> Journal doesn't just "vanish", though, so you should investigate further...

We tried putting journals as files to overcome the changes in ceph-deploy
where you can't have the journals unencrypted but only the disks itself.
(and/or you can't have the journals on an msdos-fdisk-thing disk, just gpt, but
the debian installier can't handle gpt)
(this worked when we started but was changed later)

After a crash [1] this file just wasn't there any longer.

> This log is from the new empty journal, right?

Yep.

We're slowly migrating away from the journal-as-file deployment;
I just thought that it should be able to start up with an empty
journal without dying with an assertion-failure.

Thx in advance

  Benedikt

[1] 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-December/006593.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

[Jan Schermer]

What is the setting of sysctl kernel.pid_max?
You relly need to have this:
kernel.pid_max = 4194304
(I think it also sets this as well: kernel.threads-max = 4194304)

I think you are running out of processs IDs.

Jan

> On 08 Dec 2015, at 08:10, Benedikt Fraunhofer  wrote:
> 
> Hello Cephers,
> 
> lately, our ceph-cluster started to show some weird behavior:
> 
> the osd boxes show a load of 5000-15000 before the osds get marked down.
> Usually the box is fully usable, even "apt-get dist-upgrade" runs smoothly,
> you can read and write to any disk, only things you can't do are strace the 
> osd
> processes, sync or reboot.
> 
> we only find some logs about the "xfsaild = XFS Access Item List Daemon"
> as hung_task warnings.
> 
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016108]
> [] ? kthread_create_on_node+0x1c0/0x1c0
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016112] INFO: task
> xfsaild/dm-1:1445 blocked for more than 120 seconds.
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016329]   Tainted:
> G C 3.19.0-39-generic #44~14.04.1-Ubuntu
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016558] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016802] xfsaild/dm-1
> D 8807faa03af8 0  1445  2 0x
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016805]
> 8807faa03af8 8808098989d0 00013e80 8807faa03fd8
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016808]
> 00013e80 88080bb775c0 8808098989d0 88011381b2a8
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016812]
> 8807faa03c50 7fff 8807faa03c48 8808098989d0
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016815] Call Trace:
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016819]
> [] schedule+0x29/0x70
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016823]
> [] schedule_timeout+0x20c/0x280
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016826]
> [] ? sched_clock_cpu+0x85/0xc0
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016830]
> [] ? try_to_wake_up+0x1f1/0x340
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016834]
> [] wait_for_completion+0xa4/0x170
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016836]
> [] ? wake_up_state+0x20/0x20
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016840]
> [] flush_work+0xed/0x1c0
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016846]
> [] ? destroy_worker+0x90/0x90
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016870]
> [] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016873]
> [] ? lock_timer_base.isra.36+0x2b/0x50
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016878]
> [] ? try_to_del_timer_sync+0x4f/0x70
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016901]
> [] _xfs_log_force+0x60/0x270 [xfs]
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016904]
> [] ? internal_add_timer+0x80/0x80
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016926]
> [] xfs_log_force+0x2a/0x90 [xfs]
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016948]
> [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016970]
> [] xfsaild+0x140/0x5a0 [xfs]
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016992]
> [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.016996]
> [] kthread+0xd2/0xf0
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017000]
> [] ? kthread_create_on_node+0x1c0/0x1c0
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017005]
> [] ret_from_fork+0x58/0x90
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017009]
> [] ? kthread_create_on_node+0x1c0/0x1c0
> Dec  7 15:36:32 ceph1-store204 kernel: [152066.017013] INFO: task
> xfsaild/dm-6:1616 blocked for more than 120 seconds.
> 
> kswapd is also reported as hung, but we don't have swap on the osds.
> 
> It looks like either all ceph-osd-threads are reporting in as willing to work,
> or it's the xfs-maintenance-process itself like described in [1,2]
> 
> Usually if we aint fast enough setting no{out,scrub,deep-scrub} this
> has an avalanche
> effect where we usually end up ipmi-power-cycling half of the cluster
> because all the osd-nodes
> are busy doing nothing (according to iostat or top, exept the load).
> 
> Is this a known bug for kernel 3.19.0-39 (ubuntu 14.04 with the vivid kernel)?
> Do the xfs-tweaks described here
> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg25295.html
> (i know this is for a pull request modifying the write-paths)
> look decent or worth a try?
> 
> Currently we're running with "back to defaults" and less load
> (desperate try with the filestore settings, didnt change anything)
> ceph.conf-osd section:
> 
> [osd]
>  filestore max sync interval = 15
>  filestore min sync interval = 1
>  osd max backfills = 1
>  osd recovery op priority = 1
> 
> 
> as a baffled try to get it to survive more than a day at a 

Re: [ceph-users] after loss of journal, osd fails to start with failed assert OSDMapRef OSDService::get_map(epoch_t) ret != null

[Jan Schermer]

The rule of thumb is that the data on OSD is gone if the related journal is 
gone.
Journal doesn't just "vanish", though, so you should investigate further...

This log is from the new empty journal, right?

Jan

> On 08 Dec 2015, at 08:08, Benedikt Fraunhofer  wrote:
> 
> Hello List,
> 
> after some crash of a box, the journal vanished. Creating a new one
> with --mkjournal results in the osd beeing unable to start.
> Does anyone want to dissect this any further or should I just trash
> the osd and recreate it?
> 
> Thx in advance
>  Benedikt
> 
> 2015-12-01 07:46:31.505255 7fadb7f1e900  0 ceph version 0.94.5
> (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5486
> 2015-12-01 07:46:31.628585 7fadb7f1e900  0
> filestore(/var/lib/ceph/osd/ceph-328) backend xfs (magic 0x58465342)
> 2015-12-01 07:46:31.662972 7fadb7f1e900  0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
> FIEMAP ioctl is supported and appears to work
> 2015-12-01 07:46:31.662984 7fadb7f1e900  0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
> FIEMAP ioctl is disabled via 'filestore fiemap' config option
> 2015-12-01 07:46:31.674999 7fadb7f1e900  0
> genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
> syncfs(2) syscall fully supported (by glibc and kernel)
> 2015-12-01 07:46:31.675071 7fadb7f1e900  0
> xfsfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_feature:
> extsize is supported and kernel 3.19.0-33-generic >= 3.5
> 2015-12-01 07:46:31.806490 7fadb7f1e900  0
> filestore(/var/lib/ceph/osd/ceph-328) mount: enabling WRITEAHEAD
> journal mode: checkpoint is not enabled
> 2015-12-01 07:46:35.598698 7fadb7f1e900  1 journal _open
> /var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2015-12-01 07:46:35.600956 7fadb7f1e900  1 journal _open
> /var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
> 4096 bytes, directio = 1, aio = 1
> 2015-12-01 07:46:35.619860 7fadb7f1e900  0 
> cls/hello/cls_hello.cc:271: loading cls_hello
> 2015-12-01 07:46:35.682532 7fadb7f1e900 -1 osd/OSD.h: In function
> 'OSDMapRef OSDService::get_map(epoch_t)' thread 7fadb7f1e900 time
> 2015-12-01 07:46:35.681204
> osd/OSD.h: 716: FAILED assert(ret)
> 
> ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x8b) [0xbc60eb]
> 2: (OSDService::get_map(unsigned int)+0x3f) [0x70ad5f]
> 3: (OSD::init()+0x6ad) [0x6c5e0d]
> 4: (main()+0x2860) [0x6527e0]
> 5: (__libc_start_main()+0xf5) [0x7fadb505bec5]
> 6: /usr/bin/ceph-osd() [0x66b887]
> NOTE: a copy of the executable, or `objdump -rdS ` is
> needed to interpret this.
> 
> --- begin dump of recent events ---
>   -62> 2015-12-01 07:46:31.503728 7fadb7f1e900  5 asok(0x5402000)
> register_command perfcounters_dump hook 0x53a2050
>   -61> 2015-12-01 07:46:31.503759 7fadb7f1e900  5 asok(0x5402000)
> register_command 1 hook 0x53a2050
>   -60> 2015-12-01 07:46:31.503764 7fadb7f1e900  5 asok(0x5402000)
> register_command perf dump hook 0x53a2050
>   -59> 2015-12-01 07:46:31.503768 7fadb7f1e900  5 asok(0x5402000)
> register_command perfcounters_schema hook 0x53a2050
>   -58> 2015-12-01 07:46:31.503772 7fadb7f1e900  5 asok(0x5402000)
> register_command 2 hook 0x53a2050
>   -57> 2015-12-01 07:46:31.503775 7fadb7f1e900  5 asok(0x5402000)
> register_command perf schema hook 0x53a2050
>   -56> 2015-12-01 07:46:31.503786 7fadb7f1e900  5 asok(0x5402000)
> register_command perf reset hook 0x53a2050
>   -55> 2015-12-01 07:46:31.503790 7fadb7f1e900  5 asok(0x5402000)
> register_command config show hook 0x53a2050
>   -54> 2015-12-01 07:46:31.503792 7fadb7f1e900  5 asok(0x5402000)
> register_command config set hook 0x53a2050
>   -53> 2015-12-01 07:46:31.503797 7fadb7f1e900  5 asok(0x5402000)
> register_command config get hook 0x53a2050
>   -52> 2015-12-01 07:46:31.503799 7fadb7f1e900  5 asok(0x5402000)
> register_command config diff hook 0x53a2050
>   -51> 2015-12-01 07:46:31.503802 7fadb7f1e900  5 asok(0x5402000)
> register_command log flush hook 0x53a2050
>   -50> 2015-12-01 07:46:31.503804 7fadb7f1e900  5 asok(0x5402000)
> register_command log dump hook 0x53a2050
>   -49> 2015-12-01 07:46:31.503807 7fadb7f1e900  5 asok(0x5402000)
> register_command log reopen hook 0x53a2050
>   -48> 2015-12-01 07:46:31.505255 7fadb7f1e900  0 ceph version 0.94.5
> (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5486
>   -47> 2015-12-01 07:46:31.619430 7fadb7f1e900  1 -- 10.9.246.104:0/0
> learned my addr 10.9.246.104:0/0
>   -46> 2015-12-01 07:46:31.619439 7fadb7f1e900  1
> accepter.accepter.bind my_inst.addr is 10.9.246.104:6821/5486
> need_addr=0
>   -45> 2015-12-01 07:46:31.619457 7fadb7f1e900  1
> accepter.accepter.bind my_inst.addr is 0.0.0.0:6824/5486 need_addr=1
>   -44> 2015-12-01 07:46:31.619473 7fadb7f1e900  1
> accepter.accepter.bind my_inst.addr is 0.0.0.0:6825/5486 need_addr=1
>   -43> 2015-

[ceph-users] osd become unusable, blocked by xfsaild (?) and load > 5000

[Benedikt Fraunhofer]

Hello Cephers,

lately, our ceph-cluster started to show some weird behavior:

the osd boxes show a load of 5000-15000 before the osds get marked down.
Usually the box is fully usable, even "apt-get dist-upgrade" runs smoothly,
you can read and write to any disk, only things you can't do are strace the osd
processes, sync or reboot.

we only find some logs about the "xfsaild = XFS Access Item List Daemon"
as hung_task warnings.

Dec  7 15:36:32 ceph1-store204 kernel: [152066.016108]
[] ? kthread_create_on_node+0x1c0/0x1c0
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016112] INFO: task
xfsaild/dm-1:1445 blocked for more than 120 seconds.
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016329]   Tainted:
G C 3.19.0-39-generic #44~14.04.1-Ubuntu
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016558] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016802] xfsaild/dm-1
D 8807faa03af8 0  1445  2 0x
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016805]
8807faa03af8 8808098989d0 00013e80 8807faa03fd8
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016808]
00013e80 88080bb775c0 8808098989d0 88011381b2a8
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016812]
8807faa03c50 7fff 8807faa03c48 8808098989d0
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016815] Call Trace:
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016819]
[] schedule+0x29/0x70
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016823]
[] schedule_timeout+0x20c/0x280
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016826]
[] ? sched_clock_cpu+0x85/0xc0
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016830]
[] ? try_to_wake_up+0x1f1/0x340
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016834]
[] wait_for_completion+0xa4/0x170
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016836]
[] ? wake_up_state+0x20/0x20
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016840]
[] flush_work+0xed/0x1c0
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016846]
[] ? destroy_worker+0x90/0x90
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016870]
[] xlog_cil_force_lsn+0x7e/0x1f0 [xfs]
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016873]
[] ? lock_timer_base.isra.36+0x2b/0x50
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016878]
[] ? try_to_del_timer_sync+0x4f/0x70
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016901]
[] _xfs_log_force+0x60/0x270 [xfs]
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016904]
[] ? internal_add_timer+0x80/0x80
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016926]
[] xfs_log_force+0x2a/0x90 [xfs]
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016948]
[] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016970]
[] xfsaild+0x140/0x5a0 [xfs]
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016992]
[] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Dec  7 15:36:32 ceph1-store204 kernel: [152066.016996]
[] kthread+0xd2/0xf0
Dec  7 15:36:32 ceph1-store204 kernel: [152066.017000]
[] ? kthread_create_on_node+0x1c0/0x1c0
Dec  7 15:36:32 ceph1-store204 kernel: [152066.017005]
[] ret_from_fork+0x58/0x90
Dec  7 15:36:32 ceph1-store204 kernel: [152066.017009]
[] ? kthread_create_on_node+0x1c0/0x1c0
Dec  7 15:36:32 ceph1-store204 kernel: [152066.017013] INFO: task
xfsaild/dm-6:1616 blocked for more than 120 seconds.

kswapd is also reported as hung, but we don't have swap on the osds.

It looks like either all ceph-osd-threads are reporting in as willing to work,
or it's the xfs-maintenance-process itself like described in [1,2]

Usually if we aint fast enough setting no{out,scrub,deep-scrub} this
has an avalanche
effect where we usually end up ipmi-power-cycling half of the cluster
because all the osd-nodes
are busy doing nothing (according to iostat or top, exept the load).

Is this a known bug for kernel 3.19.0-39 (ubuntu 14.04 with the vivid kernel)?
Do the xfs-tweaks described here
https://www.mail-archive.com/ceph-users@lists.ceph.com/msg25295.html
(i know this is for a pull request modifying the write-paths)
look decent or worth a try?

Currently we're running with "back to defaults" and less load
(desperate try with the filestore settings, didnt change anything)
ceph.conf-osd section:

[osd]
  filestore max sync interval = 15
  filestore min sync interval = 1
  osd max backfills = 1
  osd recovery op priority = 1


as a baffled try to get it to survive more than a day at a stretch.

Maybe kernel 4.2 is worth a try?

Thx for any input
 Benedikt


[1] 
https://www.reddit.com/r/linux/comments/18kvdb/xfsaild_is_creating_tons_of_system_threads_and/
[2] 
http://serverfault.com/questions/497049/the-xfs-filesystem-is-broken-in-rhel-centos-6-x-what-can-i-do-about-it
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd dies on pg repair with FAILED assert(!out->snaps.empty())

[Benedikt Fraunhofer]

Hello Cephers!

trying to repair an inconsistent PG results in the osd dying with an
assertion failure:

 0> 2015-12-01 07:22:13.398006 7f76d6594700 -1 osd/SnapMapper.cc:
In function 'int SnapMapper::get_snaps(const hobject_t&
, SnapMapper::object_snaps*)' thread 7f76d6594700 time 2015-12-01
07:22:13.394900
osd/SnapMapper.cc: 153: FAILED assert(!out->snaps.empty())

 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x8b) [0xbc60eb]
 2: (SnapMapper::get_snaps(hobject_t const&,
SnapMapper::object_snaps*)+0x40c) [0x72aecc]
 3: (SnapMapper::get_snaps(hobject_t const&, std::set, std::allocator >*)+0xa2) [0x72
b062]
 4: (PG::_scan_snaps(ScrubMap&)+0x454) [0x7f2f84]
 5: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool,
unsigned int, ThreadPool::TPHandle&)+0x218) [0x7f3ba8]
 6: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x480) [0x7f9da0]
 7: (PG::scrub(ThreadPool::TPHandle&)+0x2ee) [0x7fb48e]
 8: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x19) [0x6cdbf9]
 9: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xbb6b4e]
 10: (ThreadPool::WorkThread::entry()+0x10) [0xbb7bf0]
 11: (()+0x8182) [0x7f76fe072182]
 12: (clone()+0x6d) [0x7f76fc5dd47d]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent 1
  max_new 1000
  log_file /var/log/ceph/ceph-osd.339.log
--- end dump of recent events ---
2015-12-01 07:22:13.476525 7f76d6594700 -1 *** Caught signal (Aborted) **
 in thread 7f76d6594700

ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: /usr/bin/ceph-osd() [0xacd7ba]
 2: (()+0x10340) [0x7f76fe07a340]
 3: (gsignal()+0x39) [0x7f76fc519cc9]
 4: (abort()+0x148) [0x7f76fc51d0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f76fce24535]
 6: (()+0x5e6d6) [0x7f76fce226d6]
 7: (()+0x5e703) [0x7f76fce22703]
 8: (()+0x5e922) [0x7f76fce22922]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x278) [0xbc62d8]
 10: (SnapMapper::get_snaps(hobject_t const&,
SnapMapper::object_snaps*)+0x40c) [0x72aecc]
 11: (SnapMapper::get_snaps(hobject_t const&, std::set, std::allocator >*)+0xa2) [0x72b062]
 12: (PG::_scan_snaps(ScrubMap&)+0x454) [0x7f2f84]
 13: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool,
unsigned int, ThreadPool::TPHandle&)+0x218) [0x7f3ba8]
 14: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x480) [0x7f9da0]
 15: (PG::scrub(ThreadPool::TPHandle&)+0x2ee) [0x7fb48e]
 16: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x19) [0x6cdbf9]
 17: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa5e) [0xbb6b4e]
 18: (ThreadPool::WorkThread::entry()+0x10) [0xbb7bf0]
 19: (()+0x8182) [0x7f76fe072182]
 20: (clone()+0x6d) [0x7f76fc5dd47d]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.

--- begin dump of recent events ---
-4> 2015-12-01 07:22:13.403280 7f76e4db1700  1 --
10.9.246.104:6887/8548 <== osd.109 10.9.245.204:0/3407 13 
osd_ping(ping e320057 stamp 2015-12-01 07:22:13.399779) v2  47+0+0
(1340520147 0 0) 0x22456800 con 0x22340b00
-3> 2015-12-01 07:22:13.403313 7f76e4db1700  1 --
10.9.246.104:6887/8548 --> 10.9.245.204:0/3407 -- osd_ping(ping_reply
e320057 stamp 2015-12-01 07:22:13.399779) v2 -- ?+0 0x23e3be00 con
0x22340b00
-2> 2015-12-01 07:22:13.403365 7f76e35ae700  1 --
10.9.246.104:6883/8548 <== osd.109 10.9.245.204:0/3407 13 
osd_ping(ping e320057 stamp 2015-12-01 07:22:13.399779) v2  47+0+0
(1340520147 0 0) 0x22457600 con 0x22570d60
-1> 2015-12-01 07:22:13.403405 7f76e35ae700  1 --
10.9.246.104:6883/8548 --> 10.9.245.204:0/3407 -- osd_ping(ping_reply
e320057 stamp 2015-12-01 07:22:13.399779) v2 -- ?+0 0x23e3fe00 con
0x22570d60
 0> 2015-12-01 07:22:13.476525 7f76d6594700 -1 *** Caught signal
(Aborted) **
 in thread 7f76d6594700
 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: /usr/bin/ceph-osd() [0xacd7ba]
 2: (()+0x10340) [0x7f76fe07a340]
 3: (gsignal()+0x39) [0x7f76fc519cc9]
 4: (abort()+0x148) [0x7f76fc51d0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f76fce24535]
 6: (()+0x5e6d6) [0x7f76fce226d6]
 7: (()+0x5e703) [0x7f76fce2

[ceph-users] after loss of journal, osd fails to start with failed assert OSDMapRef OSDService::get_map(epoch_t) ret != null

[Benedikt Fraunhofer]

Hello List,

after some crash of a box, the journal vanished. Creating a new one
with --mkjournal results in the osd beeing unable to start.
Does anyone want to dissect this any further or should I just trash
the osd and recreate it?

Thx in advance
  Benedikt

2015-12-01 07:46:31.505255 7fadb7f1e900  0 ceph version 0.94.5
(9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5486
2015-12-01 07:46:31.628585 7fadb7f1e900  0
filestore(/var/lib/ceph/osd/ceph-328) backend xfs (magic 0x58465342)
2015-12-01 07:46:31.662972 7fadb7f1e900  0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
FIEMAP ioctl is supported and appears to work
2015-12-01 07:46:31.662984 7fadb7f1e900  0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-12-01 07:46:31.674999 7fadb7f1e900  0
genericfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2015-12-01 07:46:31.675071 7fadb7f1e900  0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-328) detect_feature:
extsize is supported and kernel 3.19.0-33-generic >= 3.5
2015-12-01 07:46:31.806490 7fadb7f1e900  0
filestore(/var/lib/ceph/osd/ceph-328) mount: enabling WRITEAHEAD
journal mode: checkpoint is not enabled
2015-12-01 07:46:35.598698 7fadb7f1e900  1 journal _open
/var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
4096 bytes, directio = 1, aio = 1
2015-12-01 07:46:35.600956 7fadb7f1e900  1 journal _open
/var/lib/ceph/osd/ceph-328/journal fd 19: 9663676416 bytes, block size
4096 bytes, directio = 1, aio = 1
2015-12-01 07:46:35.619860 7fadb7f1e900  0 
cls/hello/cls_hello.cc:271: loading cls_hello
2015-12-01 07:46:35.682532 7fadb7f1e900 -1 osd/OSD.h: In function
'OSDMapRef OSDService::get_map(epoch_t)' thread 7fadb7f1e900 time
2015-12-01 07:46:35.681204
osd/OSD.h: 716: FAILED assert(ret)

 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x8b) [0xbc60eb]
 2: (OSDService::get_map(unsigned int)+0x3f) [0x70ad5f]
 3: (OSD::init()+0x6ad) [0x6c5e0d]
 4: (main()+0x2860) [0x6527e0]
 5: (__libc_start_main()+0xf5) [0x7fadb505bec5]
 6: /usr/bin/ceph-osd() [0x66b887]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.

--- begin dump of recent events ---
   -62> 2015-12-01 07:46:31.503728 7fadb7f1e900  5 asok(0x5402000)
register_command perfcounters_dump hook 0x53a2050
   -61> 2015-12-01 07:46:31.503759 7fadb7f1e900  5 asok(0x5402000)
register_command 1 hook 0x53a2050
   -60> 2015-12-01 07:46:31.503764 7fadb7f1e900  5 asok(0x5402000)
register_command perf dump hook 0x53a2050
   -59> 2015-12-01 07:46:31.503768 7fadb7f1e900  5 asok(0x5402000)
register_command perfcounters_schema hook 0x53a2050
   -58> 2015-12-01 07:46:31.503772 7fadb7f1e900  5 asok(0x5402000)
register_command 2 hook 0x53a2050
   -57> 2015-12-01 07:46:31.503775 7fadb7f1e900  5 asok(0x5402000)
register_command perf schema hook 0x53a2050
   -56> 2015-12-01 07:46:31.503786 7fadb7f1e900  5 asok(0x5402000)
register_command perf reset hook 0x53a2050
   -55> 2015-12-01 07:46:31.503790 7fadb7f1e900  5 asok(0x5402000)
register_command config show hook 0x53a2050
   -54> 2015-12-01 07:46:31.503792 7fadb7f1e900  5 asok(0x5402000)
register_command config set hook 0x53a2050
   -53> 2015-12-01 07:46:31.503797 7fadb7f1e900  5 asok(0x5402000)
register_command config get hook 0x53a2050
   -52> 2015-12-01 07:46:31.503799 7fadb7f1e900  5 asok(0x5402000)
register_command config diff hook 0x53a2050
   -51> 2015-12-01 07:46:31.503802 7fadb7f1e900  5 asok(0x5402000)
register_command log flush hook 0x53a2050
   -50> 2015-12-01 07:46:31.503804 7fadb7f1e900  5 asok(0x5402000)
register_command log dump hook 0x53a2050
   -49> 2015-12-01 07:46:31.503807 7fadb7f1e900  5 asok(0x5402000)
register_command log reopen hook 0x53a2050
   -48> 2015-12-01 07:46:31.505255 7fadb7f1e900  0 ceph version 0.94.5
(9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5486
   -47> 2015-12-01 07:46:31.619430 7fadb7f1e900  1 -- 10.9.246.104:0/0
learned my addr 10.9.246.104:0/0
   -46> 2015-12-01 07:46:31.619439 7fadb7f1e900  1
accepter.accepter.bind my_inst.addr is 10.9.246.104:6821/5486
need_addr=0
   -45> 2015-12-01 07:46:31.619457 7fadb7f1e900  1
accepter.accepter.bind my_inst.addr is 0.0.0.0:6824/5486 need_addr=1
   -44> 2015-12-01 07:46:31.619473 7fadb7f1e900  1
accepter.accepter.bind my_inst.addr is 0.0.0.0:6825/5486 need_addr=1
   -43> 2015-12-01 07:46:31.619492 7fadb7f1e900  1 -- 10.9.246.104:0/0
learned my addr 10.9.246.104:0/0
   -42> 2015-12-01 07:46:31.619496 7fadb7f1e900  1
accepter.accepter.bind my_inst.addr is 10.9.246.104:6827/5486
need_addr=0
   -41> 2015-12-01 07:46:31.620890 7fadb7f1e900  5 asok(0x5402000)
init /var/run/ceph/ceph-osd.328.asok
   -40> 2015-12-01 07:46:31.620901 7fadb7f1e900  5 asok(0x5402000)
bind_and_listen /var/run/ceph/ceph-osd.328.a

Re: [ceph-users] scrub error with ceph

[GuangYang]

Before issuing scrub, you may check if those scrub errors would point to one 
(or a small subset of) disk/OSD, and if so, did those objects put in a 
specified interval?

It is a large amount of scrub errors in a small cluster, which might be caused 
by some hardware issue ?


> Date: Mon, 7 Dec 2015 14:15:07 -0700 
> From: erm...@ualberta.ca 
> To: ceph-users@lists.ceph.com 
> Subject: [ceph-users] scrub error with ceph 
> 
> 
> Hi, 
> 
> I found there are 128 scrub errors in my ceph system. Checked with 
> health detail and found many pgs with stuck unclean issue. Should I 
> repair all of them? Or what I should do? 
> 
> [root@gcloudnet ~]# ceph -s 
> 
> cluster a4d0879f-abdc-4f9d-8a4b-53ce57d822f1 
> 
> health HEALTH_ERR 128 pgs inconsistent; 128 scrub errors; mds1: 
> Client HTRC:cephfs_data failing to respond to cache pressure; mds0: 
> Client physics-007:cephfs_data failing to respond to cache pressure; 
> pool 'cephfs_data' is full 
> 
> monmap e3: 3 mons at 
> {gcloudnet=xxx.xxx.xxx.xxx:6789/0,gcloudsrv1=xxx.xxx.xxx.xxx:6789/0,gcloudsrv2=xxx.xxx.xxx.xxx:6789/0},
>  
> election epoch 178, quorum 0,1,2 gcloudnet,gcloudsrv1,gcloudsrv2 
> 
> mdsmap e51000: 2/2/2 up {0=gcloudsrv1=up:active,1=gcloudnet=up:active} 
> 
> osdmap e2821: 18 osds: 18 up, 18 in 
> 
> pgmap v10457877: 3648 pgs, 23 pools, 10501 GB data, 38688 kobjects 
> 
> 14097 GB used, 117 TB / 130 TB avail 
> 
> 6 active+clean+scrubbing+deep 
> 
> 3513 active+clean 
> 
> 128 active+clean+inconsistent 
> 
> 1 active+clean+scrubbing 
> 
> 
> P.S. I am increasing the pg and pgp numbers for cephfs_data pool. 
> 
> Thanks, 
> 
> Erming 
> 
> 
> 
> -- 
> 
>  
> Erming Pei, Ph.D, Senior System Analyst 
> HPC Grid/Cloud Specialist, ComputeCanada/WestGrid 
> 
> Research Computing Group, IST 
> University of Alberta, Canada T6G 2H1 
> Email: erm...@ualberta.ca 
> erming@cern.ch 
> Tel. : +1 7804929914 Fax: +1 7804921729 
> 
> ___ ceph-users mailing list 
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
  
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd wasn't marked as down/out when it's storage folder was deleted

[GuangYang]

It is actually not part of ceph.

For some files under the folder, they are only access during OSD booting up, so 
removal would not cause a problem there. For some other files, OSD would keep a 
open handle, in which case, even you remove those files from within filesystem, 
they are not erased as there is still a handle point to it (like hard link).


> From: kane.ist...@gmail.com 
> Date: Mon, 7 Dec 2015 14:01:26 -0800 
> To: ceph-users@lists.ceph.com 
> Subject: [ceph-users] osd wasn't marked as down/out when it's storage 
> folder was deleted 
> 
> I've deleted: 
> rm -rf /var/lib/ceph/osd/ceph-0/current 
> 
> folder but looks like ceph never noticed that: 
> ceph osd df 
> ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 
> 4 0.09270 1.0 97231M 54661M 42570M 56.22 1.34 95 
> 0 0.09270 1.0 97231M 139M 97091M 0.14 0.00 69 
> 5 0.09270 1.0 97231M 42603M 54627M 43.82 1.05 73 
> 1 0.09270 1.0 97231M 51950M 45280M 53.43 1.28 91 
> 6 0.09270 1.0 97231M 50656M 46575M 52.10 1.25 88 
> 2 0.09270 1.0 97231M 43896M 53334M 45.15 1.08 76 
> TOTAL 569G 238G 331G 41.81 
> MIN/MAX VAR: 0.00/1.34 STDDEV: 19.15 
> 
> ceph -w: 
> health HEALTH_OK 
> 
> Waited approximately an hour and it never reported warning, nor copied 
> data back. 
> OSD log constantly filled with: 
> 2015-12-07 15:31:06.096017 7f007505d700 -1 
> filestore(/var/lib/ceph/osd/ceph-0) could not find 
> 1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No 
> such file or directory 
> 2015-12-07 15:31:06.096023 7f007505d700 0 
> filestore(/var/lib/ceph/osd/ceph-0) write couldn't open 
> 1.4f_head/1/08ce93cf/rbd_data.853f1f358346.13ca/head: (2) 
> No such file or directory 
> 2015-12-07 15:31:06.096044 7f007505d700 -1 
> filestore(/var/lib/ceph/osd/ceph-0) could not find 
> 1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No 
> such file or directory 
> 2015-12-07 15:31:06.249584 7f007585e700 -1 
> filestore(/var/lib/ceph/osd/ceph-0) could not find 
> 1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No 
> such file or directory 
> 2015-12-07 15:31:06.249641 7f007585e700 -1 
> filestore(/var/lib/ceph/osd/ceph-0) could not find 
> 1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No 
> such file or directory 
> 2015-12-07 15:31:06.249653 7f007585e700 0 
> filestore(/var/lib/ceph/osd/ceph-0) write couldn't open 
> 1.4f_head/1/08ce93cf/rbd_data.853f1f358346.13ca/head: (2) 
> No such file or directory 
> 2015-12-07 15:31:06.249692 7f007585e700 -1 
> filestore(/var/lib/ceph/osd/ceph-0) could not find 
> 1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No 
> such file or directory 
> 
> Is it a bug or intended behavior? 
> 
> ___ ceph-users mailing list 
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
  
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer

[Shinobu Kinjo]

Thanks!

- Original Message -
From: "Gregory Farnum" 
To: "Shinobu Kinjo" 
Cc: "ceph-users" 
Sent: Tuesday, December 8, 2015 12:06:51 PM
Subject: Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer

The warning is informational -- it doesn't harm anything. Future writes to
those files/directories will generate backtraces and it'll go away.

On Monday, December 7, 2015, Shinobu Kinjo  wrote:

> Is there anything we have to do?
> or that upgrade path is not doable...
>
>  Shinobu
>
> - Original Message -
> From: "Gregory Farnum" >
> To: "Shinobu Kinjo" >
> Cc: "ceph-users" >
> Sent: Tuesday, December 8, 2015 10:36:34 AM
> Subject: Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer
>
> As that ticket indicates, older versions of the code didn't create the
> backtraces, so obviously they aren't present. That certainly includes
> Dumpling!
> -Greg
>
> On Monday, December 7, 2015, Shinobu Kinjo  > wrote:
>
> > Hello,
> >
> > Have any of you tried to upgrade the Ceph cluster through the following
> > upgrade path.
> >
> > Dumpling -> Firefly -> Hammer
> >  * Each version is newest.
> >
> > After upgrading from Dumpling, Firefly to Hammer following this:
> >
> > http://docs.ceph.com/docs/master/install/upgrading-ceph/
> >
> > I ended up with hitting ""warnings"" reported on:
> >
> > http://tracker.ceph.com/issues/12715
> >
> > mds.0 [ERR] bad backtrace on dir ino 600
> > mds.0 [ERR] bad backtrace on dir ino 601
> > mds.0 [ERR] bad backtrace on dir ino 602
> > mds.0 [ERR] bad backtrace on dir ino 603
> > mds.0 [ERR] bad backtrace on dir ino 604
> > mds.0 [ERR] bad backtrace on dir ino 605
> > mds.0 [ERR] bad backtrace on dir ino 606
> > mds.0 [ERR] bad backtrace on dir ino 607
> > mds.0 [ERR] bad backtrace on dir ino 608
> > mds.0 [ERR] bad backtrace on dir ino 609
> >
> > If I've missed anything, point it out to me.
> >
> >  Shinobu
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com  
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer

[Gregory Farnum]

The warning is informational -- it doesn't harm anything. Future writes to
those files/directories will generate backtraces and it'll go away.

On Monday, December 7, 2015, Shinobu Kinjo  wrote:

> Is there anything we have to do?
> or that upgrade path is not doable...
>
>  Shinobu
>
> - Original Message -
> From: "Gregory Farnum" >
> To: "Shinobu Kinjo" >
> Cc: "ceph-users" >
> Sent: Tuesday, December 8, 2015 10:36:34 AM
> Subject: Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer
>
> As that ticket indicates, older versions of the code didn't create the
> backtraces, so obviously they aren't present. That certainly includes
> Dumpling!
> -Greg
>
> On Monday, December 7, 2015, Shinobu Kinjo  > wrote:
>
> > Hello,
> >
> > Have any of you tried to upgrade the Ceph cluster through the following
> > upgrade path.
> >
> > Dumpling -> Firefly -> Hammer
> >  * Each version is newest.
> >
> > After upgrading from Dumpling, Firefly to Hammer following this:
> >
> > http://docs.ceph.com/docs/master/install/upgrading-ceph/
> >
> > I ended up with hitting ""warnings"" reported on:
> >
> > http://tracker.ceph.com/issues/12715
> >
> > mds.0 [ERR] bad backtrace on dir ino 600
> > mds.0 [ERR] bad backtrace on dir ino 601
> > mds.0 [ERR] bad backtrace on dir ino 602
> > mds.0 [ERR] bad backtrace on dir ino 603
> > mds.0 [ERR] bad backtrace on dir ino 604
> > mds.0 [ERR] bad backtrace on dir ino 605
> > mds.0 [ERR] bad backtrace on dir ino 606
> > mds.0 [ERR] bad backtrace on dir ino 607
> > mds.0 [ERR] bad backtrace on dir ino 608
> > mds.0 [ERR] bad backtrace on dir ino 609
> >
> > If I've missed anything, point it out to me.
> >
> >  Shinobu
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com  
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer

[Shinobu Kinjo]

Is there anything we have to do?
or that upgrade path is not doable...

 Shinobu

- Original Message -
From: "Gregory Farnum" 
To: "Shinobu Kinjo" 
Cc: "ceph-users" 
Sent: Tuesday, December 8, 2015 10:36:34 AM
Subject: Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer

As that ticket indicates, older versions of the code didn't create the
backtraces, so obviously they aren't present. That certainly includes
Dumpling!
-Greg

On Monday, December 7, 2015, Shinobu Kinjo  wrote:

> Hello,
>
> Have any of you tried to upgrade the Ceph cluster through the following
> upgrade path.
>
> Dumpling -> Firefly -> Hammer
>  * Each version is newest.
>
> After upgrading from Dumpling, Firefly to Hammer following this:
>
> http://docs.ceph.com/docs/master/install/upgrading-ceph/
>
> I ended up with hitting ""warnings"" reported on:
>
> http://tracker.ceph.com/issues/12715
>
> mds.0 [ERR] bad backtrace on dir ino 600
> mds.0 [ERR] bad backtrace on dir ino 601
> mds.0 [ERR] bad backtrace on dir ino 602
> mds.0 [ERR] bad backtrace on dir ino 603
> mds.0 [ERR] bad backtrace on dir ino 604
> mds.0 [ERR] bad backtrace on dir ino 605
> mds.0 [ERR] bad backtrace on dir ino 606
> mds.0 [ERR] bad backtrace on dir ino 607
> mds.0 [ERR] bad backtrace on dir ino 608
> mds.0 [ERR] bad backtrace on dir ino 609
>
> If I've missed anything, point it out to me.
>
>  Shinobu
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] french meetup website

[Alexandre DERUMIER]

Hi Eric, 

too bad, I was not aware about this meetup, could you make an announcement for 
the next one  next time ?

I'll glad to share my experience with my full ssd production cluster.

Regards,

Alexandre

- Mail original -
De: "eric mourgaya" 
À: "ceph-users" 
Envoyé: Lundi 7 Décembre 2015 10:59:56
Objet: [ceph-users] french meetup website

Hi, 
I glad to write that a new website ( in french ) is now available. This website 
is managed by the ceph breizh community. You can find a report of the last 
meetup on this page: 
ceph breizh (http://ceph.bzh) 

enjoy it and join us. 



-- 
Eric Mourgaya, 


Respectons la planete! 
Luttons contre la mediocrite! 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-Users] Upgrade Path to Hammer

[Gregory Farnum]

As that ticket indicates, older versions of the code didn't create the
backtraces, so obviously they aren't present. That certainly includes
Dumpling!
-Greg

On Monday, December 7, 2015, Shinobu Kinjo  wrote:

> Hello,
>
> Have any of you tried to upgrade the Ceph cluster through the following
> upgrade path.
>
> Dumpling -> Firefly -> Hammer
>  * Each version is newest.
>
> After upgrading from Dumpling, Firefly to Hammer following this:
>
> http://docs.ceph.com/docs/master/install/upgrading-ceph/
>
> I ended up with hitting ""warnings"" reported on:
>
> http://tracker.ceph.com/issues/12715
>
> mds.0 [ERR] bad backtrace on dir ino 600
> mds.0 [ERR] bad backtrace on dir ino 601
> mds.0 [ERR] bad backtrace on dir ino 602
> mds.0 [ERR] bad backtrace on dir ino 603
> mds.0 [ERR] bad backtrace on dir ino 604
> mds.0 [ERR] bad backtrace on dir ino 605
> mds.0 [ERR] bad backtrace on dir ino 606
> mds.0 [ERR] bad backtrace on dir ino 607
> mds.0 [ERR] bad backtrace on dir ino 608
> mds.0 [ERR] bad backtrace on dir ino 609
>
> If I've missed anything, point it out to me.
>
>  Shinobu
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [Ceph-Users] Upgrade Path to Hammer

[Shinobu Kinjo]

Hello,

Have any of you tried to upgrade the Ceph cluster through the following upgrade 
path.

Dumpling -> Firefly -> Hammer
 * Each version is newest.

After upgrading from Dumpling, Firefly to Hammer following this:

http://docs.ceph.com/docs/master/install/upgrading-ceph/

I ended up with hitting ""warnings"" reported on:

http://tracker.ceph.com/issues/12715

mds.0 [ERR] bad backtrace on dir ino 600
mds.0 [ERR] bad backtrace on dir ino 601
mds.0 [ERR] bad backtrace on dir ino 602
mds.0 [ERR] bad backtrace on dir ino 603
mds.0 [ERR] bad backtrace on dir ino 604
mds.0 [ERR] bad backtrace on dir ino 605
mds.0 [ERR] bad backtrace on dir ino 606
mds.0 [ERR] bad backtrace on dir ino 607
mds.0 [ERR] bad backtrace on dir ino 608
mds.0 [ERR] bad backtrace on dir ino 609

If I've missed anything, point it out to me.

 Shinobu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] poor performance when recovering

[Libin Wu]

Yeah, we will upgrade in the near future. But i'm afraid the recovery
problem also existed in the hammer version.
So, why recovery affect performance so much, any plan to improve it?

2015-12-07 22:29 GMT+08:00 Oliver Dzombic :
> Hi,
>
> maybe you should first upgrade.
>
> "
>
> Posted by sage
> November 19th, 2015
>
> This is a bugfix release for Firefly.  As the Firefly 0.80.x series is
> nearing its planned end of life in January 2016 it may also be the last.
> "
>
> I think you are wasting time, trying to analyse/fix issues on a version
> which will be EOL in 3 weeks...
>
>
> --
> Mit freundlichen Gruessen / Best regards
>
> Oliver Dzombic
>
> Am 07.12.2015 um 15:26 schrieb Libin Wu:
>> Btw, my ceph version is 0.80.11
>>
>> 2015-12-07 21:45 GMT+08:00 Libin Wu :
>>> Hi, cephers
>>>
>>> I'm doing the performance test of ceph when recovering. The scene is simple:
>>> 1. run fio on 6 krbd device
>>> 2. stop one OSD for 10 seconds
>>> 3. start that OSD
>>>
>>> However, when the OSD up and start recovering, the performance of fio
>>> drop down from 9k to 1k for about 20 seconds. At the same tiime, we
>>> found the SSD of that OSD's latency is more than 100ms, so it seems
>>> the SSD become the bottleneck。
>>>
>>> So we want to slow down the recovery speed to lighten the load of the
>>> SSD when recovery. But configuration like:
>>> osd_recovery_max_active
>>> osd_recovery_max_chunk
>>> osd_max_backfills
>>> osd_recovery_op_priority
>>> are all useless.
>>>
>>> After reading and change some code, we want add a flow control in the
>>> process of:
>>> OSD::do_recovery
>>>
>>> So, will it be possible to do so and if this solution has some
>>> potential problem?
>>>
>>> Thanks!
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 答复: How long will the logs be kept?

[David Zafman]

dout() is used for an OSD to log information about what it is doing 
locally and might become very chatty.  It is saved on the local nodes 
disk only.


clog is the cluster log and is used for major events that should be 
known by the administrator (see ceph -w).  Clog should be used sparingly 
as it sends the messages to the monitor.


David

On 12/3/15 4:36 AM, Wukongming wrote:

OK! One more question. Do you know why ceph has 2 ways outputting logs(dout && 
clog). Cause I find dout is more helpful than clog, Did ceph use clog first, and dout 
added for new version?

-
wukongming ID: 12019
Tel：0571-86760239
Dept：2014 UIS2 ONEStor

-邮件原件-
发件人: Jan Schermer [mailto:j...@schermer.cz]
发送时间: 2015年12月3日 16:58
收件人: wukongming 12019 (RD)
抄送: huang jun; ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com
主题: Re: [ceph-users] How long will the logs be kept?

You can setup logrotate however you want - not sure what the default is for 
your distro.
Usually logrotate doesn't touch files that are smaller than some size even if 
they are old. It will also not delete logs for OSDs that no longer exist.

Ceph itself has nothing to do with log rotation, logrotate does the work. Ceph 
packages likely contain default logrotate rules for the logs but you can edit 
them to your liking.

Jan


On 03 Dec 2015, at 09:38, Wukongming  wrote:

Yes, I can find ceph of rotate configure file in the directory of 
/etc/logrotate.d.
Also, I find sth. Weird.

drwxr-xr-x  2 root root   4.0K Dec  3 14:54 ./
drwxrwxr-x 19 root syslog 4.0K Dec  3 13:33 ../
-rw---  1 root root  0 Dec  2 06:25 ceph.audit.log
-rw---  1 root root85K Nov 25 09:17 ceph.audit.log.1.gz
-rw---  1 root root   228K Dec  3 16:00 ceph.log
-rw---  1 root root28K Dec  3 06:23 ceph.log.1.gz
-rw---  1 root root   374K Dec  2 06:22 ceph.log.2.gz
-rw-r--r--  1 root root   4.3M Dec  3 16:01 ceph-mon.wkm01.log
-rw-r--r--  1 root root   561K Dec  3 06:25 ceph-mon.wkm01.log.1.gz
-rw-r--r--  1 root root   2.2M Dec  2 06:25 ceph-mon.wkm01.log.2.gz
-rw-r--r--  1 root root  0 Dec  2 06:25 ceph-osd.0.log
-rw-r--r--  1 root root992 Dec  1 09:09 ceph-osd.0.log.1.gz
-rw-r--r--  1 root root19K Dec  3 10:51 ceph-osd.2.log
-rw-r--r--  1 root root   2.3K Dec  2 10:50 ceph-osd.2.log.1.gz
-rw-r--r--  1 root root27K Dec  1 10:31 ceph-osd.2.log.2.gz
-rw-r--r--  1 root root13K Dec  3 10:23 ceph-osd.5.log
-rw-r--r--  1 root root   1.6K Dec  2 09:57 ceph-osd.5.log.1.gz
-rw-r--r--  1 root root22K Dec  1 09:51 ceph-osd.5.log.2.gz
-rw-r--r--  1 root root19K Dec  3 10:51 ceph-osd.8.log
-rw-r--r--  1 root root18K Dec  2 10:50 ceph-osd.8.log.1
-rw-r--r--  1 root root   261K Dec  1 13:54 ceph-osd.8.log.2

I deployed ceph cluster on Nov 21, from that day to Dec.1, I mean the continue 
10 days' logs were compressed into one file, it is not what I want.
Does any OP affect log compressing?

Thanks!
Kongming Wu
-
wukongming ID: 12019
Tel：0571-86760239
Dept：2014 UIS2 ONEStor

-邮件原件-
发件人: huang jun [mailto:hjwsm1...@gmail.com]
发送时间: 2015年12月3日 13:19
收件人: wukongming 12019 (RD)
抄送: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com
主题: Re: How long will the logs be kept?

it will rotate every week by default, you can see the logrotate file
/etc/ceph/logrotate.d/ceph

2015-12-03 12:37 GMT+08:00 Wukongming :

Hi ,All
Is there anyone who knows How long or how many days will the logs.gz 
(mon/osd/mds)be kept, maybe before flushed?

-
wukongming ID: 12019
Tel：0571-86760239
Dept：2014 UIS2 OneStor

-
-
---
本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from
H3C, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure,
reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error,
please notify the sender by phone or email immediately and delete it!



--
thanks
huangjun
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

N�r��y���b�X��ǧv�^�)޺{.n�+���z�]z���{ay�ʇڙ�,j��f���h���z��w���
���j:+v���w�j�mzZ+�ݢj"��!tml=


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd merge-diff error

[Josh Durgin]

On 12/07/2015 03:29 PM, Alex Gorbachev wrote:

When trying to merge two results of rbd export-diff, the following error
occurs:

iss@lab2-b1:~$ rbd export-diff --from-snap autosnap120720151500
spin1/scrun1@autosnap120720151502 /data/volume1/scrun1-120720151502.bck

iss@lab2-b1:~$ rbd export-diff --from-snap autosnap120720151504
spin1/scrun1@autosnap120720151504 /data/volume1/scrun1-120720151504.bck

iss@lab2-b1:~$ rbd merge-diff /data/volume1/scrun1-120720151502.bck
/data/volume1/scrun1-120720151504.bck /data/volume1/mrg-scrun1-0204.bck
 Merging image diff: 11% complete...failed.
rbd: merge-diff error

That's all the output and I have found this link
http://tracker.ceph.com/issues/12911 but not sure if the patch should
have already been in hammer or how to get it?


That patch fixed a bug that was only present after hammer, due to
parallelizing export-diff. You're likely seeing a different (possibly
new) issue.

Unfortunately there's not much output we can enable for export-diff in
hammer. Could you try running the command via gdb to figure out where
and why it's failing? Make sure you have librbd-dbg installed, then
send the output from gdb doing:

gdb --args rbd merge-diff /data/volume1/scrun1-120720151502.bck \
/data/volume1/scrun1-120720151504.bck /data/volume1/mrg-scrun1-0204.bck
break rbd.cc:1931
break rbd.cc:1935
break rbd.cc:1967
break rbd.cc:1985
break rbd.cc:1999
break rbd.cc:2008
break rbd.cc:2021
break rbd.cc:2053
break rbd.cc:2098
run
# (it will run now, stopping when it hits the error)
info locals

Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rbd merge-diff error

[Alex Gorbachev]

When trying to merge two results of rbd export-diff, the following error
occurs:

iss@lab2-b1:~$ rbd export-diff --from-snap autosnap120720151500
spin1/scrun1@autosnap120720151502 /data/volume1/scrun1-120720151502.bck


iss@lab2-b1:~$ rbd export-diff --from-snap autosnap120720151504
spin1/scrun1@autosnap120720151504 /data/volume1/scrun1-120720151504.bck

iss@lab2-b1:~$ rbd merge-diff /data/volume1/scrun1-120720151502.bck
/data/volume1/scrun1-120720151504.bck /data/volume1/mrg-scrun1-0204.bck
Merging image diff: 11% complete...failed.
rbd: merge-diff error

That's all the output and I have found this link
http://tracker.ceph.com/issues/12911 but not sure if the patch should have
already been in hammer or how to get it?

System: ceph  version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
Ubuntu 14.04.3 kernel 4.2.1-040201-generic

Thank you
--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd wasn't marked as down/out when it's storage folder was deleted

[Kane Kim]

I've deleted:
rm -rf /var/lib/ceph/osd/ceph-0/current

folder but looks like ceph never noticed that:
ceph osd df
ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  PGS
 4 0.09270  1.0 97231M 54661M 42570M 56.22 1.34  95
 0 0.09270  1.0 97231M   139M 97091M  0.14 0.00  69
 5 0.09270  1.0 97231M 42603M 54627M 43.82 1.05  73
 1 0.09270  1.0 97231M 51950M 45280M 53.43 1.28  91
 6 0.09270  1.0 97231M 50656M 46575M 52.10 1.25  88
 2 0.09270  1.0 97231M 43896M 53334M 45.15 1.08  76
  TOTAL   569G   238G   331G 41.81
MIN/MAX VAR: 0.00/1.34  STDDEV: 19.15

ceph -w:
 health HEALTH_OK

Waited approximately an hour and it never reported warning, nor copied data
back.
OSD log constantly filled with:
2015-12-07 15:31:06.096017 7f007505d700 -1
filestore(/var/lib/ceph/osd/ceph-0) could not find
1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No
such file or directory
2015-12-07 15:31:06.096023 7f007505d700  0
filestore(/var/lib/ceph/osd/ceph-0) write couldn't open
1.4f_head/1/08ce93cf/rbd_data.853f1f358346.13ca/head: (2) No
such file or directory
2015-12-07 15:31:06.096044 7f007505d700 -1
filestore(/var/lib/ceph/osd/ceph-0) could not find
1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No
such file or directory
2015-12-07 15:31:06.249584 7f007585e700 -1
filestore(/var/lib/ceph/osd/ceph-0) could not find
1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No
such file or directory
2015-12-07 15:31:06.249641 7f007585e700 -1
filestore(/var/lib/ceph/osd/ceph-0) could not find
1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No
such file or directory
2015-12-07 15:31:06.249653 7f007585e700  0
filestore(/var/lib/ceph/osd/ceph-0) write couldn't open
1.4f_head/1/08ce93cf/rbd_data.853f1f358346.13ca/head: (2) No
such file or directory
2015-12-07 15:31:06.249692 7f007585e700 -1
filestore(/var/lib/ceph/osd/ceph-0) could not find
1/08ce93cf/rbd_data.853f1f358346.13ca/head in index: (2) No
such file or directory

Is it a bug or intended behavior?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Kernel RBD hang on OSD Failure

[Blair Bethwaite]

Hi Matt,

(CC'ing in ceph-users too - similar reports there:
http://www.spinics.net/lists/ceph-users/msg23037.html)

We've seen something similar for KVM [lib]RBD clients acting as NFS
gateways within our OpenStack cloud, the NFS services were locking up
and causing client timeouts whenever we started doing Ceph
maintenance. We eventually realised we'd somehow set the pool min_size
== size, so any single OSD outage was blocking client IO - *oops*.
Your issue sounds like something different, but NFS does seem to be
very touchy and lacking any graceful recovery from issues with the
underlying FS.


On 8 December 2015 at 07:56, Matt Conner  wrote:
> Hi,
>
> We have a Ceph cluster in which we have been having issues with RBD
> clients hanging when an OSD failure occurs. We are using a NAS gateway
> server which maps RBD images to filesystems and serves the filesystems
> out via NFS. The gateway server has close to 180 NFS clients and
> almost every time even 1 OSD goes down during heavy load, the NFS
> exports lock up and the clients are unable to access the NAS share via
> NFS. When the OSD fails, Ceph recovers without issue, but the gateway
> kernel RBD module appears to get stuck waiting on the now failed OSD.
> Note that this works correctly when under lighter loads.
>
> From what we have been able to determine, the NFS server daemon hangs
> waiting for I/O from the OSD that went out and never recovers.
> Similarly, attempting to access files from the exported FS locally on
> the gateway server will result in a similar hang. We also noticed that
> Ceph health details will continue to report blocked I/O on the now
> down OSD until either the OSD is recovered or the gateway server is
> rebooted.  Based on a few kernel logs from NFS and PVS, we were able
> to trace the problem to the RBD kernel module.
>
> Unfortunately, the only way we have been able to recover our gateway
> is by hard rebooting the server.
>
> Has anyone else encountered this issue and/or have a possible solution?
> Are there suggestions for getting more detailed debugging information
> from the RBD kernel module?
>
>
> Few notes on our setup:
> We are using Kernel RBD on a gateway server that exports filesystems via NFS
> The exported filesystems are XFS on LVMs which are each composed of 16
> striped images (NFS->LVM->XFS->PVS->RBD)
> There are currently 176 mapped RBD images on the server (11
> filesystems, 16 mapped RBD images per FS)
> Gateway Kernel: 3.18.6
> Ceph version: 0.80.9
> Note - We've tried using different kernels all the way up to 4.3.0 but
> the problem persists.
>
> Thanks,
> Matt Conner
> Keeper Technology
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd process threads stack up on osds failure

[Kostis Fardelas]

Hi Greg,
the node reboot unexpectedly. The timeline goes like this according to
ceph cluster logs:
12:36:56 - 12:37:02 osds reported down
12:42:00 - 12:42:05 osds reported out
13:50:44 - 13:50:49 osds booted again

The thread count in all other OSD nodes was ramping up from 12:36
until appr. 14:00

The cluster recovered at about 16:20. I have not restarted any OSD
till now. Nothing else happened in the cluster in the meanwhile. There
was no ERR/WRN in cluster's log.

Regards,
Kostis

On 7 December 2015 at 17:08, Gregory Farnum  wrote:
> On Mon, Dec 7, 2015 at 6:59 AM, Kostis Fardelas  wrote:
>> Hi cephers,
>> after one OSD node crash (6 OSDs in total), we experienced an increase
>> of approximately 230-260 threads for every other OSD node. We have 26
>> OSD nodes with 6 OSDs per node, so this is approximately 40 threads
>> per osd. The OSD node has joined the cluster after 15-20 minutes.
>>
>> The only workaround I have found so far is to restart the OSDs of the
>> cluster, but this is a quite heavy operation. Could you help me
>> understand if the behaviour described above is an expected one and
>> what could be the reason for this? Does ceph cleanup appropriately osd
>> processes threads?
>>
>> Extra info: all threads are in sleeping state right now and context
>> switches have been stabilized at the pre-crash levels
>
> Can you describe exactly what you observed with time intervals? Eg:
> did the OSDs get restarted after crashing, and how did the thread
> counts relate to that. Did anything else happen in the cluster while
> this was happening. How long did you wait before you began restarting
> OSDs to reduce the thread counts.
> -Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] scrub error with ceph

[Erming Pei]

Hi,

   I found there are 128 scrub errors in my ceph system.  Checked with
health detail and found many pgs with stuck unclean issue. Should I repair
all of them? Or what I should do?

[root@gcloudnet ~]# ceph -s

cluster a4d0879f-abdc-4f9d-8a4b-53ce57d822f1

 health HEALTH_ERR 128 pgs inconsistent; 128 scrub errors; mds1: Client
HTRC:cephfs_data failing to respond to cache pressure; mds0: Client
physics-007:cephfs_data failing to respond to cache pressure; pool
'cephfs_data' is full

 monmap e3: 3 mons at
{gcloudnet=xxx.xxx.xxx.xxx:6789/0,gcloudsrv1=xxx.xxx.xxx.xxx:6789/0,gcloudsrv2=xxx.xxx.xxx.xxx:6789/0},
election epoch 178, quorum 0,1,2 gcloudnet,gcloudsrv1,gcloudsrv2

 mdsmap e51000: 2/2/2 up {0=gcloudsrv1=up:active,1=gcloudnet=up:active}

 osdmap e2821: 18 osds: 18 up, 18 in

  pgmap v10457877: 3648 pgs, 23 pools, 10501 GB data, 38688 kobjects

14097 GB used, 117 TB / 130 TB avail

   6 active+clean+scrubbing+deep

3513 active+clean

 128 active+clean+inconsistent

   1 active+clean+scrubbing


P.S. I am increasing the pg and pgp numbers for cephfs_data pool.

Thanks,

Erming



-- 


Erming Pei, Ph.D, Senior System Analyst
HPC Grid/Cloud Specialist, ComputeCanada/WestGrid

Research Computing Group, IST
University of Alberta, Canada T6G 2H1
Email: erm...@ualberta.ca   erming@cern.ch
Tel. : +1 7804929914Fax: +1 7804921729
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD:s failing out after upgrade to 9.2.0 on Ubuntu 14.04

[Claes Sahlström]

Hi all,

Sorry to bother with this, I am trying my best to solve it, but I am quite 
stuck. I will continue to try to dig out more information any way I can, but I 
am quite clueless right now when it comes to why my OSDs don´t come up. Any 
help is of course very appreciated.

I increased the logging by setting the loglevel in my ceph.conf anf I got 
thousands of rows, but I have gone through them to just paste the ones that are 
not variations of the same thing in my osd-log and I hope this does npt get too 
long for the list:

2015-12-07 19:00:04.508322 7f9294de8940  0 ceph version 9.2.0 
(bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd, pid 5904
2015-12-07 19:00:04.577887 7f9294de8940  0 filestore(/ceph/osd.0) backend xfs 
(magic 0x58465342)
2015-12-07 19:00:04.582495 7f9294de8940  0 genericfilestorebackend(/ceph/osd.0) 
detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-12-07 19:00:04.582544 7f9294de8940  0 genericfilestorebackend(/ceph/osd.0) 
detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' 
config option
2015-12-07 19:00:04.582603 7f9294de8940  0 genericfilestorebackend(/ceph/osd.0) 
detect_features: splice is supported
2015-12-07 19:00:04.587960 7f9294de8940  0 genericfilestorebackend(/ceph/osd.0) 
detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2015-12-07 19:00:04.588230 7f9294de8940  0 xfsfilestorebackend(/ceph/osd.0) 
detect_features: extsize is supported and your kernel >= 3.5
2015-12-07 19:00:04.807548 7f9294de8940  0 filestore(/ceph/osd.0) mount: 
enabling WRITEAHEAD journal mode: checkpoint is not enabled
2015-12-07 19:00:39.087821 7f9294de8940  1 journal _open 
/dev/black/journal-osd.0 fd 19: 23622320128 bytes, block size 4096 bytes, 
directio = 1, aio = 1
2015-12-07 19:00:39.117817 7f9294de8940  1 journal _open 
/dev/black/journal-osd.0 fd 19: 23622320128 bytes, block size 4096 bytes, 
directio = 1, aio = 1
2015-12-07 19:00:39.166074 7f9294de8940  1 filestore(/ceph/osd.0) upgrade
2015-12-07 19:00:39.184942 7f9294de8940  0  cls/cephfs/cls_cephfs.cc:136: 
loading cephfs_size_scan
2015-12-07 19:00:39.186470 7f9294de8940  0  cls/hello/cls_hello.cc:305: 
loading cls_hello
2015-12-07 19:00:39.205219 7f9294de8940  0 osd.0 39538 crush map has features 
1107558400, adjusting msgr requires for clients
2015-12-07 19:00:39.205261 7f9294de8940  0 osd.0 39538 crush map has features 
1107558400 was 8705, adjusting msgr requires for mons
2015-12-07 19:00:39.205273 7f9294de8940  0 osd.0 39538 crush map has features 
1107558400, adjusting msgr requires for osds
2015-12-07 19:01:19.311676 7f9294de8940  0 osd.0 39538 load_pgs
2015-12-07 19:02:08.559239 7f9294de8940  0 osd.0 39538 load_pgs opened 1230 pgs
2015-12-07 19:02:08.593503 7f9294de8940 -1 osd.0 39538 log_to_monitors 
{default=true}
2015-12-07 19:02:08.609909 7f9276a63700  0 osd.0 39538 ignoring osdmap until we 
have initialized
2015-12-07 19:02:09.403148 7f9268ef3700  1 osd.0 pg_epoch: 39538 pg[8.ff8( v 
29836'11734 (15269'8693,29836'11734] local-les=19376 n=728 ec=4388 les/c/f 
19376/19392/0 39491/39491/39491) [4] r=-1 lpr=39493 pi=19328-39490/4 
crt=29836'11732 lcod 0'0 inactive NOTIFY NIBBLEWISE] state: 
transitioning to Stray
2015-12-07 19:02:09.404974 7f92696f4700  1 osd.0 pg_epoch: 39538 pg[8.f7f( v 
29322'12104 (15269'8967,29322'12104] local-les=20939 n=775 ec=4388 les/c/f 
20939/20939/0 39492/39492/39492) [5] r=-1 lpr=39493 pi=19359-39491/4 
crt=29322'12095 lcod 0'0 inactive NOTIFY NIBBLEWISE] state: 
transitioning to Stray
2015-12-07 19:02:09.405181 7f9268ef3700  1 osd.0 pg_epoch: 39538 pg[8.fe6( v 
29826'12135 (15269'9073,29826'12135] local-les=39037 n=801 ec=4388 les/c/f 
39037/39038/0 39522/39522/39522) [6] r=-1 lpr=39522 pi=36394-39521/15 
crt=29322'12125 lcod 0'0 inactive NOTIFY NIBBLEWISE] state: 
transitioning to Stray
2015-12-07 19:02:09.407047 7f92696f4700  1 osd.0 pg_epoch: 39538 pg[8.f71( v 
29826'11670 (15269'8574,29826'11670] local-les=38783 n=732 ec=4388 les/c/f 
38783/38783/0 39513/39513/39513) [5] r=-1 lpr=39513 pi=38782-39512/9 
crt=29322'11659 lcod 0'0 inactive NOTIFY NIBBLEWISE] state: 
transitioning to Stray
2015-12-07 19:02:09.408078 7f92696f4700  1 osd.0 pg_epoch: 39538 pg[8.f68( v 
29836'12136 (15269'9105,29836'12136] local-les=38443 n=795 ec=4388 les/c/f 
38443/38445/0 39527/39527/39527) [5] r=-1 lpr=39527 pi=38282-39526/16 
crt=29836'12129 lcod 0'0 inactive NOTIFY NIBBLEWISE] state: 
transitioning to Stray
2015-12-07 19:02:09.685544 7f9268ef3700  1 osd.0 pg_epoch: 39538 pg[8.fe4( v 
29836'12151 (15269'9151,29836'12151] local-les=37035 n=794 ec=4388 les/c/f 
37035/37035/0 39493/39493/39493) [6] r=-1 lpr=39493 pi=35488-39492/10 crt=0'0 
lcod 0'0 inactive NOTIFY NIBBLEWISE] state: transitioning to Stray
2015-12-07 19:02:09.688751 7f9268ef3700  1 osd.0 pg_epoch: 39538 pg[8.fdb( v 
29826'11846 (15269'8846,29826'11846] local-les=38885 n=773 ec=4388 les/c/f 
38885/38886/0 39513/39513/39513) [4] r=-1 lpr=39513 pi=38884-39512/

[ceph-users] CEPH Replication

[Le Quang Long]

Hi all,

I have one question

Why did default replication change to 3 in Ceph Firefly? I think 2 copys of
object is enough for backup. And increase the number of replication also
increase latency when object has to be replicated to secondary and tertiory
OSD.

So why default replication is 3, not 2 or 4 or other numbers?

Thanks and regards.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Another script to make backups/replication of RBD images

[Vandeir Eduardo]

Hi, just sharing a little script I made to backup/replicate RBD images:

https://github.com/vandeir/rbd-backup

It was made based on the script available in this URL:
https://www.rapide.nl/blog/item/ceph_-_rbd_replication.html

It is used to create snapshots of Ceph RBD images e then export those snaps
to be imported in images on another Ceph cluster. The difference in this
script relies on its ability to keep just a defined number of snapshots,
removing older ones.

Hope it will be helpfull.

Att.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd process threads stack up on osds failure

[Gregory Farnum]

On Mon, Dec 7, 2015 at 6:59 AM, Kostis Fardelas  wrote:
> Hi cephers,
> after one OSD node crash (6 OSDs in total), we experienced an increase
> of approximately 230-260 threads for every other OSD node. We have 26
> OSD nodes with 6 OSDs per node, so this is approximately 40 threads
> per osd. The OSD node has joined the cluster after 15-20 minutes.
>
> The only workaround I have found so far is to restart the OSDs of the
> cluster, but this is a quite heavy operation. Could you help me
> understand if the behaviour described above is an expected one and
> what could be the reason for this? Does ceph cleanup appropriately osd
> processes threads?
>
> Extra info: all threads are in sleeping state right now and context
> switches have been stabilized at the pre-crash levels

Can you describe exactly what you observed with time intervals? Eg:
did the OSDs get restarted after crashing, and how did the thread
counts relate to that. Did anything else happen in the cluster while
this was happening. How long did you wait before you began restarting
OSDs to reduce the thread counts.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd_inst.create

[Jason Dillaman]

lock_exclusive() / lock_shared() methods are not related to image watchers.  
Instead, it is tied to the advisory locking mechanism -- and list_lockers() can 
be used to query who has a lock.

-- 

Jason Dillaman 


- Original Message -
> From: "NEVEU Stephane" 
> To: "Jason Dillaman" 
> Cc: "Ceph Users" , "Gregory Farnum" 
> 
> Sent: Monday, December 7, 2015 4:09:45 AM
> Subject: RE: [ceph-users] rbd_inst.create
> 
> Hi,
> 
> One more question rbd.py / rados.py, is there a way to retrieve watchers on
> rbd images ? ImageBusy class is raised when trying to use lock_exclusive()
> or lock_shared() but I cannot find how to list watchers...
> Thanks
> 
> [@@ THALES GROUP INTERNAL @@]
> 
> -Message d'origine-
> De : Jason Dillaman [mailto:dilla...@redhat.com]
> Envoyé : lundi 30 novembre 2015 15:38
> À : NEVEU Stephane
> Cc : Ceph Users; Gregory Farnum
> Objet : Re: [ceph-users] rbd_inst.create
> 
> ... and once you create a pool-level snapshot on a pool, there is no way to
> convert that pool back to being compatible with RBD self-managed snapshots.
> 
> As for the RBD image feature bits, they are defined within rbd.py.  On
> master, they currently are as follows:
> 
> RBD_FEATURE_LAYERING = 1
> RBD_FEATURE_STRIPINGV2 = 2
> RBD_FEATURE_EXCLUSIVE_LOCK = 4
> RBD_FEATURE_OBJECT_MAP = 8
> RBD_FEATURE_FAST_DIFF = 16
> RBD_FEATURE_DEEP_FLATTEN = 32
> RBD_FEATURE_JOURNALING = 64
> 
> --
> 
> Jason Dillaman
> 
> 
> - Original Message -
> 
> > From: "Gregory Farnum" 
> > To: "NEVEU Stephane" 
> > Cc: "Ceph Users" 
> > Sent: Monday, November 30, 2015 8:17:17 AM
> > Subject: Re: [ceph-users] rbd_inst.create
> 
> > On Nov 27, 2015 3:34 AM, "NEVEU Stephane" <
> > stephane.ne...@thalesgroup.com >
> > wrote:
> > >
> > > Ok, I think I got it. It seems to come from here :
> > >
> > > tracker.ceph.com/issues/6047
> > >
> > >
> > >
> > > I’m trying to snapshot an image while I previously made a snapshot
> > > of my pool… whereas it just works fine when using a brand new pool.
> > > I’m using ceph v0.80.10 on Ubuntu 14.04. As I see, it has been
> > > patched since dumpling. Could it be a regression ?
> > Pool snapshots and the "self-managed" snapshots used by rbd are
> > incompatible.
> > You have to pick one or the other on each pool.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > De : ceph-users [mailto: ceph-users-boun...@lists.ceph.com ] De la
> > > part de NEVEU Stephane Envoyé : jeudi 26 novembre 2015 15:49 À :
> > > ceph-users@lists.ceph.com Objet : [ceph-users] rbd_inst.create
> > >
> > >
> > >
> > > Hi all,
> > >
> > >
> > >
> > > I’m using python scripts to create rbd images like described here
> > > http://docs.ceph.com/docs/giant/rbd/librbdpy/
> > >
> > > rbd_inst.create(ioctx, 'myimage', size, old_format=False,
> > > features=1) seems to create a layering image
> > >
> > > rbd_inst.create(ioctx, 'myimage', size, old_format=False,
> > > features=2) seems to create a stripped image
> > >
> > >
> > >
> > > Setting up “rbd default format =2” in ceph.conf and just using the
> > > following (without feature=x)
> > >
> > > rbd_inst.create(ioctx, 'myimage', size) seems to create a layered +
> > > stripped image
> > >
> > >
> > >
> > > If someone could point me to the documentation about those bitmasks
> > > (features), that would be great J I cannot find it.
> > >
> > >
> > >
> > > Moreover, when my images are created this way (using rbd_inst.create
> > > with python), no way to snapshot an image !
> > >
> > > #rbd snap create rbd/myimage@snap1
> > >
> > > …. -1 librbd: failed to create snap id: (22) Invalid argument
> > >
> > >
> > >
> > > Same thing with img.create_snap(snap) in python, snapshots are not
> > > created.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > [@@ THALES GROUP INTERNAL @@]
> > >
> > >
> > >
> > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> 
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd process threads stack up on osds failure

[Kostis Fardelas]

Hi cephers,
after one OSD node crash (6 OSDs in total), we experienced an increase
of approximately 230-260 threads for every other OSD node. We have 26
OSD nodes with 6 OSDs per node, so this is approximately 40 threads
per osd. The OSD node has joined the cluster after 15-20 minutes.

The only workaround I have found so far is to restart the OSDs of the
cluster, but this is a quite heavy operation. Could you help me
understand if the behaviour described above is an expected one and
what could be the reason for this? Does ceph cleanup appropriately osd
processes threads?

Extra info: all threads are in sleeping state right now and context
switches have been stabilized at the pre-crash levels

Regards,
Kostis
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] poor performance when recovering

[Oliver Dzombic]

Hi,

maybe you should first upgrade.

"

Posted by sage
November 19th, 2015

This is a bugfix release for Firefly.  As the Firefly 0.80.x series is
nearing its planned end of life in January 2016 it may also be the last.
"

I think you are wasting time, trying to analyse/fix issues on a version
which will be EOL in 3 weeks...


-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic

Am 07.12.2015 um 15:26 schrieb Libin Wu:
> Btw, my ceph version is 0.80.11
> 
> 2015-12-07 21:45 GMT+08:00 Libin Wu :
>> Hi, cephers
>>
>> I'm doing the performance test of ceph when recovering. The scene is simple:
>> 1. run fio on 6 krbd device
>> 2. stop one OSD for 10 seconds
>> 3. start that OSD
>>
>> However, when the OSD up and start recovering, the performance of fio
>> drop down from 9k to 1k for about 20 seconds. At the same tiime, we
>> found the SSD of that OSD's latency is more than 100ms, so it seems
>> the SSD become the bottleneck。
>>
>> So we want to slow down the recovery speed to lighten the load of the
>> SSD when recovery. But configuration like:
>> osd_recovery_max_active
>> osd_recovery_max_chunk
>> osd_max_backfills
>> osd_recovery_op_priority
>> are all useless.
>>
>> After reading and change some code, we want add a flow control in the
>> process of:
>> OSD::do_recovery
>>
>> So, will it be possible to do so and if this solution has some
>> potential problem?
>>
>> Thanks!
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] poor performance when recovering

[Libin Wu]

Btw, my ceph version is 0.80.11

2015-12-07 21:45 GMT+08:00 Libin Wu :
> Hi, cephers
>
> I'm doing the performance test of ceph when recovering. The scene is simple:
> 1. run fio on 6 krbd device
> 2. stop one OSD for 10 seconds
> 3. start that OSD
>
> However, when the OSD up and start recovering, the performance of fio
> drop down from 9k to 1k for about 20 seconds. At the same tiime, we
> found the SSD of that OSD's latency is more than 100ms, so it seems
> the SSD become the bottleneck。
>
> So we want to slow down the recovery speed to lighten the load of the
> SSD when recovery. But configuration like:
> osd_recovery_max_active
> osd_recovery_max_chunk
> osd_max_backfills
> osd_recovery_op_priority
> are all useless.
>
> After reading and change some code, we want add a flow control in the
> process of:
> OSD::do_recovery
>
> So, will it be possible to do so and if this solution has some
> potential problem?
>
> Thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] poor performance when recovering

[Libin Wu]

Hi, cephers

I'm doing the performance test of ceph when recovering. The scene is simple:
1. run fio on 6 krbd device
2. stop one OSD for 10 seconds
3. start that OSD

However, when the OSD up and start recovering, the performance of fio
drop down from 9k to 1k for about 20 seconds. At the same tiime, we
found the SSD of that OSD's latency is more than 100ms, so it seems
the SSD become the bottleneck。

So we want to slow down the recovery speed to lighten the load of the
SSD when recovery. But configuration like:
osd_recovery_max_active
osd_recovery_max_chunk
osd_max_backfills
osd_recovery_op_priority
are all useless.

After reading and change some code, we want add a flow control in the
process of:
OSD::do_recovery

So, will it be possible to do so and if this solution has some
potential problem?

Thanks!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-disk list crashes in infernalis

[Loic Dachary]

Thanks !

On 06/12/2015 17:50, Stolte, Felix wrote:
> Hi Loic,
> 
> output is:
> 
> /dev:
> insgesamt 0
> crw--- 1 root root 10, 235 Dez  2 17:02 autofs
> drwxr-xr-x 2 root root1000 Dez  2 17:02 block
> drwxr-xr-x 2 root root  60 Dez  2 17:02 bsg
> crw--- 1 root root 10, 234 Dez  5 06:29 btrfs-control
> drwxr-xr-x 3 root root  60 Dez  2 17:02 bus
> crw-r--r-- 1 root root255, 171 Dez  2 17:02 casr
> drwxr-xr-x 2 root root 500 Dez  2 17:02 cciss
> crw-r--r-- 1 root root255, 173 Dez  2 17:02 ccsm
> lrwxrwxrwx 1 root root   3 Dez  2 17:02 cdrom -> sr0
> crw-r--r-- 1 root root255, 178 Dez  2 17:02 cdt
> crw-r--r-- 1 root root255, 172 Dez  2 17:02 cecc
> crw-r--r-- 1 root root255, 176 Dez  2 17:02 cevt
> drwxr-xr-x 2 root root3820 Dez  5 06:29 char
> crw--- 1 root root  5,   1 Dez  2 17:04 console
> lrwxrwxrwx 1 root root  11 Dez  2 17:02 core -> /proc/kcore
> drw-r--r-- 2 root root 200 Dez  2 17:02 cpqhealth
> drwxr-xr-x 2 root root  60 Dez  2 17:02 cpu
> crw--- 1 root root 10,  60 Dez  2 17:02 cpu_dma_latency
> crw-r--r-- 1 root root255, 180 Dez  2 17:02 crom
> crw--- 1 root root 10, 203 Dez  2 17:02 cuse
> drwxr-xr-x 8 root root 160 Dez  2 17:02 disk
> drwxr-xr-x 2 root root 100 Dez  2 17:02 dri
> crw--- 1 root root 10,  61 Dez  2 17:02 ecryptfs
> crw-rw 1 root video29,   0 Dez  2 17:02 fb0
> lrwxrwxrwx 1 root root  13 Dez  2 17:02 fd -> /proc/self/fd
> crw-rw-rw- 1 root root  1,   7 Dez  2 17:02 full
> crw-rw-rw- 1 root root 10, 229 Dez  2 17:02 fuse
> crw--- 1 root root251,   0 Dez  2 17:02 hidraw0
> crw--- 1 root root251,   1 Dez  2 17:02 hidraw1
> crw--- 1 root root 10, 228 Dez  2 17:02 hpet
> drwxr-xr-x 2 root root 360 Dez  2 17:02 hpilo
> crw--- 1 root root 89,   0 Dez  2 17:02 i2c-0
> crw--- 1 root root 89,   1 Dez  2 17:02 i2c-1
> crw--- 1 root root 89,   2 Dez  2 17:02 i2c-2
> crw--- 1 root root 89,   3 Dez  2 17:02 i2c-3
> crw-r--r-- 1 root root255, 184 Dez  2 17:02 indc
> drwxr-xr-x 4 root root 200 Dez  2 17:02 input
> crw--- 1 root root248,   0 Dez  2 17:02 ipmi0
> crw--- 1 root root249,   0 Dez  2 17:02 kfd
> crw-r--r-- 1 root root  1,  11 Dez  2 17:02 kmsg
> srw-rw-rw- 1 root root   0 Dez  2 17:02 log
> brw-rw 1 root disk  7,   0 Dez  2 17:02 loop0
> brw-rw 1 root disk  7,   1 Dez  2 17:02 loop1
> brw-rw 1 root disk  7,   2 Dez  2 17:02 loop2
> brw-rw 1 root disk  7,   3 Dez  2 17:02 loop3
> brw-rw 1 root disk  7,   4 Dez  2 17:02 loop4
> brw-rw 1 root disk  7,   5 Dez  2 17:02 loop5
> brw-rw 1 root disk  7,   6 Dez  2 17:02 loop6
> brw-rw 1 root disk  7,   7 Dez  2 17:02 loop7
> crw--- 1 root root 10, 237 Dez  2 17:02 loop-control
> drwxr-xr-x 2 root root  60 Dez  2 17:02 mapper
> crw--- 1 root root 10, 227 Dez  2 17:02 mcelog
> crw-r- 1 root kmem  1,   1 Dez  2 17:02 mem
> crw--- 1 root root 10,  57 Dez  2 17:02 memory_bandwidth
> crw--- 1 root root 10, 220 Dez  2 17:02 mptctl
> drwxr-xr-x 2 root root  60 Dez  2 17:02 net
> crw--- 1 root root 10,  59 Dez  2 17:02 network_latency
> crw--- 1 root root 10,  58 Dez  2 17:02 network_throughput
> crw-rw-rw- 1 root root  1,   3 Dez  2 17:02 null
> crw-r- 1 root kmem  1,   4 Dez  2 17:02 port
> crw--- 1 root root108,   0 Dez  2 17:02 ppp
> crw-r--r-- 1 root root255, 183 Dez  2 17:02 proc
> crw--- 1 root root 10,   1 Dez  2 17:02 psaux
> crw-rw-rw- 1 root tty   5,   2 Dez  6 17:47 ptmx
> drwxr-xr-x 2 root root   0 Dez  2 17:02 pts
> brw-rw 1 root disk  1,   0 Dez  2 17:02 ram0
> brw-rw 1 root disk  1,   1 Dez  2 17:02 ram1
> brw-rw 1 root disk  1,  10 Dez  2 17:02 ram10
> brw-rw 1 root disk  1,  11 Dez  2 17:02 ram11
> brw-rw 1 root disk  1,  12 Dez  2 17:02 ram12
> brw-rw 1 root disk  1,  13 Dez  2 17:02 ram13
> brw-rw 1 root disk  1,  14 Dez  2 17:02 ram14
> brw-rw 1 root disk  1,  15 Dez  2 17:02 ram15
> brw-rw 1 root disk  1,   2 Dez  2 17:02 ram2
> brw-rw 1 root disk  1,   3 Dez  2 17:02 ram3
> brw-rw 1 root disk  1,   4 Dez  2 17:02 ram4
> brw-rw 1 root disk  1,   5 Dez  2 17:02 ram5
> brw-rw 1 root disk  1,   6 Dez  2 17:02 ram6
> brw-rw 1 root disk  1,   7 Dez  2 17:02 ram7
> brw-rw 1 root disk  1,   8 Dez  2 17:02 ram8
> brw-rw 1 root disk  1,   9 Dez  2 17:02 ram9
> crw-rw-rw- 1 root root  1,   8 Dez  2 17:02 random
> crw-r--r-- 1 root root 10,  62 Dez  2 17:02 rfkill
> lrwxrwxrwx 1 root root   4 Dez  2 17:02 rtc -> rtc0
> crw--- 1 root root254,   0 Dez  2 17:02 rtc0
> crw-rw 1 root cdrom21,   0 Dez  2 17:02 sg0
> lrwxrwxrwx 1

Re: [ceph-users] 答复: 答复: how to see file object-mappings for cephfuse client

[John Spray]

On Mon, Dec 7, 2015 at 9:13 AM, Wuxiangwei  wrote:
>
> it looks simple if everything stays as its default value. However, we do want 
> to change the stripe unit size and stripe count to achieve possible higher 
> performance. If so, it would be too troublesome to manually do the 
> calculation every time when we want to locate a given offset(and maybe a 
> certain interval). The 'cephfs map' and 'cephfs show_location' can provide 
> good information as we want, but sadly not for ceph-fuse. That's why I ask 
> for a similar tool.

If you are interested in writing this, you could look at
Client::get_file_stripe_address and Striper::file_to_extents.
Currently in libcephfs we expose methods for getting the OSDs relevant
to a particular file, in case something like hadoop wants to exploit
locality, but we don't expose the intermediate knowledge of the object
names.

I am curious about why you need this?

John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] french meetup website

[eric mourgaya]

 Hi,
 I glad to  write that a new  website ( in french ) is now available. This
website is managed by the  ceph breizh  community.  You can find  a report
of the  last meetup on this page:
ceph breizh (http://ceph.bzh) 

 enjoy it and join us.



-- 
Eric Mourgaya,


Respectons la planete!
Luttons contre la mediocrite!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 答复: 答复: how to see file object-mappings for cephfuse client

[Wuxiangwei]

it looks simple if everything stays as its default value. However, we do want 
to change the stripe unit size and stripe count to achieve possible higher 
performance. If so, it would be too troublesome to manually do the calculation 
every time when we want to locate a given offset(and maybe a certain interval). 
The 'cephfs map' and 'cephfs show_location' can provide good information as we 
want, but sadly not for ceph-fuse. That's why I ask for a similar tool. 

---
Wu Xiangwei
Tel : 0571-86760875
2014 UIS 2, TEAM BORE


-邮件原件-
发件人: Yan, Zheng [mailto:uker...@gmail.com] 
发送时间: 2015年12月7日 16:44
收件人: wuxiangwei 09660 (RD)
抄送: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com
主题: Re: 答复: how to see file object-mappings for cephfuse client

On Mon, Dec 7, 2015 at 1:52 PM, Wuxiangwei  wrote:
> Thanks Yan, what if we wanna see some more specific or detailed information? 
> E.g. with cephfs we may run 'cephfs /mnt/a.txt show_location --offset' to 
> find the location of a given offset.
>

When using default layout, object size is 4M, (offset / 4194304) is subfix of 
object name. For example. offset 0 is located in object ., offset 4194304 is located in object .001.

For fancy layout, please read
http://docs.ceph.com/docs/master/dev/file-striping/

> ---
> Wu Xiangwei
> Tel : 0571-86760875
> 2014 UIS 2, TEAM BORE
>
>
> -邮件原件-
> 发件人: Yan, Zheng [mailto:uker...@gmail.com]
> 发送时间: 2015年12月7日 11:22
> 收件人: wuxiangwei 09660 (RD)
> 抄送: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com
> 主题: Re: how to see file object-mappings for cephfuse client
>
> On Mon, Dec 7, 2015 at 10:51 AM, Wuxiangwei  wrote:
>> Hi, Everyone
>>
>> Recently I'm trying to figure out how to use ceph-fuse. If we mount cephfs 
>> as the kernel client, there is a 'cephfs' command tool (though it seems to 
>> be 'deprecated') with 'map' and 'show_location' commands to show the RADOS 
>> objects belonging to a given file. However, it doesn't work well for 
>> ceph-fuse, neither can I find a similar tool to see the mappings. Any 
>> suggestions?
>> Thank you!
>
> Rados objects  for a give inode is in form ..
>
> To get a file's layout, you can use getfattr -n ceph.file.layout  name>
>
> To find a given object is stored on which OSDs, you can use command, 
> ceph osd map  
>
>>
>>
>> ---
>> Wu Xiangwei
>> Tel : 0571-86760875
>> 2014 UIS 2, TEAM BORE
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majord...@vger.kernel.org More majordomo 
>> info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd_inst.create

[NEVEU Stephane]

Hi,

One more question rbd.py / rados.py, is there a way to retrieve watchers on rbd 
images ? ImageBusy class is raised when trying to use lock_exclusive() or 
lock_shared() but I cannot find how to list watchers...
Thanks

[@@ THALES GROUP INTERNAL @@]

-Message d'origine-
De : Jason Dillaman [mailto:dilla...@redhat.com] 
Envoyé : lundi 30 novembre 2015 15:38
À : NEVEU Stephane
Cc : Ceph Users; Gregory Farnum
Objet : Re: [ceph-users] rbd_inst.create

... and once you create a pool-level snapshot on a pool, there is no way to 
convert that pool back to being compatible with RBD self-managed snapshots.

As for the RBD image feature bits, they are defined within rbd.py.  On master, 
they currently are as follows:

RBD_FEATURE_LAYERING = 1
RBD_FEATURE_STRIPINGV2 = 2
RBD_FEATURE_EXCLUSIVE_LOCK = 4
RBD_FEATURE_OBJECT_MAP = 8
RBD_FEATURE_FAST_DIFF = 16
RBD_FEATURE_DEEP_FLATTEN = 32
RBD_FEATURE_JOURNALING = 64

-- 

Jason Dillaman 


- Original Message - 

> From: "Gregory Farnum" 
> To: "NEVEU Stephane" 
> Cc: "Ceph Users" 
> Sent: Monday, November 30, 2015 8:17:17 AM
> Subject: Re: [ceph-users] rbd_inst.create

> On Nov 27, 2015 3:34 AM, "NEVEU Stephane" < 
> stephane.ne...@thalesgroup.com >
> wrote:
> >
> > Ok, I think I got it. It seems to come from here :
> >
> > tracker.ceph.com/issues/6047
> >
> >
> >
> > I’m trying to snapshot an image while I previously made a snapshot 
> > of my pool… whereas it just works fine when using a brand new pool. 
> > I’m using ceph v0.80.10 on Ubuntu 14.04. As I see, it has been 
> > patched since dumpling. Could it be a regression ?
> Pool snapshots and the "self-managed" snapshots used by rbd are incompatible.
> You have to pick one or the other on each pool.
> >
> >
> >
> >
> >
> >
> >
> > De : ceph-users [mailto: ceph-users-boun...@lists.ceph.com ] De la 
> > part de NEVEU Stephane Envoyé : jeudi 26 novembre 2015 15:49 À : 
> > ceph-users@lists.ceph.com Objet : [ceph-users] rbd_inst.create
> >
> >
> >
> > Hi all,
> >
> >
> >
> > I’m using python scripts to create rbd images like described here 
> > http://docs.ceph.com/docs/giant/rbd/librbdpy/
> >
> > rbd_inst.create(ioctx, 'myimage', size, old_format=False, 
> > features=1) seems to create a layering image
> >
> > rbd_inst.create(ioctx, 'myimage', size, old_format=False, 
> > features=2) seems to create a stripped image
> >
> >
> >
> > Setting up “rbd default format =2” in ceph.conf and just using the 
> > following (without feature=x)
> >
> > rbd_inst.create(ioctx, 'myimage', size) seems to create a layered + 
> > stripped image
> >
> >
> >
> > If someone could point me to the documentation about those bitmasks 
> > (features), that would be great J I cannot find it.
> >
> >
> >
> > Moreover, when my images are created this way (using rbd_inst.create 
> > with python), no way to snapshot an image !
> >
> > #rbd snap create rbd/myimage@snap1
> >
> > …. -1 librbd: failed to create snap id: (22) Invalid argument
> >
> >
> >
> > Same thing with img.create_snap(snap) in python, snapshots are not created.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > [@@ THALES GROUP INTERNAL @@]
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 答复: how to see file object-mappings for cephfuse client

[Yan, Zheng]

On Mon, Dec 7, 2015 at 1:52 PM, Wuxiangwei  wrote:
> Thanks Yan, what if we wanna see some more specific or detailed information? 
> E.g. with cephfs we may run 'cephfs /mnt/a.txt show_location --offset' to 
> find the location of a given offset.
>

When using default layout, object size is 4M, (offset / 4194304) is
subfix of object name. For example. offset 0 is located in object
., offset 4194304 is located in object
.001.

For fancy layout, please read
http://docs.ceph.com/docs/master/dev/file-striping/

> ---
> Wu Xiangwei
> Tel : 0571-86760875
> 2014 UIS 2, TEAM BORE
>
>
> -邮件原件-
> 发件人: Yan, Zheng [mailto:uker...@gmail.com]
> 发送时间: 2015年12月7日 11:22
> 收件人: wuxiangwei 09660 (RD)
> 抄送: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com
> 主题: Re: how to see file object-mappings for cephfuse client
>
> On Mon, Dec 7, 2015 at 10:51 AM, Wuxiangwei  wrote:
>> Hi, Everyone
>>
>> Recently I'm trying to figure out how to use ceph-fuse. If we mount cephfs 
>> as the kernel client, there is a 'cephfs' command tool (though it seems to 
>> be 'deprecated') with 'map' and 'show_location' commands to show the RADOS 
>> objects belonging to a given file. However, it doesn't work well for 
>> ceph-fuse, neither can I find a similar tool to see the mappings. Any 
>> suggestions?
>> Thank you!
>
> Rados objects  for a give inode is in form ..
>
> To get a file's layout, you can use getfattr -n ceph.file.layout 
>
> To find a given object is stored on which OSDs, you can use command, ceph osd 
> map  
>
>>
>>
>> ---
>> Wu Xiangwei
>> Tel : 0571-86760875
>> 2014 UIS 2, TEAM BORE
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majord...@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com