On 05/15/2012 06:49 AM, Andrey Korolyov wrote:
Hi,
There are strange bug when I tried to map excessive amounts of block
devices inside the pool, like following
for vol in $(rbd ls); do rbd map $vol; [some-microsleep]; [some
operation or nothing, I have stubbed guestfs mount here] ;
[some-microsleep]; unmap /dev/rbd/rbd/$vol ; [some-microsleep]; done,
This is most likely due to a recently-fixed problem.
The fix is found in this commit, although there were
other changes that led up to it:
32eec68d2f rbd: don't drop the rbd_id too early
It is present starting in Linux kernel 3.3; it appears
you are running 2.6?
-Alex
udev or rbd seems to be somehow late and mapping fails. There is no
real-world harm at all, and such case can be easily avoided, but on
busy cluster timeout increases and I was able to catch same thing on
two-osd config in recovering state. For 0.1 second on healthy cluster,
all works okay, for 0.05 it may fail with following trace(just for me,
because I am testing on relatively old and crappy hardware, so others
may catch that on smaller intervals):
[ 2130.450044] libceph: client0 fsid 70204128-4328-47e7-9df7-c7253c833fc1
[ 2130.450643] libceph: mon0 192.168.10.129:6789 session established
[ 2130.454542] rbd0: p1 p2
[ 2130.454772] rbd: rbd0: added with size 0x80000000
[ 2137.783484] libceph: client0 fsid 70204128-4328-47e7-9df7-c7253c833fc1
[ 2137.784095] libceph: mon0 192.168.10.129:6789 session established
[ 2137.787801] rbd0: p1 p2
[ 2137.788028] rbd: rbd0: added with size 0x7d000000
[ 2138.044490] ------------[ cut here ]------------
[ 2138.044499] WARNING: at
/build/kernel/linux-2.6-3.2.15/debian/build/source_amd64_none/fs/sysfs/dir.c:481
sysfs_add_one+0x83/0x96()
[ 2138.044503] Hardware name: System Product Name
[ 2138.044505] sysfs: cannot create duplicate filename
'/devices/virtual/block/rbd0'
[ 2138.044508] Modules linked in: ip6table_filter ip6_tables
iptable_filter acpi_cpufreq mperf ip_tables cpufreq_powersave
ebtable_nat cpufreq_userspace ebtables x_tables cpufreq_conservative
cpufreq_stats cn microcode ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa
ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
fuse nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc kvm_intel kvm
bridge stp ext2 rbd libceph coretemp tcp_yeah tcp_vegas loop dm_crypt
snd_hda_codec_realtek nvidia(P) snd_hda_intel snd_hda_codec snd_hwdep
nv_tco snd_pcm psmouse i2c_nforce2 i2c_core snd_page_alloc evdev
serio_raw snd_timer snd pcspkr soundcore processor button asus_atk0110
ext4 crc16 jbd2 mbcache btrfs crc32c libcrc32c zlib_deflate dm_mod
sd_mod sr_mod cdrom crc_t10dif ohci_hcd ata_generic pata_amd sata_nv
ehci_hcd libata fan forcedeth thermal thermal_sys scsi_mod usbcore
usb_common [last unloaded: scsi_wait_scan]
[ 2138.044607] Pid: 16891, comm: rbd Tainted: P O 3.2.0-2-amd64 #1
[ 2138.044610] Call Trace:
[ 2138.044616] [<ffffffff81046811>] ? warn_slowpath_common+0x78/0x8c
[ 2138.044620] [<ffffffff810468bd>] ? warn_slowpath_fmt+0x45/0x4a
[ 2138.044624] [<ffffffff8114e918>] ? sysfs_add_one+0x83/0x96
[ 2138.044628] [<ffffffff8114e991>] ? create_dir+0x66/0xa0
[ 2138.044631] [<ffffffff8114ea66>] ? sysfs_create_dir+0x85/0x9b
[ 2138.044636] [<ffffffff811afb6b>] ? vsnprintf+0x7c/0x427
[ 2138.044640] [<ffffffff811a9aa2>] ? kobject_add_internal+0xc8/0x181
[ 2138.044643] [<ffffffff811a9e77>] ? kobject_add+0x95/0xa4
[ 2138.044647] [<ffffffff810363c7>] ? should_resched+0x5/0x23
[ 2138.044651] [<ffffffff811a99d5>] ? kobject_get+0x12/0x17
[ 2138.044655] [<ffffffff8119ccb4>] ? get_disk+0x8d/0x8d
[ 2138.044659] [<ffffffff8124bd13>] ? device_add+0xd6/0x587
[ 2138.044663] [<ffffffff8124ad4e>] ? dev_set_name+0x42/0x47
[ 2138.044667] [<ffffffff8119d8db>] ? register_disk+0x37/0x147
[ 2138.044670] [<ffffffff8119ce48>] ? blk_register_region+0x22/0x27
[ 2138.044674] [<ffffffff8119db6b>] ? add_disk+0x180/0x26c
[ 2138.044681] [<ffffffffa0e994d0>] ? rbd_add+0x7b2/0xa32 [rbd]
[ 2138.044685] [<ffffffff810363c7>] ? should_resched+0x5/0x23
[ 2138.044689] [<ffffffff8114d477>] ? sysfs_write_file+0xe0/0x11c
[ 2138.044693] [<ffffffff810f92a7>] ? vfs_write+0xa2/0xe9
[ 2138.044697] [<ffffffff810f9484>] ? sys_write+0x45/0x6b
[ 2138.044701] [<ffffffff8134e492>] ? system_call_fastpath+0x16/0x1b
[ 2138.044704] ---[ end trace b7a29490cafc363d ]---
[ 2138.044708] kobject_add_internal failed for rbd0 with -EEXIST,
don't try to register things with the same name in the same directory.
[ 2138.044723] Pid: 16891, comm: rbd Tainted: P W O 3.2.0-2-amd64 #1
[ 2138.044725] Call Trace:
[ 2138.044729] [<ffffffff811a9b31>] ? kobject_add_internal+0x157/0x181
[ 2138.044733] [<ffffffff811a9e77>] ? kobject_add+0x95/0xa4
[ 2138.044736] [<ffffffff810363c7>] ? should_resched+0x5/0x23
[ 2138.044740] [<ffffffff811a99d5>] ? kobject_get+0x12/0x17
[ 2138.044743] [<ffffffff8119ccb4>] ? get_disk+0x8d/0x8d
[ 2138.044746] [<ffffffff8124bd13>] ? device_add+0xd6/0x587
[ 2138.044750] [<ffffffff8124ad4e>] ? dev_set_name+0x42/0x47
[ 2138.044757] [<ffffffff8119d8db>] ? register_disk+0x37/0x147
[ 2138.044760] [<ffffffff8119ce48>] ? blk_register_region+0x22/0x27
[ 2138.044763] [<ffffffff8119db6b>] ? add_disk+0x180/0x26c
[ 2138.044769] [<ffffffffa0e994d0>] ? rbd_add+0x7b2/0xa32 [rbd]
[ 2138.044772] [<ffffffff810363c7>] ? should_resched+0x5/0x23
[ 2138.044776] [<ffffffff8114d477>] ? sysfs_write_file+0xe0/0x11c
[ 2138.044780] [<ffffffff810f92a7>] ? vfs_write+0xa2/0xe9
[ 2138.044783] [<ffffffff810f9484>] ? sys_write+0x45/0x6b
[ 2138.044787] [<ffffffff8134e492>] ? system_call_fastpath+0x16/0x1b
[ 2138.044928] ------------[ cut here ]------------
[ 2138.044937] kernel BUG at
/build/kernel/linux-2.6-3.2.15/debian/build/source_amd64_none/fs/sysfs/group.c:65!
[ 2138.044947] invalid opcode: 0000 [#1] SMP
[ 2138.044962] CPU 1
[ 2138.044967] Modules linked in: ip6table_filter ip6_tables
iptable_filter acpi_cpufreq mperf ip_tables cpufreq_powersave
ebtable_nat cpufreq_userspace ebtables x_tables cpufreq_conservative
cpufreq_stats cn microcode ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa
ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
fuse nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc kvm_intel kvm
bridge stp ext2 rbd libceph coretemp tcp_yeah tcp_vegas loop dm_crypt
snd_hda_codec_realtek nvidia(P) snd_hda_intel snd_hda_codec snd_hwdep
nv_tco snd_pcm psmouse i2c_nforce2 i2c_core snd_page_alloc evdev
serio_raw snd_timer snd pcspkr soundcore processor button asus_atk0110
ext4 crc16 jbd2 mbcache btrfs crc32c libcrc32c zlib_deflate dm_mod
sd_mod sr_mod cdrom crc_t10dif ohci_hcd ata_generic pata_amd sata_nv
ehci_hcd libata fan forcedeth thermal thermal_sys scsi_mod usbcore
usb_common [last unloaded: scsi_wait_scan]
[ 2138.045366]
[ 2138.045380] Pid: 16891, comm: rbd Tainted: P W O
3.2.0-2-amd64 #1 System manufacturer System Product Name/P5N-D
[ 2138.045420] RIP: 0010:[<ffffffff8114fdb7>] [<ffffffff8114fdb7>]
internal_create_group+0x27/0x11f
[ 2138.045454] RSP: 0018:ffff8800b19e3d38 EFLAGS: 00010246
[ 2138.045472] RAX: 00000000ffffffef RBX: ffff8800a19aac00 RCX: 0000000000002019
[ 2138.045491] RDX: ffffffff81624cb0 RSI: 0000000000000000 RDI: ffff8800a19aac78
[ 2138.045511] RBP: ffff8800a19aac78 R08: 0000000000000002 R09: 00000000fffffffe
[ 2138.045530] R10: 0000000000000001 R11: 0000000000000002 R12: ffffffff81624cb0
[ 2138.045549] R13: ffff8800a19aac68 R14: 0000000000000000 R15: ffff880037770038
[ 2138.045569] FS: 00007ffc5219a760(0000) GS:ffff88012fc80000(0000)
knlGS:0000000000000000
[ 2138.045598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2138.045616] CR2: 00007f16285260f2 CR3: 00000000a1946000 CR4: 00000000000006e0
[ 2138.045635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2138.045655] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2138.045675] Process rbd (pid: 16891, threadinfo ffff8800b19e2000,
task ffff880117c76830)
[ 2138.045702] Stack:
[ 2138.045716] 0000000000000000 0000000000000010 ffff8800b19e3d98
0000000069dd99c3
[ 2138.045755] ffff88006f63d3c0 ffff8800a19aac00 ffff880037770038
ffff880037770038
[ 2138.045793] ffff8800a19aac68 ffff8800a19aac00 ffff880037770038
ffffffff811990b7
[ 2138.045831] Call Trace:
[ 2138.045849] [<ffffffff811990b7>] ? blk_register_queue+0x41/0xe1
[ 2138.045868] [<ffffffff8119db73>] ? add_disk+0x188/0x26c
[ 2138.045889] [<ffffffffa0e994d0>] ? rbd_add+0x7b2/0xa32 [rbd]
[ 2138.045909] [<ffffffff810363c7>] ? should_resched+0x5/0x23
[ 2138.045928] [<ffffffff8114d477>] ? sysfs_write_file+0xe0/0x11c
[ 2138.045947] [<ffffffff810f92a7>] ? vfs_write+0xa2/0xe9
[ 2138.045965] [<ffffffff810f9484>] ? sys_write+0x45/0x6b
[ 2138.045984] [<ffffffff8134e492>] ? system_call_fastpath+0x16/0x1b
[ 2138.046002] Code: 59 5b 5d c3 41 57 41 56 41 89 f6 41 55 41 54 49
89 d4 55 48 89 fd 53 48 83 ec 28 48 85 ff 74 0b 85 f6 75 09 48 83 7f
30 00 75 12<0f> 0b 48 83 7f 30 00 b8 ea ff ff ff 0f 84 d7 00 00 00 49
8b 34
[ 2138.046248] RIP [<ffffffff8114fdb7>] internal_create_group+0x27/0x11f
[ 2138.046270] RSP<ffff8800b19e3d38>
[ 2138.046587] ---[ end trace b7a29490cafc363e ]---
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html