Re: [ceph-users] mon problems after upgrading to cuttlefish

[Bryan Stillwell]

On Thu, May 23, 2013 at 9:58 AM, Smart Weblications GmbH - Florian
Wiessner  wrote:
> you may need to update your [mon.a] section in your ceph.conf like this:
>
>
> [mon.a]
>mon data = /var/lib/ceph/mon/ceph-a/

That didn't seem to make a difference, it kept trying to use ceph-admin.

I tried adding this as well:

[mon]
mon data = /var/lib/ceph/mon/ceph-a


But it didn't work either:

# ceph-mon -d -c /etc/ceph/ceph.conf
2013-05-23 22:33:14.371601 7fb64f8cd780  0 ceph version 0.61.2
(fea782543a844bb277ae94d3391788b76c5bee60), process ceph-mon, pid
31183
IO error: lock /var/lib/ceph/mon/ceph-a/store.db/LOCK: Resource
temporarily unavailable
2013-05-23 22:33:14.374219 7fb64f8cd780  1 unable to open monitor
store at /var/lib/ceph/mon/ceph-a
2013-05-23 22:33:14.374229 7fb64f8cd780  1 check for old monitor store format
2013-05-23 22:33:14.374239 7fb64f8cd780  1 store(/var/lib/ceph/mon/ceph-a) mount
2013-05-23 22:33:14.374300 7fb64f8cd780  1 found old GV monitor store
format -- should convert!
IO error: lock /var/lib/ceph/mon/ceph-a/store.db/LOCK: Resource
temporarily unavailable
mon/Monitor.cc: In function 'int Monitor::StoreConverter::convert()'
thread 7fb64f8cd780 time 2013-05-23 22:33:14.374451
mon/Monitor.cc: 4293: FAILED assert(!db->create_and_open(std::cerr))
 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
 1: (Monitor::StoreConverter::convert()+0x467) [0x4bf257]
 2: (main()+0x44e) [0x48073e]
 3: (__libc_start_main()+0xed) [0x7fb64db2a76d]
 4: ceph-mon() [0x48417d]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.
2013-05-23 22:33:14.375019 7fb64f8cd780 -1 mon/Monitor.cc: In function
'int Monitor::StoreConverter::convert()' thread 7fb64f8cd780 time
2013-05-23 22:33:14.374451
mon/Monitor.cc: 4293: FAILED assert(!db->create_and_open(std::cerr))

 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
 1: (Monitor::StoreConverter::convert()+0x467) [0x4bf257]
 2: (main()+0x44e) [0x48073e]
 3: (__libc_start_main()+0xed) [0x7fb64db2a76d]
 4: ceph-mon() [0x48417d]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.

--- begin dump of recent events ---
   -23> 2013-05-23 22:33:14.369804 7fb64f8cd780  5 asok(0x172e000)
register_command perfcounters_dump hook 0x1722010
   -22> 2013-05-23 22:33:14.369831 7fb64f8cd780  5 asok(0x172e000)
register_command 1 hook 0x1722010
   -21> 2013-05-23 22:33:14.369840 7fb64f8cd780  5 asok(0x172e000)
register_command perf dump hook 0x1722010
   -20> 2013-05-23 22:33:14.369848 7fb64f8cd780  5 asok(0x172e000)
register_command perfcounters_schema hook 0x1722010
   -19> 2013-05-23 22:33:14.369855 7fb64f8cd780  5 asok(0x172e000)
register_command 2 hook 0x1722010
   -18> 2013-05-23 22:33:14.369857 7fb64f8cd780  5 asok(0x172e000)
register_command perf schema hook 0x1722010
   -17> 2013-05-23 22:33:14.369861 7fb64f8cd780  5 asok(0x172e000)
register_command config show hook 0x1722010
   -16> 2013-05-23 22:33:14.369867 7fb64f8cd780  5 asok(0x172e000)
register_command config set hook 0x1722010
   -15> 2013-05-23 22:33:14.369869 7fb64f8cd780  5 asok(0x172e000)
register_command log flush hook 0x1722010
   -14> 2013-05-23 22:33:14.369873 7fb64f8cd780  5 asok(0x172e000)
register_command log dump hook 0x1722010
   -13> 2013-05-23 22:33:14.369880 7fb64f8cd780  5 asok(0x172e000)
register_command log reopen hook 0x1722010
   -12> 2013-05-23 22:33:14.371601 7fb64f8cd780  0 ceph version 0.61.2
(fea782543a844bb277ae94d3391788b76c5bee60), process ceph-mon, pid
31183
   -11> 2013-05-23 22:33:14.373837 7fb64f8cd780  5 asok(0x172e000)
init /var/run/ceph/ceph-mon.admin.asok
   -10> 2013-05-23 22:33:14.373859 7fb64f8cd780  5 asok(0x172e000)
bind_and_listen /var/run/ceph/ceph-mon.admin.asok
-9> 2013-05-23 22:33:14.373910 7fb64f8cd780  5 asok(0x172e000)
register_command 0 hook 0x17210b0
-8> 2013-05-23 22:33:14.373917 7fb64f8cd780  5 asok(0x172e000)
register_command version hook 0x17210b0
-7> 2013-05-23 22:33:14.373922 7fb64f8cd780  5 asok(0x172e000)
register_command git_version hook 0x17210b0
-6> 2013-05-23 22:33:14.373929 7fb64f8cd780  5 asok(0x172e000)
register_command help hook 0x17220d0
-5> 2013-05-23 22:33:14.373965 7fb64b970700  5 asok(0x172e000) entry start
-4> 2013-05-23 22:33:14.374219 7fb64f8cd780  1 unable to open
monitor store at /var/lib/ceph/mon/ceph-a
-3> 2013-05-23 22:33:14.374229 7fb64f8cd780  1 check for old
monitor store format
-2> 2013-05-23 22:33:14.374239 7fb64f8cd780  1
store(/var/lib/ceph/mon/ceph-a) mount
-1> 2013-05-23 22:33:14.374300 7fb64f8cd780  1 found old GV
monitor store format -- should convert!
 0> 2013-05-23 22:33:14.375019 7fb64f8cd780 -1 mon/Monitor.cc: In
function 'int Monitor::StoreConverter::convert()' thread 7fb64f8cd780
time 2013-05-23 22:33:14.374451
mon/Monitor.cc: 4293: FAILED assert(!db->create_and_open(std::cerr))

 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
 1: (Monitor::StoreConverter::convert()+0x467) [0x

Re: [ceph-users] ceph-deploy

[Dewan Shamsul Alam]

I just found that

#ceph-deploy gatherkeys ceph0 ceph1 ceph2

works only if I have bobtail. cuttlefish can't find ceph.client.admin.
keyring

and then when I try this on bobtail, it says,

root@cephdeploy:~/12.04# ceph-deploy osd create ceph0:/dev/sda3
ceph1:/dev/sda3 ceph2:/dev/sda3
ceph-disk: Error: Device is mounted: /dev/sda3
Traceback (most recent call last):
  File "/usr/bin/ceph-deploy", line 22, in 
main()
  File "/usr/lib/pymodules/python2.7/ceph_deploy/cli.py", line 112, in main
return args.func(args)
  File "/usr/lib/pymodules/python2.7/ceph_deploy/osd.py", line 293, in osd
prepare(args, cfg, activate_prepared_disk=True)
  File "/usr/lib/pymodules/python2.7/ceph_deploy/osd.py", line 177, in
prepare
dmcrypt_dir=args.dmcrypt_key_dir,
  File "/usr/lib/python2.7/dist-packages/pushy/protocol/proxy.py", line
255, in 
(conn.operator(type_, self, args, kwargs))
  File "/usr/lib/python2.7/dist-packages/pushy/protocol/connection.py",
line 66, in operator
return self.send_request(type_, (object, args, kwargs))
  File "/usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py",
line 323, in send_request
return self.__handle(m)
  File "/usr/lib/python2.7/dist-packages/pushy/protocol/baseconnection.py",
line 639, in __handle
raise e
pushy.protocol.proxy.ExceptionProxy: Command '['ceph-disk-prepare', '--',
'/dev/sda3']' returned non-zero exit status 1
root@cephdeploy:~/12.04#




On Thu, May 23, 2013 at 10:49 PM, Dewan Shamsul Alam <
dewan.sham...@gmail.com> wrote:

> Hi,
>
> I tried ceph-deploy all day. Found that it has a python-setuptools as
> dependency. I knew about python-pushy. But is there any other dependency
> that I'm missing?
>
> The problem I'm getting are as follows:
>
> #ceph-deploy gatherkeys ceph0 ceph1 ceph2
> returns the following error,
> Unable to find /etc/ceph/ceph.client.admin.keyring on ['ceph0', 'ceph1',
> 'ceph2']
>
> Once I got passed this, I don't know why it works sometimes. I have been
> following the exact steps as mentioned in the blog.
>
> Then when I try to do
>
> ceph-deploy osd create ceph0:/dev/sda3 ceph1:/dev/sda3 ceph2:/dev/sda3
>
> It gets stuck.
>
> I'm using Ubuntu 13.04 for ceph-deploy and 12.04 for ceph nodes. I just
> need to get the cuttlefish working and willing to change the OS if it is
> required. Please help. :)
>
> Best Regards,
> Dewan Shamsul Alam
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mkcephfs

[Dewan Shamsul Alam]

Hi,

The previous log is based on cuttlefish. This one is based on bobtail. I'm
not using cephx, may be that's what causing the problem?

temp dir is /tmp/mkcephfs.xf5TsinRsL
preparing monmap in /tmp/mkcephfs.xf5TsinRsL/monmap
/usr/bin/monmaptool --create --clobber --add a 192.168.128.10:6789 --add b
192.168.128.11:6789 --add c 192.168.128.12:6789 --print
/tmp/mkcephfs.xf5TsinRsL/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.xf5TsinRsL/monmap
/usr/bin/monmaptool: generated fsid 1168e717-5db5-488c-a1f7-0b61e7f19138
epoch 0
fsid 1168e717-5db5-488c-a1f7-0b61e7f19138
last_changed 2013-05-24 09:51:15.012839
created 2013-05-24 09:51:15.012839
0: 192.168.128.10:6789/0 mon.a
1: 192.168.128.11:6789/0 mon.b
2: 192.168.128.12:6789/0 mon.c
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.xf5TsinRsL/monmap (3
monitors)
=== osd.0 ===
2013-05-24 09:51:16.247443 7f19b2ce5780 -1
filestore(/var/lib/ceph/osd/ceph-0) could not find
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-24 09:51:16.604586 7f19b2ce5780 -1 created object store
/var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0
fsid 1168e717-5db5-488c-a1f7-0b61e7f19138
2013-05-24 09:51:16.604667 7f19b2ce5780 -1 auth: error reading file:
/var/lib/ceph/osd/ceph-0/keyring: can't open
/var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory
2013-05-24 09:51:16.604850 7f19b2ce5780 -1 created new key in keyring
/var/lib/ceph/osd/ceph-0/keyring
=== osd.1 ===
pushing conf and monmap to
ceph1:/tmp/mkfs.ceph.f0a8d758e9f1a3f32160f67a12149281
2013-05-24 09:50:46.405722 7fd7fa8c5780 -1
filestore(/var/lib/ceph/osd/ceph-1) could not find
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-24 09:50:46.750885 7fd7fa8c5780 -1 created object store
/var/lib/ceph/osd/ceph-1 journal /var/lib/ceph/osd/ceph-1/journal for osd.1
fsid 1168e717-5db5-488c-a1f7-0b61e7f19138
2013-05-24 09:50:46.750945 7fd7fa8c5780 -1 auth: error reading file:
/var/lib/ceph/osd/ceph-1/keyring: can't open
/var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory
2013-05-24 09:50:46.751120 7fd7fa8c5780 -1 created new key in keyring
/var/lib/ceph/osd/ceph-1/keyring
collecting osd.1 key
=== osd.2 ===
pushing conf and monmap to
ceph2:/tmp/mkfs.ceph.e07be4351777982bb28d1cc7ab52e01b
2013-05-24 09:51:33.623922 7fa231b87780 -1
filestore(/var/lib/ceph/osd/ceph-2) could not find
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-24 09:51:33.859703 7fa231b87780 -1 created object store
/var/lib/ceph/osd/ceph-2 journal /var/lib/ceph/osd/ceph-2/journal for osd.2
fsid 1168e717-5db5-488c-a1f7-0b61e7f19138
2013-05-24 09:51:33.859772 7fa231b87780 -1 auth: error reading file:
/var/lib/ceph/osd/ceph-2/keyring: can't open
/var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory
2013-05-24 09:51:33.859930 7fa231b87780 -1 created new key in keyring
/var/lib/ceph/osd/ceph-2/keyring
collecting osd.2 key
=== mds.a ===
creating private key for mds.a keyring /var/lib/ceph/mds/ceph-a/keyring
creating /var/lib/ceph/mds/ceph-a/keyring
=== mds.b ===
pushing conf and monmap to
ceph1:/tmp/mkfs.ceph.5031963c92bcc7e98cd4422ee99d1220
creating private key for mds.b keyring /var/lib/ceph/mds/ceph-b/keyring
creating /var/lib/ceph/mds/ceph-b/keyring
collecting mds.b key
=== mds.c ===
pushing conf and monmap to
ceph2:/tmp/mkfs.ceph.a7475d8f01e340fc9e410fd60a3d8a80
creating private key for mds.c keyring /var/lib/ceph/mds/ceph-c/keyring
creating /var/lib/ceph/mds/ceph-c/keyring
collecting mds.c key
Building generic osdmap from /tmp/mkcephfs.xf5TsinRsL/conf
/usr/bin/osdmaptool: osdmap file '/tmp/mkcephfs.xf5TsinRsL/osdmap'
/usr/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.xf5TsinRsL/osdmap
Generating admin key at /tmp/mkcephfs.xf5TsinRsL/keyring.admin
creating /tmp/mkcephfs.xf5TsinRsL/keyring.admin
Building initial monitor keyring
added entity mds.a auth auth(auid = 18446744073709551615
key=AQC9455RSHUiFhAAcxXX2jmAnPk69KGQ7rUczA== with 0 caps)
added entity mds.b auth auth(auid = 18446744073709551615
key=AQCe455RcAEhCRAARPQhkIuPvYIXDElerq+zJg== with 0 caps)
added entity mds.c auth auth(auid = 18446744073709551615
key=AQDM455R0KRfEhAAPByEb/CBqaUK68tssLH/ug== with 0 caps)
added entity osd.0 auth auth(auid = 18446744073709551615
key=AQC0455RWKsKJBAANgtn7Xq8g3u1CPXekMCy7g== with 0 caps)
added entity osd.1 auth auth(auid = 18446744073709551615
key=AQCW455RQJ7CLBAAQzY2cAMjFwHLd36s1w8m6g== with 0 caps)
added entity osd.2 auth auth(auid = 18446744073709551615
key=AQDF455RoDM/MxAAaC/sEyCE4xFpeNHCqkyZSA== with 0 caps)
=== mon.a ===
/usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a
=== mon.b ===
pushing everything to ceph1
/usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-b for mon.b
=== mon.c ===
pushing everything to ceph2
/usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-c for mon.c
placing client.admin keyring in /etc/ceph/keyring
2013-05-24 09:51:36.973708 7f774d118700  0 -- :/21213 >>
192

Re: [ceph-users] scrub error: found clone without head

[Samuel Just]

Can you send the filenames in the pg directories for those 4 pgs?
-Sam

On Thu, May 23, 2013 at 3:27 PM, Olivier Bonvalet  wrote:
> No :
> pg 3.7c is active+clean+inconsistent, acting [24,13,39]
> pg 3.6b is active+clean+inconsistent, acting [28,23,5]
> pg 3.d is active+clean+inconsistent, acting [29,4,11]
> pg 3.1 is active+clean+inconsistent, acting [28,19,5]
>
> But I suppose that all PG *was* having the osd.25 as primary (on the
> same host), which is (disabled) buggy OSD.
>
> Question : "12d7" in object path is the snapshot id, right ? If it's the
> case, I haven't got any snapshot with this id for the
> rb.0.15c26.238e1f29 image.
>
> So, which files should I remove ?
>
> Thanks for your help.
>
>
> Le jeudi 23 mai 2013 à 15:17 -0700, Samuel Just a écrit :
>> Do all of the affected PGs share osd.28 as the primary?  I think the
>> only recovery is probably to manually remove the orphaned clones.
>> -Sam
>>
>> On Thu, May 23, 2013 at 5:00 AM, Olivier Bonvalet  
>> wrote:
>> > Not yet. I keep it for now.
>> >
>> > Le mercredi 22 mai 2013 à 15:50 -0700, Samuel Just a écrit :
>> >> rb.0.15c26.238e1f29
>> >>
>> >> Has that rbd volume been removed?
>> >> -Sam
>> >>
>> >> On Wed, May 22, 2013 at 12:18 PM, Olivier Bonvalet  
>> >> wrote:
>> >> > 0.61-11-g3b94f03 (0.61-1.1), but the bug occured with bobtail.
>> >> >
>> >> >
>> >> > Le mercredi 22 mai 2013 à 12:00 -0700, Samuel Just a écrit :
>> >> >> What version are you running?
>> >> >> -Sam
>> >> >>
>> >> >> On Wed, May 22, 2013 at 11:25 AM, Olivier Bonvalet 
>> >> >>  wrote:
>> >> >> > Is it enough ?
>> >> >> >
>> >> >> > # tail -n500 -f /var/log/ceph/osd.28.log | grep -A5 -B5 'found clone 
>> >> >> > without head'
>> >> >> > 2013-05-22 15:43:09.308352 7f707dd64700  0 log [INF] : 9.105 scrub ok
>> >> >> > 2013-05-22 15:44:21.054893 7f707dd64700  0 log [INF] : 9.451 scrub ok
>> >> >> > 2013-05-22 15:44:52.898784 7f707cd62700  0 log [INF] : 9.784 scrub ok
>> >> >> > 2013-05-22 15:47:43.148515 7f707cd62700  0 log [INF] : 9.3c3 scrub ok
>> >> >> > 2013-05-22 15:47:45.717085 7f707dd64700  0 log [INF] : 9.3d0 scrub ok
>> >> >> > 2013-05-22 15:52:14.573815 7f707dd64700  0 log [ERR] : scrub 3.6b 
>> >> >> > ade3c16b/rb.0.15c26.238e1f29.9221/12d7//3 found clone 
>> >> >> > without head
>> >> >> > 2013-05-22 15:55:07.230114 7f707d563700  0 log [ERR] : scrub 3.6b 
>> >> >> > 261cc0eb/rb.0.15c26.238e1f29.3671/12d7//3 found clone 
>> >> >> > without head
>> >> >> > 2013-05-22 15:56:56.456242 7f707d563700  0 log [ERR] : scrub 3.6b 
>> >> >> > b10deaeb/rb.0.15c26.238e1f29.86a2/12d7//3 found clone 
>> >> >> > without head
>> >> >> > 2013-05-22 15:57:51.667085 7f707dd64700  0 log [ERR] : 3.6b scrub 3 
>> >> >> > errors
>> >> >> > 2013-05-22 15:57:55.241224 7f707dd64700  0 log [INF] : 9.450 scrub ok
>> >> >> > 2013-05-22 15:57:59.800383 7f707cd62700  0 log [INF] : 9.465 scrub ok
>> >> >> > 2013-05-22 15:59:55.024065 7f707661a700  0 -- 
>> >> >> > 192.168.42.3:6803/12142 >> 192.168.42.5:6828/31490 pipe(0x2a689000 
>> >> >> > sd=108 :6803 s=2 pgs=200652 cs=73 l=0).fault with nothing to send, 
>> >> >> > going to standby
>> >> >> > 2013-05-22 16:01:45.542579 7f7022770700  0 -- 
>> >> >> > 192.168.42.3:6803/12142 >> 192.168.42.5:6828/31490 pipe(0x2a689280 
>> >> >> > sd=99 :6803 s=0 pgs=0 cs=0 l=0).accept connect_seq 74 vs existing 73 
>> >> >> > state standby
>> >> >> > --
>> >> >> > 2013-05-22 16:29:49.544310 7f707dd64700  0 log [INF] : 9.4eb scrub ok
>> >> >> > 2013-05-22 16:29:53.190233 7f707dd64700  0 log [INF] : 9.4f4 scrub ok
>> >> >> > 2013-05-22 16:29:59.478736 7f707dd64700  0 log [INF] : 8.6bb scrub ok
>> >> >> > 2013-05-22 16:35:12.240246 7f7022770700  0 -- 
>> >> >> > 192.168.42.3:6803/12142 >> 192.168.42.5:6828/31490 pipe(0x2a689280 
>> >> >> > sd=99 :6803 s=2 pgs=200667 cs=75 l=0).fault with nothing to send, 
>> >> >> > going to standby
>> >> >> > 2013-05-22 16:35:19.519019 7f707d563700  0 log [INF] : 8.700 scrub ok
>> >> >> > 2013-05-22 16:39:15.422532 7f707dd64700  0 log [ERR] : scrub 3.1 
>> >> >> > b1869301/rb.0.15c26.238e1f29.0836/12d7//3 found clone 
>> >> >> > without head
>> >> >> > 2013-05-22 16:40:04.995256 7f707cd62700  0 log [ERR] : scrub 3.1 
>> >> >> > bccad701/rb.0.15c26.238e1f29.9a00/12d7//3 found clone 
>> >> >> > without head
>> >> >> > 2013-05-22 16:41:07.008717 7f707d563700  0 log [ERR] : scrub 3.1 
>> >> >> > 8a9bec01/rb.0.15c26.238e1f29.9820/12d7//3 found clone 
>> >> >> > without head
>> >> >> > 2013-05-22 16:41:42.460280 7f707c561700  0 log [ERR] : 3.1 scrub 3 
>> >> >> > errors
>> >> >> > 2013-05-22 16:46:12.385678 7f7077735700  0 -- 
>> >> >> > 192.168.42.3:6803/12142 >> 192.168.42.5:6828/31490 pipe(0x2a689c80 
>> >> >> > sd=137 :6803 s=0 pgs=0 cs=0 l=0).accept connect_seq 76 vs existing 
>> >> >> > 75 state standby
>> >> >> > 2013-05-22 16:58:36.079010 7f707661a700  0 -- 
>> >> >> > 192.168.42.3:6803/12142 >> 192.168.42.3:6801/11745 pipe(0x2a689a00 
>> >> >> > sd=44 :6803 s=0 pgs=0 cs=0 l=0)

Re: [ceph-users] scrub error: found clone without head

[Olivier Bonvalet]

No : 
pg 3.7c is active+clean+inconsistent, acting [24,13,39]
pg 3.6b is active+clean+inconsistent, acting [28,23,5]
pg 3.d is active+clean+inconsistent, acting [29,4,11]
pg 3.1 is active+clean+inconsistent, acting [28,19,5]

But I suppose that all PG *was* having the osd.25 as primary (on the
same host), which is (disabled) buggy OSD.

Question : "12d7" in object path is the snapshot id, right ? If it's the
case, I haven't got any snapshot with this id for the
rb.0.15c26.238e1f29 image.

So, which files should I remove ?

Thanks for your help.


Le jeudi 23 mai 2013 à 15:17 -0700, Samuel Just a écrit :
> Do all of the affected PGs share osd.28 as the primary?  I think the
> only recovery is probably to manually remove the orphaned clones.
> -Sam
> 
> On Thu, May 23, 2013 at 5:00 AM, Olivier Bonvalet  wrote:
> > Not yet. I keep it for now.
> >
> > Le mercredi 22 mai 2013 à 15:50 -0700, Samuel Just a écrit :
> >> rb.0.15c26.238e1f29
> >>
> >> Has that rbd volume been removed?
> >> -Sam
> >>
> >> On Wed, May 22, 2013 at 12:18 PM, Olivier Bonvalet  
> >> wrote:
> >> > 0.61-11-g3b94f03 (0.61-1.1), but the bug occured with bobtail.
> >> >
> >> >
> >> > Le mercredi 22 mai 2013 à 12:00 -0700, Samuel Just a écrit :
> >> >> What version are you running?
> >> >> -Sam
> >> >>
> >> >> On Wed, May 22, 2013 at 11:25 AM, Olivier Bonvalet 
> >> >>  wrote:
> >> >> > Is it enough ?
> >> >> >
> >> >> > # tail -n500 -f /var/log/ceph/osd.28.log | grep -A5 -B5 'found clone 
> >> >> > without head'
> >> >> > 2013-05-22 15:43:09.308352 7f707dd64700  0 log [INF] : 9.105 scrub ok
> >> >> > 2013-05-22 15:44:21.054893 7f707dd64700  0 log [INF] : 9.451 scrub ok
> >> >> > 2013-05-22 15:44:52.898784 7f707cd62700  0 log [INF] : 9.784 scrub ok
> >> >> > 2013-05-22 15:47:43.148515 7f707cd62700  0 log [INF] : 9.3c3 scrub ok
> >> >> > 2013-05-22 15:47:45.717085 7f707dd64700  0 log [INF] : 9.3d0 scrub ok
> >> >> > 2013-05-22 15:52:14.573815 7f707dd64700  0 log [ERR] : scrub 3.6b 
> >> >> > ade3c16b/rb.0.15c26.238e1f29.9221/12d7//3 found clone without 
> >> >> > head
> >> >> > 2013-05-22 15:55:07.230114 7f707d563700  0 log [ERR] : scrub 3.6b 
> >> >> > 261cc0eb/rb.0.15c26.238e1f29.3671/12d7//3 found clone without 
> >> >> > head
> >> >> > 2013-05-22 15:56:56.456242 7f707d563700  0 log [ERR] : scrub 3.6b 
> >> >> > b10deaeb/rb.0.15c26.238e1f29.86a2/12d7//3 found clone without 
> >> >> > head
> >> >> > 2013-05-22 15:57:51.667085 7f707dd64700  0 log [ERR] : 3.6b scrub 3 
> >> >> > errors
> >> >> > 2013-05-22 15:57:55.241224 7f707dd64700  0 log [INF] : 9.450 scrub ok
> >> >> > 2013-05-22 15:57:59.800383 7f707cd62700  0 log [INF] : 9.465 scrub ok
> >> >> > 2013-05-22 15:59:55.024065 7f707661a700  0 -- 192.168.42.3:6803/12142 
> >> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689000 sd=108 :6803 s=2 
> >> >> > pgs=200652 cs=73 l=0).fault with nothing to send, going to standby
> >> >> > 2013-05-22 16:01:45.542579 7f7022770700  0 -- 192.168.42.3:6803/12142 
> >> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689280 sd=99 :6803 s=0 pgs=0 cs=0 
> >> >> > l=0).accept connect_seq 74 vs existing 73 state standby
> >> >> > --
> >> >> > 2013-05-22 16:29:49.544310 7f707dd64700  0 log [INF] : 9.4eb scrub ok
> >> >> > 2013-05-22 16:29:53.190233 7f707dd64700  0 log [INF] : 9.4f4 scrub ok
> >> >> > 2013-05-22 16:29:59.478736 7f707dd64700  0 log [INF] : 8.6bb scrub ok
> >> >> > 2013-05-22 16:35:12.240246 7f7022770700  0 -- 192.168.42.3:6803/12142 
> >> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689280 sd=99 :6803 s=2 pgs=200667 
> >> >> > cs=75 l=0).fault with nothing to send, going to standby
> >> >> > 2013-05-22 16:35:19.519019 7f707d563700  0 log [INF] : 8.700 scrub ok
> >> >> > 2013-05-22 16:39:15.422532 7f707dd64700  0 log [ERR] : scrub 3.1 
> >> >> > b1869301/rb.0.15c26.238e1f29.0836/12d7//3 found clone without 
> >> >> > head
> >> >> > 2013-05-22 16:40:04.995256 7f707cd62700  0 log [ERR] : scrub 3.1 
> >> >> > bccad701/rb.0.15c26.238e1f29.9a00/12d7//3 found clone without 
> >> >> > head
> >> >> > 2013-05-22 16:41:07.008717 7f707d563700  0 log [ERR] : scrub 3.1 
> >> >> > 8a9bec01/rb.0.15c26.238e1f29.9820/12d7//3 found clone without 
> >> >> > head
> >> >> > 2013-05-22 16:41:42.460280 7f707c561700  0 log [ERR] : 3.1 scrub 3 
> >> >> > errors
> >> >> > 2013-05-22 16:46:12.385678 7f7077735700  0 -- 192.168.42.3:6803/12142 
> >> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689c80 sd=137 :6803 s=0 pgs=0 
> >> >> > cs=0 l=0).accept connect_seq 76 vs existing 75 state standby
> >> >> > 2013-05-22 16:58:36.079010 7f707661a700  0 -- 192.168.42.3:6803/12142 
> >> >> > >> 192.168.42.3:6801/11745 pipe(0x2a689a00 sd=44 :6803 s=0 pgs=0 cs=0 
> >> >> > l=0).accept connect_seq 40 vs existing 39 state standby
> >> >> > 2013-05-22 16:58:36.798038 7f707d563700  0 log [INF] : 9.50c scrub ok
> >> >> > 2013-05-22 16:58:40.104159 7f707c561700  0 log [INF] : 9.526 scrub ok
> >> >> >
> >> >> >
> >> >> > Note : I have 8 scrub errors like that, on 4 im

Re: [ceph-users] scrub error: found clone without head

[Samuel Just]

Do all of the affected PGs share osd.28 as the primary?  I think the
only recovery is probably to manually remove the orphaned clones.
-Sam

On Thu, May 23, 2013 at 5:00 AM, Olivier Bonvalet  wrote:
> Not yet. I keep it for now.
>
> Le mercredi 22 mai 2013 à 15:50 -0700, Samuel Just a écrit :
>> rb.0.15c26.238e1f29
>>
>> Has that rbd volume been removed?
>> -Sam
>>
>> On Wed, May 22, 2013 at 12:18 PM, Olivier Bonvalet  
>> wrote:
>> > 0.61-11-g3b94f03 (0.61-1.1), but the bug occured with bobtail.
>> >
>> >
>> > Le mercredi 22 mai 2013 à 12:00 -0700, Samuel Just a écrit :
>> >> What version are you running?
>> >> -Sam
>> >>
>> >> On Wed, May 22, 2013 at 11:25 AM, Olivier Bonvalet  
>> >> wrote:
>> >> > Is it enough ?
>> >> >
>> >> > # tail -n500 -f /var/log/ceph/osd.28.log | grep -A5 -B5 'found clone 
>> >> > without head'
>> >> > 2013-05-22 15:43:09.308352 7f707dd64700  0 log [INF] : 9.105 scrub ok
>> >> > 2013-05-22 15:44:21.054893 7f707dd64700  0 log [INF] : 9.451 scrub ok
>> >> > 2013-05-22 15:44:52.898784 7f707cd62700  0 log [INF] : 9.784 scrub ok
>> >> > 2013-05-22 15:47:43.148515 7f707cd62700  0 log [INF] : 9.3c3 scrub ok
>> >> > 2013-05-22 15:47:45.717085 7f707dd64700  0 log [INF] : 9.3d0 scrub ok
>> >> > 2013-05-22 15:52:14.573815 7f707dd64700  0 log [ERR] : scrub 3.6b 
>> >> > ade3c16b/rb.0.15c26.238e1f29.9221/12d7//3 found clone without 
>> >> > head
>> >> > 2013-05-22 15:55:07.230114 7f707d563700  0 log [ERR] : scrub 3.6b 
>> >> > 261cc0eb/rb.0.15c26.238e1f29.3671/12d7//3 found clone without 
>> >> > head
>> >> > 2013-05-22 15:56:56.456242 7f707d563700  0 log [ERR] : scrub 3.6b 
>> >> > b10deaeb/rb.0.15c26.238e1f29.86a2/12d7//3 found clone without 
>> >> > head
>> >> > 2013-05-22 15:57:51.667085 7f707dd64700  0 log [ERR] : 3.6b scrub 3 
>> >> > errors
>> >> > 2013-05-22 15:57:55.241224 7f707dd64700  0 log [INF] : 9.450 scrub ok
>> >> > 2013-05-22 15:57:59.800383 7f707cd62700  0 log [INF] : 9.465 scrub ok
>> >> > 2013-05-22 15:59:55.024065 7f707661a700  0 -- 192.168.42.3:6803/12142 
>> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689000 sd=108 :6803 s=2 pgs=200652 
>> >> > cs=73 l=0).fault with nothing to send, going to standby
>> >> > 2013-05-22 16:01:45.542579 7f7022770700  0 -- 192.168.42.3:6803/12142 
>> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689280 sd=99 :6803 s=0 pgs=0 cs=0 
>> >> > l=0).accept connect_seq 74 vs existing 73 state standby
>> >> > --
>> >> > 2013-05-22 16:29:49.544310 7f707dd64700  0 log [INF] : 9.4eb scrub ok
>> >> > 2013-05-22 16:29:53.190233 7f707dd64700  0 log [INF] : 9.4f4 scrub ok
>> >> > 2013-05-22 16:29:59.478736 7f707dd64700  0 log [INF] : 8.6bb scrub ok
>> >> > 2013-05-22 16:35:12.240246 7f7022770700  0 -- 192.168.42.3:6803/12142 
>> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689280 sd=99 :6803 s=2 pgs=200667 
>> >> > cs=75 l=0).fault with nothing to send, going to standby
>> >> > 2013-05-22 16:35:19.519019 7f707d563700  0 log [INF] : 8.700 scrub ok
>> >> > 2013-05-22 16:39:15.422532 7f707dd64700  0 log [ERR] : scrub 3.1 
>> >> > b1869301/rb.0.15c26.238e1f29.0836/12d7//3 found clone without 
>> >> > head
>> >> > 2013-05-22 16:40:04.995256 7f707cd62700  0 log [ERR] : scrub 3.1 
>> >> > bccad701/rb.0.15c26.238e1f29.9a00/12d7//3 found clone without 
>> >> > head
>> >> > 2013-05-22 16:41:07.008717 7f707d563700  0 log [ERR] : scrub 3.1 
>> >> > 8a9bec01/rb.0.15c26.238e1f29.9820/12d7//3 found clone without 
>> >> > head
>> >> > 2013-05-22 16:41:42.460280 7f707c561700  0 log [ERR] : 3.1 scrub 3 
>> >> > errors
>> >> > 2013-05-22 16:46:12.385678 7f7077735700  0 -- 192.168.42.3:6803/12142 
>> >> > >> 192.168.42.5:6828/31490 pipe(0x2a689c80 sd=137 :6803 s=0 pgs=0 cs=0 
>> >> > l=0).accept connect_seq 76 vs existing 75 state standby
>> >> > 2013-05-22 16:58:36.079010 7f707661a700  0 -- 192.168.42.3:6803/12142 
>> >> > >> 192.168.42.3:6801/11745 pipe(0x2a689a00 sd=44 :6803 s=0 pgs=0 cs=0 
>> >> > l=0).accept connect_seq 40 vs existing 39 state standby
>> >> > 2013-05-22 16:58:36.798038 7f707d563700  0 log [INF] : 9.50c scrub ok
>> >> > 2013-05-22 16:58:40.104159 7f707c561700  0 log [INF] : 9.526 scrub ok
>> >> >
>> >> >
>> >> > Note : I have 8 scrub errors like that, on 4 impacted PG, and all 
>> >> > impacted objects are about the same RBD image (rb.0.15c26.238e1f29).
>> >> >
>> >> >
>> >> >
>> >> > Le mercredi 22 mai 2013 à 11:01 -0700, Samuel Just a écrit :
>> >> >> Can you post your ceph.log with the period including all of these 
>> >> >> errors?
>> >> >> -Sam
>> >> >>
>> >> >> On Wed, May 22, 2013 at 5:39 AM, Dzianis Kahanovich
>> >> >>  wrote:
>> >> >> > Olivier Bonvalet пишет:
>> >> >> >>
>> >> >> >> Le lundi 20 mai 2013 à 00:06 +0200, Olivier Bonvalet a écrit :
>> >> >> >>> Le mardi 07 mai 2013 à 15:51 +0300, Dzianis Kahanovich a écrit :
>> >> >>  I have 4 scrub errors (3 PGs - "found clone without head"), on 
>> >> >>  one OSD. Not
>> >> >>  repairing. How to repair it exclude re-creating of OSD?
>> >> >> 
>>

[ceph-users] MDS dying on cuttlefish

[Giuseppe 'Gippa' Paterno']

Hi!

I've got a cluster of two nodes on Ubuntu 12.04 with cuttlefish from the
ceph.com repo.
ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)

The MDS process is dying after a while with a stack trace, but I can't
understand why.
I reproduced the same problem on debian 7 with the same repository.

-3> 2013-05-23 23:00:42.957679 7fa39e28e700  1 --
10.123.200.189:6800/28919 <== osd.0 10.123.200.188:6802/27665 1 
osd_op_reply(5 200. [read 0~0] ack = -2 (No such file or
directory)) v4  111+0+0 (2261481792 0 0) 0x29afe00 con 0x29c4b00
-2> 2013-05-23 23:00:42.957780 7fa39e28e700  0 mds.0.journaler(ro)
error getting journal off disk
-1> 2013-05-23 23:00:42.960974 7fa39e28e700  1 --
10.123.200.189:6800/28919 <== osd.0 10.123.200.188:6802/27665 2 
osd_op_reply(1 mds0_inotable [read 0~0] ack = -2 (No such file or
directory)) v4  112+0+0 (1612134461 0 0) 0x2a1c200 con 0x29c4b00
 0> 2013-05-23 23:00:42.963326 7fa39e28e700 -1 mds/MDSTable.cc: In
function 'void MDSTable::load_2(int, ceph::bufferlist&, Context*)'
thread 7fa39e28e700 time 2013-05-23 23:00:42.961076
mds/MDSTable.cc: 150: FAILED assert(0)

 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
 1: (MDSTable::load_2(int, ceph::buffer::list&, Context*)+0x3bb) [0x6dd2db]
 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe1b) [0x7275bb]
 3: (MDS::handle_core_message(Message*)+0xae7) [0x513c57]
 4: (MDS::_dispatch(Message*)+0x33) [0x513d53]
 5: (MDS::ms_dispatch(Message*)+0xab) [0x515b3b]
 6: (DispatchQueue::entry()+0x393) [0x847ca3]
 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7caeed]
 8: (()+0x6b50) [0x7fa3a3376b50]
 9: (clone()+0x6d) [0x7fa3a1d24a7d]

Full logs here:
http://pastebin.com/C81g5jFd

I can't understand why and I'd really appreciate an hint.
Thanks!
Regards,
  Giuseppe
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ZFS on RBD?

[Tim Bishop]

Hi all,

I'm evaluating Ceph and one of my workloads is a server that provides
home directories to end users over both NFS and Samba. I'm looking at
whether this could be backed by Ceph provided storage.

So to test this I built a single node Ceph instance (Ubuntu precise,
ceph.com packages) in a VM and popped a couple of OSDs on it. I then
built another VM and used it to mount an RBD from the Ceph node. No
problems... it all worked as described in the documentation.

Then I started to look at the filesystem I was using on top of the RBD.
I'd tested ext4 without any problems. I'd been testing ZFS (from stable
zfs-native PPA) separately against local storage on the client VM too,
so I thought I'd try that on top of the RBD. This is when I hit
problems, and the VM paniced (trace at the end of this email).

Now I am just experimenting, so this isn't a huge deal right now. But
I'm wondering if this is something that should work? Am I overlooking
something? Is it a silly idea to even try it?

The trace looks to be in the ZFS code, so if there's a bug that needs
fixing it's probably over there rather than in Ceph, but I thought here
might be a good starting point for advice.

Thanks in advance everyone,

Tim.

[  504.644120] divide error:  [#1] SMP
[  504.644298] Modules linked in: coretemp(F) ppdev(F) vmw_balloon(F) 
microcode(F) psmouse(F) serio_raw(F) parport_pc(F) vmwgfx(F) i2c_piix4(F) 
mac_hid(F) ttm(F) shpchp(F) drm(F) rbd(F) libceph(F) lp(F) parport(F) zfs(POF) 
zcommon(POF) znvpair(POF) zavl(POF) zunicode(POF) spl(OF) floppy(F) e1000(F) 
mptspi(F) mptscsih(F) mptbase(F) btrfs(F) zlib_deflate(F) libcrc32c(F)
[  504.646156] CPU 0
[  504.646234] Pid: 2281, comm: txg_sync Tainted: PF   B  O 
3.8.0-21-generic #32~precise1-Ubuntu VMware, Inc. VMware Virtual Platform/440BX 
Desktop Reference Platform
[  504.646550] RIP: 0010:[]  [] 
spa_history_write+0x82/0x1d0 [zfs]
[  504.646816] RSP: 0018:88003ae3dab8  EFLAGS: 00010246
[  504.646940] RAX:  RBX:  RCX: 
[  504.647091] RDX:  RSI: 0020 RDI: 
[  504.647242] RBP: 88003ae3db28 R08: 88003b2afc00 R09: 0002
[  504.647423] R10: 88003b9a4512 R11: 6d206b6e61742066 R12: 88003add6600
[  504.647600] R13: 88003cfc2000 R14: 88003d3c9000 R15: 0008
[  504.647778] FS:  () GS:88003fc0() 
knlGS:
[  504.647997] CS:  0010 DS:  ES:  CR0: 8005003b
[  504.648153] CR2: 7fbc1ef54a38 CR3: 3bf3e000 CR4: 07f0
[  504.648380] DR0:  DR1:  DR2: 
[  504.648586] DR3:  DR6: 0ff0 DR7: 0400
[  504.648766] Process txg_sync (pid: 2281, threadinfo 88003ae3c000, task 
88003b7c45c0)
[  504.648990] Stack:
[  504.649087]  0002 a01e3360 88003b2afc00 
88003ae3dba0
[  504.649461]  88003d3c9000 0008 88003cfc2000 
5530ebc2
[  504.649835]  88003d22ac40 88003d22ac40 88003cfc2000 
88003b2afc00
[  504.650209] Call Trace:
[  504.650351]  [] spa_history_log_sync+0x235/0x650 [zfs]
[  504.650554]  [] dsl_sync_task_group_sync+0x123/0x210 [zfs]
[  504.650760]  [] dsl_pool_sync+0x41b/0x530 [zfs]
[  504.650953]  [] spa_sync+0x3a8/0xa50 [zfs]
[  504.651117]  [] ? ktime_get_ts+0x4c/0xe0
[  504.651302]  [] txg_sync_thread+0x2df/0x540 [zfs]
[  504.651501]  [] ? txg_init+0x250/0x250 [zfs]
[  504.651676]  [] thread_generic_wrapper+0x78/0x90 [spl]
[  504.651856]  [] ? __thread_create+0x310/0x310 [spl]
[  504.652029]  [] kthread+0xc0/0xd0
[  504.652174]  [] ? flush_kthread_worker+0xb0/0xb0
[  504.652339]  [] ret_from_fork+0x7c/0xb0
[  504.652492]  [] ? flush_kthread_worker+0xb0/0xb0
[  504.652655] Code: 55 b0 48 89 fa 48 29 f2 48 01 c2 48 39 55 b8 0f 82 bc 00 
00 00 4c 8b 75 b0 41 bf 08 00 00 00 48 29 c8 31 d2 49 8b b5 70 08 00 00 <48> f7 
f7 4c 8d 45 c0 4c 89 f7 48 01 ca 48 29 d3 48 83 fb 08 49
[  504.659810] RIP  [] spa_history_write+0x82/0x1d0 [zfs]
[  504.660045]  RSP 
[  504.660187] ---[ end trace e69c7eee3ba17773 ]---

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x5AE7D984
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RADOS Gateway Configuration

[John Wilkins]

It puts it in the same directory where you executed ceph-deploy.

On Thu, May 23, 2013 at 10:57 AM, Daniel Curran
 wrote:
> Hey John,
>
> Thanks for the reply. I'll check out that other doc you have there. Just for
> future reference do you know where ceph-deploy puts the ceph keyring?
>
> Daniel
>
>
> On Wed, May 22, 2013 at 7:19 PM, John Wilkins 
> wrote:
>>
>> Daniel,
>>
>> It looks like I need to update that portion of the docs too, as it
>> links back to the 5-minute quick start. Once you are up and running
>> with "HEALTH OK" on either the 5-minute Quick Start or Quick Ceph
>> Deploy, your storage cluster is running fine. The remaining issues
>> would likely be with authentication, chmod on the files, or with the
>> RGW setup. There's a quick start for RGW, which I had verified here:
>> http://ceph.com/docs/master/start/quick-rgw/. Someone else had a
>> problem with the Rewrite rule on that example reported here:
>> http://tracker.ceph.com/issues/4608. It's likely I need to run through
>> with specific Ceph and Apache versions. There are also a few
>> additional tips in the configuration section.
>> http://ceph.com/docs/master/radosgw/config/
>>
>> There is an issue in some cases where keys have forward or backslash
>> characters, and you may need to regenerate the keys.
>>
>>
>>
>> On Wed, May 22, 2013 at 4:42 PM, Daniel Curran 
>> wrote:
>> >
>> > Hello,
>> >
>> > I just started using ceph recently and was trying to get the RADOS
>> > Gateway
>> > working in order to use the Swift compatible API. I followed the install
>> > instructions found here (http://ceph.com/docs/master
>> > /start/quick-ceph-deploy/) and got to a point where "ceph health" give
>> > me
>> > HEALTH_OK. This is all well and good but near the end of the rados gw
>> > setup
>> > (found here http://ceph.com/docs/master/radosgw/manual-install/) I need
>> > to
>> > execute the following line:
>> >
>> > sudo ceph -k /etc/ceph/ceph.keyring auth add client.radosgw.gateway -i
>> > /etc/ceph/keyring.radosgw.gateway
>> >
>> > Unfortunately, I don't believe ceph-deploy places the keyring at
>> > /etc/ceph/ceph.keyring. I tried to use the one from
>> > /var/lib/ceph/bootstrap-osd/ceph.keyring but it was unable to
>> > authenticate
>> > as client.admin. Is there another location that the keyring needs to be
>> > copied from or am I doing something totally wrong?
>> >
>> > I didn't want to be held back so I restarted and did the manual install
>> > from
>> > the 5-minute quick start where I was able to find the ring. I had more
>> > issues almost immediately. I have to execute the following steps to
>> > create
>> > some users for swift:
>> >
>> > radosgw-admin user create --uid=johndoe --display-name="John Doe"
>> > --email=j...@example.com
>> > sudo radosgw-admin subuser create --uid=johndoe --subuser=johndoe:swift
>> > --access=full
>> >
>> > sudo radosgw-admin key create --subuser=johndoe:swift --key-type=swift
>> >
>> > The first two gave me output I was expecting but the very last line had
>> > some
>> > weirdness that essentially made swift unusable. The expected output is
>> > something along these lines:
>> >
>> > { "user_id": "johndoe",
>> >   "rados_uid": 0,
>> >   "display_name": "John Doe",
>> >   "email": "j...@example.com",
>> >   "suspended": 0,
>> >   "subusers": [
>> >  { "id": "johndoe:swift",
>> >"permissions": "full-control"}],
>> >   "keys": [
>> > { "user": "johndoe",
>> >   "access_key": "QFAMEDSJP5DEKJO0DDXY",
>> >   "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
>> >   "swift_keys": [
>> > { "user": "johndoe:swift",
>> >   "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]}
>> >
>> > Where that last secret key is what we hand the swift CLI as seen here:
>> >
>> > swift -V 1.0 -A http://radosgw.example.com/auth -U johndoe:swift -K
>> > E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1 post test
>> >
>> > However, my output came out like this:
>> >
>> > { "user_id": "johndoe",
>> >   "display_name": "John Doe",
>> >   "email": "j...@example.com",
>> >   "suspended": 0,
>> >   "max_buckets: 1000,
>> >   "auid": 0,
>> >  "subusers": [
>> >  { "id": "johndoe:swift",
>> >"permissions": "full-control"}],
>> >   "keys": [
>> > { "user": "johndoe",
>> >   "access_key": "SUEXWVL3WB2Z64CRAG97",
>> >   "secret_key": "C\/jHFJ3wdPv4iJ+aq4JeZ52LEC3OdnhsYEnVkhBP"}],
>> >   "swift_keys": [
>> > { "user": "johndoe:swift",
>> >   "secret_key": ""}],
>> >   "caps": []}
>> >
>> >
>> > Giving me no swift key to use. I don't believe the key is supposed to be
>> > blank because I tried that and received auth errors (to the best of my
>> > ability). I can't tell if this is my fault since I'm new nor am I able
>> > to
>> > find a way around it. It looks like there are definitely changes between
>> > the
>> > version used in the doc and mine so maybe it's all working as it should
>> > but
>> > the secret_key for swift lives somewhere else. If anyone knows anything

Re: [ceph-users] RADOS Gateway Configuration

[Daniel Curran]

Hey John,

Thanks for the reply. I'll check out that other doc you have there. Just
for future reference do you know where ceph-deploy puts the ceph keyring?

Daniel


On Wed, May 22, 2013 at 7:19 PM, John Wilkins wrote:

> Daniel,
>
> It looks like I need to update that portion of the docs too, as it
> links back to the 5-minute quick start. Once you are up and running
> with "HEALTH OK" on either the 5-minute Quick Start or Quick Ceph
> Deploy, your storage cluster is running fine. The remaining issues
> would likely be with authentication, chmod on the files, or with the
> RGW setup. There's a quick start for RGW, which I had verified here:
> http://ceph.com/docs/master/start/quick-rgw/. Someone else had a
> problem with the Rewrite rule on that example reported here:
> http://tracker.ceph.com/issues/4608. It's likely I need to run through
> with specific Ceph and Apache versions. There are also a few
> additional tips in the configuration section.
> http://ceph.com/docs/master/radosgw/config/
>
> There is an issue in some cases where keys have forward or backslash
> characters, and you may need to regenerate the keys.
>
>
>
> On Wed, May 22, 2013 at 4:42 PM, Daniel Curran 
> wrote:
> >
> > Hello,
> >
> > I just started using ceph recently and was trying to get the RADOS
> Gateway
> > working in order to use the Swift compatible API. I followed the install
> > instructions found here (http://ceph.com/docs/master
> > /start/quick-ceph-deploy/) and got to a point where "ceph health" give me
> > HEALTH_OK. This is all well and good but near the end of the rados gw
> setup
> > (found here http://ceph.com/docs/master/radosgw/manual-install/) I need
> to
> > execute the following line:
> >
> > sudo ceph -k /etc/ceph/ceph.keyring auth add client.radosgw.gateway -i
> > /etc/ceph/keyring.radosgw.gateway
> >
> > Unfortunately, I don't believe ceph-deploy places the keyring at
> > /etc/ceph/ceph.keyring. I tried to use the one from
> > /var/lib/ceph/bootstrap-osd/ceph.keyring but it was unable to
> authenticate
> > as client.admin. Is there another location that the keyring needs to be
> > copied from or am I doing something totally wrong?
> >
> > I didn't want to be held back so I restarted and did the manual install
> from
> > the 5-minute quick start where I was able to find the ring. I had more
> > issues almost immediately. I have to execute the following steps to
> create
> > some users for swift:
> >
> > radosgw-admin user create --uid=johndoe --display-name="John Doe"
> > --email=j...@example.com
> > sudo radosgw-admin subuser create --uid=johndoe --subuser=johndoe:swift
> > --access=full
> >
> > sudo radosgw-admin key create --subuser=johndoe:swift --key-type=swift
> >
> > The first two gave me output I was expecting but the very last line had
> some
> > weirdness that essentially made swift unusable. The expected output is
> > something along these lines:
> >
> > { "user_id": "johndoe",
> >   "rados_uid": 0,
> >   "display_name": "John Doe",
> >   "email": "j...@example.com",
> >   "suspended": 0,
> >   "subusers": [
> >  { "id": "johndoe:swift",
> >"permissions": "full-control"}],
> >   "keys": [
> > { "user": "johndoe",
> >   "access_key": "QFAMEDSJP5DEKJO0DDXY",
> >   "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
> >   "swift_keys": [
> > { "user": "johndoe:swift",
> >   "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]}
> >
> > Where that last secret key is what we hand the swift CLI as seen here:
> >
> > swift -V 1.0 -A http://radosgw.example.com/auth -U johndoe:swift -K
> > E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1 post test
> >
> > However, my output came out like this:
> >
> > { "user_id": "johndoe",
> >   "display_name": "John Doe",
> >   "email": "j...@example.com",
> >   "suspended": 0,
> >   "max_buckets: 1000,
> >   "auid": 0,
> >  "subusers": [
> >  { "id": "johndoe:swift",
> >"permissions": "full-control"}],
> >   "keys": [
> > { "user": "johndoe",
> >   "access_key": "SUEXWVL3WB2Z64CRAG97",
> >   "secret_key": "C\/jHFJ3wdPv4iJ+aq4JeZ52LEC3OdnhsYEnVkhBP"}],
> >   "swift_keys": [
> > { "user": "johndoe:swift",
> >   "secret_key": ""}],
> >   "caps": []}
> >
> >
> > Giving me no swift key to use. I don't believe the key is supposed to be
> > blank because I tried that and received auth errors (to the best of my
> > ability). I can't tell if this is my fault since I'm new nor am I able to
> > find a way around it. It looks like there are definitely changes between
> the
> > version used in the doc and mine so maybe it's all working as it should
> but
> > the secret_key for swift lives somewhere else. If anyone knows anything
> I'd
> > appreciate it a lot.
> >
> > Thank you,
> > Daniel
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
> --
> John Wilkins
> Senior Tec

Re: [ceph-users] mkcephfs

[Dewan Shamsul Alam]

Hi,

This is what I get while building the cluster:

#/sbin/mkcephfs -a -c /etc/ceph/ceph.conf
temp dir is /tmp/mkcephfs.yzl9PFOJYo
preparing monmap in /tmp/mkcephfs.yzl9PFOJYo/monmap
/usr/bin/monmaptool --create --clobber --add a 192.168.128.10:6789 --add b
192.168.128.11:6789 --add c 192.168.128.12:6789 --print
/tmp/mkcephfs.yzl9PFOJYo/monmap
/usr/bin/monmaptool: monmap file /tmp/mkcephfs.yzl9PFOJYo/monmap
/usr/bin/monmaptool: generated fsid 09136333-16dc-476f-8773-90262ad0b80d
epoch 0
fsid 09136333-16dc-476f-8773-90262ad0b80d
last_changed 2013-05-23 23:36:41.325667
created 2013-05-23 23:36:41.325667
0: 192.168.128.10:6789/0 mon.a
1: 192.168.128.11:6789/0 mon.b
2: 192.168.128.12:6789/0 mon.c
/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.yzl9PFOJYo/monmap (3
monitors)

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
 http://github.com/ceph/ceph-deploy
=== osd.0 ===
2013-05-23 23:36:41.851599 7fe81f143780 -1 journal FileJournal::_open:
disabling aio for non-block journal.  Use journal_force_aio to force use of
aio anyway
2013-05-23 23:36:42.576549 7fe81f143780 -1 journal FileJournal::_open:
disabling aio for non-block journal.  Use journal_force_aio to force use of
aio anyway
2013-05-23 23:36:42.577795 7fe81f143780 -1
filestore(/var/lib/ceph/osd/ceph-0) could not find
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-23 23:36:42.918456 7fe81f143780 -1 created object store
/var/lib/ceph/osd/ceph-0 journal /var/lib/ceph/osd/ceph-0/journal for osd.0
fsid 09136333-16dc-476f-8773-90262ad0b80d
2013-05-23 23:36:42.918520 7fe81f143780 -1 auth: error reading file:
/var/lib/ceph/osd/ceph-0/keyring: can't open
/var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory
2013-05-23 23:36:42.918642 7fe81f143780 -1 created new key in keyring
/var/lib/ceph/osd/ceph-0/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
 http://github.com/ceph/ceph-deploy
=== osd.1 ===
pushing conf and monmap to
ceph1:/tmp/mkfs.ceph.HWyfcu95hsnB1jxdVqxGJNAOd2u3aj5I
2013-05-23 23:36:12.380573 7ff116b63780 -1 journal FileJournal::_open:
disabling aio for non-block journal.  Use journal_force_aio to force use of
aio anyway
2013-05-23 23:36:13.026598 7ff116b63780 -1 journal FileJournal::_open:
disabling aio for non-block journal.  Use journal_force_aio to force use of
aio anyway
2013-05-23 23:36:13.037762 7ff116b63780 -1
filestore(/var/lib/ceph/osd/ceph-1) could not find
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-23 23:36:13.366445 7ff116b63780 -1 created object store
/var/lib/ceph/osd/ceph-1 journal /var/lib/ceph/osd/ceph-1/journal for osd.1
fsid 09136333-16dc-476f-8773-90262ad0b80d
2013-05-23 23:36:13.366510 7ff116b63780 -1 auth: error reading file:
/var/lib/ceph/osd/ceph-1/keyring: can't open
/var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory
2013-05-23 23:36:13.366621 7ff116b63780 -1 created new key in keyring
/var/lib/ceph/osd/ceph-1/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
 http://github.com/ceph/ceph-deploy
collecting osd.1 key
=== osd.2 ===
pushing conf and monmap to
ceph2:/tmp/mkfs.ceph.tNt36unRvZ6lVKmz65OjiOhrpUfsw7xz
2013-05-23 23:36:59.086209 7fe38a955780 -1 journal FileJournal::_open:
disabling aio for non-block journal.  Use journal_force_aio to force use of
aio anyway
2013-05-23 23:36:59.610999 7fe38a955780 -1 journal FileJournal::_open:
disabling aio for non-block journal.  Use journal_force_aio to force use of
aio anyway
2013-05-23 23:36:59.623725 7fe38a955780 -1
filestore(/var/lib/ceph/osd/ceph-2) could not find
23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-23 23:36:59.850510 7fe38a955780 -1 created object store
/var/lib/ceph/osd/ceph-2 journal /var/lib/ceph/osd/ceph-2/journal for osd.2
fsid 09136333-16dc-476f-8773-90262ad0b80d
2013-05-23 23:36:59.850574 7fe38a955780 -1 auth: error reading file:
/var/lib/ceph/osd/ceph-2/keyring: can't open
/var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory
2013-05-23 23:36:59.850688 7fe38a955780 -1 created new key in keyring
/var/lib/ceph/osd/ceph-2/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
 http://github.com/ceph/ceph-deploy
collecting osd.2 key
=== mds.a ===
creating private key for mds.a keyring /var/lib/ceph/mds/ceph-a/keyring
creating /var/lib/ceph/mds/ceph-a/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
 http://github.com/ceph/ceph-deploy
=== mds.b ===
pushing conf and monmap to
ceph1:/tmp/mkfs.ceph.QDX0IZSEBd3469OT6yw6ISlckxfmO6nu
creating private key for mds.b keyring /var/lib/ceph/mds/ceph-b/keyring
creating /var/lib/ceph/mds/ceph-b/keyring

WARNING: mkcephfs is now deprecated in favour of ceph-deploy. Please see:
 http://github.com/ceph/ceph-deploy
collecting mds.b key
=== mds.c ===
pushing conf and monmap to
ceph2:/tmp/mkfs.ceph.XimfAW4CrJR11rs8IAJhsHn0inBNJdhl
creating private 

Re: [ceph-users] FW: About RBD

[Mensah, Yao (CIV)]

Thank you very much for your prompt response…

So basically I can’t use cluster aware tool like Microsoft CSV on the RBD, is 
that correct?

What I am trying to understand is that can I have 2 physical hosts (Maybe Dell 
PowerEdge2950)

*host1 with VM#0-10
*host2 with  VM #10-20

And both of these hosts accessing one big LUN or, in this case ceph RBD?

Can host1 failed all it VMs to host2 in case that machine has trouble and still 
make it resources available to my users? This is very important to us if we 
really want to explore this new avenue of Ceph

Thank you,

Yao Mensah
Systems Administrator II
OLS Servers
yao.men...@usdoj.gov
(202) 307 0354
MCITP
MCSE NT4.0 / 2000-2003
A+

From: Dave Spano [mailto:dsp...@optogenics.com]
Sent: Thursday, May 23, 2013 1:19 PM
To: Mensah, Yao (CIV)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] FW: About RBD

Unless something changed, each RBD needs to be attached to 1 host at a time 
like an ISCSI lun.
Dave Spano
Optogenics



From: "Yao Mensah (CIV)" mailto:yao.men...@usdoj.gov>>
To: ceph-users@lists.ceph.com
Sent: Thursday, May 23, 2013 1:10:53 PM
Subject: [ceph-users] FW: About RBD
FYI

From: Mensah, Yao (CIV)
Sent: Wednesday, May 22, 2013 5:59 PM
To: 'i...@inktank.com'
Subject: About RBD

Hello,

I was doing some reading on your web site about ceph and what it capable of. I 
have one question and maybe you can help on this:

Can ceph RBD be used by 2 physical hosts at the same time? Or, is Ceph rbd 
CSV(Clustered Shared Volumes) aware?

Thank you,

Yao Mensah
Systems Administrator II
OLS Servers


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mkcephfs

[Sage Weil]

Can you be more specific?  How does it fail?

A copy of the actual output would be ideal, thanks!

sage

On Thu, 23 May 2013, Dewan Shamsul Alam wrote:

> Hi,
> 
> I had a running ceph cluster on bobtail. It was on 0.56.4. It is my test
> cluster. I upgraded it to 0.56.6, now mkcephfs doesn't work with the same
> working configuration and the following command:
> 
> /sbin/mkcephfs -a -c /etc/ceph/ceph.conf
> 
> ceph.conf
> 
> [global]
>     auth supported = none
>     auth cluster required = none
>     auth service required = none
>     auth client required = none
> [osd]
>     osd journal size = 1000
>     filestore xattr use omap = true
>     osd mkfs type = btrfs
>     osd mkfs options btrfs = -m raid0
>     osd mount options btrfs = rw, noatime
> [mon.a]
>     host = ceph0
>     mon addr = 192.168.128.10:6789
> [mon.b]
>     host = ceph1
>     mon addr = 192.168.128.11:6789
> [mon.c]
>     host = ceph2
>     mon addr = 192.168.128.12:6789
> [osd.0]
>     host = ceph0
>     devs = /dev/sda3
> [osd.1]
>     host = ceph1
>     devs = /dev/sda3
> [osd.2]
>     host = ceph2
>     devs = /dev/sda3
> [mds.a]
>     host = ceph0
> [mds.b]
>     host = ceph1
> [mds.c]
>     host = ceph2
> 
> Best Regards,
> Dewan Shamsul Alam
> 
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] FW: About RBD

[Gregory Farnum]

You can attach an RBD device to multiple volumes, and if you don't use
the cache then the RBD layer will even be coherent.
But of course that's just the disk, so unless you're using a
cluster-aware FS like OCFS2 on top then mounting from multiple places
will blow up your data.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Thu, May 23, 2013 at 10:18 AM, Dave Spano  wrote:
> Unless something changed, each RBD needs to be attached to 1 host at a time
> like an ISCSI lun.
>
> Dave Spano
> Optogenics
>
>
> 
> From: "Yao Mensah (CIV)" 
> To: ceph-users@lists.ceph.com
> Sent: Thursday, May 23, 2013 1:10:53 PM
> Subject: [ceph-users] FW: About RBD
>
>
> FYI
>
>
>
> From: Mensah, Yao (CIV)
> Sent: Wednesday, May 22, 2013 5:59 PM
> To: 'i...@inktank.com'
> Subject: About RBD
>
>
>
> Hello,
>
>
>
> I was doing some reading on your web site about ceph and what it capable of.
> I have one question and maybe you can help on this:
>
>
>
> Can ceph RBD be used by 2 physical hosts at the same time? Or, is Ceph rbd
> CSV(Clustered Shared Volumes) aware?
>
>
>
> Thank you,
>
>
>
> Yao Mensah
>
> Systems Administrator II
>
> OLS Servers
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] FW: About RBD

[Dave Spano]

Unless something changed, each RBD needs to be attached to 1 host at a time 
like an ISCSI lun. 


Dave Spano 
Optogenics 


- Original Message -

From: "Yao Mensah (CIV)"  
To: ceph-users@lists.ceph.com 
Sent: Thursday, May 23, 2013 1:10:53 PM 
Subject: [ceph-users] FW: About RBD 



FYI 



From: Mensah, Yao (CIV) 
Sent: Wednesday, May 22, 2013 5:59 PM 
To: 'i...@inktank.com' 
Subject: About RBD 

Hello, 

I was doing some reading on your web site about ceph and what it capable of. I 
have one question and maybe you can help on this: 

Can ceph RBD be used by 2 physical hosts at the same time? Or, is Ceph rbd 
CSV(Clustered Shared Volumes) aware? 

Thank you, 

Yao Mensah 
Systems Administrator II 
OLS Servers 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] FW: About RBD

[Mensah, Yao (CIV)]

FYI

From: Mensah, Yao (CIV)
Sent: Wednesday, May 22, 2013 5:59 PM
To: 'i...@inktank.com'
Subject: About RBD

Hello,

I was doing some reading on your web site about ceph and what it capable of. I 
have one question and maybe you can help on this:

Can ceph RBD be used by 2 physical hosts at the same time? Or, is Ceph rbd 
CSV(Clustered Shared Volumes) aware?

Thank you,

Yao Mensah
Systems Administrator II
OLS Servers

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] mkcephfs

[Dewan Shamsul Alam]

Hi,

I had a running ceph cluster on bobtail. It was on 0.56.4. It is my test
cluster. I upgraded it to 0.56.6, now mkcephfs doesn't work with the same
working configuration and the following command:

/sbin/mkcephfs -a -c /etc/ceph/ceph.conf

ceph.conf

[global]
auth supported = none
auth cluster required = none
auth service required = none
auth client required = none
[osd]
osd journal size = 1000
filestore xattr use omap = true
osd mkfs type = btrfs
osd mkfs options btrfs = -m raid0
osd mount options btrfs = rw, noatime
[mon.a]
host = ceph0
mon addr = 192.168.128.10:6789
[mon.b]
host = ceph1
mon addr = 192.168.128.11:6789
[mon.c]
host = ceph2
mon addr = 192.168.128.12:6789
[osd.0]
host = ceph0
devs = /dev/sda3
[osd.1]
host = ceph1
devs = /dev/sda3
[osd.2]
host = ceph2
devs = /dev/sda3
[mds.a]
host = ceph0
[mds.b]
host = ceph1
[mds.c]
host = ceph2

Best Regards,
Dewan Shamsul Alam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-deploy

[Dewan Shamsul Alam]

Hi,

I tried ceph-deploy all day. Found that it has a python-setuptools as
dependency. I knew about python-pushy. But is there any other dependency
that I'm missing?

The problem I'm getting are as follows:

#ceph-deploy gatherkeys ceph0 ceph1 ceph2
returns the following error,
Unable to find /etc/ceph/ceph.client.admin.keyring on ['ceph0', 'ceph1',
'ceph2']

Once I got passed this, I don't know why it works sometimes. I have been
following the exact steps as mentioned in the blog.

Then when I try to do

ceph-deploy osd create ceph0:/dev/sda3 ceph1:/dev/sda3 ceph2:/dev/sda3

It gets stuck.

I'm using Ubuntu 13.04 for ceph-deploy and 12.04 for ceph nodes. I just
need to get the cuttlefish working and willing to change the OS if it is
required. Please help. :)

Best Regards,
Dewan Shamsul Alam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-deploy

[Dewan Shamsul Alam]

Hi,

I tried ceph-deploy all day. Found that it has a python-setuptools as
dependency. I knew about python-pushy. But is there any other dependency
that I'm missing?

The problem I'm getting are as follows:

#ceph-deploy gatherkeys ceph0 ceph1 ceph2
returns the following error,
Unable to find /etc/ceph/ceph.client.admin.keyring on ['ceph0', 'ceph1',
'ceph2']

Once I got passed this, I don't know why it works sometimes. I have been
following the exact steps as mentioned in the blog.

Then when I try to do

ceph-deploy osd create ceph0:/dev/sda3 ceph1:/dev/sda3 ceph2:/dev/sda3

It gets stuck.

I'm using Ubuntu 13.04 for ceph-deploy and 12.04 for ceph nodes. I just
need to get the cuttlefish working and willing to change the OS if it is
required. Please help. :)

Best Regards,
Dewan Shamsul Alam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw with nginx

[Erdem Agaoglu]

Hi all,

We are trying to run radosgw with nginx.
We've found an example https://gist.github.com/guilhem/4964818
And changed our nginx.conf like below:

http {
server {
listen 0.0.0.0:80 ;
server_name _;
access_log  off;
location / {
fastcgi_pass_header Authorization;
fastcgi_pass_request_headers on;
include fastcgi_params;
fastcgi_keep_conn on;
fastcgi_pass unix:/tmp/radosgw.sock;
}
}
}

But the simplest test gives following error:

# curl -v http://x.x.x.x/bucket/test.jpg
* About to connect() to x.x.x.x port 80 (#0)
*   Trying x.x.x.x ... connected
> GET /bucket/test.jpg HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: x.x.x.x
> Accept: */*
>
< HTTP/1.1 400
< Server: nginx/1.1.19
< Date: Thu, 23 May 2013 15:34:05 GMT
< Content-Type: application/json
< Content-Length: 26
< Connection: keep-alive
< Accept-Ranges: bytes
<
* Connection #0 to host x.x.x.x left intact
* Closing connection #0
{"Code":"InvalidArgument"}

radosgw logs show these:

2013-05-23 08:34:31.074037 7f0739c33780 20 enqueued request req=0x1e78870
2013-05-23 08:34:31.074044 7f0739c33780 20 RGWWQ:
2013-05-23 08:34:31.074045 7f0739c33780 20 req: 0x1e78870
2013-05-23 08:34:31.074047 7f0739c33780 10 allocated request req=0x1ec6490
2013-05-23 08:34:31.074084 7f0720ce8700 20 dequeued request req=0x1e78870
2013-05-23 08:34:31.074093 7f0720ce8700 20 RGWWQ: empty
2013-05-23 08:34:31.074098 7f0720ce8700  1 == starting new request
req=0x1e78870 =
2013-05-23 08:34:31.074140 7f0720ce8700  2 req 4:0.42initializing
2013-05-23 08:34:31.074174 7f0720ce8700  5 nothing to log for operation
2013-05-23 08:34:31.074178 7f0720ce8700  2 req 4:0.80::GET
/bucket/test.jpg::http status=400
2013-05-23 08:34:31.074192 7f0720ce8700  1 == req done req=0x1e78870
http_status=400 ==


Normally we expect a well formed 403 (because request doesn't have
Authorization header) but we have a 400 and cannot figure out why.

Thanks in advance.

-- 
erdem agaoglu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon problems after upgrading to cuttlefish

[Smart Weblications GmbH - Florian Wiessner]

Hi,


please do not forget to respond to the list ceph-users@lists.ceph.com

find answer below.

Am 23.05.2013 17:16, schrieb Bryan Stillwell:
> This is what I currently have configured:
> 
> 
> # Ceph config file
> 
> [global]
> auth cluster required = none
> auth service required = none
> #   auth client required = cephx
> 
> [osd]
> osd journal size = 1000
> filestore xattr use omap = true
> osd mkfs type = xfs
> osd mkfs options xfs = noatime
> 
> [mon.a]
> host = a1
> mon addr = 172.24.88.50:6789
> 
> [osd.0]
> host = b1
> devs = /dev/sdb
> 
> [osd.1]
> host = b1
> devs = /dev/sdc
> 
> [osd.2]
> host = b1
> devs = /dev/sdd
> 
> [osd.3]
> host = b1
> devs = /dev/sde
> 
> [osd.4]
> host = b1
> devs = /dev/sdf
> 
> [osd.5]
> host = b2
> devs = /dev/sdb
> 
> [osd.6]
> host = b2
> devs = /dev/sdc
> 
> [osd.7]
> host = b2
> devs = /dev/sdd1
> 
> [osd.8]
> host = b2
> devs = /dev/sde
> 
> [osd.9]
> host = b2
> devs = /dev/sdf
> 
> [osd.10]
> host = b3
> devs = /dev/sdb1
> 
> [osd.11]
> host = b3
> devs = /dev/sdc1
> 
> [osd.12]
> host = b3
> devs = /dev/sdd1
> 
> [osd.13]
> host = b3
> devs = /dev/sde1
> 
> [osd.14]
> host = b3
> devs = /dev/sdf1
> 
> [osd.15]
> host = b4
> devs = /dev/sdb1
> 
> [osd.16]
> host = b4
> devs = /dev/sdc1
> 
> [osd.17]
> host = b4
> devs = /dev/sdd1
> 
> [osd.18]
> host = b4
> devs = /dev/sde1
> 
> [osd.19]
> host = b4
> devs = /dev/sdf1
> 
> [osd.20]
> host = b1
> devs = /dev/sda4
> 
> [osd.21]
> host = b2
> devs = /dev/sda4
> 
> [osd.22]
> host = b3
> devs = /dev/sda4
> 
> [osd.23]
> host = b4
> devs = /dev/sda4
> 
> [mds.a]
> host = a1
> 
> #[client]
> #   debug ms = 1
> #   debug client = 20
> 
> On Thu, May 23, 2013 at 4:00 AM, Smart Weblications GmbH - Florian
> Wiessner  wrote:
>> Am 23.05.2013 07:45, schrieb Bryan Stillwell:
>>> I attempted to upgrade my bobtail cluster to cuttlefish tonight and I
>>> believe I'm running into some mon related issues.  I did the original
>>> install manually instead of with mkcephfs or ceph-deploy, so I think
>>> that might have to do with this error:
>>>
>>> root@a1:~# ceph-mon -d -c /etc/ceph/ceph.conf
>>> 2013-05-22 23:37:29.283975 7f8fb97b3780  0 ceph version 0.61.2
>>> (fea782543a844bb277ae94d3391788b76c5bee60), process ceph-mon, pid 5531
>>> IO error: /var/lib/ceph/mon/ceph-admin/store.db/LOCK: No such file or 
>>> directory
>>> 2013-05-22 23:37:29.286534 7f8fb97b3780  1 unable to open monitor
>>> store at /var/lib/ceph/mon/ceph-admin
>>> 2013-05-22 23:37:29.286544 7f8fb97b3780  1 check for old monitor store 
>>> format
>>> 2013-05-22 23:37:29.286550 7f8fb97b3780  1
>>> store(/var/lib/ceph/mon/ceph-admin) mount
>>> 2013-05-22 23:37:29.286559 7f8fb97b3780  1
>>> store(/var/lib/ceph/mon/ceph-admin) basedir
>>> /var/lib/ceph/mon/ceph-admin dne
>>> 2013-05-22 23:37:29.286564 7f8fb97b3780 -1 unable to mount monitor
>>> store: (2) No such file or directory
>>> 2013-05-22 23:37:29.286577 7f8fb97b3780 -1 found errors while
>>> attempting to convert the monitor store: (2) No such file or directory
>>> root@a1:~# ls -l /var/lib/ceph/mon/
>>> total 4
>>> drwxr-xr-x 15 root root 4096 May 22 23:30 ceph-a
>>>
>>>
>>> I only have one mon daemon in this cluster as well.  I was planning on
>>> upgrading it to 3 tonight but when I try to run most commands they
>>> just hang now.
>>>
>>> I do see the store.db directory in the ceph-a directory if that helps:
>>>
>>> root@a1:~# ls -l  /var/lib/ceph/mon/ceph-a/
>>> total 868
>>> drwxr-xr-x 2 root root   4096 May 22 23:30 auth
>>> drwxr-xr-x 2 root root   4096 May 22 23:30 auth_gv
>>> -rw--- 1 root root 37 Feb  4 14:22 cluster_uuid
>>> -rw--- 1 root root  2 May 22 23:30 election_epoch
>>> -rw--- 1 root root120 Feb  4 14:22 feature_set
>>> -rw--- 1 root root  2 Dec 28 11:35 joined
>>> -rw--- 1 root root 77 May 22 22:30 keyring
>>> -rw--- 1 root root  0 Dec 28 11:35 lock
>>> drwxr-xr-x 2 root root  20480 May 22 23:30 logm
>>> drwxr-xr-x 2 root root  20480 May 22 23:30 logm_gv
>>> -rw--- 1 root root 21 Dec 28 11:35 magic
>>> drwxr-xr-x 2 root root  12288 May 22 23:30 mdsmap
>>> drwxr-xr-x 2 root root  12288 May 22 23:30 mdsmap_gv
>>> drwxr-xr-x 2 root root   4096 Dec 28 11:35 monmap
>>> drwxr-xr-x 2 root root 233472 May 22 23:30 osdmap
>>> drwxr-xr-x 2 root root 237568 May 22 23:30 osdmap_full
>>> drwxr-xr-x 2 root root 253952 May 22 23:30 osdmap_gv
>>> drwxr-xr-x 2 root root  20480 May 22 23:30 pgmap
>>> drwxr-xr-x 2 root root  20480 May 22 23:30 pgmap_gv
>>> drwxr-xr-x 2 root root   4096 May 

Re: [ceph-users] scrub error: found clone without head

[Olivier Bonvalet]

Not yet. I keep it for now.

Le mercredi 22 mai 2013 à 15:50 -0700, Samuel Just a écrit :
> rb.0.15c26.238e1f29
> 
> Has that rbd volume been removed?
> -Sam
> 
> On Wed, May 22, 2013 at 12:18 PM, Olivier Bonvalet  
> wrote:
> > 0.61-11-g3b94f03 (0.61-1.1), but the bug occured with bobtail.
> >
> >
> > Le mercredi 22 mai 2013 à 12:00 -0700, Samuel Just a écrit :
> >> What version are you running?
> >> -Sam
> >>
> >> On Wed, May 22, 2013 at 11:25 AM, Olivier Bonvalet  
> >> wrote:
> >> > Is it enough ?
> >> >
> >> > # tail -n500 -f /var/log/ceph/osd.28.log | grep -A5 -B5 'found clone 
> >> > without head'
> >> > 2013-05-22 15:43:09.308352 7f707dd64700  0 log [INF] : 9.105 scrub ok
> >> > 2013-05-22 15:44:21.054893 7f707dd64700  0 log [INF] : 9.451 scrub ok
> >> > 2013-05-22 15:44:52.898784 7f707cd62700  0 log [INF] : 9.784 scrub ok
> >> > 2013-05-22 15:47:43.148515 7f707cd62700  0 log [INF] : 9.3c3 scrub ok
> >> > 2013-05-22 15:47:45.717085 7f707dd64700  0 log [INF] : 9.3d0 scrub ok
> >> > 2013-05-22 15:52:14.573815 7f707dd64700  0 log [ERR] : scrub 3.6b 
> >> > ade3c16b/rb.0.15c26.238e1f29.9221/12d7//3 found clone without 
> >> > head
> >> > 2013-05-22 15:55:07.230114 7f707d563700  0 log [ERR] : scrub 3.6b 
> >> > 261cc0eb/rb.0.15c26.238e1f29.3671/12d7//3 found clone without 
> >> > head
> >> > 2013-05-22 15:56:56.456242 7f707d563700  0 log [ERR] : scrub 3.6b 
> >> > b10deaeb/rb.0.15c26.238e1f29.86a2/12d7//3 found clone without 
> >> > head
> >> > 2013-05-22 15:57:51.667085 7f707dd64700  0 log [ERR] : 3.6b scrub 3 
> >> > errors
> >> > 2013-05-22 15:57:55.241224 7f707dd64700  0 log [INF] : 9.450 scrub ok
> >> > 2013-05-22 15:57:59.800383 7f707cd62700  0 log [INF] : 9.465 scrub ok
> >> > 2013-05-22 15:59:55.024065 7f707661a700  0 -- 192.168.42.3:6803/12142 >> 
> >> > 192.168.42.5:6828/31490 pipe(0x2a689000 sd=108 :6803 s=2 pgs=200652 
> >> > cs=73 l=0).fault with nothing to send, going to standby
> >> > 2013-05-22 16:01:45.542579 7f7022770700  0 -- 192.168.42.3:6803/12142 >> 
> >> > 192.168.42.5:6828/31490 pipe(0x2a689280 sd=99 :6803 s=0 pgs=0 cs=0 
> >> > l=0).accept connect_seq 74 vs existing 73 state standby
> >> > --
> >> > 2013-05-22 16:29:49.544310 7f707dd64700  0 log [INF] : 9.4eb scrub ok
> >> > 2013-05-22 16:29:53.190233 7f707dd64700  0 log [INF] : 9.4f4 scrub ok
> >> > 2013-05-22 16:29:59.478736 7f707dd64700  0 log [INF] : 8.6bb scrub ok
> >> > 2013-05-22 16:35:12.240246 7f7022770700  0 -- 192.168.42.3:6803/12142 >> 
> >> > 192.168.42.5:6828/31490 pipe(0x2a689280 sd=99 :6803 s=2 pgs=200667 cs=75 
> >> > l=0).fault with nothing to send, going to standby
> >> > 2013-05-22 16:35:19.519019 7f707d563700  0 log [INF] : 8.700 scrub ok
> >> > 2013-05-22 16:39:15.422532 7f707dd64700  0 log [ERR] : scrub 3.1 
> >> > b1869301/rb.0.15c26.238e1f29.0836/12d7//3 found clone without 
> >> > head
> >> > 2013-05-22 16:40:04.995256 7f707cd62700  0 log [ERR] : scrub 3.1 
> >> > bccad701/rb.0.15c26.238e1f29.9a00/12d7//3 found clone without 
> >> > head
> >> > 2013-05-22 16:41:07.008717 7f707d563700  0 log [ERR] : scrub 3.1 
> >> > 8a9bec01/rb.0.15c26.238e1f29.9820/12d7//3 found clone without 
> >> > head
> >> > 2013-05-22 16:41:42.460280 7f707c561700  0 log [ERR] : 3.1 scrub 3 errors
> >> > 2013-05-22 16:46:12.385678 7f7077735700  0 -- 192.168.42.3:6803/12142 >> 
> >> > 192.168.42.5:6828/31490 pipe(0x2a689c80 sd=137 :6803 s=0 pgs=0 cs=0 
> >> > l=0).accept connect_seq 76 vs existing 75 state standby
> >> > 2013-05-22 16:58:36.079010 7f707661a700  0 -- 192.168.42.3:6803/12142 >> 
> >> > 192.168.42.3:6801/11745 pipe(0x2a689a00 sd=44 :6803 s=0 pgs=0 cs=0 
> >> > l=0).accept connect_seq 40 vs existing 39 state standby
> >> > 2013-05-22 16:58:36.798038 7f707d563700  0 log [INF] : 9.50c scrub ok
> >> > 2013-05-22 16:58:40.104159 7f707c561700  0 log [INF] : 9.526 scrub ok
> >> >
> >> >
> >> > Note : I have 8 scrub errors like that, on 4 impacted PG, and all 
> >> > impacted objects are about the same RBD image (rb.0.15c26.238e1f29).
> >> >
> >> >
> >> >
> >> > Le mercredi 22 mai 2013 à 11:01 -0700, Samuel Just a écrit :
> >> >> Can you post your ceph.log with the period including all of these 
> >> >> errors?
> >> >> -Sam
> >> >>
> >> >> On Wed, May 22, 2013 at 5:39 AM, Dzianis Kahanovich
> >> >>  wrote:
> >> >> > Olivier Bonvalet пишет:
> >> >> >>
> >> >> >> Le lundi 20 mai 2013 à 00:06 +0200, Olivier Bonvalet a écrit :
> >> >> >>> Le mardi 07 mai 2013 à 15:51 +0300, Dzianis Kahanovich a écrit :
> >> >>  I have 4 scrub errors (3 PGs - "found clone without head"), on one 
> >> >>  OSD. Not
> >> >>  repairing. How to repair it exclude re-creating of OSD?
> >> >> 
> >> >>  Now it "easy" to clean+create OSD, but in theory - in case there 
> >> >>  are multiple
> >> >>  OSDs - it may cause data lost.
> >> >> >>>
> >> >> >>> I have same problem : 8 objects (4 PG) with error "found clone 
> >> >> >>> without
> >> >> >>> head". How can I fix that ?
> >> >> 

Re: [ceph-users] v0.61.2 'ceph df' can't use

[Kelvin_Huang]

Hi Sage & Joao,

Thanks your reply !!
after checking, I missed upgrade one package on monitor node therefore ceph df 
can't use ...
I upgrade it and restart mon daemons then 'ceph df ' can be used

Thanks all 
- Kelvin 

> -Original Message-
> From: Sage Weil [mailto:s...@inktank.com]
> Sent: Tuesday, May 21, 2013 12:11 AM
> To: Kelvin Huang/WYHQ/Wiwynn
> Cc: ceph-us...@ceph.com; Victor CY Chang/WYHQ/Wiwynn; PJ
> Lin/WYHQ/Wiwynn
> Subject: Re: [ceph-users] v0.61.2 'ceph df' can't use
> 
> On Mon, 20 May 2013, kelvin_hu...@wiwynn.com wrote:
> > Hi all,
> > I upgrade Ceph to the v0.62, but I find the 'ceph df' command can't use...
> that show 'unrecognized command' , why ? that need other options ?
> 
> Make sure the ceph-mon daemons have been restarted so that they are
> running the new version.
> 
> sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com