Re: [ceph-users] radosgw and keystone version 3 domains

If any of you could provide keystone.log with me, it would be more helpful. and: keystone --version Shinobu - Original Message - From: "Shinobu Kinjo" To: "Robert Duncan" Cc: "Luis Periquito" , "Abhishek L" , "ceph-users" Sent: Saturday, September 26, 2015 12:03:17 PM Subject: Re: [c

Re: [ceph-users] radosgw and keystone version 3 domains

> and need to use openstack client. Yes, you have to for v3 anyway. Shinobu - Original Message - From: "Robert Duncan" To: "Luis Periquito" Cc: "Shinobu Kinjo" , "Abhishek L" , "ceph-users" Sent: Friday, September 25, 2015 11:29:14 PM Subject: RE: [ceph-users] radosgw and keystone ve

Re: [ceph-users] CephFS "corruption" -- Nulled bytes

It may have been. Although the timestamp on the file was almost a month ago. The typical workflow for this particular file is to copy an updated version overtop of it. i.e. 'cp qss kstat' I'm not sure if cp semantics would keep the same inode and simply truncate/overwrite the contents, or if it w

Re: [ceph-users] CephFS "corruption" -- Nulled bytes

Looks like you might be experiencing this bug: http://tracker.ceph.com/issues/12551 Fix has been merged to master and I believe it'll be part of infernalis. The original reproducer involved truncating/overwriting files. In your example, do you know if 'kstat' has been truncated/overwritten prio

[ceph-users] CephFS "corruption" -- Nulled bytes

Hello all, I've run into some sort of bug with CephFS. Client reads of a particular file return nothing but 40KB of Null bytes. Doing a rados level get of the inode returns the whole file, correctly. Tested via Linux 4.1, 4.2 kernel clients, and the 0.94.3 fuse client. Attached is a dynamic prin

Re: [ceph-users] [sepia] debian jessie repository ?

Hi, Am 2015-09-25 um 22:23 schrieb Udo Lembke: > you can use this sources-list > > cat /etc/apt/sources.list.d/ceph.list > deb http://gitbuilder.ceph.com/ceph-deb-jessie-x86_64-basic/ref/v0.94.3 > jessie main Thanks! Will test it as soon as I get back to work next week. Regards, -- j.hofmülle

Re: [ceph-users] Potential OSD deadlock?

We dropped the replication on our cluster from 4 to 3 and it looks like all the blocked I/O has stopped (no entries in the log for the last 12 hours). This makes me believe that there is some issue with the number of sockets or some other TCP issue. We have not messed with Ephemeral ports and TIME_

Re: [ceph-users] [sepia] debian jessie repository ?

Hi, you can use this sources-list cat /etc/apt/sources.list.d/ceph.list deb http://gitbuilder.ceph.com/ceph-deb-jessie-x86_64-basic/ref/v0.94.3 jessie main Udo On 25.09.2015 15:10, Jogi Hofmüller wrote: > Hi, > > Am 2015-09-11 um 13:20 schrieb Florent B: > >> Jessie repository will be available

Re: [ceph-users] occasional failure to unmap rbd

On 09/25/2015 02:28 PM, Jan Schermer wrote: What about /sys/block/krbdX/holders? Nothing in there? There is no /sys/block/krbd450, but there is /sys/block/rbd450. In our case, /sys/block/rbd450/holders is empty. Jeff ___ ceph-users mailing list ceph

Re: [ceph-users] occasional failure to unmap rbd

On Fri, Sep 25, 2015 at 7:41 PM, Jeff Epstein wrote: > On 09/25/2015 12:38 PM, Ilya Dryomov wrote: >> >> On Fri, Sep 25, 2015 at 7:17 PM, Jeff Epstein >> wrote: >>> >>> We occasionally have a situation where we are unable to unmap an rbd. >>> This >>> occurs intermittently, with no obvious cause.

Re: [ceph-users] occasional failure to unmap rbd

What about /sys/block/krbdX/holders? Nothing in there? Jan > On 25 Sep 2015, at 19:44, Jeff Epstein wrote: > > On 09/25/2015 12:53 PM, Jan Schermer wrote: >> What are you looking for in lsof? Did you try looking for the major/minor >> number of the rbd device? >> Things that could hold the dev

Re: [ceph-users] occasional failure to unmap rbd

On 09/25/2015 12:53 PM, Jan Schermer wrote: What are you looking for in lsof? Did you try looking for the major/minor number of the rbd device? Things that could hold the device are devicemapper, lvm, swraid and possibly many more, not sure if all that shows in lsof output... I searched for th

Re: [ceph-users] OSD reaching file open limit - known issues?

I get that, even though I think it should be handled more gracefuly. But is it expected to also lead to consistency issues like this? I think this is exactly what we're hitting right now http://tracker.ceph.com/issues/6101 except I have no idea why it also h

Re: [ceph-users] OSD reaching file open limit - known issues?

Yes, known issue, make sure your system open file limit is pretty high.. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Jan Schermer Sent: Friday, September 25, 2015 4:42 AM To: ceph-users@lists.ceph.com Subject: [ceph-users] OSD reaching file open limit - known issues?

Re: [ceph-users] occasional failure to unmap rbd

What are you looking for in lsof? Did you try looking for the major/minor number of the rbd device? Things that could hold the device are devicemapper, lvm, swraid and possibly many more, not sure if all that shows in lsof output... Jan > On 25 Sep 2015, at 18:41, Jeff Epstein wrote: > > On 0

Re: [ceph-users] occasional failure to unmap rbd

On 09/25/2015 12:38 PM, Ilya Dryomov wrote: On Fri, Sep 25, 2015 at 7:17 PM, Jeff Epstein wrote: We occasionally have a situation where we are unable to unmap an rbd. This occurs intermittently, with no obvious cause. For the most part, rbds can be unmapped fine, but sometimes we get this: # r

Re: [ceph-users] occasional failure to unmap rbd

On Fri, Sep 25, 2015 at 7:17 PM, Jeff Epstein wrote: > We occasionally have a situation where we are unable to unmap an rbd. This > occurs intermittently, with no obvious cause. For the most part, rbds can be > unmapped fine, but sometimes we get this: > > # rbd unmap /dev/rbd450 > rbd: sysfs writ

[ceph-users] occasional failure to unmap rbd

We occasionally have a situation where we are unable to unmap an rbd. This occurs intermittently, with no obvious cause. For the most part, rbds can be unmapped fine, but sometimes we get this: # rbd unmap /dev/rbd450 rbd: sysfs write failed rbd: unmap failed: (16) Device or resource busy Thin

Re: [ceph-users] НА: How to get RBD volume to PG mapping?

On Fri, Sep 25, 2015 at 5:53 PM, Межов Игорь Александрович wrote: > Hi! > > Thanks! > > I have some suggestions for the 1st method: > >>You could get the name prefix for each RBD from rbd info, > Yes, I did it already at the steps 1 and 2. I forgot to mention, that I grab > rbd frefix from 'rbd in

Re: [ceph-users] How to get RBD volume to PG mapping?

> On 25 Sep 2015, at 16:53, Межов Игорь Александрович wrote: > > Hi! > > Thanks! > > I have some suggestions for the 1st method: > > >You could get the name prefix for each RBD from rbd info, > Yes, I did it already at the steps 1 and 2. I forgot to mention, that I grab > rbd frefix from 'rb

Re: [ceph-users] How to get RBD volume to PG mapping?

> >then list all objects (run find on the osds?) and then you just need to > grep the OSDs for each prefix. > So, you advise to run find over ssh for all OSD hosts to traverse OSDs > filesystems and find files (objects), > named with rbd prefix? Am I right? If so, I have two thoughts: (1) it may >

[ceph-users] НА: How to get RBD volume to PG mapping?

Hi! Thanks! I have some suggestions for the 1st method: >You could get the name prefix for each RBD from rbd info, Yes, I did it already at the steps 1 and 2. I forgot to mention, that I grab rbd frefix from 'rbd info' command >then list all objects (run find on the osds?) and then you just n

Re: [ceph-users] OSD reaching file open limit - known issues?

We trashed one OSD and started backfilling it. After about 90 minutes it started crashing again: I found http://tracker.ceph.com/issues/6101 We'll disable snap trimming so it at least runs, but could someone suggest what the root cause is? Can OSD get backf

Re: [ceph-users] radosgw and keystone version 3 domains

A few other things that don’t work – -appending /v3 into the rgw.conf file (worth a try) -adding the user into the default domain - removing the v2 endpoints from the keystone catalog -using a domain scoped token in rgw.conf -using admin username and password in rgw.conf According to keystone doc

Re: [ceph-users] How to get RBD volume to PG mapping?

So I had two ideas here: 1. Use find as Jan suggested. You probably can bound it by the expected object naming and limit it to the OSDs that were impacted. This is probably the best way. 2. Use the osdmaptool against a copy of the osdmap that you pre-grab from the cluster, ala: https://www.hastexo

Re: [ceph-users] How to get RBD volume to PG mapping?

Ouch 1) I should have read it completely 2) I should have tested it :) Sorry about that... You could get the name prefix for each RBD from rbd info, then list all objects (run find on the osds?) and then you just need to grep the OSDs for each prefix... Should be much faster? Jan > On 25 Se

Re: [ceph-users] How to get RBD volume to PG mapping?

Try: ceph osd map Is that it? Jan > On 25 Sep 2015, at 15:07, Межов Игорь Александрович wrote: > > Hi! > > Last week I wrote, that one PG in our Firefly stuck in degraded state with 2 > replicas instead of 3 > and do not try to backfill or recovery. We try to investigate, what RBD vol's

Re: [ceph-users] radosgw and keystone version 3 domains

This was reported in http://tracker.ceph.com/issues/8052 about a year ago. This ticket hasn't been updated... On Fri, Sep 25, 2015 at 1:37 PM, Robert Duncan wrote: > I would be interested if anyone even has a work around to this - no matter > how arcane. > If anyone gets this to work I would be

Re: [ceph-users] [sepia] debian jessie repository ?

Hi, Am 2015-09-11 um 13:20 schrieb Florent B: > Jessie repository will be available on next Hammer release ;) An how should I continue installing ceph meanwhile? ceph-deploy new ... overwrites the /etc/apt/sources.list.d/ceph.list and hence throws an error :( Any hint appreciated. Cheers, --

[ceph-users] How to get RBD volume to PG mapping?

Hi! Last week I wrote, that one PG in our Firefly stuck in degraded state with 2 replicas instead of 3 and do not try to backfill or recovery. We try to investigate, what RBD vol's are affected. The working plan are inspired by Sebastian Han's snippet (http://www.sebastien-han.fr/blog/2013/11/

Re: [ceph-users] radosgw and keystone version 3 domains

I would be interested if anyone even has a work around to this - no matter how arcane. If anyone gets this to work I would be most obliged -Original Message- From: Shinobu Kinjo [mailto:ski...@redhat.com] Sent: 25 September 2015 13:31 To: Luis Periquito Cc: Abhishek L; Robert Duncan; ceph

Re: [ceph-users] radosgw and keystone version 3 domains

Thanks for the info. Shinobu - Original Message - From: "Luis Periquito" To: "Shinobu Kinjo" Cc: "Abhishek L" , "Robert Duncan" , "ceph-users" Sent: Friday, September 25, 2015 8:52:48 PM Subject: Re: [ceph-users] radosgw and keystone version 3 domains I'm having the exact same issue,

Re: [ceph-users] radosgw and keystone version 3 domains

I'm having the exact same issue, and after looking it seems that radosgw is hardcoded to authenticate using v2 api. from the config file: rgw keystone url = http://openstackcontrol.lab:35357/ the "/v2.0/" is hardcoded and gets appended to the authentication request. a snippet taken from radosgw

[ceph-users] OSD reaching file open limit - known issues?

Hi, we recently migrated some of our nodes to Ubuntu 12, which helped everything quite a bit. But we hit a snag where the upstart initscript would not set the file open ulimit correctly and some OSDs ran out of fds. Some roblems manifested since then on the node where this happened such as scr

Re: [ceph-users] lttng duplicate registration problem when using librados2 and libradosstriper

On 23/09/15 15:11, Jason Dillaman wrote: > It looks like the issue you are experiencing was fixed in the > Infernalis/master branches [1]. I've opened a new tracker ticket to backport > the fix to Hammer [2]. thanks for that we managed to build ceph using the SRPM (having tried and failed at

[ceph-users] nova instance cannot boot after remove cache tier--help

Hi, I have a ceph cluster as the nova backend storage, and I enabled the cache tier with readonly cache-mode for the nova_pool, now the nova instance cannot boot after remove the nova_pool cache tier, The instance show the error is "boot failed:not a bootable disk" I used the below command to

Re: [ceph-users] CephFS: Question how to debug client sessions

On Fri, Sep 25, 2015 at 2:55 AM, Goncalo Borges wrote: > Hi All... > > I have some questions about client session in CephFS. > > 1./ My setup: > > a. ceph 9.0.3 > b. 32 OSDs distributed in 4 servers (8 OSD per server). > c. 'osd pool default size = 3' and 'osd pool default min size = 2' > d. a sin