Re: [ceph-users] Error on adding Monitors (Joao Eduardo Luis)

[David J F Carradice]

My apologies Jaoa

I only attached my ceph.log instead of my conf file. Please find the global
conf file, as generated by ceph-deploy.

David

From:  David J F Carradice 
Date:  Friday, 25 October 2013 09:39
To:  
Subject:  Re: Error on adding Monitors (Joao Eduardo Luis)

Hi Joao

Please find a copy of my ceph.log from my ceph-deploy server attached.

With regards to your request for the mon.foo, could you enlighten me as to
this request. I have used the ceph-deploy and am currently trying to read
all of the Œmanual¹ installation notes, so that I may better understand how
this all works.

Regards

David

From:  
Reply-To:  
Date:  Thursday, 24 October 2013 15:48
To:  
Subject:  ceph-users Digest, Vol 9, Issue 118

Send ceph-users mailing list submissions to
ceph-users@lists.ceph.com

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
or, via email, send a message with subject or body 'help' to
ceph-users-requ...@lists.ceph.com

You can reach the person managing the list at
ceph-users-ow...@lists.ceph.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of ceph-users digest..."


Today's Topics:

   1. Re: Non-Ceph cluster name (Sage Weil)
   2. Re: Error on adding Monitors (Joao Eduardo Luis)
   3. Re: About use same SSD for OS and Journal (Kurt Bauer)
   4. Re: Hardware: SFP+ or 10GBase-T (Mark Nelson)


--

--

Message: 2
Date: Thu, 24 Oct 2013 15:23:02 +0100
From: Joao Eduardo Luis 
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Error on adding Monitors
Message-ID: <52692d46.10...@inktank.com>
Content-Type: text/plain; charset=windows-1252; format=flowed

On 10/24/2013 03:12 PM, David J F Carradice wrote:
>  Hi.
> 
>  I am getting an error on adding monitors to my cluster.
>  ceph@ceph-deploy:~/my-cluster$ ceph-deploy mon create ceph-osd01
>  [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy mon
>  create ceph-osd01
>  [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-osd01
>  [ceph_deploy.mon][DEBUG ] detecting platform for host ceph-osd01 ...
>  [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
>  [ceph_deploy.mon][INFO  ] distro info: Ubuntu 13.04 raring
>  [ceph-osd01][DEBUG ] determining if provided host has same hostname in
>  remote
>  [ceph-osd01][DEBUG ] deploying mon to ceph-osd01
>  [ceph-osd01][DEBUG ] remote hostname: ceph-osd01
>  [ceph-osd01][INFO  ] write cluster configuration to /etc/ceph/{cluster}.conf
>  [ceph-osd01][DEBUG ] checking for done path:
>  /var/lib/ceph/mon/ceph-ceph-osd01/done
>  [ceph-osd01][INFO  ] create a done file to avoid re-doing the mon deployment
>  [ceph-osd01][INFO  ] create the init path if it does not exist
>  [ceph-osd01][INFO  ] locating `service` executable...
>  [ceph-osd01][INFO  ] found `service` executable: /usr/sbin/service
>  [ceph-osd01][INFO  ] Running command: sudo initctl emit ceph-mon
>  cluster=ceph id=ceph-osd01
>  [ceph-osd01][INFO  ] Running command: sudo ceph --admin-daemon
>  /var/run/ceph/ceph-mon.ceph-osd01.asok mon_status
>  [ceph-osd01][ERROR ] admin_socket: exception getting command
>  descriptions: [Errno 2] No such file or directory
>  [ceph-osd01][WARNIN] monitor: mon.ceph-osd01, might not be running yet
>  [ceph-osd01][INFO  ] Running command: sudo ceph --admin-daemon
>  /var/run/ceph/ceph-mon.ceph-osd01.asok mon_status
>  [ceph-osd01][ERROR ] admin_socket: exception getting command
>  descriptions: [Errno 2] No such file or directory
>  [ceph-osd01][WARNIN] ceph-osd01 is not defined in `mon initial members`
>  [ceph-osd01][WARNIN] monitor ceph-osd01 does not exist in monmap
>  [ceph-osd01][WARNIN] neither `public_addr` nor `public_network` keys are
>  defined for monitors
>  [ceph-osd01][WARNIN] monitors may not be able to form quorum
> 
>  This is happening after a successful and first add of a monitor,
>  ceph-mon01. As per the ceph-deploy documentation, I added a single
>  monitor, then some disk daemons located on ceph-osd01~03, then went to
>  add more monitors, ceph-osd01 & 02 for a quorum. This is where I get the
>  issue.
> 
>  Is the issue related to the WARNING present regarding keys?

That's a warning regarding config options (public_addr/public_network)
and the lack of enough info to generate a monmap.

> 
>  It appears that when running the ceph-deploy mon create  from my
>  ceph-deploy server, it complains about there not being any
>  ceph-mon..asok (which I assume are address sockets). I looked in
>  the respective directories on the potential monitor nodes (which are
>  currently also the OSD nodes) and see that there is only an OSD.asok, no
>  MON.asok

The monitor's .asok (admin socket) will only be created at start.  If
the monitor hasn't been run yet, then there's no asok.

> 
>  I can send my ceph.conf and a brief overview if it helps.

ceph.conf, speciall

Re: [ceph-users] Problems with keyrings during deployment

[xan.peng]

On Wed, Aug 28, 2013 at 12:42 AM, Alfredo Deza  wrote:
>
>>> On Tue, Aug 27, 2013 at 12:04 PM, Francesc Alted 
 On Tue, Aug 27, 2013 at 5:29 PM, Sage Weil  wrote:
> There are several possible pitfalls here; the missing keys are just the
> most visible symptom of the monitors not forming an initial quorum.
>
>> No problem.  A possible idea for enhancing the capabilities to
>> self-detecting problems would be to implement a check in ceph-deploy (or in
>> another place) that warns (or just gives an error) when it detects that the
>> hostname is different depending on whether they do a DNS lookup or a
>> `hostname` output.
>
>
> I went ahead and created http://tracker.ceph.com/issues/6132 to track this.
>

Well, I happened to run into this gatherkeys problem today, by grep I
found a quick way to
find the cause, which is `/usr/sbin/ceph-create-keys -i $YOUR_HOST`,
bootstrap keyrings
to be gathered are created by this command, it will tell you reason.

-- 
Best,
xan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-deploy with different cluster name

[Laurent Durnez]

Hi everyone,

I'm trying to set up a second cluster specifying its name but I can't 
bring up the first mon nodes. The second cluster is on different 
hardware that my first one. I'm following the quick start : 
http://ceph.com/docs/master/start/quick-ceph-deploy


I've reproduced it with 1 node :
1) ceph-deploy --cluster cluster02 new cl02-cephnode01
2) ceph-deploy --cluster cluster02 install cl02-cephnode01
3) ceph-deploy --cluster cluster02 mon create cl02-cephnode01

And I've got the following log :

[ceph_deploy.mon][INFO  ] distro info: Debian 7.2 wheezy
[cl02-cephnode01][DEBUG ] determining if provided host has same hostname 
in remote

[cl02-cephnode01][DEBUG ] deploying mon to cl02-cephnode01
[cl02-cephnode01][DEBUG ] remote hostname: cl02-cephnode01
[cl02-cephnode01][INFO  ] write cluster configuration to 
/etc/ceph/{cluster}.conf
[cl02-cephnode01][INFO  ] creating path: 
/var/lib/ceph/mon/cluster02-cl02-cephnode01
[cl02-cephnode01][DEBUG ] checking for done path: 
/var/lib/ceph/mon/cluster02-cl02-cephnode01/done
[cl02-cephnode01][DEBUG ] done path does not exist: 
/var/lib/ceph/mon/cluster02-cl02-cephnode01/done

[cl02-cephnode01][INFO  ] creating tmp path: /var/lib/ceph/tmp
[cl02-cephnode01][INFO  ] creating keyring file: 
/var/lib/ceph/tmp/cluster02-cl02-cephnode01.mon.keyring

[cl02-cephnode01][INFO  ] create the monitor keyring file
[cl02-cephnode01][INFO  ] Running command: ceph-mon --cluster cluster02 
--mkfs -i cl02-cephnode01 --keyring 
/var/lib/ceph/tmp/cluster02-cl02-cephnode01.mon.keyring
[cl02-cephnode01][INFO  ] ceph-mon: mon.noname-a 192.168.1.111:6789/0 is 
local, renaming to mon.cl02-cephnode01
[cl02-cephnode01][INFO  ] ceph-mon: set fsid to 
6636a4f5-a839-4b9f-aff4-e8788fcd3434
[cl02-cephnode01][INFO  ] ceph-mon: created monfs at 
/var/lib/ceph/mon/cluster02-cl02-cephnode01 for mon.cl02-cephnode01
[cl02-cephnode01][INFO  ] unlinking keyring file 
/var/lib/ceph/tmp/cluster02-cl02-cephnode01.mon.keyring
[cl02-cephnode01][INFO  ] create a done file to avoid re-doing the mon 
deployment

[cl02-cephnode01][INFO  ] create the init path if it does not exist
[cl02-cephnode01][INFO  ] locating `service` executable...
[cl02-cephnode01][INFO  ] found `service` executable: /usr/sbin/service
[cl02-cephnode01][INFO  ] Running command: sudo /usr/sbin/service ceph 
-c /etc/ceph/cluster02.conf start mon.cl02-cephnode01
[cl02-cephnode01][INFO  ] Running command: sudo ceph --admin-daemon 
/var/run/ceph/ceph-mon.cl02-cephnode01.asok mon_status
[cl02-cephnode01][ERROR ] admin_socket: exception getting command 
descriptions: [Errno 2] No such file or directory
[cl02-cephnode01][WARNIN] monitor: mon.cl02-cephnode01, might not be 
running yet
[cl02-cephnode01][INFO  ] Running command: sudo ceph --admin-daemon 
/var/run/ceph/ceph-mon.cl02-cephnode01.asok mon_status
[cl02-cephnode01][ERROR ] admin_socket: exception getting command 
descriptions: [Errno 2] No such file or directory

[cl02-cephnode01][WARNIN] monitor cl02-cephnode01 does not exist in monmap
[cl02-cephnode01][WARNIN] neither `public_addr` nor `public_network` 
keys are defined for monitors

[cl02-cephnode01][WARNIN] monitors may not be able to form quorum

On the mon node, I've got this line in the log :
2013-10-25 10:22:26.427836 7f0d6d957780  0 ceph version 0.67.4 
(ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-mon, pid 2505


my cluster02.conf is the one that has been created by default :
[global]
fsid = 6636a4f5-a839-4b9f-aff4-e8788fcd3434
mon_initial_members = cl02-cephnode01
mon_host = 192.168.1.111
auth_supported = cephx
osd_journal_size = 1024
filestore_xattr_use_omap = true


However If I repeat the same steps without specifying the cluster name, 
my 1st node comes up without problem :

 [cl02-cephnode01][INFO  ] monitor: mon.cl02-cephnode01 is running

Any clue on what I'm missing?


Regards,
Laurent Durnez


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] use mkcephfs to init a specific service on a specific node

[liu jiwei]

Hi guys:
 i can init all service on all node by mkcephfs and ssh or init service
on local manually. could i use mkcephfs init a specifice service on a
specific node by mkcephfs and ssh?

 how do you deploy ceph in you environment? ceph-deploy or manually or
other tools.could you tell me ?


 thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Miguel Afonso Oliveira]

Hi,

I have a recent ceph deployment with version:

ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)

on 4 12TB OSDs:

GLOBAL:
SIZE   AVAIL RAW USED %RAW USED
49143G 8285G 40858G   83.14

POOLS:
NAME ID USED   %USED OBJECTS
data 0  20396G 41.50 7342052
metadata 1  276M   0 81826
rbd  2  0  0 0

and this morning I started to get a warning about a full OSD:

  cluster 14320bfb-8b8c-4280-afee-df63172b1d0c
   health HEALTH_WARN 1 near full osd(s)
   monmap e3: 3 mons at 
{gridio1=10.112.0.148:6789/0,gridio2=10.112.0.149:6789/0,gridio3=10.112.0.150:6789/0}, 
election epoch 44, quorum 0,1,2 gridio1,gridio2,gridio3

   osdmap e498: 4 osds: 4 up, 4 in
pgmap v485463: 6144 pgs: 6142 active+clean, 2 
active+clean+scrubbing+deep; 20396 GB data, 40858 GB used, 8285 GB / 
49143 GB avail; 2252B/s wr, 0op/s

   mdsmap e54: 1/1/1 up {0=gridio4=up:active}

However when I use a du on the mount point I get:

[root@ce01 /]# du -bsh grid/
31Ggrid/

This number is reasonable for the utilization of the FS... Even entering 
into account with replication (2 in this case) there is no way I can 
understand an underlying raw utilization

of over 40TB

What is going on here? What can cause such discrepancies? I know ceph 
uses delayed deletion but the discrepancy is too big...


Cheers,

MAO




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PG repair failing when object missing

[Harry Harrington]

Thanks Greg

> Date: Thu, 24 Oct 2013 13:30:02 -0700
> From: g...@inktank.com
> To: watering...@gmail.com; git-ha...@live.co.uk
> CC: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] PG repair failing when object missing
>
> I also created a ticket to try and handle this particular instance of bad 
> behavior:
> http://tracker.ceph.com/issues/6629
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
> On October 24, 2013 at 1:22:54 PM, Greg Farnum (gregory.far...@inktank.com) 
> wrote:
>>
>>I was also able to reproduce this, guys, but I believe it’s specific to the 
>>mode of testing rather than to anything being wrong with the OSD. In 
>>particular, after restarting the OSD whose file I removed and running repair, 
>>it did so successfully.
>>The OSD has an “fd cacher” which caches open file handles, and we believe 
>>this is what causes the observed behavior: if the removed object is among the 
>>most recent objects touched, the FileStore (an OSD subsystem) has an open fd 
>>cached, so when manually deleting the file the FileStore now has a deleted 
>>file open. When the repair happens, it finds that open file descriptor and 
>>applies the repair to it — which of course doesn’t help put it back into 
>>place!
>>-Greg
>>Software Engineer #42 @ http://inktank.com | http://ceph.com
>>
>>On October 24, 2013 at 2:52:54 AM, Matt Thompson (watering...@gmail.com) 
>>wrote:
>>>
>>>Hi Harry,
>>>
>>>I was able to replicate this.
>>>
>>>What does appear to work (for me) is to do an osd scrub followed by a pg
>>>repair. I've tried this 2x now and in each case the deleted file gets
>>>copied over to the OSD from where it was removed. However, I've tried a
>>>few pg scrub / pg repairs after manually deleting a file and have yet to
>>>see the file get copied back to the OSD on which it was deleted. Like you
>>>said, the pg repair sets the health of the PG back to active+clean, but
>>>then re-running the pg scrub detects the file as missing again and sets it
>>>back to active+clean+inconsistent.
>>>
>>>Regards,
>>>Matt
>>>
>>>
>>>On Wed, Oct 23, 2013 at 3:45 PM, Harry Harrington wrote:
>>>
 Hi,

 I've been taking a look at the repair functionality in ceph. As I
 understand it the osds should try to copy an object from another member of
 the pg if it is missing. I have been attempting to test this by manually
 removing a file from one of the osds however each time the repair
 completes the the file has not been restored. If I run another scrub on the
 pg it gets flagged as inconsistent. See below for the output from my
 testing. I assume I'm missing something obvious, any insight into this
 process would be greatly appreciated.

 Thanks,
 Harry

 # ceph --version
 ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
 # ceph status
 cluster a4e417fe-0386-46a5-4475-ca7e10294273
 health HEALTH_OK
 monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
 0 ceph1
 osdmap e13: 3 osds: 3 up, 3 in
 pgmap v232: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used,
 164 GB / 179 GB avail
 mdsmap e1: 0/0/1 up

 file removed from osd.2

 # ceph pg scrub 0.b
 instructing pg 0.b on osd.1 to scrub

 # ceph status
 cluster a4e417fe-0386-46a5-4475-ca7e10294273
 health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
 monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
 0 ceph1
 osdmap e13: 3 osds: 3 up, 3 in
 pgmap v233: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44
 bytes data, 15465 MB used, 164 GB / 179 GB avail
 mdsmap e1: 0/0/1 up

 # ceph pg repair 0.b
 instructing pg 0.b on osd.1 to repair

 # ceph status
 cluster a4e417fe-0386-46a5-4475-ca7e10294273
 health HEALTH_OK
 monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
 0 ceph1
 osdmap e13: 3 osds: 3 up, 3 in
 pgmap v234: 192 pgs: 192 active+clean; 44 bytes data, 15465 MB used,
 164 GB / 179 GB avail
 mdsmap e1: 0/0/1 up

 # ceph pg scrub 0.b
 instructing pg 0.b on osd.1 to scrub

 # ceph status
 cluster a4e417fe-0386-46a5-4475-ca7e10294273
 health HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
 monmap e1: 1 mons at {ceph1=1.2.3.4:6789/0}, election epoch 2, quorum
 0 ceph1
 osdmap e13: 3 osds: 3 up, 3 in
 pgmap v236: 192 pgs: 191 active+clean, 1 active+clean+inconsistent; 44
 bytes data, 15465 MB used, 164 GB / 179 GB avail
 mdsmap e1: 0/0/1 up



 The logs from osd.1:
 2013-10-23 14:12:31.188281 7f02a5161700 0 log [ERR] : 0.b osd.2 missing
 3a643fcb/testfile1/head//0
 2013-10-23 14:12:31.188312 7f02a5161700 0 log [ERR] : 0.b scrub 1
 missing, 0 inconsistent objects
 2013-10-23 14:12:31.188319 7f02a5161700 0 log [ERR] : 0.b scrub 1 errors
 2013-10-

Re: [ceph-users] About use same SSD for OS and Journal

[Mike Dawson]

Kurt,

When you had OS and osd journals co-located, how many osd journals were 
on the SSD containing the OS?


You mention you now use a 5:1 ratio. Was the ratio something like 11:1 
before (one SSD for OS plus 11 osd journals to 11 OSDs in a 12-disk 
chassis)?


Also, what throughput per drive were you seeing on the cluster during 
the periods where things got laggy due to backfills, etc?


Last, did you attempt to throttle using ceph config setting in the old 
setup? Do you need to throttle in your current setup?


Thanks,
Mike Dawson


On 10/24/2013 10:40 AM, Kurt Bauer wrote:

Hi,

we had a setup like this and ran into trouble, so I would strongly
discourage you from setting it up like this. Under normal circumstances
there's no problem, but when the cluster is under heavy load, for
example when it has a lot of pgs backfilling, for whatever reason
(increasing num of pgs, adding OSDs,..), there's obviously a lot of
entries written to the journals.
What we saw then was extremly laggy behavior of the cluster and when
looking at the iostats of the SSD, they were at 100% most of the time. I
don't exactly know what causes this and why the SSDs can't cope with the
amount of IOs, but seperating OS and journals did the trick. We now have
quick 15k HDDs in Raid1 for OS and Monitor journal and per 5 OSD
journals one SSD with one partition per journal (used as raw partition).

Hope that helps,
best regards,
Kurt

Martin Catudal schrieb:

Hi,
  Here my scenario :
I will have a small cluster (4 nodes) with 4 (4 TB) OSD's per node.

I will have OS installed on two SSD in raid 1 configuration.

Is one of you have successfully and efficiently a Ceph cluster that is
built with Journal on a separate partition on the OS SSD's?

I know that it may occur a lot of IO on the Journal SSD and I'm scared
of have my OS suffer from too much IO.

Any background experience?

Martin



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] use mkcephfs to init a specific service on a specific node

[Alfredo Deza]

On Fri, Oct 25, 2013 at 5:35 AM, liu jiwei  wrote:
> Hi guys:
>  i can init all service on all node by mkcephfs and ssh or init service
> on local manually. could i use mkcephfs init a specifice service on a
> specific node by mkcephfs and ssh?
>
>  how do you deploy ceph in you environment? ceph-deploy or manually or
> other tools.could you tell me ?

I would highly recommend ceph-deploy. This is what we use also
internally to deploy different ceph components (including
installing ceph).

I also would steer away from mkcephfs as it is no longer maintained.

Have you tried out ceph-deploy recently?
>
>
>  thanks
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph-deploy, sudo and proxies

[alistair.whittle]

I have an interesting problem I was hoping someone could help with.

My Red Hat servers are configured to use proxies to access the internet.   I 
have managed to successfully add the Ceph repo install ceph-deploy on the admin 
node and create the cluster.   All ceph nodes are no password sudo tested and I 
have made sure that the proxy settings are kept when trying an 'rpm' command 
using sudo.  All other preflight checks are completed with ceph being the 
default login user etc.

So, when I run the ceph-deploy install  command from the admin node, 
I get the following error:

ceph@ldtdsr02se17 PROD $ ceph-deploy install ldtdsr02se18
[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy install 
ldtdsr02se18
[ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster 
ceph hosts ldtdsr02se18
[ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
[ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
[ldtdsr02se18][ERROR ] Traceback (most recent call last):
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
23, in install
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in 
inner
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
remote_call
[ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 502, 
in check_call
[ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
[ldtdsr02se18][ERROR ] CalledProcessError: Command '['su -c \'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"\'']' 
returned non-zero exit status 1
[ldtdsr02se18][ERROR ] curl: (7) couldn't connect to host
[ldtdsr02se18][ERROR ] error: 
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc: import read 
failed(2).
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: su -c 'rpm 
--import "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'

Note that it uses sudo as it should and then complains about not being able to 
connect.   When I run the exact same command on the ceph node itself as the 
"ceph" user, it works without any errors.   This implies that the 
authentication is in place between ceph and root, and the proxy settings are 
correct.   Yet, it fails to work when initiated from the admin node via 
ceph-deploy.

Any ideas what might be going on here?   I should add that I looked at the 
github page about using the "-no-adjust-repos" flag but my version of 
ceph-deploy says it is an invalid flag...

Please help
Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy with different cluster name

[Alfredo Deza]

On Fri, Oct 25, 2013 at 5:10 AM, Laurent Durnez  wrote:
> Hi everyone,
>
> I'm trying to set up a second cluster specifying its name but I can't bring
> up the first mon nodes. The second cluster is on different hardware that my
> first one. I'm following the quick start :
> http://ceph.com/docs/master/start/quick-ceph-deploy
>
> I've reproduced it with 1 node :
> 1) ceph-deploy --cluster cluster02 new cl02-cephnode01
> 2) ceph-deploy --cluster cluster02 install cl02-cephnode01
> 3) ceph-deploy --cluster cluster02 mon create cl02-cephnode01
>
> And I've got the following log :
>
> [ceph_deploy.mon][INFO  ] distro info: Debian 7.2 wheezy
> [cl02-cephnode01][DEBUG ] determining if provided host has same hostname in
> remote
> [cl02-cephnode01][DEBUG ] deploying mon to cl02-cephnode01
> [cl02-cephnode01][DEBUG ] remote hostname: cl02-cephnode01
> [cl02-cephnode01][INFO  ] write cluster configuration to
> /etc/ceph/{cluster}.conf
> [cl02-cephnode01][INFO  ] creating path:
> /var/lib/ceph/mon/cluster02-cl02-cephnode01
> [cl02-cephnode01][DEBUG ] checking for done path:
> /var/lib/ceph/mon/cluster02-cl02-cephnode01/done
> [cl02-cephnode01][DEBUG ] done path does not exist:
> /var/lib/ceph/mon/cluster02-cl02-cephnode01/done
> [cl02-cephnode01][INFO  ] creating tmp path: /var/lib/ceph/tmp
> [cl02-cephnode01][INFO  ] creating keyring file:
> /var/lib/ceph/tmp/cluster02-cl02-cephnode01.mon.keyring
> [cl02-cephnode01][INFO  ] create the monitor keyring file
> [cl02-cephnode01][INFO  ] Running command: ceph-mon --cluster cluster02
> --mkfs -i cl02-cephnode01 --keyring
> /var/lib/ceph/tmp/cluster02-cl02-cephnode01.mon.keyring
> [cl02-cephnode01][INFO  ] ceph-mon: mon.noname-a 192.168.1.111:6789/0 is
> local, renaming to mon.cl02-cephnode01
> [cl02-cephnode01][INFO  ] ceph-mon: set fsid to
> 6636a4f5-a839-4b9f-aff4-e8788fcd3434
> [cl02-cephnode01][INFO  ] ceph-mon: created monfs at
> /var/lib/ceph/mon/cluster02-cl02-cephnode01 for mon.cl02-cephnode01
> [cl02-cephnode01][INFO  ] unlinking keyring file
> /var/lib/ceph/tmp/cluster02-cl02-cephnode01.mon.keyring
> [cl02-cephnode01][INFO  ] create a done file to avoid re-doing the mon
> deployment
> [cl02-cephnode01][INFO  ] create the init path if it does not exist
> [cl02-cephnode01][INFO  ] locating `service` executable...
> [cl02-cephnode01][INFO  ] found `service` executable: /usr/sbin/service
> [cl02-cephnode01][INFO  ] Running command: sudo /usr/sbin/service ceph -c
> /etc/ceph/cluster02.conf start mon.cl02-cephnode01
> [cl02-cephnode01][INFO  ] Running command: sudo ceph --admin-daemon
> /var/run/ceph/ceph-mon.cl02-cephnode01.asok mon_status
> [cl02-cephnode01][ERROR ] admin_socket: exception getting command
> descriptions: [Errno 2] No such file or directory
> [cl02-cephnode01][WARNIN] monitor: mon.cl02-cephnode01, might not be running
> yet
> [cl02-cephnode01][INFO  ] Running command: sudo ceph --admin-daemon
> /var/run/ceph/ceph-mon.cl02-cephnode01.asok mon_status
> [cl02-cephnode01][ERROR ] admin_socket: exception getting command
> descriptions: [Errno 2] No such file or directory
> [cl02-cephnode01][WARNIN] monitor cl02-cephnode01 does not exist in monmap
> [cl02-cephnode01][WARNIN] neither `public_addr` nor `public_network` keys
> are defined for monitors
> [cl02-cephnode01][WARNIN] monitors may not be able to form quorum
>
> On the mon node, I've got this line in the log :
> 2013-10-25 10:22:26.427836 7f0d6d957780  0 ceph version 0.67.4
> (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7), process ceph-mon, pid 2505
>
> my cluster02.conf is the one that has been created by default :
> [global]
> fsid = 6636a4f5-a839-4b9f-aff4-e8788fcd3434
> mon_initial_members = cl02-cephnode01
> mon_host = 192.168.1.111
> auth_supported = cephx
> osd_journal_size = 1024
> filestore_xattr_use_omap = true
>
>
> However If I repeat the same steps without specifying the cluster name, my
> 1st node comes up without problem :
>  [cl02-cephnode01][INFO  ] monitor: mon.cl02-cephnode01 is running
>
> Any clue on what I'm missing?

This is a problem with the recent addition of error checking and an
issue was opened for it (but not yet completed)
http://tracker.ceph.com/issues/6587

You should ignore (for now) those warnings and rely on specific checks
for the mon in that host.

This should be fixed soon though, stay tuned for that :)
>
>
> Regards,
> Laurent Durnez
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-deploy, sudo and proxies

[Alfredo Deza]

On Fri, Oct 25, 2013 at 1:25 PM,   wrote:
> I have an interesting problem I was hoping someone could help with.
>
>
>
> My Red Hat servers are configured to use proxies to access the internet.   I
> have managed to successfully add the Ceph repo install ceph-deploy on the
> admin node and create the cluster.   All ceph nodes are no password sudo
> tested and I have made sure that the proxy settings are kept when trying an
> ‘rpm’ command using sudo.  All other preflight checks are completed with
> ceph being the default login user etc.
>
>
>
> So, when I run the ceph-deploy install  command from the admin
> node, I get the following error:
>
>
>
> ceph@ldtdsr02se17 PROD $ ceph-deploy install ldtdsr02se18
>
> [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy install
> ldtdsr02se18
>
> [ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster
> ceph hosts ldtdsr02se18
>
> [ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
>
> [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
>
> [ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4
> Santiago
>
> [ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
>
> [ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import
> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
>
> [ldtdsr02se18][ERROR ] Traceback (most recent call last):
>
> [ldtdsr02se18][ERROR ]   File
> "/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line
> 23, in install
>
> [ldtdsr02se18][ERROR ]   File
> "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10,
> in inner
>
> [ldtdsr02se18][ERROR ]   File
> "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in
> remote_call
>
> [ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line
> 502, in check_call
>
> [ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
>
> [ldtdsr02se18][ERROR ] CalledProcessError: Command '['su -c \'rpm --import
> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"\'']'
> returned non-zero exit status 1
>
> [ldtdsr02se18][ERROR ] curl: (7) couldn't connect to host
>
> [ldtdsr02se18][ERROR ] error:
> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc: import
> read failed(2).
>
> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: su -c 'rpm
> --import "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
>
>
>
> Note that it uses sudo as it should and then complains about not being able
> to connect.   When I run the exact same command on the ceph node itself as
> the “ceph” user, it works without any errors.   This implies that the
> authentication is in place between ceph and root, and the proxy settings are
> correct.   Yet, it fails to work when initiated from the admin node via
> ceph-deploy.
>
>
>
> Any ideas what might be going on here?   I should add that I looked at the
> github page about using the “–no-adjust-repos” flag but my version of
> ceph-deploy says it is an invalid flag...

You should make sure you are running the latest version for
ceph-deploy (1.2.7 at the moment).

But this is a known issue, and it basically happens with proxies
because environment variables get reset on sudo (common on a lot of
distros for security concerns).

What is set for a user when it is logged in via SSH will not have a
1:1 behavior when done in a non-interactive shell which is what
ceph-deploy does.

My first recommendation is to use the latest ceph-deploy and the
`--no-adjust-repos` flag. We do have a couple of open tickets to
address the need for a
proxy flag so that ceph-deploy in turn can use those values whenever a
wget or curl is about to be used.


>
>
>
> Please help
>
> Alistair
>
>
>
> ___
>
> This message is for information purposes only, it is not a recommendation,
> advice, offer or solicitation to buy or sell a product or service nor an
> official confirmation of any transaction. It is directed at persons who are
> professionals and is not intended for retail customer use. Intended for
> recipient only. This message is subject to the terms at:
> www.barclays.com/emaildisclaimer.
>
> For important disclosures, please see:
> www.barclays.com/salesandtradingdisclaimer regarding market commentary from
> Barclays Sales and/or Trading, who are active market participants; and in
> respect of Barclays Research, including disclosures relating to specific
> issuers, please see http://publicresearch.barclays.com.
>
> ___
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Gregory Farnum]

Are you sure you're using only CephFS? Do you have any snapshots?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Oct 25, 2013 at 2:59 AM, Miguel Afonso Oliveira
 wrote:
> Hi,
>
> I have a recent ceph deployment with version:
>
> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
>
> on 4 12TB OSDs:
>
> GLOBAL:
> SIZE   AVAIL RAW USED %RAW USED
> 49143G 8285G 40858G   83.14
>
> POOLS:
> NAME ID USED   %USED OBJECTS
> data 0  20396G 41.50 7342052
> metadata 1  276M   0 81826
> rbd  2  0  0 0
>
> and this morning I started to get a warning about a full OSD:
>
>   cluster 14320bfb-8b8c-4280-afee-df63172b1d0c
>health HEALTH_WARN 1 near full osd(s)
>monmap e3: 3 mons at
> {gridio1=10.112.0.148:6789/0,gridio2=10.112.0.149:6789/0,gridio3=10.112.0.150:6789/0},
> election epoch 44, quorum 0,1,2 gridio1,gridio2,gridio3
>osdmap e498: 4 osds: 4 up, 4 in
> pgmap v485463: 6144 pgs: 6142 active+clean, 2
> active+clean+scrubbing+deep; 20396 GB data, 40858 GB used, 8285 GB / 49143
> GB avail; 2252B/s wr, 0op/s
>mdsmap e54: 1/1/1 up {0=gridio4=up:active}
>
> However when I use a du on the mount point I get:
>
> [root@ce01 /]# du -bsh grid/
> 31Ggrid/
>
> This number is reasonable for the utilization of the FS... Even entering
> into account with replication (2 in this case) there is no way I can
> understand an underlying raw utilization
> of over 40TB
>
> What is going on here? What can cause such discrepancies? I know ceph uses
> delayed deletion but the discrepancy is too big...
>
> Cheers,
>
> MAO
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-deploy, sudo and proxies

[Gruher, Joseph R]

Try configuring the curl proxy in /root/.curlrc.  I had a similar problem 
earlier this week.

Overall I have to be sure to set all these proxies individually for ceph-deploy 
to work on CentOS (Ubuntu is easier):
Curl: /root/.curlrc
rpm: /root/.rpmmacros
wget: /etc/wgetrc
yum: /etc/yum.conf

-Joe

From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
alistair.whit...@barclays.com
Sent: Friday, October 25, 2013 10:26 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Ceph-deploy, sudo and proxies

I have an interesting problem I was hoping someone could help with.

My Red Hat servers are configured to use proxies to access the internet.   I 
have managed to successfully add the Ceph repo install ceph-deploy on the admin 
node and create the cluster.   All ceph nodes are no password sudo tested and I 
have made sure that the proxy settings are kept when trying an 'rpm' command 
using sudo.  All other preflight checks are completed with ceph being the 
default login user etc.

So, when I run the ceph-deploy install  command from the admin node, 
I get the following error:

ceph@ldtdsr02se17 PROD $ ceph-deploy install ldtdsr02se18
[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy install 
ldtdsr02se18
[ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster 
ceph hosts ldtdsr02se18
[ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
[ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
[ldtdsr02se18][ERROR ] Traceback (most recent call last):
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
23, in install
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in 
inner
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
remote_call
[ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 502, 
in check_call
[ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
[ldtdsr02se18][ERROR ] CalledProcessError: Command '['su -c \'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"\'']' 
returned non-zero exit status 1
[ldtdsr02se18][ERROR ] curl: (7) couldn't connect to host
[ldtdsr02se18][ERROR ] error: 
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc: import read 
failed(2).
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: su -c 'rpm 
--import "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'

Note that it uses sudo as it should and then complains about not being able to 
connect.   When I run the exact same command on the ceph node itself as the 
"ceph" user, it works without any errors.   This implies that the 
authentication is in place between ceph and root, and the proxy settings are 
correct.   Yet, it fails to work when initiated from the admin node via 
ceph-deploy.

Any ideas what might be going on here?   I should add that I looked at the 
github page about using the "-no-adjust-repos" flag but my version of 
ceph-deploy says it is an invalid flag...

Please help
Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; and in respect of Barclays Research, including disclosures 
relating to specific issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Yan, Zheng]

On Sat, Oct 26, 2013 at 2:05 AM, Gregory Farnum  wrote:
> Are you sure you're using only CephFS? Do you have any snapshots?
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>
> On Fri, Oct 25, 2013 at 2:59 AM, Miguel Afonso Oliveira
>  wrote:
>> Hi,
>>
>> I have a recent ceph deployment with version:
>>
>> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
>>
>> on 4 12TB OSDs:
>>
>> GLOBAL:
>> SIZE   AVAIL RAW USED %RAW USED
>> 49143G 8285G 40858G   83.14
>>
>> POOLS:
>> NAME ID USED   %USED OBJECTS
>> data 0  20396G 41.50 7342052
>> metadata 1  276M   0 81826
>> rbd  2  0  0 0
>>
>> and this morning I started to get a warning about a full OSD:
>>
>>   cluster 14320bfb-8b8c-4280-afee-df63172b1d0c
>>health HEALTH_WARN 1 near full osd(s)
>>monmap e3: 3 mons at
>> {gridio1=10.112.0.148:6789/0,gridio2=10.112.0.149:6789/0,gridio3=10.112.0.150:6789/0},
>> election epoch 44, quorum 0,1,2 gridio1,gridio2,gridio3
>>osdmap e498: 4 osds: 4 up, 4 in
>> pgmap v485463: 6144 pgs: 6142 active+clean, 2
>> active+clean+scrubbing+deep; 20396 GB data, 40858 GB used, 8285 GB / 49143
>> GB avail; 2252B/s wr, 0op/s
>>mdsmap e54: 1/1/1 up {0=gridio4=up:active}
>>
>> However when I use a du on the mount point I get:
>>
>> [root@ce01 /]# du -bsh grid/
>> 31Ggrid/

what is the output of 'getfattr -d -m -  grid/' ?

>>
>> This number is reasonable for the utilization of the FS... Even entering
>> into account with replication (2 in this case) there is no way I can
>> understand an underlying raw utilization
>> of over 40TB
>>
>> What is going on here? What can cause such discrepancies? I know ceph uses
>> delayed deletion but the discrepancy is too big...
>>
>> Cheers,
>>
>> MAO
>>
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.co
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization

[Yan, Zheng]

On Sat, Oct 26, 2013 at 2:05 AM, Gregory Farnum  wrote:
> Are you sure you're using only CephFS? Do you have any snapshots?
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>
> On Fri, Oct 25, 2013 at 2:59 AM, Miguel Afonso Oliveira
>  wrote:
>> Hi,
>>
>> I have a recent ceph deployment with version:
>>
>> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7)
>>
>> on 4 12TB OSDs:
>>
>> GLOBAL:
>> SIZE   AVAIL RAW USED %RAW USED
>> 49143G 8285G 40858G   83.14
>>
>> POOLS:
>> NAME ID USED   %USED OBJECTS
>> data 0  20396G 41.50 7342052
>> metadata 1  276M   0 81826
>> rbd  2  0  0 0
>>
>> and this morning I started to get a warning about a full OSD:
>>
>>   cluster 14320bfb-8b8c-4280-afee-df63172b1d0c
>>health HEALTH_WARN 1 near full osd(s)
>>monmap e3: 3 mons at
>> {gridio1=10.112.0.148:6789/0,gridio2=10.112.0.149:6789/0,gridio3=10.112.0.150:6789/0},
>> election epoch 44, quorum 0,1,2 gridio1,gridio2,gridio3
>>osdmap e498: 4 osds: 4 up, 4 in
>> pgmap v485463: 6144 pgs: 6142 active+clean, 2
>> active+clean+scrubbing+deep; 20396 GB data, 40858 GB used, 8285 GB / 49143
>> GB avail; 2252B/s wr, 0op/s
>>mdsmap e54: 1/1/1 up {0=gridio4=up:active}
>>
>> However when I use a du on the mount point I get:
>>
>> [root@ce01 /]# du -bsh grid/
>> 31Ggrid/

sounds like the 'purge strays' bug. try umounting all clients and
restarting the mds.

>>
>> This number is reasonable for the utilization of the FS... Even entering
>> into account with replication (2 in this case) there is no way I can
>> understand an underlying raw utilization
>> of over 40TB
>>
>> What is going on here? What can cause such discrepancies? I know ceph uses
>> delayed deletion but the discrepancy is too big...
>>
>> Cheers,
>>
>> MAO
>>
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cuttlefish: pool recreation results in cluster crash

[Sage Weil]

Hi Andrey,

Just wanted to ping you on this one.  There is an open bug 
(http://tracker.ceph.com/issues/6598) but our initial attempst to 
reproduce have failed.  Can you provide more info about how you hit it 
(and ideally some logs)?

Thanks!
sage


On Mon, 21 Oct 2013, Joao Eduardo Luis wrote:

> On 10/19/2013 08:53 PM, Andrey Korolyov wrote:
> > Hello,
> > 
> > I was able to reproduce following on the top of current cuttlefish:
> > 
> > - create pool,
> > - delete it after all pgs initialized,
> > - create new pool with same name after, say, ten seconds.
> > 
> > All osds dies immediately with attached trace. The problem exists in
> > bobtail as well.
> 
> Can we have the resulting backtrace and context log?
> 
>   -Joao
> 
> 
> -- 
> Joao Eduardo Luis
> Software Engineer | http://inktank.com | http://ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Anybody going to SC13?

[asomers]

Are any Ceph developers planning to come to the SC13 supercomputing
convention in Denver CO this November?

-Alan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Anybody going to SC13?

[Patrick McGarry]

Not sure about devs going, but Ross and I will both be there.


Best Regards,

Patrick McGarry
Director, Community || Inktank
http://ceph.com  ||  http://inktank.com
@scuttlemonkey || @ceph || @inktank


On Fri, Oct 25, 2013 at 10:00 PM,   wrote:
> Are any Ceph developers planning to come to the SC13 supercomputing
> convention in Denver CO this November?
>
> -Alan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] New to Ceph.... Install Guide

[Raghavendra Lad]

Hi Need help--- Begin Message ---


Hi Cephs,

I am new to Ceph. I am planning to install CEPH.

I already have Openstack Grizzly installed and for storage thought of 
installing Ceph.


Doubts I have:

1. Does it need minimum 3 nodes and server to install Ceph?

2. Can we install Ceph and integrate with Openstack Grizzly?

3. Any simple guide or tutorial that can help install Ceph?


Please help.

Regards,
Raghavendra Lad--- End Message ---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Anybody going to SC13?

[Ian Colle]

Mark Nelson and I will be there too.

Ian R. Colle
Director of Engineering
Inktank
Cell: +1.303.601.7713 
Email: i...@inktank.com


Delivering the Future of Storage


 

 




On 10/25/13 8:07 PM, "Patrick McGarry"  wrote:

>Not sure about devs going, but Ross and I will both be there.
>
>
>Best Regards,
>
>Patrick McGarry
>Director, Community || Inktank
>http://ceph.com  ||  http://inktank.com
>@scuttlemonkey || @ceph || @inktank
>
>
>On Fri, Oct 25, 2013 at 10:00 PM,   wrote:
>> Are any Ceph developers planning to come to the SC13 supercomputing
>> convention in Denver CO this November?
>>
>> -Alan
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] New to Ceph.... Install Guide

[Raghavendra Lad]

Hi Cephs,

I am new to Ceph. I am planning to install CEPH.

I already have Openstack Grizzly installed and for storage thought of 
installing Ceph.

Doubts I have:

1. Does it need minimum 3 nodes and server to install Ceph?

2. Can we install Ceph and integrate with Openstack Grizzly?

3. Any simple guide or tutorial that can help install Ceph?


Please help.

Regards,

Raghavendra Lad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com