Docs now building again

[Dan Mick]

https://github.com/ceph/ceph/pull/7119 fixed an issue preventing docs
from building.  Master is fixed; merge that into your branches if you
want working docs again.

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tool for applying 'ceph daemon ' command to all OSDs

[Dan Mick]

On 12/22/2015 12:21 AM, igor.podo...@ts.fujitsu.com wrote:
>> -Original Message-
>> From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-
>> ow...@vger.kernel.org] On Behalf Of Dan Mick
>> Sent: Tuesday, December 22, 2015 7:00 AM
>> To: ceph-devel
>> Subject: RFC: tool for applying 'ceph daemon ' command to all OSDs
>>
>> I needed something to fetch current config values from all OSDs (sorta the
>> opposite of 'injectargs --key value), so I hacked it, and then spiffed it up 
>> a bit.
>> Does this seem like something that would be useful in this form in the
>> upstream Ceph, or does anyone have any thoughts on its design or
>> structure?
>>
> 
> You could do it using socat too:

Quite true.  That opens the permissions even more, but it works.
Maintaining a couple hundred x2 processes might be a little bit of a pain..
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RFC: tool for applying 'ceph daemon ' command to all OSDs

[Dan Mick]

On 12/21/2015 11:29 PM, Gregory Farnum wrote:
> On Mon, Dec 21, 2015 at 9:59 PM, Dan Mick  wrote:
>> I needed something to fetch current config values from all OSDs (sorta
>> the opposite of 'injectargs --key value), so I hacked it, and then
>> spiffed it up a bit.  Does this seem like something that would be useful
>> in this form in the upstream Ceph, or does anyone have any thoughts on
>> its design or structure?
>>
>> It requires a locally-installed ceph CLI and a ceph.conf that points to
>> the cluster and any required keyrings.  You can also provide it with
>> a YAML file mapping host to osds if you want to save time collecting
>> that info for a statically-defined cluster, or if you want just a subset
>> of OSDs.
>>
>> https://github.com/dmick/tools/blob/master/osd_daemon_cmd.py
>>
>> Excerpt from usage:
>>
>> Execute a Ceph osd daemon command on every OSD in a cluster with
>> one connection to each OSD host.
>>
>> Usage:
>> osd_daemon_cmd [-c CONF] [-u USER] [-f FILE] (COMMAND | -k KEY)
>>
>> Options:
>>-c CONF   ceph.conf file to use [default: ./ceph.conf]
>>-u USER   user to connect with ssh
>>-f FILE   get names and osds from yaml
>>COMMAND   command other than "config get" to execute
>>-k KEYconfig key to retrieve with config get 
> 
> I naively like the functionality being available, but if I'm skimming
> this correctly it looks like you're relying on the local node being
> able to passwordless-ssh to all of the nodes, and for that account to
> be able to access the ceph admin sockets. Granted we rely on the ssh
> for ceph-deploy as well, so maybe that's okay, but I'm not sure in
> this case since it implies a lot more network openness.

Yep; it's basically the same model and role assumed as "cluster destroyer".

> Relatedly (perhaps in an opposing direction), maybe we want anything
> exposed over the network to have some sort of explicit permissions
> model?

Well, I've heard that idea floated about the admin socket for years, but
I don't think anyone's hot to add cephx to it :)

> Maybe not and we should just ship the script for trusted users. I
> would have liked it on the long-running cluster I'm sure you built it
> for. ;)

it's like you're clairvoyant.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RFC: tool for applying 'ceph daemon ' command to all OSDs

[Dan Mick]

I needed something to fetch current config values from all OSDs (sorta
the opposite of 'injectargs --key value), so I hacked it, and then
spiffed it up a bit.  Does this seem like something that would be useful
in this form in the upstream Ceph, or does anyone have any thoughts on
its design or structure?

It requires a locally-installed ceph CLI and a ceph.conf that points to
the cluster and any required keyrings.  You can also provide it with
a YAML file mapping host to osds if you want to save time collecting
that info for a statically-defined cluster, or if you want just a subset
of OSDs.

https://github.com/dmick/tools/blob/master/osd_daemon_cmd.py

Excerpt from usage:

Execute a Ceph osd daemon command on every OSD in a cluster with
one connection to each OSD host.

Usage:
osd_daemon_cmd [-c CONF] [-u USER] [-f FILE] (COMMAND | -k KEY)

Options:
   -c CONF   ceph.conf file to use [default: ./ceph.conf]
   -u USER   user to connect with ssh
   -f FILE   get names and osds from yaml
   COMMAND   command other than "config get" to execute
   -k KEYconfig key to retrieve with config get 

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Troublesome documentation link on http://ceph.com

[Dan Mick]

On 12/02/2015 09:26 PM, Dan Mick wrote:
> On 12/01/2015 04:44 AM, Nathan Cutler wrote:
>> So I go to http://ceph.com and click on the "Documentation" link. In the
>> HTML source code the linked URL is http://ceph.com/docs but the HTTP
>> server rewrites this to:
>>
>> http://docs.ceph.com/docs/v0.80.5/
>>
>> Could someone arrange for this to be changed s/v0.80.5/master/ ?
>>
>> ( Searching for Ceph documentation is already difficult as it is,
>> because Google does not provide links to the master version, but only to
>> old outdated versions - see for example the first two search results in
>> a Google search for "troubleshoot pgs in ceph":
>> http://tinyurl.com/nfuohss )
>>
> 
> Yes, agreed; we were in the process of relocating that webserver, so had
> been neglecting the old configuration.  Hopefully we'll get it redone
> tomorrow (one in a list of a zillion things around the move, but an
> important one).
> 

I've updated DNS to point at the new server, and redirected both
docs.ceph.com/ and docs.ceph.com/docs to docs.ceph.com/docs/master.
I've (at least temporarily) removed the 'version selector' javascript
from the page as well; you can still select other versions by editing
the URL.  Barring any DNS caching past the records' TTLs on your end,
the new addresses should be active by now.

Because the site had been using 301 redirects, you may need to do
something special to make your browser realize that the redirect should
no longer be used (the browser caches redirects).  Google "clear cached
redirects" for your particular browser if you appear not to be getting
the correct info.

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Troublesome documentation link on http://ceph.com

[Dan Mick]

On 12/01/2015 04:44 AM, Nathan Cutler wrote:
> So I go to http://ceph.com and click on the "Documentation" link. In the
> HTML source code the linked URL is http://ceph.com/docs but the HTTP
> server rewrites this to:
> 
> http://docs.ceph.com/docs/v0.80.5/
> 
> Could someone arrange for this to be changed s/v0.80.5/master/ ?
> 
> ( Searching for Ceph documentation is already difficult as it is,
> because Google does not provide links to the master version, but only to
> old outdated versions - see for example the first two search results in
> a Google search for "troubleshoot pgs in ceph":
> http://tinyurl.com/nfuohss )
> 

Yes, agreed; we were in the process of relocating that webserver, so had
been neglecting the old configuration.  Hopefully we'll get it redone
tomorrow (one in a list of a zillion things around the move, but an
important one).

-- 
Dan Mick
Red Hat, Inc.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tarball for 10.0.0

[Dan Mick]

On 11/30/2015 11:57 AM, Deneau, Tom wrote:
> I did not see the source tarball for 10.0.0 at 
> http://download.ceph.com/tarballs/ceph-10.0.0.tar.gz
> 

It's there now, FWIW.


-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tracker.ceph.com downtime today

[Dan Mick]

Found that issue; reverted the database to the non-backlog-plugin state,
created a test bug.  Retry?

On 10/22/2015 06:54 PM, Dan Mick wrote:
> I see that too.  I suspect this is because of leftover database columns
> from the backlogs plugin, which is removed.  Looking into it.
> 
> On 10/22/2015 06:43 PM, Kyle Bader wrote:
>> I tried to open a new issue and got this error:
>>
>> Internal error
>>
>> An error occurred on the page you were trying to access.
>> If you continue to experience problems please contact your Redmine
>> administrator for assistance.
>>
>> If you are the Redmine administrator, check your log files for details
>> about the error.
>>
>>
>> On Thu, Oct 22, 2015 at 6:15 PM, Dan Mick  wrote:
>>> Fixed a configuration problem preventing updating issues, and switched
>>> the mailer to use ipv4; if you updated and failed, or missed an email
>>> notification, that may have been why.
>>>
>>> On 10/22/2015 04:51 PM, Dan Mick wrote:
>>>> It's back.  New DNS info is propagating its way around.  If you
>>>> absolutely must get to it, newtracker.ceph.com is the new address, but
>>>> please don't bookmark that, as it will be going away after the transition.
>>>>
>>>> Please let me know of any problems you have.
>>>
>>> ---
>>> Note: This list is intended for discussions relating to Red Hat Storage 
>>> products, customers and/or support. Discussions on GlusterFS and Ceph 
>>> architecture, design and engineering should go to relevant upstream mailing 
>>> lists.
>>
>>
>>
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tracker.ceph.com downtime today

[Dan Mick]

I see that too.  I suspect this is because of leftover database columns
from the backlogs plugin, which is removed.  Looking into it.

On 10/22/2015 06:43 PM, Kyle Bader wrote:
> I tried to open a new issue and got this error:
> 
> Internal error
> 
> An error occurred on the page you were trying to access.
> If you continue to experience problems please contact your Redmine
> administrator for assistance.
> 
> If you are the Redmine administrator, check your log files for details
> about the error.
> 
> 
> On Thu, Oct 22, 2015 at 6:15 PM, Dan Mick  wrote:
>> Fixed a configuration problem preventing updating issues, and switched
>> the mailer to use ipv4; if you updated and failed, or missed an email
>> notification, that may have been why.
>>
>> On 10/22/2015 04:51 PM, Dan Mick wrote:
>>> It's back.  New DNS info is propagating its way around.  If you
>>> absolutely must get to it, newtracker.ceph.com is the new address, but
>>> please don't bookmark that, as it will be going away after the transition.
>>>
>>> Please let me know of any problems you have.
>>
>> ---
>> Note: This list is intended for discussions relating to Red Hat Storage 
>> products, customers and/or support. Discussions on GlusterFS and Ceph 
>> architecture, design and engineering should go to relevant upstream mailing 
>> lists.
> 
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tracker.ceph.com downtime today

[Dan Mick]

Fixed a configuration problem preventing updating issues, and switched
the mailer to use ipv4; if you updated and failed, or missed an email
notification, that may have been why.

On 10/22/2015 04:51 PM, Dan Mick wrote:
> It's back.  New DNS info is propagating its way around.  If you
> absolutely must get to it, newtracker.ceph.com is the new address, but
> please don't bookmark that, as it will be going away after the transition.
> 
> Please let me know of any problems you have.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tracker.ceph.com downtime today

[Dan Mick]

It's back.  New DNS info is propagating its way around.  If you
absolutely must get to it, newtracker.ceph.com is the new address, but
please don't bookmark that, as it will be going away after the transition.

Please let me know of any problems you have.

On 10/22/2015 04:09 PM, Dan Mick wrote:
> tracker.ceph.com down now
> 
> On 10/22/2015 03:20 PM, Dan Mick wrote:
>> tracker.ceph.com will be brought down today for upgrade and move to a
>> new host.  I plan to do this at about 4PM PST (40 minutes from now).
>> Expect a downtime of about 15-20 minutes.  More notification to follow.
>>
> 
> 


-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tracker.ceph.com downtime today

[Dan Mick]

tracker.ceph.com down now

On 10/22/2015 03:20 PM, Dan Mick wrote:
> tracker.ceph.com will be brought down today for upgrade and move to a
> new host.  I plan to do this at about 4PM PST (40 minutes from now).
> Expect a downtime of about 15-20 minutes.  More notification to follow.
> 


-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

tracker.ceph.com downtime today

[Dan Mick]

tracker.ceph.com will be brought down today for upgrade and move to a
new host.  I plan to do this at about 4PM PST (40 minutes from now).
Expect a downtime of about 15-20 minutes.  More notification to follow.

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [CEPH-DEVEL] MAX_RBD_IMAGES

[Dan Mick]

Doesn't mean anything.  It was just a medium-large number of emulated
targets.  The choice had nothing to do with the kernel.

On 09/29/2015 08:58 PM, Shinobu Kinjo wrote:
> I just want to know what that number means.
> 
> Based on the Linux kenel, that number doesn't make sense but, for the Ceph 
> cluster, would make sense from performance, resource handling including 
> networking resource point of views.
> 
> So, do you remember?
> 
> Shinobu
> 
> ----- Original Message -
> From: "Dan Mick" 
> To: "Shinobu Kinjo" , "ceph-devel" 
> 
> Sent: Wednesday, September 30, 2015 9:04:35 AM
> Subject: Re: [CEPH-DEVEL] MAX_RBD_IMAGES
> 
> On 09/22/2015 02:55 PM, Shinobu Kinjo wrote:
>> Hello,
>>
>> Does any of you know why *MAX_RBD_IMAGES* was changed from 16 to 128?
>> I hope that Dan remember -;
>>
>> http://resources.ustack.com/ceph/ceph/commit/2a6dcabf7f1b7550a0fa4fd223970ffc24ad7870
>>
>>  - Shinobu
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> tohave more of them?
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [CEPH-DEVEL] MAX_RBD_IMAGES

[Dan Mick]

On 09/22/2015 02:55 PM, Shinobu Kinjo wrote:
> Hello,
> 
> Does any of you know why *MAX_RBD_IMAGES* was changed from 16 to 128?
> I hope that Dan remember -;
> 
> http://resources.ustack.com/ceph/ceph/commit/2a6dcabf7f1b7550a0fa4fd223970ffc24ad7870
> 
>  - Shinobu
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

tohave more of them?

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Email lgx...@nxtzas.com trying to subscribe to tracker.ceph.com

[Dan Mick]

Someone using the email address

lgx...@nxtzas.com

is trying to subscribe to the Ceph Redmine tracker, but neither redmine nor I 
can use that email address; it bounces with 

: Host or domain name not found. Name service error for
name=nxtzas.com type=: Host not found

If this is you, please email me privately and we'll get you fixed up.


-- 
Dan Mick Red Hat, Inc. Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: sepia gitbuilder jessie down ?

[Dan Mick]

On 08/13/2015 06:43 AM, Alexandre DERUMIER wrote:
> Hi,
> 
> since some day, sepia gitbuilder for debian jessie seem to be down.
> 
> http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-deb-jessie-amd64-basic/
> 
> 
> I don't see new update since 6 august in repository
> 
> http://gitbuilder.ceph.com/ceph-deb-jessie-x86_64-basic/ref/
> 
> 
> (BTW, is they any roadmap to release official package for jessie ? I'm 
> waiting this to include them in next proxmox version)
> 
> 
> Regards,
> 
> Alexandre
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

I've restarted it yesterday, and it's down again; I see that it's
defined on a 64GB host that has six 8GB VMs and one 16GB one, which is
.. probably doomed to failure.

I'll stop some of the 8GB machines, which aren't currently needed.


-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 答复: hello, I am confused about a question of rbd

[Dan Mick]

Adding back ceph-devel

My point was, ok, ruling out 0, then we can create a block device of
size 1 byte.  Is that useful?  No, it is not.  How about 10 bytes?
1000?  1MB?

There's no good reason to rule out zero.  Is it causing a problem somehow?

On 07/23/2015 06:06 PM, zhengbin.08...@h3c.com wrote:
> If I create a block whose size is zero, it means you absolutely can not use 
> it unless you resize it
> 
> 
> what arbitrary minimum size is too small?  -> I do not know, If I create 
> a block whose size is larger than zero,it may be can use.
> And why do we have to define a minimum size? we can just make sure the block 
> size is larger than zero
> 
> -邮件原件-
> 发件人: Dan Mick [mailto:dm...@redhat.com]
> 发送时间: 2015年7月24日 5:25
> 收件人: Jason Dillaman; zhengbin 08747 (RD)
> 抄送: ceph-devel
> 主题: Re: hello, I am confused about a question of rbd
> 
> Why not zero?
> 
> If the answer is "it can't be used", then, what arbitrary minimum size is too 
> small?
> 
> (also, given that resize exists, it can be used for storage after a resize.)
> 
> On 07/23/2015 06:03 AM, Jason Dillaman wrote:
>> According to the git history, support for zero MB images for the 
>> create/resize commands was explicitly added by commit 08f47a4.  Dan Mick or 
>> Josh Durgin could probably better explain the history behind the change 
>> since it was before my time.
>>
> 
> -
> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
> 邮件！
> This e-mail and its attachments contain confidential information from H3C, 
> which is
> intended only for the person or entity whose address is listed above. Any use 
> of the
> information contained herein in any way (including, but not limited to, total 
> or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please 
> notify the sender
> by phone or email immediately and delete it!
> 

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: registering for tracker.ceph.com

[Dan Mick]

On 07/23/2015 09:44 AM, Sage Weil wrote:
> On Thu, 23 Jul 2015, Deneau, Tom wrote:
>> I wanted to register for tracker.ceph.com to enter a few issues but never
>> got the confirming email and my registration is now in some stuck state
>> (not complete but name/email in use so can't re-register).  Any suggestions?
> 
> It does that sometimes... not sure why.  I activated your tdeneau user and 
> deleted the later tmdeneau one.

Usually the confirm email is stuck in a spam filter.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: hello, I am confused about a question of rbd

[Dan Mick]

Why not zero?

If the answer is "it can't be used", then, what arbitrary minimum size
is too small?

(also, given that resize exists, it can be used for storage after a resize.)

On 07/23/2015 06:03 AM, Jason Dillaman wrote:
> According to the git history, support for zero MB images for the 
> create/resize commands was explicitly added by commit 08f47a4.  Dan Mick or 
> Josh Durgin could probably better explain the history behind the change since 
> it was before my time.
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [sepia] debian jessie gitbuilder repositories ?

[Dan Mick]

On 07/20/2015 07:19 AM, Sage Weil wrote:
> On Mon, 20 Jul 2015, Alexandre DERUMIER wrote:
>> Hi,
>>
>> debian jessie gitbuilder is ok since 2 weeks now,
>>
>> http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-deb-jessie-amd64-basic
>>
>>
>> It is possible to push packages to repositories ?
>>
>> http://gitbuilder.ceph.com/ceph-deb-jessie-x86_64-basic/
> 
> 
> The builds are failing with this:
> 
> + GNUPGHOME=/srv/gnupg reprepro --ask-passphrase -b 
> ../out/output/sha1/6ffb1c4ae43bcde9f5fde40dd97959399135ed86.tmp -C main 
> --ignore=undef
> inedtarget --ignore=wrongdistribution include jessie 
> out~/ceph_0.94.2-50-g6ffb1c4-1jessie_amd64.changes
> Cannot find definition of distribution 'jessie'!
> There have been errors!
> 
> 
> I've seen it before a long time ago, but I forget what the resolution is.
> 
> sage
> ___
> Sepia mailing list
> se...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/sepia-ceph.com

https://github.com/ceph/ceph-build/pull/102, probably
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ansible task progress

[Dan Mick]

The overall issue here is, of course, there's a lot of vars that cephlab
relies on that are not generated by the task itself (in the case of no
inventory file).  I'm not sure what the plan was to cope with all those
vars.

On 07/07/2015 05:41 PM, Loic Dachary wrote:
> Hi Zack & Andrew,
> 
> With Dan's help we hacked passed the sudo problem with:
> 
> cat > /tmp/ansible.yaml < overrides:
>   ansible.cephlab:
> branch: wip-fix-defaults
> vars:
>   ansible_sudo: true
> interactive-on-error: true
> EOF
> 
> teuthology-suite --machine-type openstack  --suite-dir 
> $(pwd)/teuthology/test/integration --suite noop /tmp/ansible.yaml
> 
> with full logs at
> 
> http://integration.ceph.dachary.org:8081/ubuntu-2015-07-08_00:31:07-noop-master---basic-openstack/3/
> 
> Cheers
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Thoughts about metadata exposure in Calamari

[Dan Mick]

On 06/08/2015 01:02 PM, Handzik, Joe wrote:
> Ok. The code/comment I'm referring to is here: 
> https://github.com/joehandzik/calamari/blob/master/salt/srv/salt/_modules/ceph.py#L393
> 
> I guess your point is that this is all grafted to the OSDMap data anyway, so 
> the osdmap version is the relevant version anyway, yes? If so, is that 
> comment just a slightly paranoid observation?

I just meant "in terms of not having to poll it until the epoch/version
changes, for efficiency".  It would be good also to add an epoch arg to
crush dump to address the fixme.

> Just to make sure I'm understanding everyone correctly, are we saying that I 
> shouldn't bother with the 'osd metadata' call until I have a solid solution 
> to an epoch implementation for osd metadata, or just that I should target a 
> fix for that eventually?

I think the fixes can be independent, but I agree with John that it
would be nice to add an epoch/version to metadata as well (that would
probably be independent of the other versions, unless I'm missing some
coordination).  It ends up being an optimization, but probably a very
useful one.

> Joe
> 
> -Original Message-
> From: Dan Mick [mailto:dm...@redhat.com] 
> Sent: Monday, June 08, 2015 2:54 PM
> To: Handzik, Joe; John Spray; Sage Weil
> Cc: gm...@redhat.com; ceph-devel@vger.kernel.org
> Subject: Re: Thoughts about metadata exposure in Calamari
> 
> The crush map changing causes a change in the osdmap version, I'm pretty
> sure.
> 
> On 06/08/2015 12:50 PM, Handzik, Joe wrote:
>> From what I see in the source file, we'd want to fix 'osd crush dump' 
>> somehow too, right? I can take a look while I'm working in this area to see 
>> what I can accomplish. 
>>
>> Joe
>>
>> -Original Message-
>> From: John Spray [mailto:john.sp...@redhat.com] 
>> Sent: Friday, June 05, 2015 7:39 PM
>> To: Handzik, Joe; Sage Weil
>> Cc: gm...@redhat.com; ceph-devel@vger.kernel.org; Dan Mick (dm...@redhat.com)
>> Subject: Re: Thoughts about metadata exposure in Calamari
>>
>>
>>
>> On 05/06/2015 20:33, Handzik, Joe wrote:
>>> I err in the direction of calling 'osd metadata' too, but it does mean that 
>>> Calamari will need to add that call in (I'll leave it to Gregory to say if 
>>> that is particularly undesirable). Do you think it would be worthwhile to 
>>> better define the metadata bundle into a structure, or is it ok to leave it 
>>> as a set of string pairs?
>>
>> Versioning of the metadata is something to consider. The "osd metadata" 
>> stuff is outside the osdmap epochs, so anything that is consuming 
>> updates to it is stuck with doing some kind of full polling as it 
>> stands.  It might be that some better interface with versions+deltas is 
>> needed for a management layer to efficiently consume it.
>>
>> A version concept where the version is incremented when an OSD starts or 
>> updates its metadata could make synchronization with a management layer 
>> much more efficient.  Efficiency matters here when we're calling on the 
>> mons to serialize data for potentially 1s of OSDs into JSON whenever 
>> the management layer wants an update.
>>
>> John
>>
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Thoughts about metadata exposure in Calamari

[Dan Mick]

The crush map changing causes a change in the osdmap version, I'm pretty
sure.

On 06/08/2015 12:50 PM, Handzik, Joe wrote:
> From what I see in the source file, we'd want to fix 'osd crush dump' somehow 
> too, right? I can take a look while I'm working in this area to see what I 
> can accomplish. 
> 
> Joe
> 
> -Original Message-
> From: John Spray [mailto:john.sp...@redhat.com] 
> Sent: Friday, June 05, 2015 7:39 PM
> To: Handzik, Joe; Sage Weil
> Cc: gm...@redhat.com; ceph-devel@vger.kernel.org; Dan Mick (dm...@redhat.com)
> Subject: Re: Thoughts about metadata exposure in Calamari
> 
> 
> 
> On 05/06/2015 20:33, Handzik, Joe wrote:
>> I err in the direction of calling 'osd metadata' too, but it does mean that 
>> Calamari will need to add that call in (I'll leave it to Gregory to say if 
>> that is particularly undesirable). Do you think it would be worthwhile to 
>> better define the metadata bundle into a structure, or is it ok to leave it 
>> as a set of string pairs?
> 
> Versioning of the metadata is something to consider. The "osd metadata" 
> stuff is outside the osdmap epochs, so anything that is consuming 
> updates to it is stuck with doing some kind of full polling as it 
> stands.  It might be that some better interface with versions+deltas is 
> needed for a management layer to efficiently consume it.
> 
> A version concept where the version is incremented when an OSD starts or 
> updates its metadata could make synchronization with a management layer 
> much more efficient.  Efficiency matters here when we're calling on the 
> mons to serialize data for potentially 1s of OSDs into JSON whenever 
> the management layer wants an update.
> 
> John
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: tracker.ceph.com API

[Dan Mick]

Serves you right for voluntarily using XML :-P


On 02/07/2015 01:06 AM, Loic Dachary wrote:
> 
> 
> On 06/01/2015 12:15, Loic Dachary wrote:
>> Hi,
>>
>> I tried various things to update tracker.ceph.com from the command line but 
>> failed. The best result I have is the following:
>>
>> cat /tmp/a.xml
>>
>> 
>>   10281
>>   firefly: make check fails on fedora 20 (1)
>>   
>> 
>>   firefly
>> 
>>   
>> 
>>
>> curl --http1.0 --verbose -T /tmp/a.xml 
>> http://tracker.ceph.com/issues/10281.xml?key=cd9a99820b4e7543a
> 
> The content-type was missing:
> 
> curl --http1.0 --verbose --header 'Content-type: application/xml' -T 
> /tmp/a.xml http://tracker.ceph.com/issues/10281.xml?key=cd9a99820b4e7543a
> 
> works :-)
> 
>>
>> * Hostname was NOT found in DNS cache
>> *   Trying 64.90.32.38...
>> * Connected to tracker.ceph.com (64.90.32.38) port 80 (#0)
>>> PUT /issues/10281.xml?key=cd9a99820b4e7543a3c HTTP/1.0
>>> User-Agent: curl/7.35.0
>>> Host: tracker.ceph.com
>>> Accept: */*
>>> Content-Length: 239
>>>
>> * We are completely uploaded and fine
>> < HTTP/1.1 200 OK
>> < Date: Tue, 06 Jan 2015 11:05:55 GMT
>> * Server Apache/2.2.22 (Ubuntu) is not blacklisted
>> < Server: Apache/2.2.22 (Ubuntu)
>> < X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.13
>> < X-UA-Compatible: IE=Edge,chrome=1
>> < Cache-Control: no-cache
>> < X-Request-Id: eb302e17961997bafade5739f48a7183
>> < X-Runtime: 1.375926
>> < X-Rack-Cache: invalidate, pass
>> < Set-Cookie: 
>> _redmine_session=BAh7BkkiD3Nlc3Npb25faWQGOgZFRkkZTIwY2Y5NWYzYTE5BjsAVA%3D%3D--a2a2fd48f300f5c8c9;
>>  path=/; HttpOnly
>> < Set-Cookie: autologin=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
>> < Status: 200
>> < Content-Length: 0
>> < Connection: close
>> < Content-Type: application/xml; charset=utf-8
>> <
>> * Closing connection 0
>>
>> but most of the time I get:
>>
>> * Hostname was NOT found in DNS cache
>> *   Trying 64.90.32.38...
>> * Connected to tracker.ceph.com (64.90.32.38) port 80 (#0)
>>> PUT /issues/10281.xml?key=cd9a99820b4e7543a3ce82febce4970b3dfb6b86 HTTP/1.0
>>> User-Agent: curl/7.35.0
>>> Host: tracker.ceph.com
>>> Accept: */*
>>> Content-Length: 239
>>>
>> * We are completely uploaded and fine
>> * Empty reply from server
>> * Connection #0 to host tracker.ceph.com left intact
>>
>> Am I doing something wrong ?
>>
>> Cheers
>>
> 


-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Article about Object Store API

[Dan Mick]

Very cool.  Thank you!

On 01/27/2015 12:48 PM, Marcel Lauhoff wrote:
> 
> Hi,
> 
> I wrote an article about the object store API - How it works and looks
> like with a focus on how to add something new.
> 
> http://irq0.org/articles/ceph/objectstore
> 
> Maybe its useful for somebody.. Feedback is greatly appreciated!
> 
> 
> 
> [ I'm working on a Master's Thesis about Cold Storage / Archives using Ceph
> and will probably write something more about other parts of Ceph
> crossing my way in the near future :) ]
> 
> 
> ~marcel
> 
> 
> --
> Marcel Lauhoff
> Mail/XMPP: m...@irq0.org
> http://irq0.org
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Change to teuthology-suite -k

[Dan Mick]

tl;dr: after https://github.com/ceph/teuthology/pull/394 is merged, -k
testing is no longer the default for teuthology-suite; if you want it,
specify it on the CLI or in the yaml.

Further explanation:

teuthology-suite, by default, currently adds a 'kernel:' stanza to the
yaml file that requests the 'testing' kernel branch be installed on the
test machines.  Currently, the only way to not affect the installed
kernel is to specify "-k -" on the teuthology-suite commandline (to
override the default "-k testing".

Since there are some tests where the particular version of the kernel is
not important (say, Calamari tests), and since vps'es often have a
suitable kernel, and since installing a kernel and rebooting is pretty
expensive, I've changed teuthology-suite so it no longer installs a
kernel by default.  The previous behavior is available by specifying "-k
testing" explicitly (or adding a kernel stanza to the yaml for the suite
in question).

All cron jobs on teuthology and magna002 have already been changed to
explicitly specify a kernel branch.--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
-- 
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: civetweb threads

[Dan Mick]

> I tried to register with ceph issue tracker, but misspelled the email
> address (wrote @gmai.com), now I can't take the name "mustafa" again
> (I waited the request to expire, but until now I can't register).
> Who can delete that wrong registration request so I can take this
> username again?

Done; please contact me if reregistration doesn't work
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: security session at CDS

[Dan Mick]

On 10/29/2014 09:45 AM, Sage Weil wrote:
> On Wed, 29 Oct 2014, Sage Weil wrote:
>> [sage] survey: auth environments
> 
> https://www.surveymonkey.com/s/DQJPJ9J
> 
> Any suggested changes before I send this to a broader audience?  Do the 
> choices all make sense?
> 
> Thanks!
> sage

+1

-- 
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/1 linux-next] ceph: fix bool assignments

[Dan Mick]

On 10/09/2014 02:16 PM, Fabian Frederick wrote:
> Fix some coccinelle warnings:
> fs/ceph/caps.c:2400:6-10: WARNING: Assignment of bool to 0/1


> - bool wake = 0;
> + bool wake = false;

FWIW, that message is backwards: it should say "WARNING: Assignment of
0/1 to bool" (I know, it's a coccinelle issue, but..)


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ceph-maintainers] disabling updatedb

[Dan Mick]

Just adding a note in case you hadn't noticed that updatedb itself has a
CLI for managing the .conf: --add-prune{fs,names,paths}.  Sadly, there
is no "--remove", but at least it lets the conf file format be abstract.

+1 on "everything has a .d/ dir" though.

On 02/20/2014 10:47 AM, Sage Weil wrote:
> On Tue, 18 Feb 2014, bernhard glomm wrote:
>> Well IMO the cleanest solution would be to convince the 
>> mlocate devels to introduce an /etc/updatedb.d/ directory
>> so every package could drop their snippet there.
> 
> I agree. Does somebody want to take the lead on that one?  It seems like 
> the right long-term solution, even if it doesn't help us much right now.
> 
> sage
> 
>> Since that might be a rather long shot
>> I agree that it is a a job for general management 
>>
>> In cfengine it would just read something like:
>> ...
>> edit_line=>  
>> regex_replace("(^\s*PRUNEFS=\")(?!ceph)(.*$)","$(match.1)ceph $(match.2)");
>> ...
>> With a  proper  self containing bundle once called from postinst that would 
>> survive any 
>> upgrade of mlocate with minimal impact.
>>
>> Regards,
>> Bernhard
>>
>>> On Feb 18, 2014, at 7:20 PM, Dan van der Ster  
>>> wrote:
>>>
>>> Hi,
>>>
>>> On Tue, Feb 18, 2014 at 6:52 PM, Gaudenz Steinlin  
>>> wrote:

 Hi

 Sage Weil  writes:

> Dan at CERN noticed that his performance was tanking because updatedb was
> running against /var/lib/ceph.  updatedb has a PRUNE line in
> /etc/updatedb.conf that we should presumably be adding ourselves to.  One
> user pointed out a package that uses sed to rewrite this line in the init
> script on start.
>
> I have two questions:
>
> - is there no better way than sed to add ourselves to this list?
> - should we do it in the init script, or postinst, or both?
>
> Presumably this is a problem others have solved with other packages.

 At least for Debian neither solution is appropriate. Changing other
 packages conffiles in postinst scripts is forbidden by policy. There is
 also no way to preserve this reliably over upgrades without user
 interaction. I'm not sure if there is an explicit policy for init
 scripts, but this seems equally wrong. Also it's unclear how one would
 handle the case were an administrator does NOT want to exclude this
 directory.

 The only solution I see if you really want to completely exclude the
 directory is to convince the mlocate maintainers to either add the
 directory to the default configuration or to add something like an
 /etc/updatedb.d directory where packages can drop configuration file
 snippets. But the latter seems like overkill to me.

 The real question to me is why an updatedb run can drastically impact
 ceph performance. At least in Debian updatedb is run with ionice
 -c3 in the "Idle" scheduling class. According to the man page this
 means: "A program running with idle I/O priority will only get disk time
 when no other program has asked for disk I/O for a defined grace period.
 The impact of an idle I/O process on normal system activity should be
 zero. This scheduling class does not take a priority argument.
 Presently, this scheduling class is permitted for an ordinary user
 (since kernel 2.6.25)." So it should not have any negative effect.

 Maybe CERN (or the distribution they use) should also run updatedb under
 ionice.
>>>
>>> updatedb _is_ run under ionice on our systems (RHEL), but IO
>>> scheduling classes are only implemented by the cfq scheduler.
>>> We use deadline, which is recommended for an enterprise disk server,
>>> and indeed measured more stable IO latencies with deadline than with
>>> cfq. And when you run updatedb on a deadline scheduled drive, there
>>> are so many reads queued up that writes can be starved for many 10s of
>>> seconds.
>>>
>>> In our case, we've already added /var/lib/ceph to the PRUNEPATHS via
>>> their puppet configuration. Though an upstream solution is probably a
>>> good idea since I would assume that most ceph deployments use deadline
>>> and this would hit most of them eventually once they have enough
>>> files. In addition, as someone else mentioned, you'd better add ceph
>>> to the pruned fs types as well, just like /afs is pruned at the
>>> moment, lest every client spend all day stat'ing their big cephfs
>>> namespace.
>>>
>>> Cheers, Dan
>>>
>>> -- Dan van der Ster || Data & Storage Services || CERN IT Department --
>>>

 Gaudenz

 --
 Ever tried. Ever failed. No matter.
 Try again. Fail again. Fail better.
 ~ Samuel Beckett ~
 --
 To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a m

Re: a few deb updates

[Dan Mick]

One comment; otherwise looks rational to me

On 04/01/2014 02:44 PM, Sage Weil wrote:
> Can someone please review
> 
>   https://github.com/ceph/ceph/pull/1581
> 
> There are a couple of minor packages changes here that could use a second 
> look.  Mostly just shifting things into ceph-common from ceph (since they 
> can be used by clients too).
> 
> Thanks!
> sage
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: adminsocket

[Dan Mick]

On 11/22/2013 06:42 PM, James Harper wrote:

What is an adminsocket used for? Would librbd use one in normal operation?


It's a way to send administrative and informational commands directly
to a Ceph entity (usually a daemon, but sometimes a client).  Almost
all the ceph entities create one...osd, mon, mds, client.  It's not 
really "normal" operation, but you can find stats there, and often

things like status, version of the software, etc.

--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

new version of stgt pushed to ceph-extras

[Dan Mick]

Hey, all, we just refreshed stgt to its latest released version 
(1.0.41), and I also tweaked the rbd backend to be a little more

flexible and useful.

stgt is a userspace iSCSI target implementation (using tgtd) that can 
export several types of storage entities as iSCSI LUNs; backends include 
files in synchronous or asynchronous mode, "passthrough" to
real SCSI devices, tape-device emulation on a file, sheepdog block 
images, and Ceph RBD images.


New bs_rbd features include:

* fixed up tgt-admin to work with rbd images (so .conf files work)

* no more 20-rbd-image-per-tgtd limit

* tgtadm accepts --bsopts for each image
  conf= to set path to ceph.conf file
  id= to set clientid

  This means that each image can have different logging and access
  rights, or even refer to a different cluster.

The stgt source has also been refactored so that packagers can
build with or without bs_rbd, now built into an .so, and distribute
that module separately if desired; thus the base package doesn't
require Ceph libraries librbd and librados.

The source is available upstream at http://github.com/fujita/stgt.
Packages are built and available in the ceph-extras repository at 
http://ceph.com/packages/ceph-extras/.


Enjoy!

--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: possible bug in blacklist

[Dan Mick]

Pull request created with fix.

On 09/26/2013 04:51 PM, Dan Mick wrote:

Ah, yes, that's in the validator for CephEntityAddr; it's not checking
for the case that no nonce was supplied.  I've filed
http://tracker.ceph.com/issues/6425.

On 09/24/2013 04:57 PM, Mandell Degerness wrote:

See trace below.  We run this command on system restart in order to
clear any blacklist which was created while node was mis-behaving.
Now, rather than giving a reasonable error, it causes at Traceback:

[root@node-172-20-0-13 ~]# ceph osd blacklist rm 172.20.0.13
Traceback (most recent call last):
   File "/usr/bin/ceph", line 774, in 
 sys.exit(main())
   File "/usr/bin/ceph", line 717, in main
 sigdict, inbuf, verbose)
   File "/usr/bin/ceph", line 375, in new_style_command
 valid_dict = validate_command(sigdict, cmdargs, verbose)
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line
890, in validate_command
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line
715, in matchnum
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line
690, in validate_one
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line
284, in valid
ValueError: need more than 1 value to unpack
[root@node-172-20-0-13 ~]# ceph --version
ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
[root@node-172-20-0-13 ~]#
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: possible bug in blacklist

[Dan Mick]

Ah, yes, that's in the validator for CephEntityAddr; it's not checking 
for the case that no nonce was supplied.  I've filed

http://tracker.ceph.com/issues/6425.

On 09/24/2013 04:57 PM, Mandell Degerness wrote:

See trace below.  We run this command on system restart in order to
clear any blacklist which was created while node was mis-behaving.
Now, rather than giving a reasonable error, it causes at Traceback:

[root@node-172-20-0-13 ~]# ceph osd blacklist rm 172.20.0.13
Traceback (most recent call last):
   File "/usr/bin/ceph", line 774, in 
 sys.exit(main())
   File "/usr/bin/ceph", line 717, in main
 sigdict, inbuf, verbose)
   File "/usr/bin/ceph", line 375, in new_style_command
 valid_dict = validate_command(sigdict, cmdargs, verbose)
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line
890, in validate_command
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line
715, in matchnum
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line
690, in validate_one
   File "/usr/lib64/python2.7/site-packages/ceph_argparse.py", line 284, in 
valid
ValueError: need more than 1 value to unpack
[root@node-172-20-0-13 ~]# ceph --version
ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
[root@node-172-20-0-13 ~]#
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

erasure coding / jerasure: topic of interest?

[Dan Mick]

At SDC this week, Ethan Miller presented about some work he has done 
along with others including Jim Plank, who also was involved with 
jerasure; it seems like this work may be useful for optimizing the

erasure code calculations on Intel CPUs, at least:

http://bitbucket.org/ethanmiller/gf-complete

Paper here: http://web.eecs.utk.edu/~plank/plank/papers/FAST-2013-GF.pdf
looks very much like the presentation Ethan gave at the conference.

--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] v0.68 released

[Dan Mick]

Where are you looking?  ceph.com/debian-testing has 0.68

On 09/04/2013 07:12 PM, 이주헌 wrote:

Debian/Ubuntu packages is still 0.67.2.



--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: collectd plugin with cuttlefish

[Dan Mick]

It's a bit surprising that it broke with cuttlefish; something might 
have happened in dumpling, but we wouldn't expect changes in cuttlefish.

It looks like collectd just couldn't talk to the monitor properly.
Maybe look at the mon's log and see what it thinks it saw?

On 08/30/2013 05:04 AM, Damien Churchill wrote:

Hi,

Has anything changed with the admin socket that would prevent the
collectd plugin (compiled against 5.3.1 using the patches submitted to
the collectd ML) from gathering stats? I've recompiled collectd with
--enable-debug and receive the following output in the log:

ceph_init
name=mon_ceph1, asok_path=/var/run/ceph/ceph-mon.ceph1.asok
entering cconn_main_loop(request_type = 0)
did cconn_prepare(name=mon_ceph1,i=0,st=1)
cconn_handle_event(name=mon_ceph1,state=1,amt=0,ret=4)
did cconn_prepare(name=mon_ceph1,i=0,st=2)
did cconn_prepare(name=mon_ceph1,i=0,st=2)
ERROR: cconn_main_loop: timed out.
cconn_main_loop: reached all Ceph daemons :)
Initialization of plugin `ceph' failed with status -110. Plugin will
be unloaded.
plugin_unregister_read: Marked `ceph' for removal.

Thanks in advance!
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: teuthology : ulimit: error

[Dan Mick]

Ah, there's another we apply universally to our test systems, apparently:

'/etc/security/limits.d/ubuntu.conf'

ubuntu hard nofile 16384

and the tests run as user "ubuntu".  Line 4 of the script is the nofile 
setting.


On 08/10/2013 01:34 AM, Loic Dachary wrote:



On 10/08/2013 07:35, Dan Mick wrote:

IIRC we had to adjust settings in /etc/security to allow ulimit adjustment of 
at least core:

sed -i 's/^#\*.*soft.*core.*0/\*softcore unlimited/g' 
/etc/security/limits.conf

or something like that.  That seems to apply to centos/fedora/redhat systems.


Hi Dan,

This is Ubuntu and it does work with sudo. It probably is something about how 
it's called rather than a system limit.

Cheers




On 08/08/2013 02:52 PM, Loic Dachary wrote:

Hi,

Trying to use Ubuntu precise virtual machines as teuthology targets ( making 
sure they have 2GB of RAM because ceph-test-dbg will not even install with 1GB 
of RAM ;-) and installing the key with

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' 
| sudo apt-key add -

teuthology runs with

./virtualenv/bin/teuthology job.yaml

where job.yaml is

check-locks: false
roles:
- - mon.a
- mon.c
- osd.0
- - mon.b
- osd.1
- client.0
tasks:
- install:
 project: ceph
 branch: wip-5510
- ceph:
 fs: ext4
- workunit:
  clients:
all:
- filestore/filestore.sh
targets:
 ubuntu@91.121.254.229: ssh-rsa 
B3NzaC1yc2EDAQABAAABAQDOSit20EyZ2AKCvk2tnMsdQ6LmVRutvBmZb0awV9Z2EduJa0fYPrReYRb9ZhGRq2PJe0zgpFPKr8s4gGay+tL0+dFkju5uyABqMGeITlJCifd+RhM0MCVllgIzekDwVb0n6ydTS8k7GVFyYv8ZC0TPgbfcDcEtSEgqJNRJ0o1Bh8swuTn+cipanNDRVK39tOqJdfptUxak+TD+5QY8CGFdXdEQYP7VsYJ+jQHw73O2xbuPgfv5Shbmt+qGWLToxFqKca3owMtkvFeONgYUdujgg9qr7Q9p0+HhCFCXB8v4N2I7NSbWNdpGqyJLdLqwJ70gEeNlOhm5F7IsXfVxTapB
 ubuntu@91.121.254.237: ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCXVzhedORtmEmCeZJ4Ssg8wfqpYyH9W/Aa+j6CvPHSAkzZ48zXqVBATxm6S8sIIqfKkz1hWpNssx6uUotbm8k/ZatMddsd932+Di136l/HUhp6O8iIFze56kjWpyDpRPw2MM0V+OKmsiHZDfMj9ATt6ChgXfRsm23MUlmnoFHvtfwvFBnvBjcZPN7qxMpFHDamZzACNvnis/OINJrud9VprOgjhZZ7mxcTbcVZaVgcTcnyC4b9d9PRrMG2aCv0BO1eb/qnlmSogQPfoKEORJcwaUcLgabww+Taa9hJSZ9l8yEQamj+XIgr6yzGKgCvlG4lTdHM2tQdpgATZvR7/pBz

and produces the following output

INFO:teuthology.orchestra.run.err:[91.121.254.229]: 
/home/ubuntu/cephtest/lo1308082328/adjust-ulimits: 4: ulimit: error setting 
limit (Operation not permitted)

and the full output is at

http://pastealacon.com/32957


Refreshed with a longer timeout.

http://pastealacon.com/32963


as if

/home/ubuntu/cephtest/lo1308082328/adjust-ulimits ceph-coverage 
/home/ubuntu/cephtest/lo1308082328/archive/coverage monmaptool --create 
--clobber --add a 91.121.254.229:6789 --add c 91.121.254.229:6790 --add b 
91.121.254.237:6789 --print /home/ubuntu/cephtest/lo1308082328/monmap'

was run without a leading sudo. I tried running sudo adjust-ulimits echo foo 
manually on the target and it works.

I would very much appreciate a hint ;-)

Cheers




--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: teuthology : ulimit: error

[Dan Mick]

IIRC we had to adjust settings in /etc/security to allow ulimit 
adjustment of at least core:


sed -i 's/^#\*.*soft.*core.*0/\*softcore 
unlimited/g' /etc/security/limits.conf


or something like that.  That seems to apply to centos/fedora/redhat 
systems.



On 08/08/2013 02:52 PM, Loic Dachary wrote:

Hi,

Trying to use Ubuntu precise virtual machines as teuthology targets ( making 
sure they have 2GB of RAM because ceph-test-dbg will not even install with 1GB 
of RAM ;-) and installing the key with

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' 
| sudo apt-key add -

teuthology runs with

./virtualenv/bin/teuthology job.yaml

where job.yaml is

check-locks: false
roles:
- - mon.a
   - mon.c
   - osd.0
- - mon.b
   - osd.1
   - client.0
tasks:
- install:
project: ceph
branch: wip-5510
- ceph:
fs: ext4
- workunit:
 clients:
   all:
   - filestore/filestore.sh
targets:
ubuntu@91.121.254.229: ssh-rsa 
B3NzaC1yc2EDAQABAAABAQDOSit20EyZ2AKCvk2tnMsdQ6LmVRutvBmZb0awV9Z2EduJa0fYPrReYRb9ZhGRq2PJe0zgpFPKr8s4gGay+tL0+dFkju5uyABqMGeITlJCifd+RhM0MCVllgIzekDwVb0n6ydTS8k7GVFyYv8ZC0TPgbfcDcEtSEgqJNRJ0o1Bh8swuTn+cipanNDRVK39tOqJdfptUxak+TD+5QY8CGFdXdEQYP7VsYJ+jQHw73O2xbuPgfv5Shbmt+qGWLToxFqKca3owMtkvFeONgYUdujgg9qr7Q9p0+HhCFCXB8v4N2I7NSbWNdpGqyJLdLqwJ70gEeNlOhm5F7IsXfVxTapB
ubuntu@91.121.254.237: ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCXVzhedORtmEmCeZJ4Ssg8wfqpYyH9W/Aa+j6CvPHSAkzZ48zXqVBATxm6S8sIIqfKkz1hWpNssx6uUotbm8k/ZatMddsd932+Di136l/HUhp6O8iIFze56kjWpyDpRPw2MM0V+OKmsiHZDfMj9ATt6ChgXfRsm23MUlmnoFHvtfwvFBnvBjcZPN7qxMpFHDamZzACNvnis/OINJrud9VprOgjhZZ7mxcTbcVZaVgcTcnyC4b9d9PRrMG2aCv0BO1eb/qnlmSogQPfoKEORJcwaUcLgabww+Taa9hJSZ9l8yEQamj+XIgr6yzGKgCvlG4lTdHM2tQdpgATZvR7/pBz

and produces the following output

INFO:teuthology.orchestra.run.err:[91.121.254.229]: 
/home/ubuntu/cephtest/lo1308082328/adjust-ulimits: 4: ulimit: error setting 
limit (Operation not permitted)

and the full output is at

http://pastealacon.com/32957

as if

/home/ubuntu/cephtest/lo1308082328/adjust-ulimits ceph-coverage 
/home/ubuntu/cephtest/lo1308082328/archive/coverage monmaptool --create 
--clobber --add a 91.121.254.229:6789 --add c 91.121.254.229:6790 --add b 
91.121.254.237:6789 --print /home/ubuntu/cephtest/lo1308082328/monmap'

was run without a leading sudo. I tried running sudo adjust-ulimits echo foo 
manually on the target and it works.

I would very much appreciate a hint ;-)

Cheers


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph admin command: debuggging

[Dan Mick]

There are a number of subsystems that the clients use, so a number of 
knobs that matter.  The "logging/authentication name" (the type.id 
name), by default, is 'client.admin'.  Some of the relevant logging 
knobs are ms/monc; of course there are usually effects caused by the 
daemons too, so the mon/osd logs will contain things triggered by the 
client.


What sort of things are you looking for specifically?

On 08/08/2013 07:04 AM, Andreas Bluemle wrote:

Hi,

maybe this is the wrong list - but I am looking for
logging support for the /usr/bin/ceph adminstration
comamand.

This is build from the same source as the ceph daemons are.

For the daemons, logging is controlled for the different
comoponents of a daemon on behalf of configuration file
directives (e.g. "debug ms" for the message I/O).

Can this be applied to the /usr/bin/ceph command, too?
And where would the log file go?


Thanks for any help

Andreas Bluemle



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Rados Protocoll

[Dan Mick]

On 08/04/2013 10:51 PM, Noah Watkins wrote:

On Fri, Aug 2, 2013 at 1:58 AM, Niklas Goerke  wrote:


As for the documentation you referenced: I didn't find a documentation of
the RADOS Protocol which could be used to base an implementation of librados
upon. Does anything like this exist, or would I need to "translate" the c
implementation?


I do not know of any detailed documentation of the protocol except for
the source :(


...nor, it should be pointed out, is it to be considered at all stable...

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Optimize Ceph cluster (kernel, osd, rbd)

[Dan Mick]

I would get the cluster up and running and do some experiments before I 
spent any time on optimization, much less all this.


On 07/20/2013 09:35 AM, Ta Ba Tuan wrote:

Please help me!


On 07/20/2013 02:11 AM, Ta Ba Tuan wrote:

Hi everyone,

I have *3 nodes (running MON and MDS)*
and *6 data nodes ( 84 OSDs**)*
Each data nodes has configuraions:
  - CPU: 24 processor * Core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
  - RAM: 32GB
  - Disk: 14*4TB
(14disks *4TB *6 data nodes= 84 OSDs)

To optimize Ceph Cluster, *I adjusted some kernel arguments*
(nr_request in queue and increated read throughput):

#Adjust nr_request in queue (staying in mem - default is 128)
echo 1024 > /sys/block/sdb/queue/nr_requests
echo noop > /sys/block/sda/queue/scheduler   (default= noop
deadline [cfq])
#Increase read throughput  (default: 128)
echo "512" > /sys/block/*/queue/read_ahead_kb

And, *tuning Ceph configuraion options below:*

[client]

 rbd cache = true
 rbd cache size = 536870912
 rbd cache max dirty = 134217728
 rbd cache target dirty = 33554432
 rbd cache max dirty age = 5

[osd]
osd data = /var/lib/ceph/osd/cloud-$id
osd journal = /var/lib/ceph/osd/cloud-$id/journal
osd journal size = 1
osd mkfs type = xfs
osd mkfs options xfs = "-f -i size=2048"
osd mount options xfs = "rw,noatime,inode64,logbsize=250k"

keyring = /var/lib/ceph/osd/cloud-$id/keyring.osd.$id
#increasing the number may increase the request processing rate
osd op threads = 24
#The number of disk threads, which are used to perform background disk
intensive OSD operations such as scrubbing and snap trimming
osd disk threads =24
#The number of active recovery requests per OSD at one time. More
requests will accelerate recovery, but the requests places an
increased load on the cluster.
osd recovery max active =1
#writing direct to the journal.
#Allow use of libaio to do asynchronous writes
journal dio = true
journal aio = true
#Synchronization interval:
#The maximum/minimum interval in seconds for synchronizing the filestore.
filestore max sync interval = 100
filestore min sync interval = 50
#Defines the maximum number of in progress operations the file store
accepts before blocking on queuing new operations.
filestore queue max ops = 2000
#The maximum number of bytes for an operation
filestore queue max bytes = 536870912
#The maximum number of operations the filestore can commit.
filestore queue committing max ops = 2000 (default =500)
#The maximum number of bytes the filestore can commit.
filestore queue committing max bytes = 536870912
#When you add or remove Ceph OSD Daemons to a cluster, the CRUSH
algorithm will want to rebalance the cluster by moving placement
groups to or from Ceph OSD Daemons to restore the balance. The process
of migrating placement groups and the objects they contain can reduce
the cluster’s operational performance considerably. To maintain
operational performance, Ceph performs this migration with
‘backfilling’, which allows Ceph to set backfill operations to a lower
priority than requests to read or write data.
osd max backfills = 1


Tomorrow, I'm going to implement Ceph Cluster,
I have very little experience in managing Ceph. So, I hope someone
give me advices about above arguments and guide me how to best
optimize ceph cluster?

Thank you so much!
--tuantaba







___
ceph-users mailing list
ceph-us...@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Ceph developers, please note: changes to 'ceph' CLI tool in master branch

[Dan Mick]

A large restructuring of the 'ceph' command-line tool has been pushed to 
the master branch (and will be present in v0.65 as well).  The ceph tool 
you execute is now a Python script that talks to the cluster through 
rados.py, the Python binding to librados.so (and, of course, then, with 
librados.so).


Those who install/upgrade using packages will get the coordinated 
versions of all the pieces, and can stop reading except as a matter of 
interest.


However, those who build from source will need to be aware that 
PYTHONPATH must include the source versions of src/pybind/rados.py, and 
LD_LIBRARY_PATH must include .libs.  Currently, you must set these in 
your environment before running ./ceph.


I've just pushed a commit that will try to automatically determine this 
situation: if the path to the ceph tool ends in src/, and there exist 
.libs and pybind directories there, the tool will reset LD_LIBRARY_PATH 
and PYTHONPATH and re-exec itself so that LD_LIBRARY_PATH takes effect.

It also issues a message to stderr noting this "developer mode":

*** DEVELOPER MODE: setting PYTHONPATH and LD_LIBRARY_PATH

This latest convenience feature for developers is in
commit e5184ea95031b7bea4264062de083045767d5dc3 in master.

--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Finding out via librados if a cluster is near full

[Dan Mick]

On 05/29/2013 01:24 AM, Wido den Hollander wrote:

On 05/29/2013 10:14 AM, Wido den Hollander wrote:

Hi,

Is there a way to find out if a cluster is near full via librados?



Yes, there is. (Thanks tnt on IRC!)

There is ofcourse rados_cluster_stat which will give you:

struct rados_cluster_stat_t {
   uint64_t kb, kb_used, kb_avail;
   uint64_t num_objects;
};


One thing here is however that you don't know to what the (near)full
ratio has been set to. So you have to do your own guessing.


Aside: Soon there will be a librados interface for issuing commands like
ceph pg dump and getting the response (in JSON if desired), if that's
going to be helpful.




The reason I'm asking is that I'm working on a deployment of the RADOS
Gateway and one of the potential problems I see is that the RADOS
Gateway can keep writing and filling up the whole cluster.



So, with the cluster stat information we could implement:

rgw_refuse_write_above_ratio DOUBLE 0.90


It's obvious that it's the admin's responsibility to make sure that
doesn't happen, but what I'd like to do is make the RADOS Gateway deny
PUT requests when the cluster is near full.

That way normal meta data operations (thus writes) can continue, but new
objects won't be accepted until the cluster has enough space available.

This might also be useful for librbd. Refuse the creation of new RBD
images so that existing images can continue operating.





--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [doc] strange navigation error

[Dan Mick]

On 05/24/2013 02:16 PM, Travis Rhoden wrote:

On Fri, May 24, 2013 at 4:55 PM, Dan Mick  wrote:



On 05/24/2013 10:05 AM, Travis Rhoden wrote:


I stumbled across a weird TOC/navigation error in the docs.

On the page:
http://ceph.com/docs/master/rados/deployment/ceph-deploy-mds/,
if you hover over "next" on the top right, it wants to go to
mds-config-ref, even though the TOC on the left says the next entry
should be ceph-deploy-purge.



Ah.  That's because the TOC *also* says that the next is MDS Configuration
(under CephFS).  I guess we're sharing the Add/Remove MDS page at two places
in the hierarchy.


Yes, that is exactly what is happening.  Since the Add/Remove MDS page
is specific to ceph-deploy, I would think you want to keep it in the
ceph-deploy section, and not add it ot the TOC a second time from the
CephFS section.  We could instead a reference/link to it, but a page
should really only ever be in one TOC.  If someone says that is okay
to do, I can send a pull-request with the offending line removed (and
potentially a reference/link instead).

  - Travis





Sure.  I'm gonna go look at your earlier pull request now, too.  Thanks.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [doc] strange navigation error

[Dan Mick]

On 05/24/2013 10:05 AM, Travis Rhoden wrote:

I stumbled across a weird TOC/navigation error in the docs.

On the page: http://ceph.com/docs/master/rados/deployment/ceph-deploy-mds/,
if you hover over "next" on the top right, it wants to go to
mds-config-ref, even though the TOC on the left says the next entry
should be ceph-deploy-purge.


Ah.  That's because the TOC *also* says that the next is MDS 
Configuration (under CephFS).  I guess we're sharing the Add/Remove MDS 
page at two places in the hierarchy.


--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: found some issues on ceph v0.61.2

[Dan Mick]

On 05/20/2013 05:00 PM, ymorita...@gmail.com wrote:

Hi,

I have found some issues on ceph v0.61.2 on Ubuntu 12.10.

(1) "ceph-deploy osd create" command fails when using --cluster  option.

[root@host3 yuji_ceph]# ceph-deploy --cluster yuji osd create host1:sdb
Traceback (most recent call last):
   File "/usr/sbin/ceph-deploy", line 8, in 
 load_entry_point('ceph-deploy==0.1', 'console_scripts', 'ceph-deploy')()
   File "/root/ceph-deploy/ceph_deploy/cli.py", line 112, in main
 return args.func(args)
   File "/root/ceph-deploy/ceph_deploy/osd.py", line 428, in osd
 prepare(args, cfg, activate_prepared_disk=True)
   File "/root/ceph-deploy/ceph_deploy/osd.py", line 273, in prepare
 s = '{} returned {}\n{}\n{}'.format(cmd, ret, out, err)
ValueError: zero length field name in format


This is a symptom of Python < 2.7, but you say Ubuntu 12.10; are all the 
hosts running 12.10?  Have you manually configured the hosts to use 
Python < 2.7?


If the hosts are running < 2.7 on purpose, there's a fix for this in 
git; see http://tracker.ceph.com/issues/5086, but it shouldn't be 
happening on 12.10.



(2) "service ceph -a start/stop" command is accepted, but nothing happens.

root@host1:~# ceph osd tree

# idweight  type name   up/down reweight
-1  0.16root default
-2  0.16host host1
0   0.03osd.0   up  1
1   0.03osd.1   up  1
2   0.03osd.2   up  1
3   0.06999 osd.3   up  1

root@host1:~# /etc/init.d/ceph -a stop
root@host1:~# ceph osd tree

# idweight  type name   up/down reweight
-1  0.16root default
-2  0.16host host1
0   0.03osd.0   up  1
1   0.03osd.1   up  1
2   0.03osd.2   up  1
3   0.06999 osd.3   up  1


If you're running on 12.10 with ceph-deploy, you're using Upstart, so 
you probably want upstart commands (start ceph/stop ceph)



By the way, I could not find [osd], [host] and [mon] entry in the
/etc/ceph/ceph.conf anymore when I create ceph cluster using
"ceph-deploy". Is this information still stored somewhere?


Not directly; it's inferred from the presence of directories in 
/var/lib/ceph.  You can, of course, still add configuration options to 
ceph.conf, it's just that you don't have to specify daemon 
location/address there as you used to.



Thank you.
Yuji
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Urgent Notice to Users on 0.61.1

[Dan Mick]

I also just pushed a fix to the cuttlefish branch, so if you want 
packages that fix this, you can get them from gitbuilders using the 
"testing" versions, branch "cuttlefish".


Thanks, Mike, for pointing this out!

On 05/10/2013 08:27 PM, Mike Dawson wrote:

Anyone running 0.61.1,

Watch out for high disk usage due to a file likely located at
/var/log/ceph/ceph-mon..tdump. This file contains debugging
for monitor transactions. This debugging was added in the past week or
so to track down another anomaly. It is not necessary (or useful unless
you are debugging monitor transactions).

Depending on the load of your cluster, the tdump file may become large
quickly. I've seen it grow over 50GB/day.

IMHO, the default setting should be "false", but it was merged in prior
to 0.61.1 with a default of "true". If you are on 0.61.1, you can
confirm the settings with:

ceph --admin-daemon /var/run/ceph/ceph-mon..asok config show |
grep mon_debug_dump

To disable it, add "mon debug dump transactions = false" to ceph.conf,
and restart your monitors. Alternatively, you can inject the argument or
add the cmdline flag.

dmick has an issue started at:
http://tracker.ceph.com/issues/5024

Thanks,
Mike
___
ceph-users mailing list
ceph-us...@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Authentication question with RBD+STGT

[Dan Mick]

On 04/22/2013 11:09 AM, Scott Sullivan wrote:

Referring to this:
http://ceph.com/dev-notes/adding-support-for-rbd-to-stgt/

I compiled the latest tgt with RBD support. My question is when using
this method to access RBD volumes, where do you tell it what user to
authenticate to the cluster with? I do see the above linked page
mentions it will read a local ceph.conf.


Yes, as Wido hints at, it passes NULL to rados_create()'s second arg; 
that means it will authenticate as client.admin.  This could certainly 
be easily fixed.


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bug in rbd rename command?

[Dan Mick]

It's arguable, but we wanted to treat source and destination pools 
separately in general.


Note that you can also specify images as POOLNAME/a and POOLNAME/b.

On 04/19/2013 12:28 AM, Stefan Priebe - Profihost AG wrote:

Hi,

if i issue rbd -p POOLNAME ... rename a b

It uses the POOL for a but not for b.

Output is then:
rbd: mv/rename across pools not supported
source pool: vmstorssd1 dest pool: rbd

Shouldn't it use the pool provided by -p for both?

Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: skip / disable journal

[Dan Mick]

On 04/15/2013 01:24 PM, Stefan Priebe wrote:


is there a possibility to disable / skip the jornal? Stuff like
FlashCache, BCache and others seem to work much better but sequential
I/O becomes a bottleneck if the ceph journal is on a single ssd.


As far as I know, the journal is a required element for consistency. 
Sequential I/O is a benefit of the journal.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph and efficient access of distributed resources

[Dan Mick]

On 04/15/2013 01:06 PM, Gandalf Corvotempesta wrote:

2013/4/12 Mark Nelson 


Currently reads always come from the primary OSD in the placement group
rather than a secondary even if the secondary is closer to the client.



In this way, only one OSD will be involved in reading an object, this will
result in a bottleneck if multiple clients needs to access to the same file.

For example, a 3KB CSS file served by a webserver to 400 users, will be
read just from one OSD. 400 users directed to 1 OSD  while (in case of
replica 3) other 2 OSDs are available?


Yes.  Consistency across the cluster is dependent on this scheme, currently.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pg dump --format crashes all mon nodes

[Dan Mick]

Indeed.  http://tracker.ceph.com/issues/4678.  Thanks for the report.

On 4/5/2013 5:49 PM, Lorieri wrote:

Hi,

My colleague run ceph pg dump --format with no other arguments.
It crashes all mon daemons in the cluster.

happens in ceph 0.60 and 0.56
3 mon, 3 osd, default crush map

# ceph pg dump --format
2013-04-05 21:21:42.198843 7f3b1e58b700  0 monclient: hunting for new mon
2013-04-05 21:21:42.210685 7f3b1e58b700  0 monclient: hunting for new mon
2013-04-05 21:21:42.229632 7f3b1e58b700  0 monclient: hunting for new mon
2013-04-05 21:21:42.230233 7f3b22ca4700  0 -- 10.134.35.70:0/3005 >>
10.134.35.68:6789/0 pipe(0x7f3b14005130 sd=3 :0 pgs=0 cs=0 l=1).fault
2013-04-05 21:21:45.178005 7f3b1c486700  0 -- 10.134.35.70:0/3005 >>
10.134.35.69:6789/0 pipe(0x7f3b18000f90 sd=3 :0 pgs=0 cs=0 l=1).fault
2013-04-05 21:21:48.178089 7f3b22ca4700  0 -- 10.134.35.70:0/3005 >>
10.134.35.70:6789/0 pipe(0x7f3b18003010 sd=4 :0 pgs=0 cs=0 l=1).fault

# ceph health
2013-04-05 21:24:22.365023 7f59371ec700  0 -- :/3114 >>
10.134.35.70:6789/0 pipe(0x1e51540 sd=3 :0 pgs=0 cs=0 l=1).fault
2013-04-05 21:24:25.365819 7f593d909700  0 -- :/3114 >>
10.134.35.69:6789/0 pipe(0x7f592c000c00 sd=4 :0 pgs=0 cs=0 l=1).fault
2013-04-05 21:24:28.365786 7f59371ec700  0 -- :/3114 >>
10.134.35.70:6789/0 pipe(0x7f592c003010 sd=4 :0 pgs=0 cs=0 l=1).fault
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Puppet modules for Ceph finally landed!

[Dan Mick]

This is pretty cool, Sébastien.

On 03/28/2013 02:34 AM, Sebastien Han wrote:

Hello everybody,

Quite recently François Charlier and I worked together on the Puppet
modules for Ceph on behalf of our employer eNovance. In fact, François
started to work on them last summer, back then he achieved the Monitor
manifests. So basically, we worked on the OSD manifest. Modules are in
pretty good shape thus we thought it was important to communicate to the
community. That's enough talk, let's dive into these modules and explain
what do they do. See below what's available:

* Testing environment is Vagrant ready.
* Bobtail Debian latest stable version will be installed
* The module only supports CephX, at least for now
* Generic deployment for 3 monitors based on a template file
examples/common.sh which respectively includes mon.sh, osd.sh, mds.sh.
* Generic deployment for N OSDs. OSD disks need to be set from the
examples/site.pp file (line 71). Puppet will format specified disks in
XFS (only filesystem implemented) using these options: `-f -d
agcount= -l size=1024m -n size=64k` and finally mounted
with: `rw,noatime,inode64`. Then it will mount all of them and append
the appropriate lines in the fstab file of each storage node. Finally
the OSDs will be added into Ceph.

All the necessary materials (sources and how-to) are publicly available
(and for free) under AGPL license on Github at
https://github.com/enovance/puppet-ceph . Those manifests do the job
quite nicely, although we still need to work on MDS (90% done, just need
a validation),  RGW (0% done) and a more flexible implementation
(authentication and filesystem support). Obviously comments,
constructive critics and feedback are more then welcome. Thus don't
hesitate to drop an email to either François (f.charl...@enovance.com
) or I (sebast...@enovance.com
) if you have further questions.

Cheers!


Sébastien Han
Cloud Engineer

"Always give 100%. Unless you're giving blood."









PHONE : +33 (0)1 49 70 99 72 – MOBILE : +33 (0)6 52 84 44 70
EMAIL : sebastien@enovance.com
 – SKYPE : han.sbastien
ADDRESS : 10, rue de la Victoire – 75009 Paris
WEB : www.enovance.com – TWITTER : @enovance



___
ceph-users mailing list
ceph-us...@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rados_pool_list usage

[Dan Mick]

There is a typo though: "List objects in a pool"

That should be: "List all pools"


yep.
--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rados_pool_list usage

[Dan Mick]

It looks like it attempts to behave as documented (somewhat surprisingly 
and fragile-ly, IMO; I would have made it all-or-nothing and return 
nothing in buf if len is too small).


The Python binding retries/resizes until it can return them all.

What do you think is wrong, Wido?

On 03/27/2013 08:59 AM, Gregory Farnum wrote:



On Wednesday, March 27, 2013 at 1:59 AM, Wido den Hollander wrote:


Hi,

While working with rados_pool_list I stumbled upon what I think is a
documentation issue.

librados.h tells me this:

/**
* List objects in a pool
*
* Gets a list of pool names as NULL-terminated strings. The pool
* names will be placed in the supplied buffer one after another.
* After the last pool name, there will be two 0 bytes in a row.
*
* If len is too short to fit all the pool name entries we need, we
will fill
* as much as we can.
*
* @param cluster cluster handle
* @param buf output buffer
* @param len output buffer length
* @returns length of the buffer we would need to list all pools
*/
int rados_pool_list(rados_t cluster, char *buf, size_t len);

"If len is too short to fit all the pool name entries we need, we will
fill as much as we can."

 From what I could remember it would return the length required if "len"
isn't long enough. Looking at the Python and PHP bindings (which I
wrote) it seems that is correct.

It also says: "@returns length of the buffer we would need to list all
pools"

Docs issue I guess?

It certainly returns the amount of space needed; does it not also fill in the 
given buffer with however many pools it could? (It might not, but those 
sentences aren't contradictory in my mind.)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: FW: errno=Connection refused

[Dan Mick]

Everything else was coming from github.com; ceph-object-corpus was the 
one thing talking to ceph.com.


I just did
git clone git://ceph.com/git/ceph-object-corpus.git

and it worked for me.

On 03/27/2013 12:52 AM, charles L wrote:

Hi Joao,

I am able to access ceph.com without any issue and if you see the log, ceph 
clone was successful until the point of Cloning into 'ceph-object-corpus'.

I really don't understand the reason for this, but it has been successful in 
the past.

Pls i still need help about this.

Thanks,

Charles.


git://ceph.com/git/ceph-object-corpus.git

Date: Tue, 26 Mar 2013 15:09:11 +
From: joao.l...@inktank.com
To: charlesboy...@hotmail.com
CC: ceph-devel@vger.kernel.org
Subject: Re: FW: errno=Connection refused

On 03/26/2013 02:33 PM, charles L wrote:



From: charlesboy...@hotmail.com
To: ceph-us...@ceph.com
Subject: errno=Connection refused
Date: Tue, 26 Mar 2013 11:26:03 +0100

Hi,

Can somebody help?

git clone --recursive https://github.com/ceph/ceph.git
Cloning into 'ceph'...
remote: Counting objects: 179817, done.
remote: Compressing objects: 100% (34122/34122), done.
remote: Total 179817 (delta 146077), reused 177377 (delta 143959)
Receiving objects: 100% (179817/179817), 35.42 MiB | 136 KiB/s, done.
Resolving deltas: 100% (146077/146077), done.
Submodule 'ceph-object-corpus' (git://ceph.com/git/ceph-object-corpus.git) 
registered for path 'ceph-object-corpus'
Submodule 'src/libs3' (git://github.com/ceph/libs3.git) registered for path 
'src/libs3'
Cloning into 'ceph-object-corpus'...
fatal: unable to connect to ceph.com:
ceph.com[0: 208.113.241.137]: errno=Connection refused

Clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path 
'ceph-object-cor pus' failed



Works for me. Maybe some temporary connection error? Are you able to
access 'ceph.com' from wherever you are?

-Joao

$ git clone --recursive https://github.com/ceph/ceph.git
Cloning into 'ceph'...
remote: Counting objects: 180035, done.
remote: Compressing objects: 100% (34122/34122), done.
remote: Total 180035 (delta 146255), reused 177644 (delta 144177)
Receiving objects: 100% (180035/180035), 35.46 MiB | 1.51 MiB/s, done.
Resolving deltas: 100% (146255/146255), done.
Submodule 'ceph-object-corpus'
(git://ceph.com/git/ceph-object-corpus.git) registered for path
'ceph-object-corpus'
Submodule 'src/libs3' (git://github.com/ceph/libs3.git) registered for
path 'src/libs3'
Cloning into 'ceph-object-corpus'...
remote: Counting objects: 6250, done.
remote: Compressing objects: 100% (3459/3459), done.
remote: Total 6250 (delta 772), reused 6101 (delta 654)
Receiving objects: 100% (6250/6250), 735.47 KiB | 305 KiB/s, done.
Resolving deltas: 100% (772/772), done.
Submodule path 'ceph-object-corpus': checked out
'0655429f62bc3366250496da3513abec15c34856'
Cloning into 'src/libs3'...
remote: Counting objects: 1000, done.
remote: Compressing objects: 100% (289/289), done.
remote: Total 1000 (delta 720), reused 976 (delta 696)
Receiving objects: 100% (1000/1000), 347.92 KiB | 262 KiB/s, done.
Resolving deltas: 100% (720/720), done.
Submodule path 'src/libs3': checked out
'9dc3a9c683385abfe4ad92b7c6ff30719acc3c13'





Thanks. --

To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html   
  --

To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: crush changes via cli

[Dan Mick]

That's right. The "remove" versus "unlink" verbs make that pretty clear
to me, at least... Are you suggesting this be clarified in the docs, or
that the command set change? I think once we settle on the CLI, John can
make a pass through the crush docs and make sure these commands are
explained.



That makes sense if you know the internal representation of the
CRUSH  map; less so if you're thinking of it in terms from related fields like,
say, a filesystem. ;) I admit I don't have any better suggestions and
strong docs might be just fine, but the distinction is one that's very
visible to users *afterwards* but not beforehand. I'm imagining a lot of
crush maps with 5 unlinked parallel trees in our future.


I'm not saying it's better, but perhaps/possibly connect/disconnect?
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: docs

[Dan Mick]

On 03/22/2013 05:37 AM, Jerker Nyberg wrote:


There seem to be a missing argument to ceph osd lost (also in help for
the command).

http://ceph.com/docs/master/rados/operations/control/#osd-subsystem



Indeed, it seems to be missing the id.  The CLI is getting a big rework 
right now, but the docs should be corrected.  Patch or file an issue, 
either way.



src/tools/ceph.cc
src/test/cli/ceph/help.t
doc/rados/operations/control.rst

The documentation for development release packages is slightly confused.
Should it not refer to http://ceph.com/rpm-testing for development
release packages? (Also, the ceph-release package in the development
release does not refer to itself (in /etc/yum.repos.d/ceph.repo) but to
(http://ceph.com/rpms) packages.)

http://ceph.com/docs/master/install/rpm/


Do you want patches?

--jerker
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: deb/rpm package purge

[Dan Mick]

As a point of comparison, mysql removes the config files but not
/var/lib/mysql.
The question is, is that okay/typical/desireable/recommended/a bad idea?

  I should have asked this sooner. Do you know _any_ program that removes
your favorite music collection, your family photos or your business
emails when you do uninstall it?
I suspect that your question was theoretical instead.


It's somewhat different in that the data is not owned by one user, but 
there are clear parallels.  The thing to be careful about here, IMO, is 
not only to preserve the data, but the associated files that allow

(reasonably-easy) access to that data.  (It's no good preserving the OSD
filestore if the keys, monmap, or osdmap are gone or hard to recover.)


- remove /etc/ceph if it has been untouched

  This is an other case. dpkg itself handle package files here, called
conffiles. I should check the method (md5sum and/or sha1 variants) used
for the checksum on these files. On upgrade it's used not to overwrite
local changes by the user. It may worth to read a bit more about it[1]
from Raphaël Hertzog. He is the co-author of Debian Administrator's
handbook[2] BTW.


Excellent reference; thanks for the pointer.


  On purge dpkg will remove the package conffiles no matter what. It
won't check if those were changed or not. You may not mark the files
under /etc as conffiles, but then you'll lose the mentioned
merge logic on upgrades; dpkg will just overwrite those.
  In short, files under /var/lib/ceph are the only candidates for
in-package checksumming. How many files under there that essential for
the packages?

Laszlo/GCS
[1] 
http://raphaelhertzog.com/2010/09/21/debian-conffile-configuration-file-managed-by-dpkg/
[2] http://debian-handbook.info/


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: deb/rpm package purge

[Dan Mick]

Is there a way out if you remove the keyrings?

On 03/19/2013 01:05 PM, Mark Nelson wrote:



On 03/19/2013 02:48 PM, Sage Weil wrote:

Should the package purge remove /var/lib/ceph/* (potential mon data, osd
data) and/or /var/log/ceph/* (logs)?  Right now it does, but mysql, for
example, leaves /var/lib/mysql where it is (not sure about logs).


I'm definitely for leaving mon/osd data in place.  Those files are
created at cluster creation time, not when the packages are installed.
They may have been created by a totally different installation of Ceph
than the packaged version.

What's worse is that you can't get the files back simply by reinstalling
the package.  With the way things currently are, a package purge will
effectively permanently destroy your cluster.  Purge should get rid of
configuration files, but I don't think it should destroy user data which
is what it effectively is doing now.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Gateway quick start

[Dan Mick]

That depends on what functionality you want to test...

On 03/11/2013 04:03 AM, waed Albataineh wrote:


Hello there,
For the quick start of Ceph, do i need to continue the RESTful
Gateway quick start ??
even when i will just be testing the basic functions of Ceph!

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: librbd bug?

[Dan Mick]

On 03/07/2013 02:16 AM, Wolfgang Hennerbichler wrote:

Hi,

I've a libvirt-VM that gets format 2 rbd-childs 'fed' by the superhost.
It crashed recently with this in the logs:

osdc/ObjectCacher.cc: In function 'void
ObjectCacher::bh_write_commit(int64_t, sobject_t, loff_t, uint64_t,
tid_t, int)' thread 7f0cab5fd700 time 2013-03-01 22:02:37.374410
osdc/ObjectCacher.cc: 834: FAILED assert(ob->last_commit_tid < tid)
  ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
  1: (ObjectCacher::bh_write_commit(long, sobject_t, long, unsigned long,
unsigned long, int)+0xd68) [0x7f0d087cda28]
  2: (ObjectCacher::C_WriteCommit::finish(int)+0x6b) [0x7f0d087d460b]
  3: (Context::complete(int)+0xa) [0x7f0d0878c9fa]
  4: (librbd::C_Request::finish(int)+0x85) [0x7f0d087bc325]
  5: (Context::complete(int)+0xa) [0x7f0d0878c9fa]
  6: (librbd::rados_req_cb(void*, void*)+0x47) [0x7f0d087a1387]
  7: (librados::C_AioSafe::finish(int)+0x1d) [0x7f0d07b5834d]
  8: (Finisher::finisher_thread_entry()+0x1c0) [0x7f0d07bc20d0]
  9: (()+0x7e9a) [0x7f0d0546be9a]
  10: (clone()+0x6d) [0x7f0d05198cbd]
  NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'

Any clue why that happened?



This looks like

http://tracker.ceph.com/issues/4271

--
Dan Mick, Filesystem Engineering
Inktank Storage, Inc.   http://inktank.com
Ceph docs: http://ceph.com/docs
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Usable Space

[Dan Mick]

You're aware of the just-added ceph df?  I don't know it well enough to know if 
it's a solution, but it's in that space...

On Mar 6, 2013, at 6:48 AM, Patrick McGarry  wrote:

> Proxy-ing this in for a user I had a discussion with on irc this morning:
> 
> The question is "is there a way to display usable space based on
> replication level?"
> 
> Ultimately what would be nice is to see something like the following:
> 
> ---
> $: sudo ceph --usable-space
> 
> Total Space: X / Y  ||  Usable Space: A / B
> 
> By Pools:
> rbd -- J / K
> foo -- F / G
> bar -- H / I
> baz -- C / D
> 
> -
> 
> Would it be possible to add this in at some point? Seems like a great
> addition to go with some of the other 'usability enhancements' that
> are planned.  Or would this get computationally sticky based on having
> many pools with different replication levels?
> 
> 
> Best Regards,
> 
> -- 
> Patrick McGarry
> Director, Community
> Inktank
> 
> http://ceph.com  ||  http://inktank.com
> @scuttlemonkey || @ceph || @inktank
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph mkfs failed

[Dan Mick]

leveldb is required.  What filesystem are you using that doesn't support 
mmap?


On 02/07/2013 02:11 PM, sheng qiu wrote:

Is it possible comment the leveldb setup codes during mkfs call path?
i find the problem is because i am trying to use a bunch of memory as
the OSD, when the leveldb mmap() the file on OSD, it failed, as the
file actually located on memory.

Thanks,
Sheng

On Thu, Feb 7, 2013 at 2:53 PM, Gregory Farnum  wrote:

On Thu, Feb 7, 2013 at 12:42 PM, sheng qiu  wrote:

Hi Dan,

thanks for your reply.

after some code tracking, i found it failed at this point :
in file leveldb/db/db_impl.cc  --> NewDB()

log::Writer log(file);
 std::string record;
 new_db.EncodeTo(&record);
 s = log.AddRecord(record);
 if (s.ok()) {
   fprintf(test, "NewDB: 2\n");
   s = file->Close();
 }else
   fprintf(test, "NewDB: 2.5\n");

the log.AddRecord return s which is not ok().

can you provide some hint why it fails?  i am reading the AddRecord()
function now.


LevelDB is a generic library which we don't develop. My understanding
is that it's expected to work on any POSIX-compliant filesystem, but
you can check out the docs
(http://leveldb.googlecode.com/svn/trunk/doc/index.html) or source
code for more info.
-Greg





--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph mkfs failed

[Dan Mick]

On 02/07/2013 09:59 AM, sheng qiu wrote:

Hi,

i am trying to port a local file system other than ext3,ext4,btrfs to
manage the OSD. There are some reasons for us to do it.
the mkcephfs stopped at this point:

2013-02-07 11:39:32.009889 7effe43d17a0 -1 filestore(/mnt/osd.0) mkfs
failed to create leveldb: IO error:
/mnt/osd.0/current/omap/MANIFEST-01: Invalid argument
2013-02-07 11:39:32.009907 7effe43d17a0 -1 OSD::mkfs: FileStore::mkfs
failed with error -1
2013-02-07 11:39:32.009949 7effe43d17a0 -1  ** ERROR: error creating
empty object store in /mnt/osd.0: (1) Operation not permitted


Does your ssh user have permission to create dirs in /mnt?
(I'll grant you it looks like you're doing root, but, still)


failed: 'ssh root@mon-mds /usr/local/sbin/mkcephfs -d
/tmp/mkfs.ceph.2686 --init-daemon osd.0'


You could certainly try this command on the failing system as root,
and look with strace to see what system call is actually failing.



look into the code, it failed at os/FileStore.cc -->  leveldb::Status
status = leveldb::DB::Open(options, omap_dir, &db);

Does anyone have any suggestions?

Thanks,
Sheng


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CEPHFS mount error !!!

[Dan Mick]

On 02/07/2013 03:05 AM, femi anjorin wrote:

Hi ... I am now testing cephfs on an ubuntu client before going back to centos.

A quick question about this command:
mount -t ceph 192.168.0.1:6789:/ /mnt/mycephfs -o
name=admin,secretfile=/etc/ceph/admin.secret ?
mount -t ceph 192.168.0.1:6789:/ /mnt/mycephfs -o
name=ceph,secretfile=/etc/ceph/ceph.keyring ?

Does it need admin.secret  or can i use ceph.keyring to mount with
cephfs file system?


It needs the unadorned base64 key; it doesn't parse keyring files.
You can either supply the key on the commandline or in the file
(which is a good idea to keep the key string out of your command line
and command history).


Just a note: At the initial setup of the cluster i used:
mkcephfs -c /etc/ceph/ceph.conf --mkfs -a -k  /etc/ceph/ceph.keyring.

So i only have ceph.keyring NOT admin.secret 


Right.  admin.secret is a file containing only the key string.  You
can create that file with your own editing, or you can use
ceph-authtool --name client.admin  --print-key


if i do:

#mount -t ceph 172.16.0.25:6789:/ /mnt/mycephfs
mount error 22 = Invalid argument
#dmesg
[ 7182.027665] libceph: client0 fsid 31c4d83e-cb0a-4fed-ab6f-581e8d5356f5
[ 7182.027715] libceph: no secret set (for auth_x protocol)
[ 7182.027719] libceph: error -22 on auth protocol 2 init


#mount -t ceph 172.16.0.25:6789:/ /mnt/mycephfs -o
name=ceph,secretfile=/etc/ceph/ceph.keyring
secret is not valid base64: Invalid argument.
adding ceph secret key to kernel failed: Invalid argument.
failed to parse ceph_options
#dmesg
[ 7639.377337] libceph: client0 fsid 31c4d83e-cb0a-4fed-ab6f-581e8d5356f5
[ 7639.378047] libceph: auth method 'x' error -1


Regards,
Femi







On Wed, Feb 6, 2013 at 4:29 PM, Dimitri Maziuk  wrote:

On 2/6/2013 5:54 AM, Dennis Jacobfeuerborn wrote:
...


To mount cephfs like that you need to have kernel support. As the Linux
kernel on CentOS 6.3 is version 2.6.32 and Ceph support wasn't added
until
2.6.34, you need to compile your own kernel.



The better alternative is probably to install a kernel from
http://elrepo.org/tiki/kernel-lt . "lt" stand for "long term" and should
be
fairly stable and "ml" is "mainline" which is even more current but
because
of that not quite as stable (currently 3.7.6).



I had problems booting ml on some/most (dep. on the version) our machines,
plus it's a pain to track: there's a new one every day.

I do have lt running without problems and mounting cephfs, however, I
haven't gotten around to the actual ceph testing on it yet so I can't say
anything about ceph client's performance/stability on it. ("lt" is 3.0, as
I
understand it doesn't have the latest and greatest ceph module(?))

Dimitri


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph RBD Client Filesystem Benchmarks

[Dan Mick]

Thanks for the work, Adam!

On 02/06/2013 03:18 PM, Michel, Adam K - (amichel) wrote:

I did some testing against an RBD device in my local environment to suss out 
differences between a few filesystem options. Folks in #ceph thought this data 
might be interesting to some on the list, so I'm posting a link to my 
spreadsheet here.

http://mirrors.arizona.edu/ceph-devel/oxide-rbd-benches.xlsx

My ceph cluster layout is pretty simple. I have one server node with all the 
OSDs. It's a backblaze-style pod with 40 drives, 1 OSD per drive, dual-core i3 
at 3GHz, 16GB of RAM and small SSD for system disk. It's running Ubuntu 12.04 
and attached with 10GbE to a switch in my lab. I have three mons on older Dell 
hardware, 2950s, dual quad-core E5345 Xeon, 32GB of RAM. They're running 
CentOS6 (needed support for the QL CNAs I had available) attached with 10GbE 
internal to the lab as well. I can provide ceph configuration data if anyone 
would like, but I followed the documentation pretty religiously so I expect it 
is quite vanilla. The client I used for testing is a VMware VM running Ubuntu 
12.04 in a different environment (ran out of hardware in the lab) which has a 
1GbE bottleneck in its path to the lab.

I created a 5TB rbd (it is theoretically going to end up in use in a pilot for 
a dropbox-style service for our campus, thus the rather strangely large size 
for testing) and mapped it to the client, created a GPT partition table with a 
single primary partition starting at sector 8192 (on guidance from someone in 
#ceph) and then formatted it with filesystem defaults for each of ext4, xfs and 
btrfs. I mounted with no special options in all cases.

I ran bonnie++ v1.97 with the option to skip char tests. For iozone I tested 
record sizes up to 4096 on default max file size of 512M. I've generated all 
the standard 3d charts for the iozone results in their respective sheets to the 
right of their matching data tables.

I make no particular claims to have done this as well as possible and would 
appreciate any feedback from the list on any kind of testing rubric that might 
generate data of more use to the community at large. This particular round of 
testing was mostly for my own edification.

Hopefully some of you find this useful!

Cheers,
Adam

--
Adam Michel
Systems Administrator
UITS, University of Arizona
amic...@arizona.edu
5206262189


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rest mgmt api

[Dan Mick]

On 02/06/2013 12:14 PM, Sage Weil wrote:

On Wed, 6 Feb 2013, Dimitri Maziuk wrote:

On 02/06/2013 01:34 PM, Sage Weil wrote:


I think the one caveat here is that having a single registry for commands
in the monitor means that commands can come in two flavors: vector
(cli) and URL (presumably in json form).  But a single command
dispatch/registry framework will make that distinction pretty simple...


Any reason you can't have your CLI json-encode the commands (or,
conversely, your cgi/wsgi/php/servlet URL handler decode them into
vector) before passing them on to the monitor?


We can, but they won't necessarily look the same, because it is unlikely
we can make a sane 1:1 translation of the CLI to REST that makes sense,
and it would be nice to avoid baking knowledge about the individual
commands into the client side.

  ceph osd pool create  
vs
  /osd/pool/?op=create&poolname=foo&numpgs=bar

or whatever.  I know next to nothing about REST API design best practices,
but I'm guessing it doesn't look like a CLI.

sage


Well we could easily package, say, "mon getmap" into a json
vector-of-strings for transmission as the http payload of the PUT
request, *or* encode them as a simple "s1=mon&s2=getmap", but again, the
data format is so simple that I'm not sure it buys much. We definitely
want all the interpretation centralized in the daemons, I think, so this
is just string marshalling however we choose to do it.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rest mgmt api

[Dan Mick]

My take on this is to keep the current behaviour (client issues a
command and the monitor handles it as it sees fit), but all
communication should be done in json, either to or from the monitors.
This would allow us to provide more information on each result, getting
rid of all the annoying format on the reply messages and simplify a
great deal of code on the monitor end by removing the silly need of
returning on either plain-text or json.  We would then let the
client-side libraries deal with converting it to whichever format the
user wants (plain-text, xml, w/e).  As for new commands on the monitor
that are not present on the library, replies to said commands could then
be presented just in json, or we could come up with a standardized way
to always convert json into a human-readable format/any other format.

This would however mean to be able to parse json on the monitors (which
we do not currently do, although we do produce json output).  I can't
say I have strong feeling for the client->monitor communication to be
done in json, but for sake of coherency I do think it would be best.


Sage and I talked briefly about this yesterday; my take is that the 
input doesn't do very much mechanical validation at all; most of it is 
semantic and based on partial results (i.e. "you can't ask for that from 
this object" or "that object you want to operate on doesn't exist"), so 
I'm not sure we win much from JSON input.  There are a few places where 
numbers need to be validated but that's about it; the input is not 
complex, it's pretty much just a list of words for the most part.


but there's also no reason to rule it out for future 
perhaps-more-complex command inputs.


I think having the commands be self-describing and table-driven will 
solve a whole lot of ugliness, and one of the self-description vectors 
can be "this parameter should be validated in this standard way".

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rest mgmt api

[Dan Mick]

On 02/06/2013 10:15 AM, Yehuda Sadeh wrote:

On Wed, Feb 6, 2013 at 9:25 AM, Sage Weil  wrote:

One of the goals for cuttlefish is to improvement manageability of the
system.  This will invovle both cleaning up the CLI and adding a REST API
to do everything the CLI current does.  There are a few implementation
choices to make.

Currenty the 'ceph' tool has a bunch of code to send messages to the
monitor and wait for replies.  This is 90% of what users currently can do.
For the most part, the commands are interpreted by the monitor.  A small
subset of commands (ceph tell ..., ceph pg  ...) will send commands
directory to OSDs.


There are two main options for a REST endpoint that we've discussed so
far:

1- Wrap the above in a clean library (probably integrating most of the
code into Objecter/MonClient.. see wip-monc for a start on this).  Wrap
libcephadmin in python and make a simple HTTP/REST front-end.  Users would
deploy mgmt endpoints in addition to/alongside monitors and everything
else.  If they want the rest api at all.

2- Embed a web server in the ceph-mon daemons, and push the current admin
'client' functionality there.  Come up with some basic authentication so
that this doesn't break the current security model.


I'm in favor of the more modular and flexible approach, #1.


The more I think about it, the more I am too.  I can imagine admin 
flexibility being key: not just redundancy/failover, but also access 
perms, and the more independent the REST channel is from the normal auth 
channels, the better, I think.



Note that neither of these solves the HA issue directly; HTTP is a
client/server protocol, so whoever is using the API can specify only one
server endpoint.  If it is a monitor, they'll need to be prepare to fail
over to another in their code, or set up a load balancer.  Same goes for
the restful endpoint, if it fails.  The difference is that a single
endpoint can proxy to whichever monitors are in quorum, so a much smaller
set of errors (endpoint machine crash, buggy endpoint) affect availability
of the API.


Right. However, that's really an orthogonal issue. It'll be easier
scaling the HTTP endpoints if they're decoupled from the monitors.


and easier to rewrite/redeploy, and authenticate, etc.


The somewhat orthogonal question is how to clean up the CLI usage,
parsing, and get equivalence in the new REST API.

One option is to create a basic framework in the monitor so that there is
a table of api commands.  The 'parsing' would be regularized and validated
in a generic way.  The rest endpoint would pass the URI through in a
generic way (sanitized json?) that can be matched against the same table.

Another option is to support a single set of commands on the monitor side,
and do the REST->CLI or CLI->REST or CLI,REST->json translation on the
client side.  The command registry framework above would live in the CLI
utility and REST endpoints instead (or libcephadmin).  This means that the
monitor code is simpler, but also means that the libcephadmin or ceph tool
and REST endpoint need to be running the latest code to be able to send
the latest commands to the monitor.  It also means multiple places where
the command set is defined (mon and endpoint and cli).


I'd rather keep the clients dumb, not involve them with the actual
configuration logic. Will make our lives easier in the long run.


Yes, and I really really want to simplify/table-ify the command parsing 
code.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CEPHFS mount error !!!

[Dan Mick]

Yes; as Martin said last night, you don't have the ceph module.
Did you build your own kernel?

See
http://ceph.com/docs/master/install/os-recommendations/#linux-kernel

On 02/05/2013 09:37 PM, femi anjorin wrote:

Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS 6.3

Pls can somebody help ... This command not working on CentOS 6.3 

mount -t ceph 172.16.0.25:6789:/ /mnt/mycephfs -o
name=admin,secret=AQCqnw9RmBPpOBAAeJjkIgKGYvyRGlekTpUPog==

  FATAL: Module ceph not found.
  mount.ceph: modprobe failed, exit status 1
  mount error: ceph filesystem not supported by the system


Regards,
Femi





On Tue, Feb 5, 2013 at 1:49 PM, femi anjorin  wrote:


Hi ...

Thanks. i set --debug ms = 0. The result is HEALTH_OK ...but i get an
error when trying to setup an client access to the cluster CEPHFS!!!

I tried setting up another server which should act as client..
- i install ceph on it.
- got the configuration file from the cluster servers to the new
server ...  /etc/ceph/ceph.conf
-i did ...  mkdir /mnt/mycephfs
-i copied the key from ceph.keyring and used it in the command below
-i try to run this command: mount -t ceph 172.16.0.25:6789:/
/mnt/mycephfs -o
name=admin,secret=AQCqnw9RmBPpOBAAeJjkIgKGYvyRGlekTpUPog==
Here is the result i got:::

[root@testclient]# mount -t ceph 172.16.0.25:6789:/ /mnt/mycephfs -o
name=admin,secret=AQCqnw9RmBPpOBAAeJjkIgKGYvyRGlekTpUPog==
FATAL: Module ceph not found.
mount.ceph: modprobe failed, exit status 1
mount error: ceph filesystem not supported by the system

Regards,
Femi.

On Mon, Feb 4, 2013 at 3:27 PM, Joao Eduardo Luis  wrote:

This wasn't obvious due to all the debug being outputted, but here's why
'ceph health' wasn't replying with HEALTH_OK:


On 02/04/2013 12:21 PM, femi anjorin wrote:


2013-02-04 12:56:15.818985 7f149bfff700  1 HEALTH_WARN 4987 pgs
peering; 4987 pgs stuck inactive; 5109 pgs stuck unclean



Furthermore, on your other email in which you ran 'ceph health detail', this
appear to have gone away, as it is replying with HEALTH_OK again.

You might want to set '--debug-ms 0' when you run 'ceph', or set it in your
ceph.conf, leaving it at a higher level only for daemons (i.e., under [mds],
[mon], [osd]...).  The resulting output will be clearer and more easily
understandable.

   -Joao


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Trigger to create PGs ?

[Dan Mick]

(rbd was set to 2, which meant it didn't match, which I'm sure is what 
Sage meant.  Just correcting the record for those scoring at home.)



On 02/04/2013 06:36 PM, Yasuhiro Ohara wrote:


Thanks Sage, it instantly fixed the problem.
:)

regards,
Yasu

From: Sage Weil 
Subject: Re: Trigger to create PGs ?
Date: Mon, 4 Feb 2013 18:19:46 -0800 (PST)
Message-ID: 


All of the stuck pgs are in pool 2.  My guess is that that pool is
referncing a broken crush rule.

Your CRUSH map has min and max_size of 3 for all of the rules; it should
be min 1 and max 10.  Probably the RBD pool is set to 3 replicas, which
means it matches no existing CRUSH rule and you get no OSDs.  You can fix
the CRUSH rules (that is a good idea anyway), or also change the pool 2
(rbd) to 3x replication:

ceph osd pool set rbd size 3

sage


On Mon, 4 Feb 2013, Yasuhiro Ohara wrote:


Umm, I mean, my system stuck like:

health HEALTH_WARN 1088 pgs stuck inactive; 1088 pgs stuck unclean
monmap e1: 5 mons at 
{0=128.114.52.59:6789/0,1=128.114.52.67:6789/0,2=128.114.52.68:6789/0,3=128.114.52.69:6789/0,4=128.114.52.70:6789/0},
 election epoch 72, quorum 0,1,2,3,4 0,1,2,3,4
osdmap e295: 16 osds: 16 up, 16 in
 pgmap v83184: 3264 pgs: 1088 creating, 2176 active+clean; 672 GB data, 
2042 GB used, 5107 GB / 7452 GB avail
mdsmap e20: 1/1/1 up {0=1=up:active}, 4 up:standby

and I am asking how to bring it to HEALTHY state.

regards,
Yasu

From: Yasuhiro Ohara 
Subject: Trigger to create PGs ?
Date: Mon, 04 Feb 2013 14:50:38 -0800 (PST)
Message-ID: <20130204.145038.210467743.y...@soe.ucsc.edu>



Hi,

I happened to have an incorrect crush map in the start-up of
my system, but even after fixing it manually, the PGs do not
seem to be created properly. Is there any way to trigger the
system to start creating the PGs again ?

Here's my configurations:
ceph.conf: http://pastebin.com/EwwdQrf9
crush map: http://pastebin.com/UYNFvvQx
ceph osd tree: http://pastebin.com/u2Z4Hppn
ceph pg dump: http://pastebin.com/JfE146WJ

FYI, in the first, I had mistakenly osd.0 in all host clauses.

regards,
Yasu


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Trigger to create PGs ?

[Dan Mick]

I see now that there are a bunch of PGs in creating state; sorry for 
missing that.


I think you might try "ceph pg send_pg_creates" to kick the cluster in 
the head and get it to create those PGs.


On 02/04/2013 04:22 PM, Yasuhiro Ohara wrote:


Dan,

In fact it's been more than a week since it is in the status.
ceph -w can show us the other progress (like backfilling on the
down/up osds), but has not shown any progress on the 'creating'.

regards,
Yasu

From: Dan Mick 
Subject: Re: Trigger to create PGs ?
Date: Mon, 04 Feb 2013 15:53:25 -0800
Message-ID: <511049f5@inktank.com>




On 02/04/2013 03:26 PM, Yasuhiro Ohara wrote:

3264 pgs: 1088 creating, 2176 active+clean


This means, I believe, that the cluster has healed 2176 of 3264 PGs,
and is working on the remaining 1088.  You can use 'ceph -w' to
observe the progress, but I think your cluster is backfilling the
newly-configured OSDs as it should be.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Trigger to create PGs ?

[Dan Mick]

On 02/04/2013 03:26 PM, Yasuhiro Ohara wrote:

3264 pgs: 1088 creating, 2176 active+clean


This means, I believe, that the cluster has healed 2176 of 3264 PGs, and 
is working on the remaining 1088.  You can use 'ceph -w' to observe the 
progress, but I think your cluster is backfilling the newly-configured 
OSDs as it should be.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [0.48.3] OSD memory leak when scrubbing

[Dan Mick]

...and/or do you have the corepath set interestingly, or one of the 
core-trapping mechanisms turned on?


On 02/04/2013 11:29 AM, Sage Weil wrote:

On Mon, 4 Feb 2013, S?bastien Han wrote:

Hum just tried several times on my test cluster and I can't get any
core dump. Does Ceph commit suicide or something? Is it expected
behavior?


SIGSEGV should trigger the usual path that dumps a stack trace and then
dumps core.  Was your ulimit -c set before the daemon was started?

sage




--
Regards,
S?bastien Han.


On Sun, Feb 3, 2013 at 10:03 PM, S?bastien Han  wrote:

Hi Lo?c,

Thanks for bringing our discussion on the ML. I'll check that tomorrow :-).

Cheer
--
Regards,
S?bastien Han.


On Sun, Feb 3, 2013 at 10:01 PM, S?bastien Han  wrote:

Hi Lo?c,

Thanks for bringing our discussion on the ML. I'll check that tomorrow :-).

Cheers

--
Regards,
S?bastien Han.


On Sun, Feb 3, 2013 at 7:17 PM, Loic Dachary  wrote:


Hi,

As discussed during FOSDEM, the script you wrote to kill the OSD when it
grows too much could be amended to core dump instead of just being killed &
restarted. The binary + core could probably be used to figure out where the
leak is.

You should make sure the OSD current working directory is in a file system
with enough free disk space to accomodate for the dump and set

ulimit -c unlimited

before running it ( your system default is probably ulimit -c 0 which
inhibits core dumps ). When you detect that OSD grows too much kill it with

kill -SEGV $pid

and upload the core found in the working directory, together with the
binary in a public place. If the osd binary is compiled with -g but without
changing the -O settings, you should have a larger binary file but no
negative impact on performances. Forensics analysis will be made a lot
easier with the debugging symbols.

My 2cts

On 01/31/2013 08:57 PM, Sage Weil wrote:

On Thu, 31 Jan 2013, Sylvain Munaut wrote:

Hi,

I disabled scrubbing using


ceph osd tell \* injectargs '--osd-scrub-min-interval 100'
ceph osd tell \* injectargs '--osd-scrub-max-interval 1000'


and the leak seems to be gone.

See the graph at  http://i.imgur.com/A0KmVot.png  with the OSD memory
for the 12 osd processes over the last 3.5 days.
Memory was rising every 24h. I did the change yesterday around 13h00
and OSDs stopped growing. OSD memory even seems to go down slowly by
small blocks.

Of course I assume disabling scrubbing is not a long term solution and
I should re-enable it ... (how do I do that btw ? what were the
default values for those parameters)


It depends on the exact commit you're on.  You can see the defaults if
you
do

  ceph-osd --show-config | grep osd_scrub

Thanks for testing this... I have a few other ideas to try to reproduce.

sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
Lo?c Dachary, Artisan Logiciel Libre







--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph client

[Dan Mick]

You might want to ask a more-specific question.

On 01/31/2013 08:24 AM, charles L wrote:



I need some help and guide on compiling ceph client on Eclipse..
  --

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/6] fix build (ceph.spec)

[Dan Mick]

On 01/30/2013 09:08 PM, Dan Mick wrote:



On 01/30/2013 11:55 AM, Sage Weil wrote:


I think the end goal is to build an rbd-fuse package, just like
ceph-fuse.
I'm not sure it matters how mature the tool is before it goes into the
package; as long as it is separate, people can not install it, and the
distros can keep it out of their supported repos as they like.


I like that plan a lot better.


and the major reason it wasn't yet done is that there are no manpages, 
tests, etc., but I wanted this to be available to experimenters.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/6] fix build (ceph.spec)

[Dan Mick]

On 01/30/2013 11:55 AM, Sage Weil wrote:


I think the end goal is to build an rbd-fuse package, just like ceph-fuse.
I'm not sure it matters how mature the tool is before it goes into the
package; as long as it is separate, people can not install it, and the
distros can keep it out of their supported repos as they like.


I like that plan a lot better.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Testing nonsubscriber posts

[Dan Mick]

ignore
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] fix some compiler warnings

[Dan Mick]

Sage merged these into master.  Thanks!

On 01/27/2013 12:57 PM, Danny Al-Gaaf wrote:

Attached two patches to fix some compiler warnings.

Danny Al-Gaaf (2):
   utime: fix narrowing conversion compiler warning in sleep()
   rbd: don't ignore return value of system()

  src/include/utime.h |  2 +-
  src/rbd.cc  | 36 ++--
  2 files changed, 31 insertions(+), 7 deletions(-)


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] fix some rbd-fuse related issues

[Dan Mick]

Actually Sage merged them into master.  Thanks again.

On 01/28/2013 09:45 AM, Dan Mick wrote:

Thanks Danny, I'll look at these today.

On Jan 28, 2013, at 7:33 AM, Danny Al-Gaaf  wrote:


Here three patches to fix some issues with the new rbd-fuse
code and an issues with the fuse handling in configure.

Danny Al-Gaaf (3):
  configure: fix check for fuse_getgroups()
  rbd-fuse: fix usage of conn->want
  rbd-fuse: fix printf format for off_t and size_t

configure.ac|  8 
src/rbd_fuse/rbd-fuse.c | 12 +++-
2 files changed, 11 insertions(+), 9 deletions(-)

--
1.8.1.1


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] fix some compiler warnings

[Dan Mick]

I'd just noticed utime on my laptop 32-bit build and was trying to figure out 
why our 32-bit nightly didn't see it. And Greg had seen the system build 
problem where I didn't, and I was isolating differences there as well. 

I purposely didn't spend time on the system() error handling because I was 
thinking of those calls as best-effort, if they fail the map will likely fail 
anyway, but there's no harm in handling errors, particularly if it'll shit the 
compiler up :)

On Jan 27, 2013, at 12:57 PM, Danny Al-Gaaf  wrote:

> Attached two patches to fix some compiler warnings.
> 
> Danny Al-Gaaf (2):
>  utime: fix narrowing conversion compiler warning in sleep()
>  rbd: don't ignore return value of system()
> 
> src/include/utime.h |  2 +-
> src/rbd.cc  | 36 ++--
> 2 files changed, 31 insertions(+), 7 deletions(-)
> 
> -- 
> 1.8.1.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/3] fix some rbd-fuse related issues

[Dan Mick]

Thanks Danny, I'll look at these today. 

On Jan 28, 2013, at 7:33 AM, Danny Al-Gaaf  wrote:

> Here three patches to fix some issues with the new rbd-fuse
> code and an issues with the fuse handling in configure.
> 
> Danny Al-Gaaf (3):
>  configure: fix check for fuse_getgroups()
>  rbd-fuse: fix usage of conn->want
>  rbd-fuse: fix printf format for off_t and size_t
> 
> configure.ac|  8 
> src/rbd_fuse/rbd-fuse.c | 12 +++-
> 2 files changed, 11 insertions(+), 9 deletions(-)
> 
> -- 
> 1.8.1.1
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RadosGW performance and disk space usage

[Dan Mick]

On 1/25/2013 9:35 PM, Dan Mick wrote:



If the S3 API is not well suited to my scenario, then my effort should
be better directed to porting or writing a native ceph client for
Windows. I just need an API to read and write/append blocks to files.
Any comments are really appreciated.


Hopefully someone with more windows experience will give you better
info/advice than I can.


The python bindings should Just Work(tm).  Just a thought.


...but of course, as Sam points out, they rely on the C++ libs.  Duh. 
Sorry to mislead.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RadosGW performance and disk space usage

[Dan Mick]

If the S3 API is not well suited to my scenario, then my effort should
be better directed to porting or writing a native ceph client for
Windows. I just need an API to read and write/append blocks to files.
Any comments are really appreciated.


Hopefully someone with more windows experience will give you better
info/advice than I can.


The python bindings should Just Work(tm).  Just a thought.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: recoverying from 95% full osd

[Dan Mick]

what is pool 2 (rbd) for? looks like it's absolutely empty.


by default it's for rbd images (see the rbd command etc.).  It being 
empty or not has no effect on the other pools.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Understanding Ceph

[Dan Mick]

You'd think that only one [osd] section in ceph.conf implies
nrep = 1, though. (And then you can go on adding OSDs and changing nrep
accordingly -- that was my plan.)



Yeah; it's probably mostly just that one-OSD configurations are so 
uncommon that we never special-cased that small user set.  Also, you can 
run with a cluster in that state forever (well, until that one OSD dies 
at least); I do that regularly with the default vstart.sh local test cluster


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Understanding Ceph

[Dan Mick]

On 01/20/2013 08:32 AM, Dimitri Maziuk wrote:

On 1/19/2013 11:13 AM, Sage Weil wrote:

If you want to use the kernel client(s), that is true: there are no 
plans
to backport the client code to the ancient RHEL kernels. Nothing 
prevents

you from running the server side, though, or the userland clients
(ceph-fuse, librbd, qemu/KVM, radosgw, etc.)


mkcephfs form 5-minute start fails without rbd.ko. I already reported 
that.


Dima


This is an apparently-unique problem, and we'd love to see details.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Understanding Ceph

[Dan Mick]

On 01/24/2013 07:28 AM, Dimitri Maziuk wrote:

On 1/24/2013 8:20 AM, Sam Lang wrote:


Yep it means that you only have one OSD with replication level of 2.
If you had a rep level of 3, you would see degraded (66.667%). If you
just want to make the message go away (for testing purposes), you can
set the rep level to 1
(http://ceph.com/w/index.php?title=Adjusting_replication_level&redirect=no). 



OK, thanks Sam and Dino -- I kinda suspected that but didn't find any 
docs.


This looks like it's not adjustable via ceph.conf, I can only do it at 
runtime, correct?


or you could just add another OSD.


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: /etc/init.d/ceph bug for multi-host when using -a option

[Dan Mick]

On 01/22/2013 11:18 PM, Chen, Xiaoxi wrote:

Hi List,
  Here is part of /etc/init.d/ceph script:
case "$command" in
 start)
 # Increase max_open_files, if the configuration calls for it.
 get_conf max_open_files "8192" "max open files"
 if [ $max_open_files != "0" ]; then
 # Note: Don't try to do math with these numbers, because POSIX 
shells
 # can't do 64-bit math (natively). Just treat them as strings.
 cur=`ulimit -n`xjkk
 if [ "x$max_open_files" != "x$cur" ]; then
Line253:  ulimit -n $max_open_files
 fi
 fi

   When using with -a option, for **remote** osd , this script also run 
ulimit on **local**, and results in ulimit -n didn't change in remote nodes. I 
think the Line 253 shoud use do_cmd instead of run directly,also line 251.


I think you're right.  I opened http://tracker.newdream.net/issues/3900 
to track this.


Here is the output of local osd daemon's limits:
root@ceph-4:~# cat /proc/13131/limits  | grep file
Max file size unlimitedunlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max open files131072   131072   files
Max file locksunlimitedunlimitedlocks

Here is a remote one:
root@snb-15:~# cat /proc/23709/limits | grep file
Max file size unlimitedunlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max open files1024 4096 files
Max file locksunlimitedunlimitedlocks




Xiaoxi
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/3] rbd: check for overflow in rbd_get_num_segments()

[Dan Mick]

Reviewed-by: Dan Mick 

On 01/22/2013 01:58 PM, Alex Elder wrote:

The return type of rbd_get_num_segments() is int, but the values it
operates on are u64.  Although it's not likely, there's no guarantee
the result won't exceed what can be respresented in an int.  The
function is already designed to return -ERANGE on error, so just add
this possible overflow as another reason to return that.

Signed-off-by: Alex Elder 
---
  drivers/block/rbd.c |7 ++-
  1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 4ed0741..58d01e3 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -820,6 +820,7 @@ static int rbd_get_num_segments(struct
rbd_image_header *header,
  {
u64 start_seg;
u64 end_seg;
+   u64 result;

if (!len)
return 0;
@@ -829,7 +830,11 @@ static int rbd_get_num_segments(struct
rbd_image_header *header,
start_seg = ofs >> header->obj_order;
end_seg = (ofs + len - 1) >> header->obj_order;

-   return end_seg - start_seg + 1;
+   result = end_seg - start_seg + 1;
+   if (result > (u64) INT_MAX)
+   return -ERANGE;
+
+   return (int) result;
  }

  /*


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] rbd: small changes

[Dan Mick]

Reviewed-by: Dan Mick 

On 01/22/2013 01:57 PM, Alex Elder wrote:

A few very minor changes to the rbd code:
 - RBD_MAX_OPT_LEN is unused, so get rid of it
 - Consolidate rbd options definitions
 - Make rbd_segment_name() return pointer to const char

Signed-off-by: Alex Elder 
---
  drivers/block/rbd.c |   17 -
  1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 007b726..4ed0741 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -69,7 +69,6 @@
(NAME_MAX - (sizeof (RBD_SNAP_DEV_NAME_PREFIX) - 1))

  #define RBD_MAX_SNAP_COUNT510 /* allows max snapc to fit in 4KB */
-#define RBD_MAX_OPT_LEN1024

  #define RBD_SNAP_HEAD_NAME"-"

@@ -96,8 +95,6 @@
  #define DEV_NAME_LEN  32
  #define MAX_INT_FORMAT_WIDTH  ((5 * sizeof (int)) / 2 + 1)

-#define RBD_READ_ONLY_DEFAULT  false
-
  /*
   * block device image metadata (in-memory version)
   */
@@ -156,10 +153,6 @@ struct rbd_spec {
struct kref kref;
  };

-struct rbd_options {
-   boolread_only;
-};
-
  /*
   * an instance of the client.  multiple devices may share an rbd client.
   */
@@ -475,6 +468,12 @@ static match_table_t rbd_opts_tokens = {
{-1, NULL}
  };

+struct rbd_options {
+   boolread_only;
+};
+
+#define RBD_READ_ONLY_DEFAULT  false
+
  static int parse_rbd_opts_token(char *c, void *private)
  {
struct rbd_options *rbd_opts = private;
@@ -773,7 +772,7 @@ static void rbd_header_free(struct rbd_image_header
*header)
header->snapc = NULL;
  }

-static char *rbd_segment_name(struct rbd_device *rbd_dev, u64 offset)
+static const char *rbd_segment_name(struct rbd_device *rbd_dev, u64 offset)
  {
char *name;
u64 segment;
@@ -1338,7 +1337,7 @@ static int rbd_do_op(struct request *rq,
 struct rbd_req_coll *coll,
 int coll_index)
  {
-   char *seg_name;
+   const char *seg_name;
u64 seg_ofs;
u64 seg_len;
int ret;


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ssh passwords

[Dan Mick]

The '-a/--allhosts' parameter is to spread the command across the 
cluster...that is, service ceph -a start will start across the cluster.


On 01/22/2013 01:01 PM, Xing Lin wrote:

I like the current approach. I think it is more convenient to run
commands once at one host to do all the setup work. When the first time
I deployed a ceph cluster with 4 hosts, I thought 'service ceph start'
would start the whole ceph cluster. But as it turns out, it only starts
local osd, mon processes. So, currently, I am using polysh to run the
same commands at all hosts (mostly, to restart ceph service before every
measurement.). Thanks.

Xing

On 01/22/2013 12:35 PM, Neil Levine wrote:

Out of interest, would people prefer that the Ceph deployment script
didn't try to handle server-server file copy and just did the local
setup only, or is it useful that it tries to be a mini-config
management tool at the same time?

Neil


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: questions on networks and hardware

[Dan Mick]

On 01/21/2013 12:19 AM, Gandalf Corvotempesta wrote:

2013/1/21 Gregory Farnum :

I'm not quite sure what you mean…the use of the "cluster network" and "public 
network" are really just intended as conveniences for people with multiple NICs on their box. 
There's nothing preventing you from running everything on the same network…(and more specifically, 
from different speed grades to different boxes, but keeping them all on the same network).


I mean using two cluster network and two pubic networks. 4 NICs in total.
Our cluster network will be 10GBe or Infiniband DDR/QDR, making a
fully redundany cluster network (two nics on each server, two
switches) will double our costs.


Well, again, you needn't use two different networks for Ceph; that 
capability is there so that you *can* only use the faster gear for the 
cluster traffic if you want to limit costs but still segregate traffic. 
 If you're just after a fully-redundant network (by which I assume you 
also mean fully-redundant fabric, power, etc.), there's no reason it 
can't all be fast gear.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: radosgw boto issue

[Dan Mick]

Derek, see

http://ceph.com/docs/master/radosgw/manual-install/

for details.

On 01/17/2013 04:01 PM, Yehuda Sadeh wrote:

The vanilla mod_fastcgi doesn't handle 100-continue at all. We've
created a modified version that does handle it.

Yehuda

On Thu, Jan 17, 2013 at 3:26 PM, Derek Yarnell  wrote:

On 1/17/13 4:47 PM, Yehuda Sadeh wrote:

rgw print continue = false


Hi Yehuda,

Thanks once again for a very helpful answer.  I am running apache with
mod_fastcgi.  Is this a deficiency in mod_fastcgi/apache that doesn't
deal with the 100-continue correctly?

Thanks,
derek

--
---
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: understanding cephx

[Dan Mick]

Glad you got it working!  Sorry for missing your original message..

On 01/16/2013 01:47 AM, Michael Menge wrote:

Quoting Michael Menge :



So i guess, by trying to get some more informations I somehow
manged to delete the mon. key. I was unable the retieve the history
because of the full filesystem.

So I tried to use "ceph auth" and ceph-authtool to (re-)add the mon. key
but only managed that mon.d is now too unable the authenticate.



I was able to insert the mon. key by receating the mon servers

ceph auth get mon. -o /tmp/monkey.new
ceph mon getmap -o /tmp/monmap.new
ceph-mon -i d --mkfs --monmap /tmp/monmap.new --keyring /tmp/monkey.new

Restarting the mon server didn't seem to have worked at first.
Restarting with --debug_mon 10 showed that I was too impatient
as the mon server had to update his mdsmap which took quiet some time

Regards

Michael Menge




M.MengeTel.: (49) 7071/29-70316
Universität Tübingen   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung  mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ceph-osd can`t start after crush server

[Dan Mick]

Did you have any logging on at the time?  If not, can you turn on
debug ms = 1, debug osd = 20 and try again, and collect the osd log?


On 01/15/2013 09:53 PM, aleo...@nazarianin.com wrote:

 Hello All
Ceph-osd can`t start after crush server. I use XFS for osd folder.

Best regards,
  Aleksey Leonov

/usr/bin/ceph-osd -i 1 --pid-file /var/run/ceph/osd.1.pid -c
/etc/ceph/ceph.conf -d
2013-01-16 11:20:57.994965 7f73b089e780  0 ceph version 0.56.1
(e4a541624df62ef353e754391cbbb707f54b16f7), process ceph-osd, pid 20960
starting osd.1 at :/0 osd_data /ceph/osd/osd.1 /ceph/osd/osd.1/journal
2013-01-16 11:20:58.068506 7f73b089e780  0 filestore(/ceph/osd/osd.1)
mount FIEMAP ioctl is supported and appears to work
2013-01-16 11:20:58.068538 7f73b089e780  0 filestore(/ceph/osd/osd.1)
mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-01-16 11:20:58.068940 7f73b089e780  0 filestore(/ceph/osd/osd.1)
mount did NOT detect btrfs
2013-01-16 11:20:58.081290 7f73b089e780  0 filestore(/ceph/osd/osd.1)
mount syncfs(2) syscall fully supported (by glibc and kernel)
2013-01-16 11:20:58.081424 7f73b089e780  0 filestore(/ceph/osd/osd.1)
mount found snaps <>
2013-01-16 11:20:58.113247 7f73b089e780  0 filestore(/ceph/osd/osd.1)
mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-01-16 11:20:58.118473 7f73b089e780  1 journal _open
/ceph/osd/osd.1/journal fd 19: 1048576000 bytes, block size 4096 bytes,
directio = 1, aio = 0
2013-01-16 11:20:58.118632 7f73b089e780  1 journal _open
/ceph/osd/osd.1/journal fd 19: 1048576000 bytes, block size 4096 bytes,
directio = 1, aio = 0
2013-01-16 11:20:58.120640 7f73b089e780  1 journal close
/ceph/osd/osd.1/journal
*** Caught signal (Segmentation fault) **
  in thread 7f73b089e780
  ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
  1: /usr/bin/ceph-osd() [0x7edcc4]
  2: (()+0x10460) [0x7f73b028c460]
  3: (std::vector<__gnu_cxx::_Hashtable_node
 >*, std::allocator<__gnu_cxx::_Hashtable_node >*>
 >::_M_fill_insert(__gnu_cxx::__normal_iterator<__gnu_cxx::_Hashtable_node >**, 
std::vector<__gnu_cxx::_Hashtable_node >*, std::allocator<__gnu_cxx::_Hashtable_node >*> > >, unsigned long, __gnu_cxx::_Hashtable_node >* const&)+0x70) [0x69bdd0]
  4: (OSD::OSD(int, Messenger*, Messenger*, Messenger*, Messenger*,
MonClient*, std::string const&, std::string const&)+0xdd9) [0x67d889]
  5: (main()+0x3df4) [0x59bb84]
  6: (__libc_start_main()+0xfd) [0x7f73aebd]
  7: /usr/bin/ceph-osd() [0x597989]
Segmentation fault

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html