Re: [ceph-users] Ceph monitoring / stats and troubleshooting tools
For the second issue, I got the answer from within: http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/#finding-an-object-location. Thanks, Guang On Oct 8, 2013, at 8:43 PM, Guang wrote: > Hi ceph-users, > After walking through the operations document, I still have several questions > in terms of operation / monitoring for ceph which need you help. Thanks! > > 1. Does ceph provide build in monitoring mechanism for Rados and RadosGW? > Taking Rados for example, is it possible to monitor the health / latency / > storage on regular basis and ideally have an web UI? > > 2. One common trouble shooting requirement would be, given an object name, > how to locate the PG / OSD / physical file path for this object? Does Ceph > provide such type of utility? > > Thanks, > Guang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Same journal device for multiple OSDs?
Hello, I have a Ceph test cluster with 88 OSDs running well. In ceph.conf I found multiple OSDs that are using the same SSD block device (without a file system) for their journal: [osd.214] osd journal = /dev/fioa1 ... [osd.314] osd journal = /dev/fioa1 ... Is this a allowed configuration? Regards Andreas Friedrich -- FUJITSU Fujitsu Technology Solutions GmbH Heinz-Nixdorf-Ring 1, 33106 Paderborn, Germany Tel: +49 (5251) 525-1512 Fax: +49 (5251) 525-321512 Email: andreas.friedr...@ts.fujitsu.com Web: ts.fujitsu.com Company details: de.ts.fujitsu.com/imprint -- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Radosgw Admin REST API
Hello to all, I can't succeed in using the Admin Ops REST API for radosgw. Where can I find an example, in any language (Perl, Python, Bash) ? For instance, how to proceed to get info for user xxx ? Via cli, i do radosgw user info --uid=xxx but with the REST API ? Thanks for your answers. Alexis ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Dumpling ceph.conf looks different
Hi When I compare the /etc/ceph.conf for the latest release-dumpling and previous releases I find they are different. in the older release we had [osd],[mon],[msd] in the ceph.conf now i dont seem them.Where are these values stored now? How does ceph figure out the partitions of osd and journal Thanks Su ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Dumpling ceph.conf looks different
I too have noticed this as well when using ceph-deploy to configure ceph. >From what I can tell it just creates symlinks from the default osd location at /var/lib/ceph. Same for the journal. If it on a different device a symlink is created from the dir. Then it appears the osds are just defined in the cruchmap. This is just what I have been able to gleam from observation so someone please correct me if I am wrong. On Oct 9, 2013 7:55 AM, "su kucherova" wrote: > Hi > > When I compare the /etc/ceph.conf for the latest release-dumpling and > previous releases I find they are different. > > in the older release we had [osd],[mon],[msd] in the ceph.conf > now i dont seem them.Where are these values stored now? > How does ceph figure out the partitions of osd and journal > > Thanks > Su > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 另j
发自我的 iPhone ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] disk usage monitor database
Hi, What is the estimated storage usage for a monitor (i.e. the amount of data stored in /var/lib/ceph/mon/ceph-mon01) Currently in my starting test system it's something like 40M (du -s -h /var/lib/ceph/mon/ceph-mon01), but that will probably grow with the number of osds. Are there some numbers available? -Kees ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph stats and monitoring
Hi, Can someone share your experience with monitoring the Ceph cluster? How is going with the work mentioned here: http://wiki.ceph.com/01Planning/02Blueprints/Dumpling/ceph_stats_and_monitoring_tools Thanks, Guang___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] disk usage monitor database
On 09/10/13 13:38, Kees Bos wrote: Hi, What is the estimated storage usage for a monitor (i.e. the amount of data stored in /var/lib/ceph/mon/ceph-mon01) Currently in my starting test system it's something like 40M (du -s -h /var/lib/ceph/mon/ceph-mon01), but that will probably grow with the number of osds. Are there some numbers available? We don't have hard numbers for you unfortunately. From my experience however I can tell you that it will grow to several hundred MB to few GB. It mostly depends on your cluster size, OSD-wise, and the healthiness of your cluster (if you spend a considerable time in an unhealthy OSD state, then the monitors will keep more OSD maps that they usually do, thus increasing disk usage). As a rule of thumbs, if your monitors' stores hit double-digits GBs and keep growing, you should definitely look into what's going on and maybe drop in #ceph @ OFTC, but that has been becoming unlikely as of latest releases. -Joao -- Joao Eduardo Luis Software Engineer | http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] help
help ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] disk usage monitor database
On Wed, 2013-10-09 at 15:18 +0200, Joao Eduardo Luis wrote: > On 09/10/13 13:38, Kees Bos wrote: > > Hi, > > > > What is the estimated storage usage for a monitor (i.e. the amount of > > data stored in /var/lib/ceph/mon/ceph-mon01) > > > > Currently in my starting test system it's something like 40M (du -s > > -h /var/lib/ceph/mon/ceph-mon01), but that will probably grow with the > > number of osds. > > > > Are there some numbers available? > > We don't have hard numbers for you unfortunately. From my experience > however I can tell you that it will grow to several hundred MB to few > GB. It mostly depends on your cluster size, OSD-wise, and the > healthiness of your cluster (if you spend a considerable time in an > unhealthy OSD state, then the monitors will keep more OSD maps that they > usually do, thus increasing disk usage). > > As a rule of thumbs, if your monitors' stores hit double-digits GBs and > keep growing, you should definitely look into what's going on and maybe > drop in #ceph @ OFTC, but that has been becoming unlikely as of latest > releases. Thanks a lot. I'm playing with the idea to put a monitor database on 55Gb (ext4 formated) SSD, so at least the size will be OK for a while... ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] help
On 09/10/2013 15:24, Erwan Velu wrote: help _ Sorry... was about to send it to ceph-users-requ...@lists.ceph.com /me hides ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw Admin REST API
> Via cli, i do radosgw user info --uid=xxx
> but with the REST API ?
Hi Alexis,
Here is a simple python example on how to use the admin api. You will
need to get a few packages from the cheese shop (virtualenv + pip makes
this easy).
pip install requests-aws
You will also need to set the appropriate key, secret and server for
your local instance. If you have changed where the admin api lives in
the URI for your radosgw (/admin/ by default) you will need to update
that. You will also need to ensure you set the appropriate 'caps' on
the radosgw for what information you are looking for. Initially this is
done with the 'radosgw-admin caps add' command. The current supported
caps are buckets, metadata, usage and users. You can have either read,
write or * (both).
radosgw-admin caps add --uid=test --caps="users=*"
The last thing you need to understand is that you have to sign each
request in the same way you would a S3 request (S3Auth in the following
example).
import requests
import logging
from awsauth import S3Auth
logging.basicConfig(level=logging.DEBUG)
aws_key = 'DFJKJDKLSsdafd321s'
secret = 'j23141kjalJLKDFDM:FDFD+dfDKJHNK'
server = 'radosgw.example.com'
url = 'https://%s/admin/user?format=json&uid=test' % server
r = requests.get(url, auth=S3Auth(aws_key, secret, server))
print r.content
This produces something that should look like this,
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS
connection (1): radosgw.example.com
DEBUG:requests.packages.urllib3.connectionpool:"GET
/admin/user?format=json&uid=test HTTP/1.1" 200 None
{"user_id":"test","display_name":"Testing","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"test","access_key":"3FV6LBDJ80D3kI3I9TH8","secret_key":"z+/Rx0Osb3culGMCHL2E3Vg68gpFUSgSxcgf8BG9m"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"metadata","perm":"*"},{"type":"usage","perm":"*"},{"type":"users","perm":"*"}]}
Thanks,
derek
--
---
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw Admin REST API
Great !
Thanks a lot. It works.
I didn't know awsauth module.
Thanks again.
2013/10/9 Derek Yarnell :
>> Via cli, i do radosgw user info --uid=xxx
>> but with the REST API ?
>
> Hi Alexis,
>
> Here is a simple python example on how to use the admin api. You will
> need to get a few packages from the cheese shop (virtualenv + pip makes
> this easy).
>
> pip install requests-aws
>
> You will also need to set the appropriate key, secret and server for
> your local instance. If you have changed where the admin api lives in
> the URI for your radosgw (/admin/ by default) you will need to update
> that. You will also need to ensure you set the appropriate 'caps' on
> the radosgw for what information you are looking for. Initially this is
> done with the 'radosgw-admin caps add' command. The current supported
> caps are buckets, metadata, usage and users. You can have either read,
> write or * (both).
>
> radosgw-admin caps add --uid=test --caps="users=*"
>
> The last thing you need to understand is that you have to sign each
> request in the same way you would a S3 request (S3Auth in the following
> example).
>
> import requests
> import logging
> from awsauth import S3Auth
>
> logging.basicConfig(level=logging.DEBUG)
>
> aws_key = 'DFJKJDKLSsdafd321s'
> secret = 'j23141kjalJLKDFDM:FDFD+dfDKJHNK'
> server = 'radosgw.example.com'
>
> url = 'https://%s/admin/user?format=json&uid=test' % server
> r = requests.get(url, auth=S3Auth(aws_key, secret, server))
> print r.content
>
>
> This produces something that should look like this,
>
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS
> connection (1): radosgw.example.com
> DEBUG:requests.packages.urllib3.connectionpool:"GET
> /admin/user?format=json&uid=test HTTP/1.1" 200 None
> {"user_id":"test","display_name":"Testing","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"test","access_key":"3FV6LBDJ80D3kI3I9TH8","secret_key":"z+/Rx0Osb3culGMCHL2E3Vg68gpFUSgSxcgf8BG9m"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"metadata","perm":"*"},{"type":"usage","perm":"*"},{"type":"users","perm":"*"}]}
>
> Thanks,
> derek
>
> --
> ---
> Derek T. Yarnell
> University of Maryland
> Institute for Advanced Computer Studies
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Expanding ceph cluster by adding more OSDs
You can add PGs, the process is called splitting. I don't think PG merging, the reduction in the number of PGs, is ready yet. On Oct 8, 2013, at 11:58 PM, Guang wrote: > Hi ceph-users, > Ceph recommends the PGs number of a pool is (100 * OSDs) / Replicas, per my > understanding, the number of PGs for a pool should be fixed even we scale out > / in the cluster by adding / removing OSDs, does that mean if we double the > OSD numbers, the PG number for a pool is not optimal any more and there is no > chance to correct it? > > > Thanks, > Guang > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Same journal device for multiple OSDs?
Hi, to avoid confusion: the configuration did *not* contain multiple osds referring to the same journal device (or file). The snippet from ceph.conf suggests osd.214 and osd.314 both use the same journal - but it doesn't show that these osds run on different hosts. Regards Andreas Bluemle On Wed, 9 Oct 2013 11:23:18 +0200 Andreas Friedrich wrote: > Hello, > > I have a Ceph test cluster with 88 OSDs running well. > > In ceph.conf I found multiple OSDs that are using the same SSD block > device (without a file system) for their journal: > > [osd.214] > osd journal = /dev/fioa1 > ... > [osd.314] > osd journal = /dev/fioa1 > ... > > Is this a allowed configuration? > > Regards > Andreas Friedrich > -- > FUJITSU > Fujitsu Technology Solutions GmbH > Heinz-Nixdorf-Ring 1, 33106 Paderborn, Germany > Tel: +49 (5251) 525-1512 > Fax: +49 (5251) 525-321512 > Email: andreas.friedr...@ts.fujitsu.com > Web: ts.fujitsu.com > Company details: de.ts.fujitsu.com/imprint > -- > > -- Andreas Bluemle mailto:andreas.blue...@itxperts.de ITXperts GmbH http://www.itxperts.de Balanstrasse 73, Geb. 08Phone: (+49) 89 89044917 D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 Company details: http://www.itxperts.de/imprint.htm ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] indexing object store with SOLR
All, I have been prototyping an object store and am looking at a way to index content and metadata. Has anyone looked at doing anything similar? I would be interested in kicking around some ideas. I'd really like to implement something with Apache Solr or something similar. Thanks, Mike ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Dumpling ceph.conf looks different
I would also love to see this answered, this is sometimes asked during my geek on duty shift and I don't know a real answer to this, and I myself always do it old-(bobtail)-style. Wolfgang -- http://www.wogri.at On Oct 9, 2013, at 13:54 , su kucherova wrote: > Hi > > When I compare the /etc/ceph.conf for the latest release-dumpling and > previous releases I find they are different. > > in the older release we had [osd],[mon],[msd] in the ceph.conf > now i dont seem them.Where are these values stored now? > How does ceph figure out the partitions of osd and journal > > Thanks > Su > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Dumpling ceph.conf looks different
Previously the Ceph startup scripts required an enumeration of the daemons in the ceph.conf in order to start them. We've been doing a lot of incremental work since last October or so to make the system do more self-bootstrapping, and by the time we released Dumpling that got far enough to be used as the default configuration system (although it wasn't only for Dumpling). Now all the startup scripts will look in the default paths (or those specified by a ceph.conf in /etc/ceph, I believe) for anything that "looks like" a daemon, and start them up if they're present. This was the natural evolution of doing things like setting up udev rules that automatically ran any OSDs that have been created on a freshly plugged-in disk (which is how all the ceph-disk-[prepare|activate] stuff works). In general we find that this is easier to maintain since users are less likely to have conflicting ceph.confs across multiple nodes, and it doesn't present the illusion that a monolithic config file is necessary — but you are of course free to do otherwise if you prefer! -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Wed, Oct 9, 2013 at 12:30 PM, Wolfgang Hennerbichler wrote: > I would also love to see this answered, this is sometimes asked during my > geek on duty shift and I don't know a real answer to this, and I myself > always do it old-(bobtail)-style. > > Wolfgang > -- > http://www.wogri.at > > On Oct 9, 2013, at 13:54 , su kucherova wrote: > >> Hi >> >> When I compare the /etc/ceph.conf for the latest release-dumpling and >> previous releases I find they are different. >> >> in the older release we had [osd],[mon],[msd] in the ceph.conf >> now i dont seem them.Where are these values stored now? >> How does ceph figure out the partitions of osd and journal >> >> Thanks >> Su >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Dumpling ceph.conf looks different
Ok, thanks for the detailed answer, I already assumed so. But how do the OSD's then find their mon's? I believe this again has to be in ceph.conf, right? wogri -- http://www.wogri.at On Oct 9, 2013, at 21:36 , Gregory Farnum wrote: > Previously the Ceph startup scripts required an enumeration of the > daemons in the ceph.conf in order to start them. We've been doing a > lot of incremental work since last October or so to make the system do > more self-bootstrapping, and by the time we released Dumpling that got > far enough to be used as the default configuration system (although it > wasn't only for Dumpling). > Now all the startup scripts will look in the default paths (or those > specified by a ceph.conf in /etc/ceph, I believe) for anything that > "looks like" a daemon, and start them up if they're present. This was > the natural evolution of doing things like setting up udev rules that > automatically ran any OSDs that have been created on a freshly > plugged-in disk (which is how all the ceph-disk-[prepare|activate] > stuff works). > In general we find that this is easier to maintain since users are > less likely to have conflicting ceph.confs across multiple nodes, and > it doesn't present the illusion that a monolithic config file is > necessary — but you are of course free to do otherwise if you prefer! > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Wed, Oct 9, 2013 at 12:30 PM, Wolfgang Hennerbichler > wrote: >> I would also love to see this answered, this is sometimes asked during my >> geek on duty shift and I don't know a real answer to this, and I myself >> always do it old-(bobtail)-style. >> >> Wolfgang >> -- >> http://www.wogri.at >> >> On Oct 9, 2013, at 13:54 , su kucherova wrote: >> >>> Hi >>> >>> When I compare the /etc/ceph.conf for the latest release-dumpling and >>> previous releases I find they are different. >>> >>> in the older release we had [osd],[mon],[msd] in the ceph.conf >>> now i dont seem them.Where are these values stored now? >>> How does ceph figure out the partitions of osd and journal >>> >>> Thanks >>> Su >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Dumpling ceph.conf looks different
Yes, the monitors need to be specified in the ceph.conf still. ceph-deploy and similar systems make sure to do so. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Wed, Oct 9, 2013 at 12:42 PM, Wolfgang Hennerbichler wrote: > Ok, thanks for the detailed answer, I already assumed so. > But how do the OSD's then find their mon's? I believe this again has to be in > ceph.conf, right? > > wogri > -- > http://www.wogri.at > > On Oct 9, 2013, at 21:36 , Gregory Farnum wrote: > >> Previously the Ceph startup scripts required an enumeration of the >> daemons in the ceph.conf in order to start them. We've been doing a >> lot of incremental work since last October or so to make the system do >> more self-bootstrapping, and by the time we released Dumpling that got >> far enough to be used as the default configuration system (although it >> wasn't only for Dumpling). >> Now all the startup scripts will look in the default paths (or those >> specified by a ceph.conf in /etc/ceph, I believe) for anything that >> "looks like" a daemon, and start them up if they're present. This was >> the natural evolution of doing things like setting up udev rules that >> automatically ran any OSDs that have been created on a freshly >> plugged-in disk (which is how all the ceph-disk-[prepare|activate] >> stuff works). >> In general we find that this is easier to maintain since users are >> less likely to have conflicting ceph.confs across multiple nodes, and >> it doesn't present the illusion that a monolithic config file is >> necessary — but you are of course free to do otherwise if you prefer! >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> >> On Wed, Oct 9, 2013 at 12:30 PM, Wolfgang Hennerbichler >> wrote: >>> I would also love to see this answered, this is sometimes asked during my >>> geek on duty shift and I don't know a real answer to this, and I myself >>> always do it old-(bobtail)-style. >>> >>> Wolfgang >>> -- >>> http://www.wogri.at >>> >>> On Oct 9, 2013, at 13:54 , su kucherova wrote: >>> Hi When I compare the /etc/ceph.conf for the latest release-dumpling and previous releases I find they are different. in the older release we had [osd],[mon],[msd] in the ceph.conf now i dont seem them.Where are these values stored now? How does ceph figure out the partitions of osd and journal Thanks Su ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ...-all-starter documentation available?
I was poking around on a node, found the following executables: radosgw-all-starter ceph-mds-all-starter ceph-mon-all-starter ceph-osd-all-starter A Ceph web page search yielded no results. Does documentation exist? Where? Tim ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] About Ceph SSD and HDD strategy
Journal on SSD should effectively double your throughput because data will not be written to the same device twice to ensure transactional integrity. Additionally, by placing the OSD journal on an SSD you should see less latency, the disk head no longer has to seek back and forth between the journal and data partitions. For large writes it's not as critical to have a device that supports high IOPs or throughput because large writes are striped across many 4MB rados objects, relatively evenly distributed across the cluster. Small write operations will benefit the most from an OSD data partition with a writeback cache like btier/flashcache because it can absorbs an order of magnitude more IOPs and allow a slower spinning device catch up when there is less activity. On Tue, Oct 8, 2013 at 12:09 AM, Robert van Leeuwen < robert.vanleeu...@spilgames.com> wrote: > > I tried putting Flashcache on my spindle OSDs using an Intel SSL and > it works great. > > This is getting me read and write SSD caching instead of just write > performance on the journal. > > It should also allow me to protect the OSD journal on the same drive as > the OSD data and still get benefits of SSD caching for writes. > > Small note that on Red Hat based distro's + Flashcache + XFS: > There is a major issue (kernel panics) running xfs + flashcache on a 6.4 > kernel. (anything higher then 2.6.32-279) > It should be fixed in kernel 2.6.32-387.el6 which, I assume, will be 6.5 > which only just entered Beta. > > Fore more info, take a look here: > https://github.com/facebook/flashcache/issues/113 > > Since I've hit this issue (thankfully in our dev environment) we are > slightly less enthusiastic about running flashcache :( > It also adds a layer of complexity so I would rather just run the journals > on SSD, at least on Redhat. > I'm not sure about the performance difference of just journals v.s. > Flashcache but I'd be happy to read any such comparison :) > > Also, if you want to make use of the SSD trim func > > P.S. My experience with Flashcache is on Openstack Swift & Nova not Ceph. > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- Kyle ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Same journal device for multiple OSDs?
You can certainly use a similarly named device to back an OSD journal if the OSDs are on separate hosts. If you want to take a single SSD device and utilize it as a journal for many OSDs on the same machine then you would want to partition the SSD device and use a different partition for each OSD journal. You might consider using /dev/disk/by-id/foo instead of /dev/fioa1 to avoid potential device reordering issues after a reboot. Hope that helps, sorry if I misunderstood. On Wed, Oct 9, 2013 at 7:03 AM, Andreas Bluemle wrote: > Hi, > > to avoid confusion: the configuration did *not* contain > multiple osds referring to the same journal device (or file). > > The snippet from ceph.conf suggests osd.214 and osd.314 > both use the same journal - > but it doesn't show that these osds run on different hosts. > > > Regards > > Andreas Bluemle > > > On Wed, 9 Oct 2013 11:23:18 +0200 > Andreas Friedrich wrote: > > > Hello, > > > > I have a Ceph test cluster with 88 OSDs running well. > > > > In ceph.conf I found multiple OSDs that are using the same SSD block > > device (without a file system) for their journal: > > > > [osd.214] > > osd journal = /dev/fioa1 > > ... > > [osd.314] > > osd journal = /dev/fioa1 > > ... > > > > Is this a allowed configuration? > > > > Regards > > Andreas Friedrich > > -- > > FUJITSU > > Fujitsu Technology Solutions GmbH > > Heinz-Nixdorf-Ring 1, 33106 Paderborn, Germany > > Tel: +49 (5251) 525-1512 > > Fax: +49 (5251) 525-321512 > > Email: andreas.friedr...@ts.fujitsu.com > > Web: ts.fujitsu.com > > Company details: de.ts.fujitsu.com/imprint > > -- > > > > > > > > -- > Andreas Bluemle mailto:andreas.blue...@itxperts.de > ITXperts GmbH http://www.itxperts.de > Balanstrasse 73, Geb. 08Phone: (+49) 89 89044917 > D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 > > Company details: http://www.itxperts.de/imprint.htm > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Kyle ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] About Ceph SSD and HDD strategy
While in theory this should be true, I'm not finding it to be the case for a typical enterprise LSI card with 24 drives attached. We tried a variety of ratios and went back to collocated journals on the spinning drives. Eagerly awaiting the tiered performance changes to implement a faster tier via SSD. -- Warren On Oct 9, 2013, at 5:52 PM, Kyle Bader wrote: > Journal on SSD should effectively double your throughput because data will > not be written to the same device twice to ensure transactional integrity. > Additionally, by placing the OSD journal on an SSD you should see less > latency, the disk head no longer has to seek back and forth between the > journal and data partitions. For large writes it's not as critical to have a > device that supports high IOPs or throughput because large writes are striped > across many 4MB rados objects, relatively evenly distributed across the > cluster. Small write operations will benefit the most from an OSD data > partition with a writeback cache like btier/flashcache because it can absorbs > an order of magnitude more IOPs and allow a slower spinning device catch up > when there is less activity. > > > On Tue, Oct 8, 2013 at 12:09 AM, Robert van Leeuwen > wrote: >> > I tried putting Flashcache on my spindle OSDs using an Intel SSL and it >> > works great. >> > This is getting me read and write SSD caching instead of just write >> > performance on the journal. >> > It should also allow me to protect the OSD journal on the same drive as >> > the OSD data and still get benefits of SSD caching for writes. >> >> Small note that on Red Hat based distro's + Flashcache + XFS: >> There is a major issue (kernel panics) running xfs + flashcache on a 6.4 >> kernel. (anything higher then 2.6.32-279) >> It should be fixed in kernel 2.6.32-387.el6 which, I assume, will be 6.5 >> which only just entered Beta. >> >> Fore more info, take a look here: >> https://github.com/facebook/flashcache/issues/113 >> >> Since I've hit this issue (thankfully in our dev environment) we are >> slightly less enthusiastic about running flashcache :( >> It also adds a layer of complexity so I would rather just run the journals >> on SSD, at least on Redhat. >> I'm not sure about the performance difference of just journals v.s. >> Flashcache but I'd be happy to read any such comparison :) >> >> Also, if you want to make use of the SSD trim func >> >> P.S. My experience with Flashcache is on Openstack Swift & Nova not Ceph. >> >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > > -- > > Kyle > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Expanding ceph cluster by adding more OSDs
Thanks Mike. Is there any documentation for that? Thanks, Guang On Oct 9, 2013, at 9:58 PM, Mike Lowe wrote: > You can add PGs, the process is called splitting. I don't think PG merging, > the reduction in the number of PGs, is ready yet. > > On Oct 8, 2013, at 11:58 PM, Guang wrote: > >> Hi ceph-users, >> Ceph recommends the PGs number of a pool is (100 * OSDs) / Replicas, per my >> understanding, the number of PGs for a pool should be fixed even we scale >> out / in the cluster by adding / removing OSDs, does that mean if we double >> the OSD numbers, the PG number for a pool is not optimal any more and there >> is no chance to correct it? >> >> >> Thanks, >> Guang >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Expanding ceph cluster by adding more OSDs
There used to be, can't find it right now. Something like 'ceph osd set pg_num ' then 'ceph osd set pgp_num ' to actually move your data into the new pg's. I successfully did it several months ago, when bobtail was current. Sent from my iPad > On Oct 9, 2013, at 10:30 PM, Guang wrote: > > Thanks Mike. > > Is there any documentation for that? > > Thanks, > Guang > >> On Oct 9, 2013, at 9:58 PM, Mike Lowe wrote: >> >> You can add PGs, the process is called splitting. I don't think PG >> merging, the reduction in the number of PGs, is ready yet. >> >>> On Oct 8, 2013, at 11:58 PM, Guang wrote: >>> >>> Hi ceph-users, >>> Ceph recommends the PGs number of a pool is (100 * OSDs) / Replicas, per my >>> understanding, the number of PGs for a pool should be fixed even we scale >>> out / in the cluster by adding / removing OSDs, does that mean if we double >>> the OSD numbers, the PG number for a pool is not optimal any more and there >>> is no chance to correct it? >>> >>> >>> Thanks, >>> Guang >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Expanding ceph cluster by adding more OSDs
Thanks Mike. I get your point. There are still a few things confusing me: 1) We expand Ceph cluster by adding more OSDs, which will trigger re-balance PGs across the old & new OSDs, and likely it will break the optimized PG numbers for the cluster. 2) We can add more PGs which will trigger re-balance objects across old & new PGs. So: 1) What is the recommended way to expand the cluster by adding OSDs (and potentially adding PGs), should we do them at the same time? 2) What is the recommended way to scale a cluster from like 1PB to 2PB, should we scale it to like 1.1PB to 1.2PB or move to 2PB directly? Thanks, Guang On Oct 10, 2013, at 11:10 AM, Michael Lowe wrote: > There used to be, can't find it right now. Something like 'ceph osd set > pg_num ' then 'ceph osd set pgp_num ' to actually move your data > into the new pg's. I successfully did it several months ago, when bobtail > was current. > > Sent from my iPad > >> On Oct 9, 2013, at 10:30 PM, Guang wrote: >> >> Thanks Mike. >> >> Is there any documentation for that? >> >> Thanks, >> Guang >> >>> On Oct 9, 2013, at 9:58 PM, Mike Lowe wrote: >>> >>> You can add PGs, the process is called splitting. I don't think PG >>> merging, the reduction in the number of PGs, is ready yet. >>> On Oct 8, 2013, at 11:58 PM, Guang wrote: Hi ceph-users, Ceph recommends the PGs number of a pool is (100 * OSDs) / Replicas, per my understanding, the number of PGs for a pool should be fixed even we scale out / in the cluster by adding / removing OSDs, does that mean if we double the OSD numbers, the PG number for a pool is not optimal any more and there is no chance to correct it? Thanks, Guang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Expanding ceph cluster by adding more OSDs
I had those same questions, I think the answer I got was that it was better to have too few pg's than to have overloaded osd's. So add osd's then add pg's. I don't know the best increments to grow in, probably depends largely on the hardware in your osd's. Sent from my iPad > On Oct 9, 2013, at 11:34 PM, Guang wrote: > > Thanks Mike. I get your point. > > There are still a few things confusing me: > 1) We expand Ceph cluster by adding more OSDs, which will trigger re-balance > PGs across the old & new OSDs, and likely it will break the optimized PG > numbers for the cluster. > 2) We can add more PGs which will trigger re-balance objects across old & > new PGs. > > So: > 1) What is the recommended way to expand the cluster by adding OSDs (and > potentially adding PGs), should we do them at the same time? > 2) What is the recommended way to scale a cluster from like 1PB to 2PB, > should we scale it to like 1.1PB to 1.2PB or move to 2PB directly? > > Thanks, > Guang > >> On Oct 10, 2013, at 11:10 AM, Michael Lowe wrote: >> >> There used to be, can't find it right now. Something like 'ceph osd set >> pg_num ' then 'ceph osd set pgp_num ' to actually move your data >> into the new pg's. I successfully did it several months ago, when bobtail >> was current. >> >> Sent from my iPad >> >>> On Oct 9, 2013, at 10:30 PM, Guang wrote: >>> >>> Thanks Mike. >>> >>> Is there any documentation for that? >>> >>> Thanks, >>> Guang >>> On Oct 9, 2013, at 9:58 PM, Mike Lowe wrote: You can add PGs, the process is called splitting. I don't think PG merging, the reduction in the number of PGs, is ready yet. > On Oct 8, 2013, at 11:58 PM, Guang wrote: > > Hi ceph-users, > Ceph recommends the PGs number of a pool is (100 * OSDs) / Replicas, per > my understanding, the number of PGs for a pool should be fixed even we > scale out / in the cluster by adding / removing OSDs, does that mean if > we double the OSD numbers, the PG number for a pool is not optimal any > more and there is no chance to correct it? > > > Thanks, > Guang > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ...-all-starter documentation available?
Ceph deployed by ceph-deploy on Ubuntu uses upstart. On Wed, Oct 9, 2013 at 1:48 PM, Snider, Tim wrote: > I was poking around on a node, found the following executables: > > radosgw-all-starter > > ceph-mds-all-starter > > ceph-mon-all-starter > > ceph-osd-all-starter > > A Ceph web page search yielded no results. Does documentation exist? Where? > > > > Tim > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- John Wilkins Senior Technical Writer Intank john.wilk...@inktank.com (415) 425-9599 http://inktank.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ...-all-starter documentation available?
Upstart itself could do with better docs :-( I'd recommend starting with 'man initctl', should help clarify things a bit! Cheers Mark On 10/10/13 17:50, John Wilkins wrote: Ceph deployed by ceph-deploy on Ubuntu uses upstart. On Wed, Oct 9, 2013 at 1:48 PM, Snider, Tim wrote: I was poking around on a node, found the following executables: radosgw-all-starter ceph-mds-all-starter ceph-mon-all-starter ceph-osd-all-starter A Ceph web page search yielded no results. Does documentation exist? Where? Tim ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] recovering from unhealthy state
Hi,
I've managed to get cepth in a unhealthy state, from which it will not
recover automatically. I've done some 'ceph osd out X' and stopped
ceph-osd processes before the rebalancing was completed. (All in a test
environment :-) )
Now I see:
# ceph -w
cluster 7fac9ad3-455e-4570-ae24-5c4311763bf9
health HEALTH_WARN 12 pgs degraded; 9 pgs stale; 9 pgs stuck stale; 964 pgs
stuck unclean; recovery 617/50262 degraded (1.228%)
monmap e4: 3 mons at
{n2=192.168.5.12:6789/0,node01=192.168.5.10:6789/0,node03=192.168.5.11:6789/0},
election epoch 126, quorum 0,1,2 n2,node01,node03
osdmap e1462: 17 osds: 17 up, 10 in
pgmap v198793: 4416 pgs: 3452 active+clean, 2 stale+active, 943
active+remapped, 12 active+degraded, 7 stale+active+remapped; 95639 MB data,
192 GB used, 15628 GB / 15821 GB avail; 0B/s rd, 110KB/s wr, 9op/s; 617/50262
degraded (1.228%)
mdsmap e1: 0/0/1 up
2013-10-10 07:02:57.741031 mon.0 [INF] pgmap v198792: 4416 pgs: 3452
active+clean, 2 stale+active, 943 active+remapped, 12 active+degraded, 7
stale+active+remapped; 95639 MB data, 192 GB used, 15628 GB / 15821 GB avail;
0B/s rd, 17492B/s wr, 2op/s; 617/50262 degraded (1.228%)
I've seen some documentation at
http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/
* inactive - The placement group has not been active for too long
(i.e., it hasn’t been able to service read/write requests).
* unclean - The placement group has not been clean for too long
(i.e., it hasn’t been able to completely recover from a previous
failure).
* stale - The placement group status has not been updated by a
ceph-osd, indicating that all nodes storing this placement group
may be down.
Which leaves 'remapped' and 'degraded' unexplained (though I can imagine
what they mean).
I presume I've lost some data. Alas. How to get to a clean state again?
I mean, if you're stuck with lost data, you don't want to have the
cluster in a unhealthy state forever. I'd like to just cut my losses an
get on.
- Kees
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] show warning when make source code of ceph
warning infomation: common/Preforker.h: In member function ‘void Preforker::daemonize()’: common/Preforker.h:97:40: warning: ignoring return value of ‘ssize_t write(int, const void*, size_t)’, declared with attribute warn_unused_result [-Wunused-result] test/encoding/ceph_dencoder.cc: In function ‘int main(int, const char**)’: test/encoding/ceph_dencoder.cc:196:5: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without version: 0.61.8 package download from http://ceph.com/download/ceph-0.61.8.tar.gz configure parameter: ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var/lib/ceph --with-radosgw Is it normal? do it influence ceph operation?? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
