Re: [ceph-users] Generic Tuning parameters?
Hi Erich, I'm also on searching for improvements. You should use the right mountoptions, to prevent fragmentation (for XFS). [osd] osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M osd_op_threads = 4 osd_disk_threads = 4 With 45 OSDs per node you need an powerfull system... AFAIK 12 OSDs/node is recomended. You should think about what happens if one node die... I use an monitoring-script which do an set noout if more than N OSDs are down. Then I must decide, if it's faster to get the failed node back, or do an rebuild (normaly the first choice). Udo On 27.06.2014 20:00, Erich Weiler wrote: Hi Folks, We're going to spin up a ceph cluster with the following general specs: * Six 10Gb/s connected servers, each with 45 4TB disks in a JBOD * Each disk is an OSD, so 45 OSDs per server * So 45*6 = 270 OSDs total * Three separate, dedicated monitor nodes The files stored on this storage cluster will be large file, each file will be several GB in size at the minimum, with some files being over 100GB. Generically, are there any tuning parameters out there that would be good to drop in for this hardware profile and file size? We plan on growing this filesystem as we go, to 10 servers, then 15, then 20, etc. Thanks a bunch for any hints!! cheers, erich ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSD Data not evenly distributed
Hi, all My cluster has been running for about 4 month now. I have about 108 osds and all are 600G SAS Disk. Their disk usage is between 70% and 85%. It seems that ceph cannot distribute data evenly by default settings. Is there any configuration that helps distribute data more evenly? Thanks very much ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Data not evenly distributed
Did you also increase the pgp_num? On Saturday, June 28, 2014, Jianing Yang jianingy.y...@gmail.com wrote: Actually, I did increase PG number to 32768 (120 osds) and I also use tunable optimal. But the data still not distribute evenly. On Sun, Jun 29, 2014 at 3:42 AM, Konrad Gutkowski konrad.gutkow...@ffs.pl javascript:_e(%7B%7D,'cvml','konrad.gutkow...@ffs.pl'); wrote: Hi, Increasing PG number for pools that hold data might help if you didn't do that already. Check out this thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/ 2014-January/027094.html You might find some tips there (although it was pre firefly). W dniu 28.06.2014 o 14:44 Jianing Yang jianingy.y...@gmail.com javascript:_e(%7B%7D,'cvml','jianingy.y...@gmail.com'); pisze: Hi, all My cluster has been running for about 4 month now. I have about 108 osds and all are 600G SAS Disk. Their disk usage is between 70% and 85%. It seems that ceph cannot distribute data evenly by default settings. Is there any configuration that helps distribute data more evenly? Thanks very much ___ ceph-users mailing list ceph-users@lists.ceph.com javascript:_e(%7B%7D,'cvml','ceph-users@lists.ceph.com'); http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com javascript:_e(%7B%7D,'cvml','ceph-users@lists.ceph.com'); http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Data not evenly distributed
Of course, both to 32768. On Sun, Jun 29, 2014 at 9:15 AM, Gregory Farnum g...@inktank.com wrote: Did you also increase the pgp_num? On Saturday, June 28, 2014, Jianing Yang jianingy.y...@gmail.com wrote: Actually, I did increase PG number to 32768 (120 osds) and I also use tunable optimal. But the data still not distribute evenly. On Sun, Jun 29, 2014 at 3:42 AM, Konrad Gutkowski konrad.gutkow...@ffs.pl wrote: Hi, Increasing PG number for pools that hold data might help if you didn't do that already. Check out this thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/ 2014-January/027094.html You might find some tips there (although it was pre firefly). W dniu 28.06.2014 o 14:44 Jianing Yang jianingy.y...@gmail.com pisze: Hi, all My cluster has been running for about 4 month now. I have about 108 osds and all are 600G SAS Disk. Their disk usage is between 70% and 85%. It seems that ceph cannot distribute data evenly by default settings. Is there any configuration that helps distribute data more evenly? Thanks very much ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] uWSGI native rados plugin
Hi all, in the last few days i have worked on improving the uWSGI plugin for rados (its first version was released a year ago, but was buggy and little integrated with the whole project). http://uwsgi-docs.readthedocs.org/en/latest/Rados.html (for those who do not know uWSGI, it is a platform/library for building application servers) you currently need to clone the github repository (2.0.6 will be released next week). In the uwsgi-2.1 branch the plugin includes support for some additional webdav methods (like PROPFIND) thanks to the new (still work in progress) uWSGI webdav api. To build the plugin (or the monolithic binary) you need gcc/clang, pcre development headers (to use the internal routing support), openssl (if you need https) and librados. A working python/python3 executable is required to run the build system. The sources are here: https://github.com/unbit/uwsgi/blob/master/plugins/rados/ (docs source is at https://github.com/unbit/uwsgi-docs/blob/master/Rados.rst) the only complex part is async support, as we need pipes and mutexes to synchronize aio functions with the non-blocking (stack-switching) uWSGI core. (but you can obviously use it in classic multiprocess/multithread blocking mode) Currently i have started stress testing it with seven nodes (each node has a mon, a osd and uWSGI instance [sorry, i know it is not the best approach]) behind a nginx proxy. (uWSGI caching is not enabled as i want to get statistics of the rados response times) Planned new features are: - improving webdav coverage in 2.1 branch (using xattr as dav properties, and implementing locking on top of rados locks) - adding support for watch/notify (could be useful for auto-cache invalidation and for signaling) - integration with uWSGI alarm subsystem (for triggering events on failed rados pings and so on) Hope it will be useful for someone Every report will be wellcomed. Thanks -- Roberto De Ioris http://unbit.it ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
