[ceph-users] It works !Re?? //?? // ceph-mon is blocked after shutting down and ip address changed
It works , after I remove v2 address in ceph.conf.
Hope I see. Thank you !
[root@ceph-node1 ceph]# ceph -s
cluster:
id: e384e8e6-94d5-4812-bfbb-d1b0468bdef5
health: HEALTH_WARN
1 MDSs report slow metadata IOs
noout,nobackfill,norecover flag(s) set
9 osds down
no active mgr
Reduced data availability: 102 pgs
inactive, 128 pgs down, 8 pgs stale
Degraded data redundancy:
3664/29810955 objects degraded (0.012%), 28 pgs degraded, 29 pgs undersized
1 monitors have not enabled msgr2
services:
mon: 1 daemons, quorum ceph-node1 (age 13h)
mgr: no daemons active (since 3d)
mds: cephfs:1 {0=ceph-node1=up:active(laggy or crashed)}
osd: 12 osds: 3 up (since 3d), 12 in (since 3d)
flags noout,nobackfill,norecover
rgw: 1 daemon active (ceph-node1.rgw0)
data:
pools: 6 pools, 168 pgs
objects: 2.49M objects, 9.3 TiB
usage: 14 TiB used, 74 TiB / 88 TiB avail
pgs: 97.024% pgs not active
3664/29810955 objects degraded
(0.012%)
128 down
13
stale+undersized+degraded+peered
11
undersized+degraded+peered
6
stale+undersized+peered
5 undersized+peered
4
active+undersized+degraded
1 active+undersized
-- --
??: "Stefan Kooman"https://www.bit.nl/ ;
Kamer van Koophandel 09090351
| GPG:
0xD14839C6
+31 318 648 688 / i...@bit.nl___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Deploying a Ceph storage cluster using Warewulf on Centos-7
Hi, all I don't know which automation tool is best for deploying Ceph and I'd like to know about. I'm comfortable with Warewulf since I've been using it for HPC clusters. I find it quite convenient for Ceph too. I wrote a set of scripts that can deploy a Ceph cluster quickly. Here is how I did it just using virtualbox: http://ruilinchu.blogspot.com/2015/09/deploying-ceph-storage-cluster-using.html comments are welcome! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to observed civetweb. (Kobi Laredo)
Hi, Kobi, >> We are currently testing tuning civetweb's num_threads >> and request_timeout_ms to improve radosgw performance I wonder how's your testing of radosgw perofrmance going? We hit some limitation of single radosgw instance, about ~300MB/s. I'm just thinking how num_threads & request_timeout_ms affects rgw thoughput. Cheers, Po On Fri, Sep 11, 2015 at 3:05 AM, wrote: > > Message: 9 > Date: Wed, 9 Sep 2015 23:36:33 -0700 > From: Kobi Laredo > To: Vickie ch > Cc: ceph-users > Subject: Re: [ceph-users] How to observed civetweb. > Message-ID: > < > cagsygfczjs5c_dsytkaa0rjyqhqyksz+h3jchbhrmqf3m5h...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > We haven't had the need to explore civetweb's SSL termination ability, so I > don't know the answer to your question. > Either way, haproxy is a safer bet. > > *Kobi Laredo* > *Cloud Systems Engineer* | (*408) 409-KOBI* > > On Tue, Sep 8, 2015 at 8:50 PM, Vickie ch wrote: > > > Thanks a lot!! > > One more question. I can understand use haproxy is a better way for > > loadbalance. > > And github say civetweb already support https. > > But I found some documents mention that civetweb need haproxy for https. > > Which one is true? > > > > > > > > Best wishes, > > Mika > > > > > > 2015-09-09 2:21 GMT+08:00 Kobi Laredo : > > > >> Vickie, > >> > >> You can add: > >> *access_log_file=/var/log/civetweb/access.log > >> error_log_file=/var/log/civetweb/error.log* > >> > >> to *rgw frontends* in ceph.conf though these logs are thin on info > >> (Source IP, date, and request) > >> > >> Check out > >> https://github.com/civetweb/civetweb/blob/master/docs/UserManual.md for > >> more civetweb configs you can inject through *rgw frontends* config > >> attribute in ceph.conf > >> > >> We are currently testing tuning civetweb's num_threads > >> and request_timeout_ms to improve radosgw performance > >> > >> *Kobi Laredo* > >> *Cloud Systems Engineer* | (*408) 409-KOBI* > >> > >> On Tue, Sep 8, 2015 at 8:20 AM, Yehuda Sadeh-Weinraub < > yeh...@redhat.com> > >> wrote: > >> > >>> You can increase the civetweb logs by adding 'debug civetweb = 10' in > >>> your ceph.conf. The output will go into the rgw logs. > >>> > >>> Yehuda > >>> > >>> On Tue, Sep 8, 2015 at 2:24 AM, Vickie ch > >>> wrote: > >>> > Dear cephers, > >>> >Just upgrade radosgw from apache to civetweb. > >>> > It's really simple to installed and used. But I can't find any > >>> parameters or > >>> > logs to adjust(or observe) civetweb. (Like apache log). I'm really > >>> confuse. > >>> > Any ideas? > >>> > > >>> > > >>> > Best wishes, > >>> > Mika > >>> > > >>> > > >>> > ___ > >>> > ceph-users mailing list > >>> > ceph-users@lists.ceph.com > >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > > >>> ___ > >>> ceph-users mailing list > >>> ceph-users@lists.ceph.com > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >> > >> > > > -- next part -- > An HTML attachment was scrubbed... > URL: < > http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20150909/65384530/attachment.html > > > > -- > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cache-tier do not evict
What ceph version do you use? Regards, On 9 Apr 2015 18:58, "Patrik Plank" wrote: > Hi, > > > i have build a cach-tier pool (replica 2) with 3 x 512gb ssd for my kvm > pool. > > these are my settings : > > > ceph osd tier add kvm cache-pool > > ceph osd tier cache-mode cache-pool writeback > > ceph osd tier set-overlay kvm cache-pool > > > ceph osd pool set cache-pool hit_set_type bloom > > ceph osd pool set cache-pool hit_set_count 1 > > ceph osd pool set cache-pool hit set period 3600 > > > ceph osd pool set cache-pool target_max_bytes 751619276800 > > ceph osd pool set cache-pool target_max_objects 100 > > > ceph osd pool set cache-pool cache_min_flush_age 1800 > > ceph osd pool set cache-pool cache_min_evict_age 600 > > > ceph osd pool cache-pool cache_target_dirty_ratio .4 > > ceph osd pool cache-pool cache target_full_ratio .8 > > > So the problem is, the cache-tier do no evict automatically. > > If i copy some kvm images to the ceph cluster, the cache osds always run > full. > > > Is that normal? > > Is there a miss configuration? > > > thanks > > best regards > > Patrik > > > > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph -s slow return result
Thank you very much! On 29 Mar 2015 11:25, "Kobi Laredo" wrote: > I'm glad it worked. > You can set a warning to catch this early next time (1GB) > > *mon leveldb size warn = 10* > > > > *Kobi Laredo* > *Cloud Systems Engineer* | (*408) 409-KOBI* > > On Fri, Mar 27, 2015 at 5:45 PM, Chu Duc Minh > wrote: > >> @Kobi Laredo: thank you! It's exactly my problem. >> # du -sh /var/lib/ceph/mon/ >> *2.6G * /var/lib/ceph/mon/ >> # ceph tell mon.a compact >> compacted leveldb in 10.197506 >> # du -sh /var/lib/ceph/mon/ >> *461M*/var/lib/ceph/mon/ >> Now my "ceph -s" return result immediately. >> >> Maybe monitors' LevelDB store grow so big because i pushed 13 millions >> file into a bucket (over radosgw). >> When have extreme large number of files in a bucket, the state of ceph >> cluster could become unstable? (I'm running Giant) >> >> Regards, >> >> On Sat, Mar 28, 2015 at 12:57 AM, Kobi Laredo >> wrote: >> >>> What's the current health of the cluster? >>> It may help to compact the monitors' LevelDB store if they have grown in >>> size >>> http://www.sebastien-han.fr/blog/2014/10/27/ceph-mon-store-taking-up-a-lot-of-space/ >>> Depends on the size of the mon's store size it may take some time to >>> compact, make sure to do only one at a time. >>> >>> *Kobi Laredo* >>> *Cloud Systems Engineer* | (*408) 409-KOBI* >>> >>> On Fri, Mar 27, 2015 at 10:31 AM, Chu Duc Minh >>> wrote: >>> >>>> All my monitors running. >>>> But i deleting pool .rgw.buckets, now having 13 million objects (just >>>> test data). >>>> The reason that i must delete this pool is my cluster become unstable, >>>> and sometimes an OSD down, PG peering, incomplete,... >>>> Therefore i must delete this pool to re-stablize my cluster. (radosgw >>>> is too slow for delete objects when one of my bucket reachs few million >>>> objects). >>>> >>>> Regards, >>>> >>>> >>>> On Sat, Mar 28, 2015 at 12:23 AM, Gregory Farnum >>>> wrote: >>>> >>>>> Are all your monitors running? Usually a temporary hang means that the >>>>> Ceph client tries to reach a monitor that isn't up, then times out and >>>>> contacts a different one. >>>>> >>>>> I have also seen it just be slow if the monitors are processing so >>>>> many updates that they're behind, but that's usually on a very unhappy >>>>> cluster. >>>>> -Greg >>>>> On Fri, Mar 27, 2015 at 8:50 AM Chu Duc Minh >>>>> wrote: >>>>> >>>>>> On my CEPH cluster, "ceph -s" return result quite slow. >>>>>> Sometimes it return result immediately, sometimes i hang few seconds >>>>>> before return result. >>>>>> >>>>>> Do you think this problem (ceph -s slow return) only relate to >>>>>> ceph-mon(s) process? or maybe it relate to ceph-osd(s) too? >>>>>> (i deleting a big bucket, .rgw.buckets, and ceph-osd(s) disk util >>>>>> quite high) >>>>>> >>>>>> Regards, >>>>>> ___ >>>>>> ceph-users mailing list >>>>>> ceph-users@lists.ceph.com >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>> >>>>> >>>> >>>> ___ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>> >>> >> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph -s slow return result
@Kobi Laredo: thank you! It's exactly my problem. # du -sh /var/lib/ceph/mon/ *2.6G * /var/lib/ceph/mon/ # ceph tell mon.a compact compacted leveldb in 10.197506 # du -sh /var/lib/ceph/mon/ *461M*/var/lib/ceph/mon/ Now my "ceph -s" return result immediately. Maybe monitors' LevelDB store grow so big because i pushed 13 millions file into a bucket (over radosgw). When have extreme large number of files in a bucket, the state of ceph cluster could become unstable? (I'm running Giant) Regards, On Sat, Mar 28, 2015 at 12:57 AM, Kobi Laredo wrote: > What's the current health of the cluster? > It may help to compact the monitors' LevelDB store if they have grown in > size > http://www.sebastien-han.fr/blog/2014/10/27/ceph-mon-store-taking-up-a-lot-of-space/ > Depends on the size of the mon's store size it may take some time to > compact, make sure to do only one at a time. > > *Kobi Laredo* > *Cloud Systems Engineer* | (*408) 409-KOBI* > > On Fri, Mar 27, 2015 at 10:31 AM, Chu Duc Minh > wrote: > >> All my monitors running. >> But i deleting pool .rgw.buckets, now having 13 million objects (just >> test data). >> The reason that i must delete this pool is my cluster become unstable, >> and sometimes an OSD down, PG peering, incomplete,... >> Therefore i must delete this pool to re-stablize my cluster. (radosgw is >> too slow for delete objects when one of my bucket reachs few million >> objects). >> >> Regards, >> >> >> On Sat, Mar 28, 2015 at 12:23 AM, Gregory Farnum >> wrote: >> >>> Are all your monitors running? Usually a temporary hang means that the >>> Ceph client tries to reach a monitor that isn't up, then times out and >>> contacts a different one. >>> >>> I have also seen it just be slow if the monitors are processing so many >>> updates that they're behind, but that's usually on a very unhappy cluster. >>> -Greg >>> On Fri, Mar 27, 2015 at 8:50 AM Chu Duc Minh >>> wrote: >>> >>>> On my CEPH cluster, "ceph -s" return result quite slow. >>>> Sometimes it return result immediately, sometimes i hang few seconds >>>> before return result. >>>> >>>> Do you think this problem (ceph -s slow return) only relate to >>>> ceph-mon(s) process? or maybe it relate to ceph-osd(s) too? >>>> (i deleting a big bucket, .rgw.buckets, and ceph-osd(s) disk util quite >>>> high) >>>> >>>> Regards, >>>> ___ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph -s slow return result
All my monitors running. But i deleting pool .rgw.buckets, now having 13 million objects (just test data). The reason that i must delete this pool is my cluster become unstable, and sometimes an OSD down, PG peering, incomplete,... Therefore i must delete this pool to re-stablize my cluster. (radosgw is too slow for delete objects when one of my bucket reachs few million objects). Regards, On Sat, Mar 28, 2015 at 12:23 AM, Gregory Farnum wrote: > Are all your monitors running? Usually a temporary hang means that the > Ceph client tries to reach a monitor that isn't up, then times out and > contacts a different one. > > I have also seen it just be slow if the monitors are processing so many > updates that they're behind, but that's usually on a very unhappy cluster. > -Greg > On Fri, Mar 27, 2015 at 8:50 AM Chu Duc Minh > wrote: > >> On my CEPH cluster, "ceph -s" return result quite slow. >> Sometimes it return result immediately, sometimes i hang few seconds >> before return result. >> >> Do you think this problem (ceph -s slow return) only relate to >> ceph-mon(s) process? or maybe it relate to ceph-osd(s) too? >> (i deleting a big bucket, .rgw.buckets, and ceph-osd(s) disk util quite >> high) >> >> Regards, >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph -s slow return result
On my CEPH cluster, "ceph -s" return result quite slow. Sometimes it return result immediately, sometimes i hang few seconds before return result. Do you think this problem (ceph -s slow return) only relate to ceph-mon(s) process? or maybe it relate to ceph-osd(s) too? (i deleting a big bucket, .rgw.buckets, and ceph-osd(s) disk util quite high) Regards, ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !
@Michael Kuriger: when ceph/librbd operate normally, i know that double the pg_num is the safe way. But when it has problem, i think double it can make many many VMs die (maybe >= 50%?) On Mon, Mar 16, 2015 at 9:53 PM, Michael Kuriger wrote: > I always keep my pg number a power of 2. So I’d go from 2048 to 4096. > I’m not sure if this is the safest way, but it’s worked for me. > > > > [image: yp] > > > > Michael Kuriger > > Sr. Unix Systems Engineer > > * mk7...@yp.com |( 818-649-7235 > > From: Chu Duc Minh > Date: Monday, March 16, 2015 at 7:49 AM > To: Florent B > Cc: "ceph-users@lists.ceph.com" > Subject: Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down ! > >I'm using the latest Giant and have the same issue. When i increase > PG_num of a pool from 2048 to 2148, my VMs is still ok. When i increase > from 2148 to 2400, some VMs die (Qemu-kvm process die). > My physical servers (host VMs) running kernel 3.13 and use librbd. > I think it's a bug in librbd with crushmap. > (I set crush_tunables3 on my ceph cluster, does it make sense?) > > Do you know a way to safely increase PG_num? (I don't think increase > PG_num 100 each times is a safe & good way) > > Regards, > > On Mon, Mar 16, 2015 at 8:50 PM, Florent B wrote: > >> We are on Giant. >> >> On 03/16/2015 02:03 PM, Azad Aliyar wrote: >> > >> > May I know your ceph version.?. The latest version of firefly 80.9 has >> > patches to avoid excessive data migrations during rewighting osds. You >> > may need set a tunable inorder make this patch active. >> > >> > This is a bugfix release for firefly. It fixes a performance regression >> > in librbd, an important CRUSH misbehavior (see below), and several RGW >> > bugs. We have also backported support for flock/fcntl locks to >> ceph-fuse >> > and libcephfs. >> > >> > We recommend that all Firefly users upgrade. >> > >> > For more detailed information, see >> > http://docs.ceph.com/docs/master/_downloads/v0.80.9.txt >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.ceph.com_docs_master_-5Fdownloads_v0.80.9.txt&d=AwMFaQ&c=lXkdEK1PC7UK9oKA-BBSI8p1AamzLOSncm6Vfn0C_UQ&r=CSYA9OS6Qd7fQySI2LDvlQ&m=0MEOMMXqQGLq4weFd85B2Bxn5uBH9V9uMiuajNVb7o0&s=-HHkWm2cMQZ06FKpWF4Ai-YkFb9lUR_tH_KR0eITbuU&e=> >> > >> > Adjusting CRUSH maps >> > >> > >> > * This point release fixes several issues with CRUSH that trigger >> > excessive data migration when adjusting OSD weights. These are most >> > obvious when a very small weight change (e.g., a change from 0 to >> > .01) triggers a large amount of movement, but the same set of bugs >> > can also lead to excessive (though less noticeable) movement in >> > other cases. >> > >> > However, because the bug may already have affected your cluster, >> > fixing it may trigger movement *back* to the more correct location. >> > For this reason, you must manually opt-in to the fixed behavior. >> > >> > In order to set the new tunable to correct the behavior:: >> > >> > ceph osd crush set-tunable straw_calc_version 1 >> > >> > Note that this change will have no immediate effect. However, from >> > this point forward, any 'straw' bucket in your CRUSH map that is >> > adjusted will get non-buggy internal weights, and that transition >> > may trigger some rebalancing. >> > >> > You can estimate how much rebalancing will eventually be necessary >> > on your cluster with:: >> > >> > ceph osd getcrushmap -o /tmp/cm >> > crushtool -i /tmp/cm --num-rep 3 --test --show-mappings > /tmp/a >> 2>&1 >> > crushtool -i /tmp/cm --set-straw-calc-version 1 -o /tmp/cm2 >> > crushtool -i /tmp/cm2 --reweight -o /tmp/cm2 >> > crushtool -i /tmp/cm2 --num-rep 3 --test --show-mappings > /tmp/b >> > 2>&1 >> > wc -l /tmp/a # num total mappings >> > diff -u /tmp/a /tmp/b | grep -c ^+# num changed mappings >> > >> >Divide the total number of lines in /tmp/a with the number of lines >> >changed. We've found that most clusters are under 10%. >> > >> >You can force all of this rebalancing to happen at once with:: >> > >> > ceph osd crush reweight-all >> > >> >Otherwise, it will happen at some unknown
Re: [ceph-users] [SPAM] Changing pg_num => RBD VM down !
I'm using the latest Giant and have the same issue. When i increase PG_num of a pool from 2048 to 2148, my VMs is still ok. When i increase from 2148 to 2400, some VMs die (Qemu-kvm process die). My physical servers (host VMs) running kernel 3.13 and use librbd. I think it's a bug in librbd with crushmap. (I set crush_tunables3 on my ceph cluster, does it make sense?) Do you know a way to safely increase PG_num? (I don't think increase PG_num 100 each times is a safe & good way) Regards, On Mon, Mar 16, 2015 at 8:50 PM, Florent B wrote: > We are on Giant. > > On 03/16/2015 02:03 PM, Azad Aliyar wrote: > > > > May I know your ceph version.?. The latest version of firefly 80.9 has > > patches to avoid excessive data migrations during rewighting osds. You > > may need set a tunable inorder make this patch active. > > > > This is a bugfix release for firefly. It fixes a performance regression > > in librbd, an important CRUSH misbehavior (see below), and several RGW > > bugs. We have also backported support for flock/fcntl locks to ceph-fuse > > and libcephfs. > > > > We recommend that all Firefly users upgrade. > > > > For more detailed information, see > > http://docs.ceph.com/docs/master/_downloads/v0.80.9.txt > > > > Adjusting CRUSH maps > > > > > > * This point release fixes several issues with CRUSH that trigger > > excessive data migration when adjusting OSD weights. These are most > > obvious when a very small weight change (e.g., a change from 0 to > > .01) triggers a large amount of movement, but the same set of bugs > > can also lead to excessive (though less noticeable) movement in > > other cases. > > > > However, because the bug may already have affected your cluster, > > fixing it may trigger movement *back* to the more correct location. > > For this reason, you must manually opt-in to the fixed behavior. > > > > In order to set the new tunable to correct the behavior:: > > > > ceph osd crush set-tunable straw_calc_version 1 > > > > Note that this change will have no immediate effect. However, from > > this point forward, any 'straw' bucket in your CRUSH map that is > > adjusted will get non-buggy internal weights, and that transition > > may trigger some rebalancing. > > > > You can estimate how much rebalancing will eventually be necessary > > on your cluster with:: > > > > ceph osd getcrushmap -o /tmp/cm > > crushtool -i /tmp/cm --num-rep 3 --test --show-mappings > /tmp/a > 2>&1 > > crushtool -i /tmp/cm --set-straw-calc-version 1 -o /tmp/cm2 > > crushtool -i /tmp/cm2 --reweight -o /tmp/cm2 > > crushtool -i /tmp/cm2 --num-rep 3 --test --show-mappings > /tmp/b > > 2>&1 > > wc -l /tmp/a # num total mappings > > diff -u /tmp/a /tmp/b | grep -c ^+# num changed mappings > > > >Divide the total number of lines in /tmp/a with the number of lines > >changed. We've found that most clusters are under 10%. > > > >You can force all of this rebalancing to happen at once with:: > > > > ceph osd crush reweight-all > > > >Otherwise, it will happen at some unknown point in the future when > >CRUSH weights are next adjusted. > > > > Notable Changes > > --- > > > > * ceph-fuse: flock, fcntl lock support (Yan, Zheng, Greg Farnum) > > * crush: fix straw bucket weight calculation, add straw_calc_version > > tunable (#10095 Sage Weil) > > * crush: fix tree bucket (Rongzu Zhu) > > * crush: fix underflow of tree weights (Loic Dachary, Sage Weil) > > * crushtool: add --reweight (Sage Weil) > > * librbd: complete pending operations before losing image (#10299 Jason > > Dillaman) > > * librbd: fix read caching performance regression (#9854 Jason Dillaman) > > * librbd: gracefully handle deleted/renamed pools (#10270 Jason Dillaman) > > * mon: fix dump of chooseleaf_vary_r tunable (Sage Weil) > > * osd: fix PG ref leak in snaptrimmer on peering (#10421 Kefu Chai) > > * osd: handle no-op write with snapshot (#10262 Sage Weil) > > * radosgw-admi > > > > > > > > > > On 03/16/2015 12:37 PM, Alexandre DERUMIER wrote: > > >>> VMs are running on the same nodes than OSD > > > Are you sure that you didn't some kind of out of memory. > > > pg rebalance can be memory hungry. (depend how many osd you have). > > > > 2 OSD per host, and 5 hosts in this cluster. > > hosts h > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Running ceph in Deis/Docker
Hi, This is a followup question to my previous question. When the last monitor in a ceph monitor set is down, what is the proper way to boot up the ceph monitor set again? On one hand, we could try not to make this happen, but on the other hand, as Murphy Law states, I am sure it will happen sooner or later. Thanks - JC On 16/12/14 12:14 pm, Christian Balzer wrote: Hello, your subject is misleading, as this is not really related to Deis/Docker. Find the very recent "Is mon initial members used after the first quorum?" thread in this ML. In short, list all your 3 mons in the initial members section. And yes, rebooting things all at the same time can be "fun", I managed to get into a similar situation like yours once (though that cluster really had only one mon) and it took a hard reset to fix things eventually. Christian On Tue, 16 Dec 2014 08:52:15 +0800 Jimmy Chu wrote: Hi, I installed ceph on 3 nodes, having one monitor, and one OSD running on each node. After rebooting them all at once (I see this may be a bad move now), the ceph monitors refuse to connect to each other. When I run: ceph mon getmap -o /etc/ceph/monmap or even ceph -s It only shows the following: Dec 14 16:38:44 deis-1 sh[933]: 2014-12-14 08:38:44.265419 7f5cec71f700 0 -- :/121 >> 10.132.183.191:6789/0 pipe(0x7f5ce40296a0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5ce4029930).fault Dec 14 16:38:44 deis-1 sh[933]: 2014-12-14 08:38:44.265419 7f5cec71f700 0 -- :/121 >> 10.132.183.192:6789/0 pipe(0x7f5ce40296a0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5ce4029930).fault Dec 14 16:38:50 deis-1 sh[933]: 2014-12-14 08:38:50.267398 7f5cec71f700 0 -- :/121 >> 10.132.183.190:6789/0 pipe(0x7f5cd40030e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5cd4003370).fault ...keep repeating... So, there is no quorum formed, and ceph admin socket file is not there for connection. What should be my next step to recover the storage? This is *my /etc/ceph/ceph.conf file*: [global] fsid = cc368515-9dc6-48e2-9526-58ac4cbb3ec9 mon initial members = deis-3 auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min_size = 1 osd pool default pg_num = 128 osd pool default pgp_num = 128 osd recovery delay start = 15 log file = /dev/stdout [mon.deis-3] host = deis-3 mon addr = 10.132.183.190:6789 [mon.deis-1] host = deis-1 mon addr = 10.132.183.191:6789 [mon.deis-2] host = deis-2 mon addr = 10.132.183.192:6789 [client.radosgw.gateway] host = deis-store-gateway keyring = /etc/ceph/ceph.client.radosgw.keyring rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock log file = /dev/stdout Thank you. - Jimmy Chu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Running ceph in Deis/Docker
Hi, I installed ceph on 3 nodes, having one monitor, and one OSD running on each node. After rebooting them all at once (I see this may be a bad move now), the ceph monitors refuse to connect to each other. When I run: ceph mon getmap -o /etc/ceph/monmap or even ceph -s It only shows the following: Dec 14 16:38:44 deis-1 sh[933]: 2014-12-14 08:38:44.265419 7f5cec71f700 0 -- :/121 >> 10.132.183.191:6789/0 pipe(0x7f5ce40296a0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5ce4029930).fault Dec 14 16:38:44 deis-1 sh[933]: 2014-12-14 08:38:44.265419 7f5cec71f700 0 -- :/121 >> 10.132.183.192:6789/0 pipe(0x7f5ce40296a0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5ce4029930).fault Dec 14 16:38:50 deis-1 sh[933]: 2014-12-14 08:38:50.267398 7f5cec71f700 0 -- :/121 >> 10.132.183.190:6789/0 pipe(0x7f5cd40030e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5cd4003370).fault ...keep repeating... So, there is no quorum formed, and ceph admin socket file is not there for connection. What should be my next step to recover the storage? This is *my /etc/ceph/ceph.conf file*: [global] fsid = cc368515-9dc6-48e2-9526-58ac4cbb3ec9 mon initial members = deis-3 auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min_size = 1 osd pool default pg_num = 128 osd pool default pgp_num = 128 osd recovery delay start = 15 log file = /dev/stdout [mon.deis-3] host = deis-3 mon addr = 10.132.183.190:6789 [mon.deis-1] host = deis-1 mon addr = 10.132.183.191:6789 [mon.deis-2] host = deis-2 mon addr = 10.132.183.192:6789 [client.radosgw.gateway] host = deis-store-gateway keyring = /etc/ceph/ceph.client.radosgw.keyring rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock log file = /dev/stdout Thank you. - Jimmy Chu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Running ceph in Deis/Docker
Hi, I am running a 3-node Deis cluster with ceph as underlying FS. So it is ceph running inside Docker containers running in three separate servers. I rebooted all three nodes (almost at once). After rebooted, the ceph monitor refuse to connect to each other. Symptoms are: - no quorum formed, - ceph admin socket file does not exist - only the following in ceph log: Dec 14 16:38:44 deis-1 sh[933]: 2014-12-14 08:38:44.265419 7f5cec71f700 0 -- :/121 >> 10.132.183.191:6789/0 pipe(0x7f5ce40296a0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5ce4029930).fault Dec 14 16:38:44 deis-1 sh[933]: 2014-12-14 08:38:44.265419 7f5cec71f700 0 -- :/121 >> 10.132.183.192:6789/0 pipe(0x7f5ce40296a0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5ce4029930).fault Dec 14 16:38:50 deis-1 sh[933]: 2014-12-14 08:38:50.267398 7f5cec71f700 0 -- :/121 >> 10.132.183.190:6789/0 pipe(0x7f5cd40030e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f5cd4003370).fault ...keep repeating... This is *my /etc/ceph/ceph.conf file*: [global] fsid = cc368515-9dc6-48e2-9526-58ac4cbb3ec9 mon initial members = deis-3 auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min_size = 1 osd pool default pg_num = 128 osd pool default pgp_num = 128 osd recovery delay start = 15 log file = /dev/stdout [mon.deis-3] host = deis-3 mon addr = 10.132.183.190:6789 [mon.deis-1] host = deis-1 mon addr = 10.132.183.191:6789 [mon.deis-2] host = deis-2 mon addr = 10.132.183.192:6789 [client.radosgw.gateway] host = deis-store-gateway keyring = /etc/ceph/ceph.client.radosgw.keyring rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock log file = /dev/stdout *IP table of the docker host:** *core@deis-3 ~ $ sudo iptables --list Chain INPUT (policy DROP) target prot opt source destination Firewall-INPUT all -- anywhere anywhere Chain FORWARD (policy DROP) target prot opt source destination ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:http ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:https ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt: ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Firewall-INPUT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain Firewall-INPUT (2 references) target prot opt source destination ACCEPT all -- anywhere anywhere ACCEPT icmp -- anywhere anywhere icmp echo-reply ACCEPT icmp -- anywhere anywhere icmp destination-unreachable ACCEPT icmp -- anywhere anywhere icmp time-exceeded ACCEPT icmp -- anywhere anywhere icmp echo-request ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- 10.132.183.190 anywhere ACCEPT all -- 10.132.183.192 anywhere ACCEPT all -- 10.132.183.191 anywhere ACCEPT all -- anywhere anywhere ACCEPT tcp -- anywhere anywhere ctstate NEW multiport dports ssh,,http,https LOGall -- anywhere anywhere LOG level warning REJECT all -- anywhere anywhere reject-with icmp-host-prohibited All private IPs are ping-gable within the ceph monitor container. What could I do next to troubleshoot this issue? Thanks a lot! - Jimmy Chu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [URGENT] My CEPH cluster is dying (due to "incomplete" PG)
I have no choice except re-create this PG: # ceph pg force_create_pg 6.9d8 But it still stuck at creating: # ceph pg dump | grep creating dumped all in format plain 6.9d8 0 0 0 0 0 0 0 0 creating2014-11-09 03:27:23.611838 0'0 0:0 [] -1 [] -1 0'0 0.000'0 0.00 Do you have any suggestion? Thank you very much indeed! On Sun, Nov 9, 2014 at 12:52 AM, Chu Duc Minh wrote: > My ceph cluster have a pg in state "incomplete" and i can not query them > any more. > > *# ceph pg 6.9d8 query* (hang forever) > > All my volumes may be lost data because of this PG. > > # ceph pg dump_stuck inactive > ok > pg_stat objects mip degrmispunf bytes log disklog > state state_stamp v reportedup up_primary > acting acting_primary last_scrub scrub_stamp last_deep_scrub > deep_scrub_stamp > 6.9d8 30730 0 0 0 12678708736 4865 > 4865incomplete 2014-11-09 00:07:20.838531 109334'8040089 > 110376:1118 *[88,43,32] *88 [88,43,32] 88 > 91654'7460936 2014-10-10 10:36:25.433016 81667'5815892 2014-08-29 > 09:44:14.012219 > > My folders .../current/6.9d8_head still have data in some OSDs (osd.43), > how i can force CEPH to use data in osd.43 for this PG? I tried repair, > mark lost osd, v.v... > But i can't help. > > Can you give me some suggesstions. My data is dying :( > > Thank you very much! > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] [URGENT] My CEPH cluster is dying (due to "incomplete" PG)
My ceph cluster have a pg in state "incomplete" and i can not query them any more. *# ceph pg 6.9d8 query* (hang forever) All my volumes may be lost data because of this PG. # ceph pg dump_stuck inactive ok pg_stat objects mip degrmispunf bytes log disklog state state_stamp v reportedup up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 6.9d8 30730 0 0 0 12678708736 4865 4865incomplete 2014-11-09 00:07:20.838531 109334'8040089 110376:1118 *[88,43,32] *88 [88,43,32] 88 91654'7460936 2014-10-10 10:36:25.433016 81667'5815892 2014-08-29 09:44:14.012219 My folders .../current/6.9d8_head still have data in some OSDs (osd.43), how i can force CEPH to use data in osd.43 for this PG? I tried repair, mark lost osd, v.v... But i can't help. Can you give me some suggesstions. My data is dying :( Thank you very much! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RBD command crash & can't delete volume!
Hi, i will start a new email thread, but i think it related to this rbd bug. Do you have any suggestion about quick fix for this buggy volume (eg: way to safely delete it,...)? Maybe it is a reason to make me can not start the last OSD. Thank you very much! On Fri, Nov 7, 2014 at 10:14 PM, Jason Dillaman wrote: > It appears you have discovered a bug in librbd that occurs when a child's > parent image is missing or corrupt. I have opened the following ticket for > this issue: http://tracker.ceph.com/issues/10030. > > For the OSD failure, can you start a new email thread with the supporting > details of that issue? > > -- > > Jason Dillaman > Red Hat > dilla...@redhat.com > http://www.redhat.com > > > -- > *From: *"Chu Duc Minh" > *To: *ceph-de...@vger.kernel.org, "ceph-users@lists.ceph.com >> > ceph-users@lists.ceph.com" > *Sent: *Friday, November 7, 2014 7:05:58 AM > *Subject: *[ceph-users] RBD command crash & can't delete volume! > > > Hi folks, some volumes in my ceph cluster have problem and I can NOT > delete it by rbd command. When i show info or try to delete it, rbd comand > crash. > > Command i used: > > *# rbd -p volumes info volume-e110b0a5-5116-46f2-99c7-84bb546f15c2# rbd -p > volumes rm volume-e110b0a5-5116-46f2-99c7-84bb546f15c2* > > I attach crash-dump log in this email, pls help me investigate it. > > PS: This problem makes me unable to start an osd too. (If i star this osd, > it die few minutes after that. I retried many times but got the same > result.) > > Thank you! > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] RBD command crash & can't delete volume!
Hi folks, some volumes in my ceph cluster have problem and I can NOT delete it by rbd command. When i show info or try to delete it, rbd comand crash. Command i used: *# rbd -p volumes info volume-e110b0a5-5116-46f2-99c7-84bb546f15c2# rbd -p volumes rm volume-e110b0a5-5116-46f2-99c7-84bb546f15c2* I attach crash-dump log in this email, pls help me investigate it. PS: This problem makes me unable to start an osd too. (If i star this osd, it die few minutes after that. I retried many times but got the same result.) Thank you! root@ceph-mon-02:~# rbd -p volumes info volume-e110b0a5-5116-46f2-99c7-84bb546f15c2 [331/331] 2014-11-07 18:53:28.075090 7fa78f90d780 -1 librbd::ImageCtx: error reading immutable metadata: (2) No such file or directory 2014-11-07 18:53:28.075397 7fa78f90d780 -1 librbd: error opening parent image: (2) No such file or directory ./log/SubsystemMap.h: In function 'bool ceph::log::SubsystemMap::should_gather(unsigned int, int)' thread 7fa78f90d780 time 2014-11-07 18:53:28. 075425 ./log/SubsystemMap.h: 62: FAILED assert(sub < m_subsys.size()) ceph version 0.87-6-gdba7def (dba7defc623474ad17263c9fccfec60fe7a439f0) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0x7fa78c9f39ff] 2: (()+0x2eca0) [0x7fa78edf8ca0] 3: (()+0x36679) [0x7fa78ee00679] 4: (librbd::close_image(librbd::ImageCtx*)+0x36) [0x7fa78ee13146] 5: (librbd::open_parent(librbd::ImageCtx*)+0x663) [0x7fa78ee1a7f3] 6: (librbd::refresh_parent(librbd::ImageCtx*)+0x138) [0x7fa78ee1aca8] 7: (librbd::ictx_refresh(librbd::ImageCtx*)+0xa41) [0x7fa78ee0e361] 8: (librbd::open_image(librbd::ImageCtx*)+0x1a4) [0x7fa78ee19fc4] 9: (librbd::RBD::open_read_only(librados::IoCtx&, librbd::Image&, char const*, char const*)+0x8a) [0x7fa78edf989a] 10: (main()+0x14b3) [0x40e0b3] 11: (__libc_start_main()+0xed) [0x7fa78bad276d] 12: rbd() [0x413d49] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2014-11-07 18:53:28.076481 7fa78f90d780 -1 ./log/SubsystemMap.h: In function 'bool ceph::log::SubsystemMap::should_gather(unsigned int, int)' th read 7fa78f90d780 time 2014-11-07 18:53:28.075425 ./log/SubsystemMap.h: 62: FAILED assert(sub < m_subsys.size()) ceph version 0.87-6-gdba7def (dba7defc623474ad17263c9fccfec60fe7a439f0) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0x7fa78c9f39ff] 2: (()+0x2eca0) [0x7fa78edf8ca0] 3: (()+0x36679) [0x7fa78ee00679] 4: (librbd::close_image(librbd::ImageCtx*)+0x36) [0x7fa78ee13146] 5: (librbd::open_parent(librbd::ImageCtx*)+0x663) [0x7fa78ee1a7f3] 6: (librbd::refresh_parent(librbd::ImageCtx*)+0x138) [0x7fa78ee1aca8] 7: (librbd::ictx_refresh(librbd::ImageCtx*)+0xa41) [0x7fa78ee0e361] 8: (librbd::open_image(librbd::ImageCtx*)+0x1a4) [0x7fa78ee19fc4] 9: (librbd::RBD::open_read_only(librados::IoCtx&, librbd::Image&, char const*, char const*)+0x8a) [0x7fa78edf989a] 10: (main()+0x14b3) [0x40e0b3] 11: (__libc_start_main()+0xed) [0x7fa78bad276d] 12: rbd() [0x413d49] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -82> 2014-11-07 18:53:27.989029 7fa78f90d780 5 asok(0x23d6c60) register_command perfcounters_dump hook 0x23d8af0 -81> 2014-11-07 18:53:27.989059 7fa78f90d780 5 asok(0x23d6c60) register_command 1 hook 0x23d8af0 -80> 2014-11-07 18:53:27.989063 7fa78f90d780 5 asok(0x23d6c60) register_command perf dump hook 0x23d8af0 -79> 2014-11-07 18:53:27.989069 7fa78f90d780 5 asok(0x23d6c60) register_command perfcounters_schema hook 0x23d8af0 -78> 2014-11-07 18:53:27.989074 7fa78f90d780 5 asok(0x23d6c60) register_command 2 hook 0x23d8af0 -77> 2014-11-07 18:53:27.989082 7fa78f90d780 5 asok(0x23d6c60) register_command perf schema hook 0x23d8af0 -76> 2014-11-07 18:53:27.989092 7fa78f90d780 5 asok(0x23d6c60) register_command config show hook 0x23d8af0 -75> 2014-11-07 18:53:27.989095 7fa78f90d780 5 asok(0x23d6c60) register_command config set hook 0x23d8af0 -74> 2014-11-07 18:53:27.989102 7fa78f90d780 5 asok(0x23d6c60) register_command config get hook 0x23d8af0 -73> 2014-11-07 18:53:27.989107 7fa78f90d780 5 asok(0x23d6c60) register_command config diff hook 0x23d8af0 -72> 2014-11-07 18:53:27.989110 7fa78f90d780 5 asok(0x23d6c60) register_command log flush hook 0x23d8af0 -71> 2014-11-07 18:53:27.989116 7fa78f90d780 5 asok(0x23d6c60) register_command log dump hook 0x23d8af0 -70> 2014-11-07 18:53:27.989120 7fa78f90d780 5 asok(0x23d6c60) register_command log reopen hook 0x23d8af0 -69> 2014-11-07 18:53:27.995981 7fa78f90d780 10 monclient(hunting): build_initial_monmap -68> 2014-11-07 18:53:27.996113 7fa78f90d780 1 librados: starting msgr at :/0 -67> 2014-11-07 18:53:27.996121 7fa78f90d780 1 librados: starting objecter -66> 2014-11-07 18:53:27.996189 7fa78f90d780 5 asok(0x23d6c60) register_command objecter_requests hook
Re: [ceph-users] OSDs vanishing from Ceph cluster?
@font-face{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2
4;}You should run "ceph osd tree" and post the
output here.
BR,On March 28, 2014, at 5:54AM, Dan Koren
wrote:Just
ran into this problem: a week ago I set up a Ceph cluster on 4
systems, with one admin node and 3 mon+osd nodes, then ran a fewcasual
IO tests. I returned to work after a few days out of town at
a
conference, and now my Ceph cluster appears to have no OSDs!
root@rts24:/var/log/ceph# ceph status
cluster
284dbfe0-e612-4732-9d26-2c5909f0fbd1
health HEALTH_ERR 119 pgs
degraded; 192 pgs stale; 192 pgs stuck stale; 119 pgs stuck unclean; recovery
2/4 objects degraded (50.000%); no osds
monmap e1: 3 mons at {rts21=http://172.29.0.21:6789/0,rts22=172.29.0.22:6789/0,rts23=172.29.0.23:6789/0";>172.29.0.21:6789/0,rts22=172.29.0.22:6789/0,rts23=172.29.0.23:6789/0},
election epoch 32, quorum 0,1,2 rts21,rts22,rts23
osdmap e33: 0 osds: 0 up, 0
in
pgmap v2774: 192 pgs, 3 pools, 135 bytes data, 2
objects
0 kB used, 0 kB / 0 kB avail
2/4 objects degraded
(50.000%)
73
stale+active+clean
119 stale+active+degraded
I would appreciate if anyone could explain how can something likethis
happen, or where to look for any evidence that might help me
understand
what happened. The log files in /var/log/ceph/ show no
activity except for the monitors' Paxos chatter.
Thx,Dan
KorenDirector of SoftwareDATERA |
650.210.7910 | @dateranews
mailto:d...@datera.io";
target="_blank">dnk@datera.io
This email and any attachments thereto may
contain private,confidential, and privileged material for the sole use of
the
intended recipient. Any review, copying, or distribution ofthis email (or
any attachments thereto) by others is strictlyprohibited. If you are not
the intended recipient, pleasecontact the sender immediately and
permanently delete the
original and any copies of this email and any
attachmentsthereto.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] "rbd import" so slow
When using RBD backend for Openstack volume, i can easily surpass 200MB/s. But when using "rbd import" command, eg: # rbd import --pool test Debian.raw volume-Debian-1 --new-format --id volumes I only can import at speed ~ 30MB/s I don't know why rbd import slow? What can i do to improve import speed? Thanks you very much! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Speed limit on RadosGW?
My cluster has 3 MON nodes & 6 DATA nodes, all nodes have 2Gbps connectivity (bonding). Each Data node has 14 SATA HDD (osd), each journal on the same disk as OSD. Each MON node run RadosGW too. On Oct 15, 2013 12:34 AM, "Kyle Bader" wrote: > I've personally saturated 1Gbps links on multiple radosgw nodes on a large > cluster, if I remember correctly, Yehuda has tested it up into the 7Gbps > range with 10Gbps gear. Could you describe your clusters hardware and > connectivity? > > > On Mon, Oct 14, 2013 at 3:34 AM, Chu Duc Minh wrote: > >> Hi sorry, i missed this mail. >> >> >> > During writes, does the CPU usage on your RadosGW node go way up? >> No, CPU stay the same & very low (< 10%) >> >> When upload small files(300KB/file) over RadosGW: >> - using 1 process: upload bandwidth ~ 3MB/s >> - using 100 processes: upload bandwidth ~ 15MB/s >> >> When upload big files(3GB/file) over RadosGW: >> - using 1 process: upload bandwidth ~ 70MB/s >> (Therefore i don't upload big files using multi-processes any more :D) >> >> Maybe, RadosGW have a problem when write many smail files. Or it's a >> problem of CEPH when simultaneously write many smail files into a bucket, >> that already have millions files? >> >> >> On Wed, Sep 25, 2013 at 7:24 PM, Mark Nelson wrote: >> >>> On 09/25/2013 02:49 AM, Chu Duc Minh wrote: >>> >>>> I have a CEPH cluster with 9 nodes (6 data nodes & 3 mon/mds nodes) >>>> And i setup 4 separate nodes to test performance of Rados-GW: >>>> - 2 node run Rados-GW >>>> - 2 node run multi-process put file to [multi] Rados-GW >>>> >>>> Result: >>>> a) When i use 1 RadosGW node & 1 upload-node, speed upload = 50MB/s >>>> /upload-node, Rados-GW input/output speed = 50MB/s >>>> >>>> b) When i use 2 RadosGW node & 1 upload-node, speed upload = 50MB/s >>>> /upload-node; each RadosGW have input/output = 25MB/s ==> sum >>>> input/ouput of 2 Rados-GW = 50MB/s >>>> >>>> c) When i use 1 RadosGW node & 2 upload-node, speed upload = 25MB/s >>>> /upload-node ==> sum output of 2 upload-node = 50MB/s, RadosGW have >>>> input/output = 50MB/s >>>> >>>> d) When i use 2 RadosGW node & 2 upload-node, speed upload = 25MB/s >>>> /upload-node ==> sum output of 2 upload-node = 50MB/s; each RadosGW have >>>> input/output = 25MB/s ==> sum input/ouput of 2 Rados-GW = 50MB/s >>>> >>>> _*Problem*_: i can pass limit 50MB/s when put file over Rados-GW, >>>> >>>> regardless of the number Rados-GW nodes and upload-nodes. >>>> When i use this CEPH cluster over librados (openstack/kvm), i can easily >>>> achieve > 300MB/s >>>> >>>> I don't know why performance of RadosGW is so low. What's bottleneck? >>>> >>> >>> During writes, does the CPU usage on your RadosGW node go way up? >>> >>> If this is a test cluster, you might want to try the wip-6286 build from >>> our gitbuilder site. There is a fix that depending on the size of your >>> objects, could have a big impact on performance. We're currently >>> investigating some other radosgw performance issues as well, so stay tuned. >>> :) >>> >>> Mark >>> >>> >>> >>>> Thank you very much! >>>> >>>> >>>> >>>> >>>> __**_ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >>>> >>>> >>> __**_ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >>> >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > > > -- > > Kyle > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Speed limit on RadosGW?
Hi sorry, i missed this mail. > During writes, does the CPU usage on your RadosGW node go way up? No, CPU stay the same & very low (< 10%) When upload small files(300KB/file) over RadosGW: - using 1 process: upload bandwidth ~ 3MB/s - using 100 processes: upload bandwidth ~ 15MB/s When upload big files(3GB/file) over RadosGW: - using 1 process: upload bandwidth ~ 70MB/s (Therefore i don't upload big files using multi-processes any more :D) Maybe, RadosGW have a problem when write many smail files. Or it's a problem of CEPH when simultaneously write many smail files into a bucket, that already have millions files? On Wed, Sep 25, 2013 at 7:24 PM, Mark Nelson wrote: > On 09/25/2013 02:49 AM, Chu Duc Minh wrote: > >> I have a CEPH cluster with 9 nodes (6 data nodes & 3 mon/mds nodes) >> And i setup 4 separate nodes to test performance of Rados-GW: >> - 2 node run Rados-GW >> - 2 node run multi-process put file to [multi] Rados-GW >> >> Result: >> a) When i use 1 RadosGW node & 1 upload-node, speed upload = 50MB/s >> /upload-node, Rados-GW input/output speed = 50MB/s >> >> b) When i use 2 RadosGW node & 1 upload-node, speed upload = 50MB/s >> /upload-node; each RadosGW have input/output = 25MB/s ==> sum >> input/ouput of 2 Rados-GW = 50MB/s >> >> c) When i use 1 RadosGW node & 2 upload-node, speed upload = 25MB/s >> /upload-node ==> sum output of 2 upload-node = 50MB/s, RadosGW have >> input/output = 50MB/s >> >> d) When i use 2 RadosGW node & 2 upload-node, speed upload = 25MB/s >> /upload-node ==> sum output of 2 upload-node = 50MB/s; each RadosGW have >> input/output = 25MB/s ==> sum input/ouput of 2 Rados-GW = 50MB/s >> >> _*Problem*_: i can pass limit 50MB/s when put file over Rados-GW, >> >> regardless of the number Rados-GW nodes and upload-nodes. >> When i use this CEPH cluster over librados (openstack/kvm), i can easily >> achieve > 300MB/s >> >> I don't know why performance of RadosGW is so low. What's bottleneck? >> > > During writes, does the CPU usage on your RadosGW node go way up? > > If this is a test cluster, you might want to try the wip-6286 build from > our gitbuilder site. There is a fix that depending on the size of your > objects, could have a big impact on performance. We're currently > investigating some other radosgw performance issues as well, so stay tuned. > :) > > Mark > > > >> Thank you very much! >> >> >> >> >> __**_ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >> >> > __**_ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com<http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Speed limit on RadosGW?
I have a CEPH cluster with 9 nodes (6 data nodes & 3 mon/mds nodes) And i setup 4 separate nodes to test performance of Rados-GW: - 2 node run Rados-GW - 2 node run multi-process put file to [multi] Rados-GW Result: a) When i use 1 RadosGW node & 1 upload-node, speed upload = 50MB/s /upload-node, Rados-GW input/output speed = 50MB/s b) When i use 2 RadosGW node & 1 upload-node, speed upload = 50MB/s /upload-node; each RadosGW have input/output = 25MB/s ==> sum input/ouput of 2 Rados-GW = 50MB/s c) When i use 1 RadosGW node & 2 upload-node, speed upload = 25MB/s /upload-node ==> sum output of 2 upload-node = 50MB/s, RadosGW have input/output = 50MB/s d) When i use 2 RadosGW node & 2 upload-node, speed upload = 25MB/s /upload-node ==> sum output of 2 upload-node = 50MB/s; each RadosGW have input/output = 25MB/s ==> sum input/ouput of 2 Rados-GW = 50MB/s *Problem*: i can pass limit 50MB/s when put file over Rados-GW, regardless of the number Rados-GW nodes and upload-nodes. When i use this CEPH cluster over librados (openstack/kvm), i can easily achieve > 300MB/s I don't know why performance of RadosGW is so low. What's bottleneck? Thank you very much! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] xattr on cephfs with dosattrib
於 2013年05月16日 20:45, Ulrich Schinz 提到: Hi there, today i setup my ceph-cluster. it's up and running fine. mounting cephfs to my "client"-machine works fine as well. ~# mount 172.17.50.71:6789:/ on /samba/ceph type ceph (rw,relatime,name=admin,secret=) touching a file in that directory and setting xattr works perfect. ~# touch /samba/ceph/test.txt ~# setfattr -n user.test -v test /samba/ceph/test.txt ~# getfattr -d /samba/ceph/test.txt getfattr: Entferne führenden '/' von absoluten Pfadnamen # file: samba/ceph/test.txt user.test="test" so this seems to work fine. but when i try to copy a file from a samba share into ceph-dir, i get following: ~# cp -a /samba/data/test.txt /samba/ceph/ cp: Erhalten der Zugriffsrechte für „./test.txt“: Die Operation wird nicht unterstützt (trying to translate: cp: preserving attributes for "Read..": operation not supported) also trying to change attributes from windows-clients via samba is not working. windows-error is: access denied dumping the attributes of the copied file (cp -a...) the attributes are identical. original file: # file: test.txt user.DOSATTRIB=0sMHgyMwADEQAAACAAAJzI1eupK84BAAA= copied file: # file: test.txt user.DOSATTRIB=0sMHgyMwADEQAAACAAAJzI1eupK84BAAA= only thing that changed are/is acl-settings: original file: ~# getfacl /samba/data/test.txt getfacl: Entferne führende '/' von absoluten Pfadnamen # file: samba/data/test.txt # owner: 300 # group: users user::rwx user:root:rwx group::--- group:users:--- group:300:rwx mask::rwx other::--- copied file: getfacl test.txt # file: test.txt # owner: 300 # group: users user::rwx group::rwx other::--- trying to set acl i get error: operation not supported maybe there is a problem with mount-options, but i couldn't find options for mount.ceph which could solve this. in ceph-logfiles i can't find any errors. syslog, messages, dmesg everything looks ok on client and on server(s). someone with some hints where i could search for errors, or maybe an idea what the problem could be... kind regards uli Hi Ulrich, I encounter this issue too recently. My current understanding is cephfs client (I use the kernel client but I think this applies to fuse too) does not support ACL operations which samba uses (http://tracker.ceph.com/issues/27). If you turn on samba debug you'll find log similar to the following. [2013/05/17 11:18:35.219348, 10] modules/vfs_posixacl.c:110(posixacl_sys_acl_set_file) acl_set_file failed: Operation not supported I'm still investigating on how to make samba store ACl in xattr *ONLY* and do not try to set posix ACL. "acl_xattr: ignore system acls = yes" doesn't seems to work for me. Best regards, Alex ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
