[ceph-users] How does ceph pg repair work in jewel or later versions of ceph?

I have read that the pg repair is simply copy the data from the primary osd to other osds.Is that true?or the later version of ceph has improved that? 2017-05-05 lin.yunfan___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/li

Re: [ceph-users] Ceph memory overhead when used with KVM

Hi, I used one of the fio example files and changed it a bit: """ # This job file tries to mimic the Intel IOMeter File Server Access Pattern [global] description=Emulation of Intel IOmeter File Server Access Pattern randrepeat=0 filename=/root/test.dat # IOMeter defines the server loads as the fo

Re: [ceph-users] Intel power tuning - 30% throughput performance increase

Sounds good, but could also have a config option to set it before dropping root? On 4 May 2017 20:28, "Brad Hubbard" wrote: On Thu, May 4, 2017 at 10:58 AM, Haomai Wang wrote: > refer to https://github.com/ceph/ceph/pull/5013 How about we issue a warning about possible performance implications

Re: [ceph-users] Limit bandwidth on RadosGW?

Yes, but I have not yet found an open source reverse proxy can achieve it. haproxy blocks requests instead of limit bandwidth in a fixed Mbps; nginx can only limit the download speed (by the option proxy_limit_rate), and has a negative side effect that it buffers the response body, it cause huge pe

Re: [ceph-users] Limit bandwidth on RadosGW?

According to the link you provided, haproxy seems can only block requests instead of limit bandwidth. If people violate the rate limiting in this example they are redirected to the backend ease-up-y0 which gives them 503 error page that can be customized. On Fri, May 5, 2017 at 7:38 AM, George

Re: [ceph-users] Intel power tuning - 30% throughput performance increase

On Thu, May 4, 2017 at 10:58 AM, Haomai Wang wrote: > refer to https://github.com/ceph/ceph/pull/5013 How about we issue a warning about possible performance implications if we detect this is not set to 1 *or* 0 at startup? > > On Thu, May 4, 2017 at 7:56 AM, Brad Hubbard wrote: >> +ceph-devel

Re: [ceph-users] Limit bandwidth on RadosGW?

Terminate the connections on haproxy which is great for ssl as well, and use these instructions to set qos per connection and data transferred: http://blog.serverfault.com/2010/08/26/1016491873/ > On May 4, 2017, at 04:35, hrchu wrote: > > Thanks for reply. > > tc can only do limit on int

Re: [ceph-users] Limit bandwidth on RadosGW?

On Thu, May 04, 2017 at 04:35:21PM +0800, hrchu wrote: > Thanks for reply. > > tc can only do limit on interfaces or given IPs, but what I am talking > about is "per connection", e.g., each put object could be 5MB/s, get > object could be 1MB/s. To achieve your required level of control, you need

Re: [ceph-users] Extremely high OSD memory utilization on Kraken 11.2.0 (with XFS -or- bluestore)

Hi Aaron- Sorry, lost track of this one. In order to get backtraces out of the core you need the matching executables. Can you make sure the ceph-osd-dbg or ceph-debuginfo package is installed on the machine (depending on if it's deb or rpm) and then gdb ceph-osd corefile and 'thr app all bt'

[ceph-users] RS vs LRC - abnormal results

Hi, I'm comparing different configurations of LRC with Reed-Solomon. Specifically I'm comparing the total data read in all OSDs during a reconstruction of a single node (I drop a single OSD and measure until the system is stable again). While most of the configurations output the desired result, a

Re: [ceph-users] Extremely high OSD memory utilization on Kraken 11.2.0 (with XFS -or- bluestore)

Were the backtraces we obtained not useful? Is there anything else we can try to get the OSDs up again? On Wed, Apr 19, 2017 at 4:18 PM, Aaron Ten Clay wrote: > I'm new to doing this all via systemd and systemd-coredump, but I appear to > have gotten cores from two OSD processes. When xzipped the

Re: [ceph-users] Replication (k=1) in LRC

Hi Loic Commenting out the sanity check did the trick. The code is working as I'd expected. Thanks On Fri, Apr 28, 2017 at 1:48 AM, Loic Dachary wrote: > > > On 04/27/2017 11:43 PM, Oleg Kolosov wrote: > > Hi Loic, > > Of course. > > I'm implementing a version of Pyramid Code. In Pyramid you re

[ceph-users] Monitor issues

Hello, So at some point during the night, our monitor 1 server rebooted for so far unknown reason. When it came back up, the clock was skewed by 6 hours. There were no right happening when I got alerted to the issue. ceph shows all OSD's up and in, but no op/s and 600+ blocked requests. I logge

Re: [ceph-users] How to calculate the nearfull ratio ?

On Thu, May 4, 2017 at 5:30 AM Loic Dachary wrote: > Hi, > > In a cluster where the failure domain is the host and dozens of hosts, the > 85% default for nearfull ratio is fine. A host failing won't suddenly make > the cluster 99% full. In smaller clusters, with 10 hosts or less, it is > likely t

Re: [ceph-users] corrupted rbd filesystems since jewel

Hi Stefan - we simply disabled exclusive-lock on all older (pre-jewel) images. We still allow the default jewel featuresets for newly created images because as you mention - the issue does not seem to affect them. On Thu, May 4, 2017 at 10:19 AM, Stefan Priebe - Profihost AG < s.pri...@profihost.a

Re: [ceph-users] corrupted rbd filesystems since jewel

and yes i also see hung tasks in those VMs until they crash. Stefan Am 04.05.2017 um 19:11 schrieb Brian Andrus: > Sounds familiar... and discussed in "disk timeouts in libvirt/qemu VMs..." > > We have not had this issue since reverting exclusive-lock, but it was > suggested this was not the iss

Re: [ceph-users] corrupted rbd filesystems since jewel

Hello Brian, this really sounds the same. I don't see this on a cluster with only images created AFTER jewel. And it seems to start happening after i enabled exclusive lock on all images. Did just use feature disable, exclusive-lock,fast-diff,object-map or did you also restart all those vms? Gre

Re: [ceph-users] corrupted rbd filesystems since jewel

Sounds familiar... and discussed in "disk timeouts in libvirt/qemu VMs..." We have not had this issue since reverting exclusive-lock, but it was suggested this was not the issue. So far it's held up for us with not a single corrupt filesystem since then. On some images (ones created post-Jewel up

Re: [ceph-users] Reg: PG

If you delete and recreate the pools you will indeed lose data. Your cephfs_metadata pool will have almost no data in it. I have a 9TB cephfs_data pool and 40MB in the cephfs_metadata pool. It shouldn't have anywhere near 128 PGs in it based on a cluster this size. When you increase your cluste

Re: [ceph-users] How to calculate the nearfull ratio ?

On 05/04/2017 03:58 PM, Xavier Villaneau wrote: > Hello Loïc, > > On Thu, May 4, 2017 at 8:30 AM Loic Dachary > wrote: > > Is there a way to calculate the optimum nearfull ratio for a given > crushmap ? > > > This is a question that I was planning to cover in th

Re: [ceph-users] Reg: PG

Hi David, Thanks for your explanation. I have ran following command to create pg pool. ceph osd pool create cephfs_data 128 ceph osd pool create cephfs_metadata 128 ceph fs new dev-ceph-setup cephfs_metadata cephfs_data Is it a proper way for 3 osd? Does delete and recreate pg

Re: [ceph-users] Checking the current full and nearfull ratio

On Thu, 4 May 2017, Adam Carheden wrote: > How do I check the full ratio and nearfull ratio of a running cluster? > > I know i can set 'mon osd full ratio' and 'mon osd nearfull ratio' in > the [global] setting of ceph.conf. But things work fine without those > lines (uses defaults, obviously). >

[ceph-users] Checking the current full and nearfull ratio

How do I check the full ratio and nearfull ratio of a running cluster? I know i can set 'mon osd full ratio' and 'mon osd nearfull ratio' in the [global] setting of ceph.conf. But things work fine without those lines (uses defaults, obviously). They can also be changed with `ceph tell mon.* injec

Re: [ceph-users] Reg: PG

The extra pools are probably the data and metadata pools that are automatically created for cephfs. http://ceph.com/pgcalc/ is a useful tool for helping to work out how many PGs your pools should have. Rich On 04/05/17 15:41, David Turner wrote: > I'm guessing you have more than just the 1 pool

Re: [ceph-users] Reg: PG

I'm guessing you have more than just the 1 pool with 128 PGs in your cluster (seeing as you have 320 PGs total, I would guess 2 pools with 128 PGs and 1 pool with 64 PGs). The combined total number of PGs for all of your pools is 320 and with only 3 OSDs and most likely replica size 3... that leav

Re: [ceph-users] Ceph Performance

On 05/04/17 13:37, Fuxion Cloud wrote: > Hi, > > Our ceph version is 0.80.7. We used it with the openstack as a block > storage RBD. The ceph storage configured with 3 replication of data. > I'm getting low IOPS (400) from fio benchmark in random readwrite. > Please advise how to improve it. Thank

[ceph-users] Reg: PG

Hi, I'm running 3 osd in my test setup. I have created PG pool with 128 as per the ceph documentation. But i'm getting too many PGs warning. Can anyone clarify? why i'm getting this warning. Each OSD contain 240GB disk. cluster 9d325da2-3d87-4b6b-8cca-e52a4b65aa08 h

Re: [ceph-users] How to calculate the nearfull ratio ?

Hello Loïc, On Thu, May 4, 2017 at 8:30 AM Loic Dachary wrote: > Is there a way to calculate the optimum nearfull ratio for a given > crushmap ? > This is a question that I was planning to cover in those calculations I was working on for python-crush. I've currently shelved the work for a few w

Re: [ceph-users] How to calculate the nearfull ratio ?

from our experience even 80% is a 'dangerous zone' ( unfortunately this is how it goes ... and it is quiet wasteful comparing to other solutions ) On Thu, May 4, 2017 at 3:57 PM, David Turner wrote: > The Ceph Enterprise default is 65% nearfull. Do not go above 85% nearfull > unless y

Re: [ceph-users] How to calculate the nearfull ratio ?

The Ceph Enterprise default is 65% nearfull. Do not go above 85% nearfull unless you are stuck while backfilling and need to increase it to add/remove storage. Ceph needs overhead to be able to recover from situations where disks are lost. I always take into account what would happen to the %full i

Re: [ceph-users] corrupted rbd filesystems since jewel

Hi Jason, > Odd. Can you re-run "rbd rm" with "--debug-rbd=20" added to the > command and post the resulting log to a new ticket at [1]? will do so next time. I was able to solve this by restarting all osds. After that i was able to successfuly delete the image. > I'd also be interested if you c

Re: [ceph-users] Ceph newbie thoughts and questions

For gluster, when files are written into it as a mounted network gluster filesystem, it word a lot of metadata for each object to know everything it needs to about it for replication purposes. If you put the data manually on the brick then it wouldn't be able to sync. Correct, 3 mons, 2 mds, and 3

[ceph-users] Rebalancing causing IO Stall/IO Drops to zero

Hi Everyone, We keep running into stalled IOs / they also drop almost to zero, whenever a node suddenly would go down or if there was a large amount of rebalancing going on and once rebalancing is completed, we would also get stalled io for 2-10 mins. Has anyone seen this behaviour before an

[ceph-users] How to calculate the nearfull ratio ?

Hi, In a cluster where the failure domain is the host and dozens of hosts, the 85% default for nearfull ratio is fine. A host failing won't suddenly make the cluster 99% full. In smaller clusters, with 10 hosts or less, it is likely to not be enough. And in larger clusters 85% may be too much t

Re: [ceph-users] corrupted rbd filesystems since jewel

Odd. Can you re-run "rbd rm" with "--debug-rbd=20" added to the command and post the resulting log to a new ticket at [1]? I'd also be interested if you could re-create that "librbd::object_map::InvalidateRequest" issue repeatably. [1] http://tracker.ceph.com/projects/rbd/issues On Thu, May 4, 20

Re: [ceph-users] Ceph Performance

Hi, Our ceph version is 0.80.7. We used it with the openstack as a block storage RBD. The ceph storage configured with 3 replication of data. I'm getting low IOPS (400) from fio benchmark in random readwrite. Please advise how to improve it. Thanks. Here's the hardware info. 12 x storage nodes -

Re: [ceph-users] Ceph health warn MDS failing to respond to cache pressure

I have faced the same problem many times. Usually it doesn't cause anything bad, but I had a 30 min system outage twice because of this. It might be because of the number of inodes on your ceph filesystem. Go to the MDS server and do (supposing your mds server id is intcfs-osd1): ceph daemon mds.

Re: [ceph-users] Limit bandwidth on RadosGW?

Mandi! Marc Roos In chel di` si favelave... > Just a thought, what about marking connections with iptables and using > that mark with tc? Surely, but many things have to be taken into account: a) doing traffic control mean disabling ALL network hardware optimizations (queue, offline checksu

Re: [ceph-users] Limit bandwidth on RadosGW?

Just a thought, what about marking connections with iptables and using that mark with tc? -Original Message- From: hrchu [mailto:petertc@gmail.com] Sent: donderdag 4 mei 2017 10:35 To: Marc Roos; ceph-users Subject: Re: [ceph-users] Limit bandwidth on RadosGW? Thanks for re

Re: [ceph-users] Ceph Performance

On Thu, May 4, 2017 at 7:53 PM, Fuxion Cloud wrote: > Hi all, > > Im newbie in ceph technology. We have ceph deployed by vendor 2 years ago > with Ubuntu 14.04LTS without fine tuned the performance. I noticed that the > performance of storage is very slow. Can someone please help to advise how >

Re: [ceph-users] Limit bandwidth on RadosGW?

Thanks for reply. tc can only do limit on interfaces or given IPs, but what I am talking about is "per connection", e.g., each put object could be 5MB/s, get object could be 1MB/s. Correct me if anything wrong. Regards, Chu, Hua-Rong (曲華榮), +886-3-4227151 #57968 Networklab, Computer Science &

Re: [ceph-users] Ceph newbie thoughts and questions

Thank you very much for your answer David, just what I was after! Just some additional questions to make it clear to me. The mds do not need to be in odd numbers? They can be set up 1,2,3,4 aso. as needed? You made the basics clear to me so when I set up my first ceph fs I need as a start: 3 m

Re: [ceph-users] Limit bandwidth on RadosGW?

No experience with it. But why not use linux for it? Maybe this solution on every RGW is sufficient, I cannot imagine you need 3rd party for this. https://unix.stackexchange.com/questions/28198/how-to-limit-network-bandwidth https://wiki.archlinux.org/index.php/Advanced_traffic_control --

[ceph-users] Ceph Performance

Hi all, Im newbie in ceph technology. We have ceph deployed by vendor 2 years ago with Ubuntu 14.04LTS without fine tuned the performance. I noticed that the performance of storage is very slow. Can someone please help to advise how to improve the performance? Any changes or configuration requir

Re: [ceph-users] corrupted rbd filesystems since jewel

There are no watchers involved: # rbd status cephstor2/vm-136-disk-1 Watchers: none Greets, Stefan Am 04.05.2017 um 09:45 schrieb Stefan Priebe - Profihost AG: > Example: > # rbd rm cephstor2/vm-136-disk-1 > Removing image: 99% complete... > > Stuck at 99% and never completes. This is an image w

Re: [ceph-users] corrupted rbd filesystems since jewel

Example: # rbd rm cephstor2/vm-136-disk-1 Removing image: 99% complete... Stuck at 99% and never completes. This is an image which got corrupted for an unknown reason. Greets, Stefan Am 04.05.2017 um 08:32 schrieb Stefan Priebe - Profihost AG: > I'm not sure whether this is related but our backu

[ceph-users] Limit bandwidth on RadosGW?

Hi all, I want to limit RadosGW per connection upload/download speed for QoS. There is no build-in option for this, so maybe a 3rd party reverse proxy in front of Radosgw is needed. Does anyone have experience about this? ___ ceph-users mailing list ceph-