Re: [ceph-users] MDSs report damaged metadata

Hi there! Does anyone else have an idea what I could do to get rid of this error? BTW: it is the third time that the pg 20.0 is gone inconsistent. This is a pg from the metadata pool (cephfs). May this be related anyhow? # ceph health detail HEALTH_ERR 1 MDSs report damaged metadata; 1 scrub

Re: [ceph-users] How RBD tcp connection works

Hi, Thanks Br. Yafeng On Tue, 20 Aug 2019 at 11:14, Eliza wrote: > Hi > > on 2019/8/20 11:00, fengyd wrote: > > I think you're right. > > I am not so sure about it. But I think ceph client always wants to know > the cluster's topology, so it needs to communicate with cluster all the > time.

Re: [ceph-users] How RBD tcp connection works

Hi on 2019/8/20 11:00, fengyd wrote: I think you're right. I am not so sure about it. But I think ceph client always wants to know the cluster's topology, so it needs to communicate with cluster all the time. The big difference for ceph to other distributed storage is clients participate

Re: [ceph-users] How RBD tcp connection works

Hi, I think you're right. thanks. Br. Yafeng On Tue, 20 Aug 2019 at 10:59, Eliza wrote: > > > on 2019/8/20 10:57, fengyd wrote: > > Long connections means new tcp connection which connect the same targets > > is reestablished after timeout? > > yes, once timeouted, then reconnecting. >

Re: [ceph-users] How RBD tcp connection works

on 2019/8/20 10:57, fengyd wrote: Long connections means new tcp connection which connect the same targets is reestablished after timeout? yes, once timeouted, then reconnecting. ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] How RBD tcp connection works

Hi, Long connections means new tcp connection which connect the same targets is reestablished after timeout? On Tue, 20 Aug 2019 at 10:37, Eliza wrote: > Hi > > on 2019/8/20 10:30, fengyd wrote: > > If the creation timestamp of the FD is not changed, but the socket > > information to which

Re: [ceph-users] How RBD tcp connection works

Hi on 2019/8/20 10:30, fengyd wrote: If the creation timestamp of  the FD is not changed, but the socket information to which the FD was linked is changed, it means new tcp connection is established. If there's no reading/wring ongoing,  why new tcp connection is still established and the FD

Re: [ceph-users] How RBD tcp connection works

Hi, 1. Create a VM and a volume, and attach the volume to VM. Check the FD count with lsof and the FD count is increased by 10. 2. Fill the volume with dd command on the VM Check the FD count with lsof and the FD count is increased dramatically and stable after the FD count is increased by

[ceph-users] 答复: How does CephFS find a file?

Thank you very much! I understand it now. 发件人: Patrick Donnelly 发送时间: 2019年8月20日 4:35 收件人: Robert LeBlanc 抄送: aot...@outlook.com; ceph-users@lists.ceph.com 主题: Re: [ceph-users]

Re: [ceph-users] How RBD tcp connection works

on 2019/8/20 9:54, fengyd wrote: I checked the FD information with the command "ls -l /proc/25977/fd"  // here 25977 is Qemu process. I found that the creation timestamp of  the FD was not changed, but the socket information to which the FD was linked was changed. So, I guess the FD is reused

Re: [ceph-users] How RBD tcp connection works

Hi, I checked the FD information with the command "ls -l /proc/25977/fd" // here 25977 is Qemu process. I found that the creation timestamp of the FD was not changed, but the socket information to which the FD was linked was changed. So, I guess the FD is reused when establishing new tcp

Re: [ceph-users] RESOLVED: Sudden loss of all SSD OSDs in a cluster, immedaite abort on restart [Mimic 13.2.6]

While I'm still unsure how this happened, this is what was done to solve this. Started OSD in foreground with debug 10, watched for the most recent osdmap epoch mentioned before abort(). For example, if it mentioned that it just tried to load 80896 and then crashed # ceph osd getmap -o

Re: [ceph-users] latency on OSD

We recommend you SSD Hi all, I have installed ceph luminous, witch 5 nodes(45 OSD) * 5 ceph-osd network: bond lacp 10GB RAM: 96GB HD: 9 disk SATA-3TB (bluestore) I wanted to ask for help to fix the latency of the osd "ceph osd perf" You who recommend me? My config is:

[ceph-users] latency on OSD

Hi all, I have installed ceph luminous, witch 5 nodes(45 OSD) * 5 ceph-osd network: bond lacp 10GB RAM: 96GB HD: 9 disk SATA-3TB (bluestore) I wanted to ask for help to fix the latency of the osd "ceph osd perf" You who recommend me? My config is: /etc/ceph/ceph.conf [global] fsid =

Re: [ceph-users] How does CephFS find a file?

On Mon, Aug 19, 2019 at 7:50 AM Robert LeBlanc wrote: > The MDS manages dentries as omap (simple key/value database) entries in the > metadada pool. Each dentry keeps a list of filenames and some metadata about > the file such as inode number and some other info such as size I presume > (can't

[ceph-users] lz4 compression?

Dear all, I've not seen posts from people using LZ4 compression, and wondered what other peoples experiences are if they have tried LZ4 on Nautilus. Since enabling LZ4 we have copied 1.9 PB into a pool without problem. However, and if "ceph df detail" is accurate, we are not getting much

Re: [ceph-users] How does CephFS find a file?

I'm fairly new to CephFS, but in my poking around with it, this is what I understand. The MDS manages dentries as omap (simple key/value database) entries in the metadada pool. Each dentry keeps a list of filenames and some metadata about the file such as inode number and some other info such as

Re: [ceph-users] MDSs report damaged metadata - "return_code": -116

Hi Paul, thanks for the hint. I did a recursive scrub from "/". The log says there where some inodes with bad backtraces repaired. But the error remains. May this have something to do with a deleted file? Or a file within a snapshot? The path told by # ceph tell mds.mds3 damage ls 2019-08-19

Re: [ceph-users] Ceph Balancer code

Hi, On 8/18/19 12:06 AM, EDH - Manuel Rios Fernandez wrote: Hi , Whats the reason for not allow balancer PG if objects are inactive/misplaced at least in nautilus 14.2.2 ? https://github.com/ceph/ceph/blob/master/src/pybind/mgr/balancer/module.py#L874 *snipsnap* We can understood

Re: [ceph-users] MDSs report damaged metadata - "return_code": -116

Hi, that error just says that the path is wrong. I unfortunately don't know the correct way to instruct it to scrub a stray path off the top of my head; you can always run a recursive scrub on / to go over everything, though Paul -- Paul Emmerich Looking for help with your Ceph cluster?

Re: [ceph-users] MDSs report damaged metadata - "return_code": -116

Hi all! Where can I look up what the error number means? Or did I something wrong in my command line? Thanks in advance, Lars Fri, 16 Aug 2019 13:31:38 +0200 Lars Täuber ==> Paul Emmerich : > Hi Paul, > > thank you for your help. But I get the following error: > > # ceph tell mds.mds3 scrub

Re: [ceph-users] Correct number of pg

Wonderful, we will leave our pg at 4096 :) many thanks for the advice Paul :) have a good day, Jake On 8/19/19 11:03 AM, Paul Emmerich wrote: > On Mon, Aug 19, 2019 at 10:51 AM Jake Grimmett wrote: >> >> Dear All, >> >> We have a new Nautilus cluster, used for cephfs, with pg_autoscaler in >>

Re: [ceph-users] Correct number of pg

On Mon, Aug 19, 2019 at 10:51 AM Jake Grimmett wrote: > > Dear All, > > We have a new Nautilus cluster, used for cephfs, with pg_autoscaler in > warn mode. > > Shortly after hitting 62% full, the autoscaler started warning that we > have too few pg: > >

[ceph-users] How does CephFS find a file?

I am a student new to cephfs. I think there are 2 steps to finding a file: 1.Find out which objects belong to this file. 2.Use CRUSH to find out OSDs. What I don’t know is how does CephFS get the object list of the file. Does MDS save all object list of all files? Or CRUSH can use some

Re: [ceph-users] How RBD tcp connection works

I collected the lsof at different time and found that: The total number of open FD is stable at a fixed value, and some of tcp connection are changed. On Mon, 19 Aug 2019 at 16:42, fengyd wrote: > -how long do you monitor after r/w finish? > More than 900 seconds. > > I executed the following

[ceph-users] Correct number of pg

Dear All, We have a new Nautilus cluster, used for cephfs, with pg_autoscaler in warn mode. Shortly after hitting 62% full, the autoscaler started warning that we have too few pg: * Pool ec82pool has 4096 placement groups, should have

Re: [ceph-users] How RBD tcp connection works

-how long do you monitor after r/w finish? More than 900 seconds. I executed the following command last Saturday and today, the output was same. sudo lsof -p 5509 | wc -l And the result from /proc: ls -ltr /proc/5509/fd | grep socket | grep "Aug 13" | wc -l 134 sudo ls -ltr /proc/5509/fd | grep

Re: [ceph-users] How RBD tcp connection works

how long do you monitor after r/w finish? there is a configure item named 'ms_connection_idle_timeout' which default value is 900 fengyd 于2019年8月19日周一 下午4:10写道： > > Hi, > > I have a question about tcp connection. > In the test environment, openstack uses ceph RBD as backend storage. > I created

Re: [ceph-users] How RBD tcp connection works

Hi, on 2019/8/19 16:10, fengyd wrote: I think when reading/writing to volume/image, tcp connection needs to be established which needs FD, then the FD count may increase. But after reading/writing, why the FD count doesn't descrease? The tcp may be long connections.

[ceph-users] How RBD tcp connection works

Hi, I have a question about tcp connection. In the test environment, openstack uses ceph RBD as backend storage. I created a VM and attache a volume/image to the VM. I monitored how many fd was used by Qemu process. I used the command dd to fill the whole volume/image. I found that the FD count

Re: [ceph-users] Sudden loss of all SSD OSDs in a cluster, immedaite abort on restart [Mimic 13.2.6]

For me, it was the .rgw.meta pool that had very dense placement groups. The OSDs would fail to start and would then commit suicide while trying to scan the PGs. We had to remove all references of those placement groups just to get the OSDs to start. It wasn't pretty. On Mon, Aug 19, 2019, 2:09

Re: [ceph-users] Sudden loss of all SSD OSDs in a cluster, immedaite abort on restart [Mimic 13.2.6]

Yes, it's possible that they do, but since all of the affected OSDs are still down and the monitors have been restarted since, all of those pools have pgs that are in unknown state and don't return anything in ceph pg ls. There weren't that many placement groups for the SSDs, but also I don't