Re: [ceph-users] ceph-users Digest, Vol 67, Issue 6

Am 06.08.2018 um 22:01 schrieb Pawel S: ## ceph-ansible ### Section Releases (URL: http://docs.ceph.com/ceph-ansible/master/index.html#releases) This section says that supported ansible versions are 2.4 and 2.5. Is the current version 2.6.2 supported as well or do I have to use one of the earli

Re: [ceph-users] Upgrading journals to BlueStore: a conundrum

Oh, I see. Then there's probably no way around replacing the entire host. Zitat von Robert Stanford : Eugen: I've tried similar approaches in the past and it seems like it won't work like that. I have to zap the entire journal disk. Also I plan to use the configuration tunable for making the

[ceph-users] cephfs kernel client hangs

Hi, I have a CEPH 12.2.5 cluster running on 4 CentOS 7.3 servers with kernel 4.17.0, Including 3 mons, 16 osds, 2 mds(1active+1backup). I have some cllients mounted cephfs in kernel mode. Client A is using kernel 4.4.145, and others are using kernel 4.12.8. All of them are using ceph client versi

Re: [ceph-users] Whole cluster flapping

It’s been over a week now and the whole cluster keeps flapping, it is never the same OSDs that go down. Is there a way to get the progress of this recovery ? (The pool hat I deleted is no longer present (for a while now)) In fact, there is a lot of i/o activity on the server where osds go down.

Re: [ceph-users] cephfs kernel client hangs

Hi, you are using the kernel implementation of CephFS. In this case some information can be retrieved from the /sys/kernel/debug/ceph/ directory. Especially the mdsc, monc and osdc files are important, since they contain pending operations on mds, mon and osds. We have a similar problem in

Re: [ceph-users] cephfs kernel client hangs

Hi Burkhard, Files located in /sys/kernel/debug/ceph/ are all new files generated after I reboot the server. The clients were in blacklist and I manully remove them from the blacklist. But the clients hung still. Thanks Burkhard Linke 于2018年8月7日周二 下午4:54写道： > Hi, > > > you are using the kernel

Re: [ceph-users] Beginner's questions regarding Ceph, Deployment with ceph-ansible

Am 06.08.2018 um 22:01 schrieb Pawel S: > On Mon, Aug 6, 2018 at 3:08 PM J?rg Kastning wrote: But what are agents, rgws, nfss, restapis, rbdmirrors, clients and iscsi-gws? Where could I found additional information about them? Where for and how do I use them? Please, put me to the right section

Re: [ceph-users] cephfs kernel client hangs

did you mount cephfs on the same machines that run ceph-osd? On Tue, Aug 7, 2018 at 5:14 PM Zhenshi Zhou wrote: > > Hi Burkhard, > Files located in /sys/kernel/debug/ceph/ are all new files generated > after I reboot the server. > The clients were in blacklist and I manully remove them from the

Re: [ceph-users] ceph-mgr dashboard behind reverse proxy

Hi, Am 04.08.2018 um 09:04 schrieb Tobias Florek: > I want to set up the dashboard behind a reverse proxy. How do >>> people determine which ceph-mgr is active? Is there any simple and >>> elegant solution? >> You can use haproxy. It supports periodic check for the availability >> of the configu

Re: [ceph-users] cephfs kernel client hangs

Yes, some osd servers mount cephfs Yan, Zheng 于2018年8月7日 周二19:12写道： > did you mount cephfs on the same machines that run ceph-osd? > > On Tue, Aug 7, 2018 at 5:14 PM Zhenshi Zhou wrote: > > > > Hi Burkhard, > > Files located in /sys/kernel/debug/ceph/ are all new files > generated after I reboot

Re: [ceph-users] Beginner's questions regarding Ceph, Deployment with ceph-ansible

On 08/07/2018 11:23 AM, Jörg Kastning wrote: > Am 06.08.2018 um 22:01 schrieb Pawel S: >> On Mon, Aug 6, 2018 at 3:08 PM J?rg Kastning > wrote: >>> But what are agents, rgws, nfss, restapis, rbdmirrors, clients and >>> iscsi-gws? Where could I found additional information about them? Where >>> f

Re: [ceph-users] cephfs kernel client hangs

On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou wrote: > > Yes, some osd servers mount cephfs > this can cause memory deadlock. you should avoid doing this > Yan, Zheng 于2018年8月7日 周二19:12写道： >> >> did you mount cephfs on the same machines that run ceph-osd? >> >> On Tue, Aug 7, 2018 at 5:14 PM Zhens

Re: [ceph-users] OSD had suicide timed out

To follow up, I did some further digging with debug_osd=20/20 and it appears as if there's no traffic to the OSD, even though it comes UP for the cluster (this started happening on another OSD in the cluster today, same stuff):    -27> 2018-08-07 14:10:55.146531 7f9fce3cd700 10 osd.0 12560 ha

Re: [ceph-users] Ceph MDS and hard links

I switched configs to use ms_type: simple and restarted all of our MDS (there are 3 but only 1 active). It looks like the memory usage crept back up to the same levels as before. I've included new mempool dump and heap stat. If I can provide other debug info let me know. ceph daemon mds.xxx co

[ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

Hi, 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of records like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the corresponding log files, e.g. 2018-08-07 04:34:06.000585 7fdd8f012700 0 /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries end_key=1_015336164

Re: [ceph-users] Whole cluster flapping

Frédéric, see if the number of objects is decreasing in the pool with `ceph df [detail]` Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil* *IRC NICK - WebertRLZ* On Tue, Aug 7, 2018 at 5:46 AM CUZA Frédéric wrote: > It’s been over a week now and the whole clust

Re: [ceph-users] Whole cluster flapping

Pool is already deleted and no longer present in stats. Regards, De : ceph-users De la part de Webert de Souza Lima Envoyé : 07 August 2018 15:08 À : ceph-users Objet : Re: [ceph-users] Whole cluster flapping Frédéric, see if the number of objects is decreasing in the pool with `ceph df [det

Re: [ceph-users] cephfs kernel client hangs

Hi Yan, thanks for the advice. I will umount the cephfs on osd servers and keep an eye on it.:) Yan, Zheng 于2018年8月7日周二 下午7:51写道： > On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou wrote: > > > > Yes, some osd servers mount cephfs > > > > this can cause memory deadlock. you should avoid doing this >

[ceph-users] Recovering from broken sharding: fill_status OVER 100%

Hi, On my test servers, I created a bucket using 12.2.5, turned on versioning, uploaded 100,000 objects, and the bucket broke, as expected. Autosharding said it was running but didn't complete. Then I upgraded that cluster to 12.2.7. Resharding seems to have finished, but now that cluster s

Re: [ceph-users] Whole cluster flapping

oops, my bad, you're right. I don't know much you can see but maybe you can dig around performance counters and see what's happening on those OSDs, try these: ~# ceph daemonperf osd.XX ~# ceph daemon osd.XX perf dump change XX to your OSD numbers. Regards, Webert Lima DevOps Engineer at MAV Te

Re: [ceph-users] cephfs kernel client hangs

Yan, Zheng 于2018年8月7日周二 下午7:51写道： > On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou wrote: > this can cause memory deadlock. you should avoid doing this > > > Yan, Zheng 于2018年8月7日 周二19:12写道： > >> > >> did you mount cephfs on the same machines that run ceph-osd? > >> I didn't know about this. I ru

Re: [ceph-users] Inconsistent PG could not be repaired

Last time I had an inconsistent PG that could not be repaired using the repair command, I looked at which OSDs hosted the PG, then restarted them one by one(usually stopping, waiting a few seconds, then starting them back up ). You could also stop them, flush the journal, then start them back u

[ceph-users] pg count question

Folks, I am little confused so just need clarification, I have 14 osd in my cluster and i want to create two pool (pool-1 & pool-2) how do i device pg between two pool with replication 3 Question: 1 Is this correct formula? 14 * 100 / 3 / 2 = 233 ( power of 2 would be 256) So should i give

Re: [ceph-users] pg count question

Question 2: ceph osd pool set-quota max_objects|max_bytes set object or byte limit on pool > Le 7 août 2018 à 16:50, Satish Patel a écrit : > > Folks, > > I am little confused so just need clarification, I have 14 osd in my > clu

[ceph-users] Bluestore OSD Segfaults (12.2.5/12.2.7)

Hi all, We have recently begun switching over to Bluestore on our Ceph cluster, currently on 12.2.7. We first began encountering segfaults on Bluestore during 12.2.5, but strangely these segfaults apply exclusively to our SSD pools and not the PCIE/HDD disks. We upgraded to 12.2.7 last week to

Re: [ceph-users] Least impact when adding PG's

Hi All, Any advice? Thanks, John ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] BlueStore performance: SSD vs on the same spinning disk

I was surprised to see an email on this list a couple of days ago, which said that write performance would actually fall with BlueStore. I thought the reason BlueStore existed was to increase performance. Nevertheless, it seems like filestore is going away and everyone should upgrade. My quest

Re: [ceph-users] BlueStore performance: SSD vs on the same spinning disk

n 07/08/18 17:10, Robert Stanford wrote: > >  I was surprised to see an email on this list a couple of days ago, > which said that write performance would actually fall with BlueStore.  I > thought the reason BlueStore existed was to increase performance.  > Nevertheless, it seems like filestore i

Re: [ceph-users] cephfs kernel client hangs

This is the first I am hearing about this as well. Granted, I am using ceph-fuse rather than the kernel client at this point, but that isn’t etched in stone. Curious if there is more to share. Reed > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima > wrote: > > > Yan, Zheng mailto:uker...@

Re: [ceph-users] Broken multipart uploads

But still, I get No Such key !! s3cmd abortmp s3://weird_bucket 2~CDIJMxZvy8aQejBGBPeNyQK-AJ1lmO4 ERROR: S3 error: 404 (NoSuchKey) s3cmd abortmp s3://weird_bucket 2~alvAZmF5tAlSeiJrUjOwXV7Io22uH0H ERROR: S3 error: 404 (NoSuchKey) Regards, Harry. On 06/08/2018 04:29 πμ, Konstantin Shalygin

Re: [ceph-users] cephfs kernel client hangs

On Tue, Aug 7, 2018 at 5:42 PM Reed Dier wrote: > > This is the first I am hearing about this as well. This is not a Ceph-specific thing -- it can also affect similar systems like Lustre. The classic case is when under some memory pressure, the kernel tries to free memory by flushing the client'

Re: [ceph-users] [Ceph-community] How much RAM and CPU cores would you recommend when using ceph only as block storage for KVM?

ceph-users is a better place to ask this kind of question. Anyway the 1GB RAM per TB storage recommendation still stands as far as I know plus you want some for the OS and some safety margin so in your case 64GB seem sensible On Wed, 8 Aug 2018, 01:51 Cheyenne Forbes, wrote: > The case is 28TB

Re: [ceph-users] cephfs kernel client hangs

That's good to know, thanks for the explanation. Fortunately we are in the process of cluster redesign and we can definitely fix that scenario. Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil* *IRC NICK - WebertRLZ* On Tue, Aug 7, 2018 at 4:37 PM John Spray wrot

Re: [ceph-users] cephfs kernel client hangs

Hi, I'm not sure if it just mounts the cephfs without using or doing any operation within the mounted directory would be affected by flushing cache. I mounted cephfs on osd servers only for testing and then left it there. Anyway I will umount it. Thanks John Spray 于2018年8月8日 周三03:37写道： > On Tue,

Re: [ceph-users] CephFS - Mounting a second Ceph file system

We are using kernel 4.15.17 and we keep receiving this error mount.ceph: unrecognized mount option "mds_namespace", passing to kernel. Running Luminous 12.2.7 on the client. It seems like mds_namespace is not a recognized option. The actual command is as follows: mount -t ceph -v -o name=admin,s

Re: [ceph-users] Bluestore OSD Segfaults (12.2.5/12.2.7)

Looks like https://tracker.ceph.com/issues/21826 which is a dup of https://tracker.ceph.com/issues/20557 On Wed, Aug 8, 2018 at 1:49 AM, Thomas White wrote: > Hi all, > > We have recently begun switching over to Bluestore on our Ceph cluster, > currently on 12.2.7. We first began encountering se

Re: [ceph-users] OSD had suicide timed out

Try to work out why the other osds are saying this one is down. Is it because this osd is too busy to respond or something else. debug_ms = 1 will show you some message debugging which may help. On Tue, Aug 7, 2018 at 10:34 PM, Josef Zelenka wrote: > To follow up, I did some further digging with

[ceph-users] permission errors rolling back ceph cluster to v13

Hi, I have been running into some connection issues with the latest ceph-14 version, so we thought the feasible solution would be to roll back the cluster to previous version (ceph-13.0.1) where things are known to work properly. I'm wondering if rollback/downgrade is supported at all ? After

Re: [ceph-users] pg count question

Thanks! Do you have any comments on Question: 1 ? On Tue, Aug 7, 2018 at 10:59 AM, Sébastien VIGNERON wrote: > Question 2: > > ceph osd pool set-quota max_objects|max_bytes > set object or byte limit on pool > > >> Le 7 août 2018 à

Re: [ceph-users] cephfs kernel client hangs

Hi, I check all my ceph servers and they are not mount cephfs on each of them(maybe I umount after testing). As a result, the cluster didn't encounter a memory deadlock. Besides, I check the monitoring system and the memory and cpu usage were at common level while the clients hung. Back to my quest

Re: [ceph-users] Broken multipart uploads

On 08/08/2018 01:31 AM, DHD.KOHA wrote: But still, I get No Such key !! s3cmd abortmp s3://weird_bucket 2~CDIJMxZvy8aQejBGBPeNyQK-AJ1lmO4 ERROR: S3 error: 404 (NoSuchKey) s3cmd abortmp s3://weird_bucket 2~alvAZmF5tAlSeiJrUjOwXV7Io22uH0H ERROR: S3 error: 404 (NoSuchKey) Try like this: s3cmd

Re: [ceph-users] cephfs kernel client hangs

On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou wrote: > > Hi, > I check all my ceph servers and they are not mount cephfs on each of > them(maybe I umount after testing). As a result, the cluster didn't encounter > a memory deadlock. Besides, I check the monitoring system and the memory and > cpu

Re: [ceph-users] Inconsistent PGs every few days

Hi, I run a cluster with 7 OSD. The cluster has no much traffic on it. But every few days, I get a HEALTH_ERR, because of inconsistent PGs: root at Sam ~ # ceph status cluster: id: c4bfc288-8ba8-4c3a-b3a6-ed95503f50b7

Re: [ceph-users] cephfs kernel client hangs

I restarted the client server so that there's no file in that directory. I will take care of it if the client hangs next time. Thanks Yan, Zheng 于2018年8月8日周三 上午11:23写道： > On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou wrote: > > > > Hi, > > I check all my ceph servers and they are not mount ceph