Re: [ceph-users] 13.2.4 odd memory leak?

> On 8 Mar 2019, at 14.30, Mark Nelson wrote: > > > On 3/8/19 5:56 AM, Steffen Winther Sørensen wrote: >> >>> On 5 Mar 2019, at 10.02, Paul Emmerich >> <mailto:paul.emmer...@croit.io>> wrote: >>> >>> Yeah, there's a bug

Re: [ceph-users] 13.2.4 odd memory leak?

r 5, 2019 at 9:00 AM Steffen Winther Sørensen > wrote: >> >> >> >> On 4 Mar 2019, at 16.09, Paul Emmerich wrote: >> >> Bloated to ~4 GB per OSD and you are on HDDs? >> >> Something like that yes. >> >> >> 13.2.3 backported

Re: [ceph-users] 13.2.4 odd memory leak?

sizes cache_size 1073741824 Any hints appreciated! /Steffen > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 189658

[ceph-users] 13.2.4 odd memory leak?

List Members, patched a centos 7 based cluster from 13.2.2 to 13.2.4 last monday, everything appeared working fine. Only this morning I found all OSDs in the cluster to be bloated in memory foot print, possible after weekend backup through MDS. Anyone else seeing possible memory leak in 13.2.

Re: [ceph-users] Mimic 13.2.1 release date

> On 9 Jul 2018, at 17.11, Wido den Hollander wrote: > > Hi, > > Is there a release date for Mimic 13.2.1 yet? > > There are a few issues which currently make deploying with Mimic 13.2.0 > a bit difficult, for example: > > - https://tracker.ceph.com/issues/24423 > - https://github.com/ceph/c

Re: [ceph-users] FYI - Mimic segv in OSD

> On 9 Jul 2018, at 15.49, John Spray wrote: > > On Mon, Jul 9, 2018 at 2:37 PM Steffen Winther Sørensen > wrote: >> >> Dunno if this has been seen before so just for info, 1 in 24 OSD just did >> this: >> >> Jul 9 15:13:35 n4 ceph-osd: *** Caugh

[ceph-users] FYI - Mimic segv in OSD

Dunno if this has been seen before so just for info, 1 in 24 OSD just did this: Jul 9 15:13:35 n4 ceph-osd: *** Caught signal (Segmentation fault) ** Jul 9 15:13:35 n4 ceph-osd: in thread 7ff209282700 thread_name:msgr-worker-2 Jul 9 15:13:35 n4 kernel: msgr-worker-2[4697]: segfault at 0 ip 000

Re: [ceph-users] Performance tuning for SAN SSD config

> On 6 Jul 2018, at 17.55, Matthew Stroud wrote: > > We have changed the IO scheduler to NOOP, which seems to yield the best > results. However, I haven’t look into messing around with tuned. Let me play > with that and see if I get different results. > > On 5 Jul 2018, at 16.51, Matthew Str

Re: [ceph-users] Spurious empty files in CephFS root pool when multiple pools associated

> On 6 Jul 2018, at 14.45, Steffen Winther Sørensen wrote: > > > >> On 3 Jul 2018, at 15.36, Jesus Cea wrote: >> >> Signed PGP part >> On 03/07/18 15:09, Steffen Winther Sørensen wrote: >>> >>> >>>> On 3 Jul 2018, at

Re: [ceph-users] Spurious empty files in CephFS root pool when multiple pools associated

> On 3 Jul 2018, at 15.36, Jesus Cea wrote: > > Signed PGP part > On 03/07/18 15:09, Steffen Winther Sørensen wrote: >> >> >>> On 3 Jul 2018, at 12.53, Jesus Cea wrote: >>> >>> Hi there. >>> >>> I have an issue wit

Re: [ceph-users] Performance tuning for SAN SSD config

> On 5 Jul 2018, at 16.51, Matthew Stroud wrote: > > Bump. I’m hoping I can get people more knowledgeable than me to take a look. > We back some of our ceph clusters with SAN SSD disk, particularly VSP G/F and > Purestorage. I’m curious what are some settings we should look into modifying > t

Re: [ceph-users] Spurious empty files in CephFS root pool when multiple pools associated

> On 3 Jul 2018, at 12.53, Jesus Cea wrote: > > Hi there. > > I have an issue with cephfs and multiple datapools inside. I have like > SIX datapools inside the cephfs, I control where files are stored using > xattrs in the directories. Couldn’t you just use 6xCephFS each w/metadata + data pool

Re: [ceph-users] Recreating a purged OSD fails

> On 27 Jun 2018, at 08.57, Paul Emmerich wrote: > > You are running into https://tracker.ceph.com/issues/24423 > In above You said: Updated by Paul Emmerich 18 days

[ceph-users] Recreating a purged OSD fails

List, Had a failed disk behind an OSD in a Mimic Cluster 13.2.0, so I tried following the doc on removal of an OSD. I did: # ceph osd crush reweight osd.19 0 waited for rebalancing to finish and cont.: # ceph osd out 19 # systemctl stop ceph-osd@19 # ceph osd purge 19 --yes-i-really-mean-it ve

Re: [ceph-users] Increase queue_depth in KVM

> On 26 Jun 2018, at 14.04, Damian Dabrowski wrote: > > Hi Stefan, thanks for reply. > > Unfortunately it didn't work. > > disk config: > >discard='unmap'/> > > > >name='volumes-nvme/volume-ce247187-a625-49f1-bacd-fc03df215395'> > >

Re: [ceph-users] Ceph Mimic on CentOS 7.5 dependency issue (liboath)

On 24 Jun 2018, at 06.57, Brad Hubbard wrote: > > As Brian pointed out > > # yum -y install > https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm > Yeap has worked fine for me in CentOS 7.5 w/EPEL

[ceph-users] Howto add another client user id to a cluster

Anyone, We’ve ceph clients that we want to let mount two cephfs from each their own ceph clusters. Both cluster are standard created w/ceph-deploy and possible only has knowledge of each their client.admin. How could we allow a new client id to access the 2. cluster eg. as admin2? On ceph cli

Re: [ceph-users] Minimal MDS for CephFS on OSD hosts

> Den 19. jun. 2018 kl. 16.50 skrev Webert de Souza Lima > : > > Keep in mind that the mds server is cpu-bound, so during heavy workloads it > will eat up CPU usage, so the OSD daemons can affect or be affected by the > MDS daemon. > But it does work well. We've been running a few clusters wi

[ceph-users] Mimic 13.2 - Segv in ceph-osd

List, Just a heads up, I found an osd that did a segv on a CentOS 7.5 node: # ceph --version ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) # cat /etc/centos-release CentOS Linux release 7.5.1804 (Core) Jun 17 07:01:18 n3 ceph-osd: *** Caught signal (Segmentation

Re: [ceph-users] Ceph bonding vs separate provate public network

> On 12 Jun 2018, at 14.00, Steven Vacaroaia wrote: > > Hi, > > I am designing a new ceph cluster and was wondering whether I should bond the > 10 GB adapters or use one for public one for private > > The advantage of bonding is simplicity and, maybe, performance > The catch though is that I

[ceph-users] Debian GPG key for Luminous

Community, Where would I find the GPG release key for Debian Luminous? as I’m getting: W: GPG error: http://download.ceph.com/debian-luminous stretch InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY E84AC2C0460F3994 when attempting ap

Re: [ceph-users] Move data from Hammer to Mimic

> On 29 May 2018, at 23.29, Gregory Farnum wrote: > > On Tue, May 29, 2018 at 3:59 AM Steffen Winther Sørensen <mailto:ste...@gmail.com>> wrote: > (ie. would Jewel be able to connect to both clusters)? > > Yes; that should work without any issues Great, that

[ceph-users] Move data from Hammer to Mimic

List, Got an old Hammer Cluster where I would like to migrate it’s data (rbd images) to a newly installed Mimic Cluster. Would this be possible if I could upgrade the clients from Hammer to Jewel (ie. would Jewel be able to connect to both clusters)? /Steffen ___

Re: [ceph-users] Mimic EPERM doing rm pool

> On 29 May 2018, at 10.08, Eugen Block wrote: > > Hi, > >> [root@n1 ~]# ceph osd pool rm mytestpool mytestpool --yes-i-really-mean-it >> Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored > > if the command you posted is complete then you forgot one "really" in the > --y

[ceph-users] Mimic EPERM doing rm pool

List, I’ve just installed a new mimic cluster and wonder why I can’t remove a initial test pool like this: [root@n1 ~]# ceph -s cluster: id: 2284bf30-a27e-4543-af8f-b2726207762a health: HEALTH_OK services: mon: 3 daemons, quorum n1,n2,n3 mgr: n1.ceph(active), standbys:

[ceph-users] Hammer patching on Wheezy?

Ceph users, Got a Hammer cluster installed on old debian wheezy (7.11) boxes (I know :) root@node4:~# dpkg -l | grep -i ceph ii ceph 0.94.9-1~bpo70+1 amd64 distributed storage and file system ii ceph-common 0.94.9-1~bpo70

Re: [ceph-users] Hammer patch on Wheezy + CephFS leaking space?

> On 27 Jun 2017, at 11.45, John Spray wrote: > > On Tue, Jun 27, 2017 at 9:52 AM, Steffen Winther Sørensen > mailto:ste...@gmail.com>> wrote: >> Ceph users, >> >> Got an old Hammer CephFS installed on old debian wheezy (7.11) boxes (I know >> :) >

[ceph-users] Hammer patch on Wheezy + CephFS leaking space?

Ceph users, Got an old Hammer CephFS installed on old debian wheezy (7.11) boxes (I know :) root@node4:~# dpkg -l | grep -i ceph ii ceph 0.94.9-1~bpo70+1 amd64 distributed storage and file system ii ceph-common 0.94.9-1~b

Re: [ceph-users] Hammer OSD crash during deep scrub

Steffen Winther Soerensen writes: > Looks like an IO error during read maybe, > only nothing logged in syslog messages at the time. :) but it was logged in syslog at the time: Feb 15 01:28:14 node2 kernel: cciss :46:00.0: cmd 88003a900280 has CHECK CONDITION sense key = 0x3 Feb

[ceph-users] Hammer OSD crash during deep scrub

I've had few OSD crash from time to time, latest like this: --- begin dump of recent events --- -12> 2016-02-15 01:28:15.386412 7f29c8828700 1 -- 10.0.3.2:6819/448052 <== osd.17 10.0.3.1:0/6746 181211 osd_ping(ping e12542 stamp 2016-02-15 01:28:15.385759) v2 47+0+0 (1302847072 0 0

Re: [ceph-users] Hammer on Debian Wheezy not pulling in update +0.94.5

Christian Balzer writes: > > > Hello, > > On Tue, 16 Feb 2016 08:49:00 +0000 (UTC) Steffen Winther Soerensen wrote: > > > Got few OSDs crash from time to time in my Hammer 0.94.5 cluster and > > it seems Hammer is at 0.94.7 > > Where do

[ceph-users] Hammer on Debian Wheezy not pulling in update +0.94.5

Got few OSDs crash from time to time in my Hammer 0.94.5 cluster and it seems Hammer is at 0.94.7 but why doesn't My Debian Wheezy nodes that pull patches above 0.94.5? root@node2:~# apt-get update ... Hit http://ceph.com wheezy Release Hit

Re: [ceph-users] S3 RadosGW - Create bucket OP

Yehuda Sadeh-Weinraub writes: > According to the api specified here > http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUT.html, > there's no response expected. I can only assume that the application > tries to decode the xml if xml content type is returned. Also what I hinted App vendor

Re: [ceph-users] EC Pool and Cache Tier Tuning

Nick Fisk writes: > My Ceph cluster comprises of 4 Nodes each with the following:- > 10x 3TB WD Red Pro disks - EC pool k=3 m=3 (7200rpm) > 2x S3700 100GB SSD's (20k Write IOPs) for HDD Journals > 1x S3700 400GB SSD (35k Write IOPs) for cache tier - 3x replica If I have following 4x node config:

Re: [ceph-users] S3 RadosGW - Create bucket OP

Yehuda Sadeh-Weinraub writes: > If you're using apache, then it filters out zero Content-Length. > Nothing much radosgw can do about it. > You can try using the radosgw civetweb frontend, see if it changes anything. Thanks, only no difference... Req: PUT /mssCl/ HTTP/1.1 Host: rgw.gsp.sprawl.dk

Re: [ceph-users] S3 RadosGW - Create bucket OP

Steffen W Sørensen writes: > Response: > HTTP/1.1 200 OK > Date: Fri, 06 Mar 2015 10:41:14 GMT > Server: Apache/2.2.22 (Fedora) > Connection: close > Transfer-Encoding: chunked > Content-Type: application/xml > > This response makes the App say: > > S3.createBucket, class , code , > message re

Re: [ceph-users] initially conf calamari to know about my Ceph cluster(s)

Steffen Winther writes: > Trying to figure out how to initially configure > calamari clients to know about my > Ceph Cluster(s) when such aint install through ceph.deploy > but through Proxmox pveceph. > > Assume I possible need to copy some client admin keys and > c

Re: [ceph-users] Calamari build in vagrants

Dan Mick writes: > > 0cbcfbaa791baa3ee25c4f1a135f005c1d568512 on the 1.2.3 branch has the > change to yo 1.1.0. I've just cherry-picked that to v1.3 and master. Do you mean that you merged 1.2.3 into master and branch 1.3? BTW I managed to clone and built branch 1.2.3 in my vagrant env. _

[ceph-users] initially conf calamari to know about my Ceph cluster(s)

Trying to figure out how to initially configure calamari clients to know about my Ceph Cluster(s) when such aint install through ceph.deploy but through Proxmox pveceph. Assume I possible need to copy some client admin keys and configure my MON hosts somehow, any pointers to doc on this? ___

Re: [ceph-users] Calamari build in vagrants

Dan Mick writes: > > 0cbcfbaa791baa3ee25c4f1a135f005c1d568512 on the 1.2.3 branch has the > change to yo 1.1.0. I've just cherry-picked that to v1.3 and master. Thanks, but sorry for my newbie question :) How/where do I find this ID from the 1.2.3 branch? on git I guess... will look there... m

Re: [ceph-users] Calamari build in vagrants

Steffen Winther writes: > > Trying to build calamari rpm+deb packages following this guide: > http://karan-mj.blogspot.fi/2014/09/ceph-calamari-survival-guide.html > > Server packages works fine, but fails in clients for: > dashboard manage admin login due to: > > yo

[ceph-users] Calamari build in vagrants

Trying to build calamari rpm+deb packages following this guide: http://karan-mj.blogspot.fi/2014/09/ceph-calamari-survival-guide.html Server packages works fine, but fails in clients for: dashboard manage admin login due to: yo < 1.1.0 seems needed to build the clients, but can't found this with