Re: [ceph-users] Adding disks -> getting unfound objects [Luminous]

Good morning, the osd.61 actually just crashed and the disk is still intact. However, after 8 hours of rebuilding, the unfound objects are still missing: root@server1:~# ceph -s cluster: id: 26c0c5a8-d7ce-49ac-b5a7-bfd9d0ba81ab health: HEALTH_WARN noscrub,nodeep-scrub

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

On 2018-01-23 08:27, Blair Bethwaite wrote: > Firstly, the OP's premise in asking, "Or should there be a differnce > of 10x", is fundamentally incorrect. Greater bandwidth does not mean > lower latency, though the latter almost always results in the former. > Unfortunately, changing the speed of

[ceph-users] Replication count - demo

Hello, What is best and simple way to showcase that - each ceph image replicated 3 (ie size=3)? A few ideas: - Use "ceph osd map image_id If any simple way to showcase the above, please share. Thanks Swami ___ ceph-users mailing list

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

Firstly, the OP's premise in asking, "Or should there be a differnce of 10x", is fundamentally incorrect. Greater bandwidth does not mean lower latency, though the latter almost always results in the former. Unfortunately, changing the speed of light remains a difficult engineering challenge :-).

Re: [ceph-users] OSD doesn't start - fresh installation

Hello, Sorry for wasting your time; below mistakes originated from unproperly removing OSDs (specifically forgetting to run “ceph auth del osd.#” ) from the cluster. Thank you for attention though. Best regards, Atatür From: Brad Hubbard [mailto:bhubb...@redhat.com] Sent: Tuesday, January

[ceph-users] PG inactive, peering

Hi, We added a new host to cluster and it was rebalancing. And one PG became "inactive, peering" for very long time which created lot of slow requests and poor performance to the whole cluster. When I queried that PG, it showed this : "recovery_state": [ { "name":

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

ping -c 10 -f 10.0.1.12 Intel X710-DA2 -> Switch -> Intel X710-DA2: --- 172.16.16.3 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 1932ms rtt min/avg/max/mdev = 0.013/0.014/0.131/0.004 ms, ipg/ewma 0.019/0.014 ms k

Re: [ceph-users] ghost degraded objects

Yes, the pending backport for what we have so far is in https://github.com/ceph/ceph/pull/20055 With this changes a backfill caused by marking an osd out has the results as shown:     health: HEALTH_WARN     115/600 objects misplaced (19.167%) ...   data:     pools:   1 pools, 1

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

25Gbe network. Servers have ConnectX-4 Pro, across a router, since L2 is terminated as the ToR: 10 packets transmitted, 10 received, 0% packet loss, time 1926ms rtt min/avg/max/mdev = 0.013/0.013/0.205/0.004 ms, ipg/ewma 0.019/0.014 ms Warren Wang On 1/22/18, 4:06 PM, "ceph-users on

Re: [ceph-users] OSD doesn't start - fresh installation

On Mon, Jan 22, 2018 at 10:37 PM, Hüseyin Atatür YILDIRIM < hyildi...@havelsan.com.tr> wrote: > > Hi again, > > > > In the “journalctl –xe” output: > > > > Jan 22 15:29:18 mon02 ceph-osd-prestart.sh[1526]: OSD data directory > /var/lib/ceph/osd/ceph-1 does not exist; bailing out. > > > > Also in

Re: [ceph-users] After Luminous upgrade: ceph-fuse clients failing to respond to cache pressure

Just to close this thread up - it looks like all the problems were related to setting the "mds cache size" option in Luminous instead of using "mds cache memory limit".  The "mds cache size" option documentation says that "it is recommended to use mds_cache_memory_limit ...", but it looks more

Re: [ceph-users] Luminous upgrade with existing EC pools

On Mon, Jan 22, 2018 at 9:23 PM, David Turner wrote: > I ran into a problem removing the cache tier. I tried everything I could to > get past it, but I ended up having to re-enable it. I'm running on 12.2.2 > with all bluestore OSDs. > > I successfully set

Re: [ceph-users] Stuck pgs (activating+remapped) and slow requests after adding OSD node via ceph-ansible

Did you find out anything about this? We are also getting pgs stuck "activating+remapped". I have to manually alter bucket weights so that they are basically the same everywhere, even if disks aren't the same size to fix the problem, but it is a real hassle every time we add a new node or

Re: [ceph-users] Luminous upgrade with existing EC pools

I ran into a problem removing the cache tier. I tried everything I could to get past it, but I ended up having to re-enable it. I'm running on 12.2.2 with all bluestore OSDs. I successfully set allow_ec_overwrites to true, I set the cache-mode to forward, I flushed/evicted the entire cache, and

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

ping -c 10 -f ping -M do -s 8972 10Gb ConnectX-3 Pro, DAC + Vlan rtt min/avg/max/mdev = 0.010/0.013/0.200/0.003 ms, ipg/ewma 0.025/0.014 ms 8980 bytes from 10.0.0.11: icmp_seq=3 ttl=64 time=0.144 ms 8980 bytes from 10.0.0.11: icmp_seq=4 ttl=64 time=0.205 ms 8980 bytes from 10.0.0.11:

Re: [ceph-users] Ceph Future

On 01/22/2018 08:38 PM, Massimiliano Cuttini wrote: > The web interface is needed because:*cmd-lines are prune to typos.* And you never misclick, indeed; > SMART is widely used. SMART has never, and will never be any useful for failure prediction. > My opinion is pretty simple: the more a

Re: [ceph-users] Luminous: example of a single down osd taking out a cluster

Here's a bit more info as I read the logs. Firstly, these are in fact Filestore OSDs... I was confused, but I don't think it makes a big difference. Next, all the other OSDs had indeed noticed that osd.2 had failed: 2018-01-22 18:37:20.456535 7f831728e700 -1 osd.0 598 heartbeat_check: no reply

Re: [ceph-users] Ceph Future

I have to disagree with you Marc, Hmmm, I have to disagree with 'too many services' What do you mean, there is a process for each osd, mon, mgr and mds. There are less processes running than on a default windows fileserver. What is the complaint here? I wrote: "/_Ceph is amazing_, but is

[ceph-users] Luminous: example of a single down osd taking out a cluster

Hi all, We just saw an example of one single down OSD taking down a whole (small) luminous 12.2.2 cluster. The cluster has only 5 OSDs, on 5 different servers. Three of those servers also run a mon/mgr combo. First, we had one server (mon+osd) go down legitimately [1] -- I can tell when it went

Re: [ceph-users] Adding disks -> getting unfound objects [Luminous]

Weight the remaining disks you added to 0.0. They seem to be a bad batch. This will start moving their data off of them and back onto the rest of the cluster. I generally suggest not to add storage in more than what you can afford to lose, unless you trust your burn-in process. So if you have a

Re: [ceph-users] Adding disks -> getting unfound objects [Luminous]

While writing, yet another disk (osd.61 now) died and now we have 172 pgs down: [19:32:35] server2:~# ceph -s cluster: id: 26c0c5a8-d7ce-49ac-b5a7-bfd9d0ba81ab health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 21033/2263701 objects misplaced (0.929%)

[ceph-users] Ideal Bluestore setup

Hi folks, I’m not sure the ideal setup for bluestore given the set of hardware I have to work with so I figured I would ask the collective wisdom of the ceph community. It is a small deployment so the hardware is not all that impressive, but I’d still like to get some feedback on what would be

Re: [ceph-users] Adding disks -> getting unfound objects [Luminous]

I do remember seeing that exactly. As the number of recovery_wait pgs decreased, the number of unfound objects decreased until they were all found. Unfortunately it blocked some IO from happening during the recovery, but in the long run we ended up with full data integrity again. On Mon, Jan 22,

Re: [ceph-users] Adding disks -> getting unfound objects [Luminous]

Hey David, thanks for the fast answer. All our pools are running with size=3, min_size=2 and the two disks were in 2 different hosts. What I am a bit worried about is the output of "ceph pg 4.fa query" (see below) that indicates that ceph already queried all other hosts and did not find the

Re: [ceph-users] Adding disks -> getting unfound objects [Luminous]

I have had the same problem before with unfound objects that happened while backfilling after losing a drive. We didn't lose drives outside of the failure domains and ultimately didn't lose any data, but we did have to wait until after all of the PGs in recovery_wait state were caught up. So if

Re: [ceph-users] Luminous - bad performance

Hi David, I noticed the public interface of the server I am running the test from is heavily used so I will bond that one too I doubt though that this explains the poor performance Thanks for your advice Steven On 22 January 2018 at 12:02, David Turner wrote: > I'm

Re: [ceph-users] Luminous - bad performance

I'm not speaking to anything other than your configuration. "I am using 2 x 10 GB bonded ( BONDING_OPTS="mode=4 miimon=100 xmit_hash_policy=1 lacp_rate=1") for cluster and 1 x 1GB for public" It might not be a bad idea for you to forgo the public network on the 1Gb interfaces and either put

[ceph-users] Adding disks -> getting unfound objects [Luminous]

Hello, we added about 7 new disks yesterday/today and our cluster became very slow. While the rebalancing took place, 2 of the 7 new added disks died. Our cluster is still recovering, however we spotted that there are a lot of unfound objects. We lost osd.63 and osd.64, which seem not to be

Re: [ceph-users] Luminous - bad performance

I did test with rados bench ..here are the results rados bench -p ssdpool 300 -t 12 write --no-cleanup && rados bench -p ssdpool 300 -t 12 seq Total time run: 300.322608 Total writes made: 10632 Write size: 4194304 Object size:4194304 Bandwidth (MB/sec):

Re: [ceph-users] Luminous - bad performance

sorry ..send the message too soon Here is more info Vendor Id : SEAGATE Product Id : ST600MM0006 State : Online Disk Type : SAS,Hard Disk Device Capacity : 558.375 GB

Re: [ceph-users] Luminous - bad performance

Hi David, Yes, I meant no separate partitions for WAL and DB I am using 2 x 10 GB bonded ( BONDING_OPTS="mode=4 miimon=100 xmit_hash_policy=1 lacp_rate=1") for cluster and 1 x 1GB for public Disks are Vendor Id : TOSHIBA Product Id : PX05SMB040Y

Re: [ceph-users] Luminous - bad performance

On Mon, 22 Jan 2018, Steven Vacaroaia wrote: > Hi, > > I'll appreciate if you can provide some guidance / suggestions regarding > perfomance issues on a test cluster ( 3 x DELL R620, 1 Entreprise SSD, 3 x > 600 GB ,Entreprise HDD, 8 cores, 64 GB RAM) > > I created 2 pools ( replication factor 2)

Re: [ceph-users] Luminous - bad performance

Disk models, other hardware information including CPU, network config? You say you're using Luminous, but then say journal on same device. I'm assuming you mean that you just have the bluestore OSD configured without a separate WAL or DB partition? Any more specifics you can give will be

[ceph-users] Luminous - bad performance

Hi, I'll appreciate if you can provide some guidance / suggestions regarding perfomance issues on a test cluster ( 3 x DELL R620, 1 Entreprise SSD, 3 x 600 GB ,Entreprise HDD, 8 cores, 64 GB RAM) I created 2 pools ( replication factor 2) one with only SSD and the other with only HDD ( journal on

Re: [ceph-users] iSCSI over RBD

Phew ..right ... thanks for your patience Again, my apologies for waiting your time Steven On 22 January 2018 at 09:53, Jason Dillaman wrote: > Point yum at it -- those "repodata" files are for yum/dnf not you. The > packages are in the x86_64 / noarch directories as per

Re: [ceph-users] iSCSI over RBD

Point yum at it -- those "repodata" files are for yum/dnf not you. The packages are in the x86_64 / noarch directories as per the standard layout of repos. On Mon, Jan 22, 2018 at 9:52 AM, Steven Vacaroaia wrote: > Clicking on the link provided , I get this > > ../ > SRPMS/

Re: [ceph-users] iSCSI over RBD

Clicking on the link provided , I get this ../ SRPMS/

Re: [ceph-users] iSCSI over RBD

Which URL isn't working for you? You should follow the links in a web browser, select the most recent build, and then click the "Repo URL" button to get the URL to provide yum. On Mon, Jan 22, 2018 at 9:30 AM, Steven Vacaroaia wrote: > > Thanks again for your prompt response >

Re: [ceph-users] iSCSI over RBD

Thanks again for your prompt response My apologies for wasting your time with trivial question but the repo provided does not contain rpms but a bunch of compressed files ( like 2f241a8387cf35372fd709be4ef6ec83b8a00cc744bb90f31d82bb27bdd80531-other.sqlite.bz2 ) and a repomd.xml How/what exactly

Re: [ceph-users] iSCSI over RBD

You can use these repos [1][2][3] [1] https://shaman.ceph.com/repos/python-rtslib/master/ [2] https://shaman.ceph.com/repos/ceph-iscsi-config/master/ [3] https://shaman.ceph.com/repos/ceph-iscsi-cli/master/ targetcli isn't used for iSCSI over RBD (gwcli from ceph-iscsi-cli replaces it), so you

Re: [ceph-users] iSCSI over RBD

Excellent news Many thanks for all your efforts If you do not mind, please confirm the following steps ( for centos 7, kernel version 3.10.0-693.11.6.el7.x86_64) - download and Install the RPMs from x86_64 repositories you provided - do a git clone and, if new version available "pip install .

Re: [ceph-users] iSCSI over RBD

The v4.13-based kernel with the necessary bug fixes and TCMU changes is available here [1] and tcmu-runner v1.3.0 is available here [2]. [1] https://shaman.ceph.com/repos/kernel/ceph-iscsi-test/ [2] https://shaman.ceph.com/repos/tcmu-runner/master/ On Sat, Jan 20, 2018 at 7:33 AM, Marc Roos

Re: [ceph-users] udev rule or script to auto add bcache devices?

On Mon, Jan 22, 2018 at 1:37 AM, Wido den Hollander wrote: > > > On 01/20/2018 07:56 PM, Stefan Priebe - Profihost AG wrote: >> >> Hello, >> >> bcache didn't supported partitions on the past so that a lot of our osds >> have their data directly on: >> /dev/bcache[0-9] >> >> But

Re: [ceph-users] OSD doesn't start - fresh installation

Hi again, In the “journalctl –xe” output: Jan 22 15:29:18 mon02 ceph-osd-prestart.sh[1526]: OSD data directory /var/lib/ceph/osd/ceph-1 does not exist; bailing out. Also in my previous post, I forgot to say that “ceph-deploy osd create” command doesn’t fail and appears to be successful,

Re: [ceph-users] Luminous upgrade with existing EC pools

I've already migrated all osds to bluestore and changed my pools to use a crush rule specifying them to use an HDD class (forced about half of my data to move). This week I'm planning to add in some new SSDs to move the metadata pool to. I have experience with adding and removing cache tiers

[ceph-users] OSD doesn't start - fresh installation

Hi all, Fresh installation but already ısed disks. I zapped all the disks and ran “ceph-deploy ods create” again but got same results. Log is attached. Can you please help? Thank you, Atatur [cid:image028a8d.PNG@94a00751.438f095a]

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

Anyone with 25G ethernet willing to do the test? Would love to see what the latency figures are for that. From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Maged Mokhtar Sent: 22 January 2018 11:28 To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] What is the

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

On 2018-01-22 08:39, Wido den Hollander wrote: > On 01/20/2018 02:02 PM, Marc Roos wrote: > >> If I test my connections with sockperf via a 1Gbit switch I get around >> 25usec, when I test the 10Gbit connection via the switch I have around >> 12usec is that normal? Or should there be a

Re: [ceph-users] Luminous upgrade with existing EC pools

On Sat, Jan 20, 2018 at 6:26 PM, David Turner wrote: > I am not able to find documentation for how to convert an existing cephfs > filesystem to use allow_ec_overwrites. The documentation says that the > metadata pool needs to be replicated, but that the data pool can be

Re: [ceph-users] ceph df shows 100% used

Hi, On Fri, Jan 19, 2018 at 8:31 PM, zhangbingyin wrote: > 'MAX AVAIL' in the 'ceph df' output represents the amount of data that can > be used before the first OSD becomes full, and not the sum of all free > space across a set of OSDs. > Thank you very much. I

Re: [ceph-users] Missing udev rule for FC disks (Re: mkjournal error creating journal ... : (13) Permission denied)

I believe I've recently spent some time with this issue, so I hope this is helpful. Apologies if it's an unrelated dm/udev/ceph-disk problem. https://lists.freedesktop.org/archives/systemd-devel/2017-July/039222.html The above email from last July explains the situation somewhat, with the