Re: [ceph-users] Is Ceph appropriate for small installations?

>>True, true. But I personally think that Ceph doesn't perform well on >>small <10 node clusters. Hi, I can reach 60 iops 4k read with 3 nodes (6ssd each). - Mail original - De: "Lindsay Mathieson" À: "Tony Nelson" Cc:

Re: [ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

Hi Christian, for my setup "b" takes too long - too much data movement and stress to all nodes. I have simply (with replica 3) "set noout", reinstall one node (with new filesystem on the OSDs, but leave them in the crushmap) and start all OSDs (at friday night) - takes app. less than one day

Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance

Not much suggestion from my side (I never ran Ceph on complete HDDs), other than probably running more OSDs/HDDs. More OSDs should help specially if you can spread these on many nodes. I would say try with fio-rbd (librbd) first (rbd_cache = false) as it may give you some boost over kernel rbd

[ceph-users] Firefly to Hammer Upgrade -- HEALTH_WARN; too many PGs per OSD (480 > max 300)

Hi , I 'm in the process of upgrading my ceph cluster from Firefly to Hammer. The ceph cluster has 12 OSD spread across 4 nodes. Mons have been upgraded to hammer, since I have created pools with value 512 and 256 , so am bit confused with the warning message. --snip-- ceph -s cluster

Re: [ceph-users] OSD won't go up after node reboot

No, it really was in the cluster. Before reboot cluster had HEALTH_OK. Though now I've checked `current` directory and it doesn't contain any data: root@staging-coreos-1:/var/lib/ceph/osd/ceph-0# ls current commit_op_seq meta nosnap omap while other OSDs do. It really looks

Re: [ceph-users] Monitor segfault

On 08/31/2015 10:37 AM, Eino Tuominen wrote: > Hi Greg, > > Sure, should have gathered that myself... > > (gdb) bt > #0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x009a996d in reraise_fatal (signum=11) at > global/signal_handler.cc:59 > #2

[ceph-users] ceph version for productive clusters?

Hi cephers, What's the recommended version for new productive clusters? Thanks in advanced, Best regards, *German* ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Hello, On Mon, 31 Aug 2015 08:31:57 -0500 Kenneth Van Alstyne wrote: > Sorry about the repost from the cbt list, but it was suggested I post > here as well: > I wasn't even aware a CBT (what the heck does that acronym stand for?) existed... > I am attempting to track down some performance

Re: [ceph-users] a couple of radosgw questions

I see that the objects that were deleted last Friday are indeed gone now (via gc I guess). gc list does not show anything even after right after objects are deleted. I couldn't get temp remove to do anything. -- Tom > -Original Message- > From: Ben Hines [mailto:bhi...@gmail.com] >

Re: [ceph-users] ceph version for productive clusters?

Thanks a lot Kobi *German* 2015-08-31 14:20 GMT-03:00 Kobi Laredo : > Hammer should be very stable at this point. > > *Kobi Laredo* > *Cloud Systems Engineer* | (*408) 409-KOBI* > > On Mon, Aug 31, 2015 at 8:51 AM, German Anders > wrote: > >> Hi

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Christian, et al: Sorry for the lack of information. I wasn’t sure what of our hardware specifications or Ceph configuration was useful information at this point. Thanks for the feedback — any feedback, is appreciated at this point, as I’ve been beating my head against a wall trying to

Re: [ceph-users] Rados: Undefined symbol error

Hello Brad, Sorry for the delay in replying. As you mentioned earlier, $ `which rados` -v Returns a command not found error $ strings `which rados`|grep "^ceph version" -A5 Returns no results. Thanks a lot! Aakanksha -Original Message- From: Brad Hubbard

[ceph-users] Inconsistency in 'ceph df' stats

On one of our staging ceph clusters (firefly 0.80.10) I've noticed that some of the statistics in the 'ceph df' output don't seem to match up. For example in the output below the amount of raw used is 8,402G, which with triple replication would be 2,800.7G used (all the pools are triple

Re: [ceph-users] Rados: Undefined symbol error

- Original Message - > From: "Aakanksha Pudipeddi-SSI" > To: "Brad Hubbard" > Cc: ceph-us...@ceph.com > Sent: Tuesday, 1 September, 2015 3:33:38 AM > Subject: RE: [ceph-users] Rados: Undefined symbol error > > Hello Brad, > > Sorry for

Re: [ceph-users] Rados: Undefined symbol error

Hello Brad, When I type "which rados" it returns /usr/bin/rados. I am using Ubuntu 14.04. I follow these steps in installing ceph from source: 1. ./autogen.sh 2. Going to be using rocksdb, so: ./configure --with-librocksdb-static 3. make 4. sudo dpkg-buildpackage Then I use ceph-deploy to

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Hey Kenneth, it looks like you¹re just down the tollroad from me. I¹m in Reston Town Center. Just as a really rough estimate, I¹d say this is your max IOPS: 80 IOPS/spinner * 6 drives / 3 replicas = 160ish max sustained IOPS It¹s more complicated than that, since you have a reasonable solid

Re: [ceph-users] PGs stuck stale during data migration and OSD restart

I tried pg query, but it doesn't return, it hungs forever. As I understand it, when the PG is stale, there is no OSD to get the query. Am I right? I did the tunables in 2 steps, but didn't wait for all the data being moved before doing the second step. I rolled back to intermediate tunables

Re: [ceph-users] Still have orphaned rgw shadow files, ceph 0.94.3

No input, eh? (or maybe TL,DR for everyone) Short version: Presuming the bucket index shows blank/empty, which it does and is fine, would me manually deleting the rados objects with the prefix matching the former bucket's ID cause any problems? thanks, -Ben On Fri, Aug 28, 2015 at 4:22 PM, Ben

Re: [ceph-users] Rados: Undefined symbol error

- Original Message - > From: "Aakanksha Pudipeddi-SSI" > To: "Brad Hubbard" > Cc: ceph-us...@ceph.com > Sent: Tuesday, 1 September, 2015 7:27:04 AM > Subject: RE: [ceph-users] Rados: Undefined symbol error > > Hello Brad, > > When I

Re: [ceph-users] Still have orphaned rgw shadow files, ceph 0.94.3

As long as you're 100% sure that the prefix is only being used for the specific bucket that was previously removed, then it is safe to remove these objects. But please do double check and make sure that there's no other bucket that matches this prefix somehow. Yehuda On Mon, Aug 31, 2015 at 2:42

Re: [ceph-users] Rados: Undefined symbol error

Brad, Yes, you are right. Sorry about that! This is what I get when I try with the back ticks: $ `which rados` -v /usr/bin/rados: symbol lookup error: /usr/bin/rados: undefined symbol: _ZN5MutexC1ERKSsbbbP11CephContext $ strings `which rados`|grep "^ceph version" $ $ strings $(which rados)|grep

Re: [ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

When we know we need to off a node, we weight it down over time. Depending on your cluster, you may need to do this over days or hours. In theory, you could do the same when putting OSDs in, by setting noin, and then setting weight to something very low, and going up over time. I haven¹t tried

Re: [ceph-users] Still have orphaned rgw shadow files, ceph 0.94.3

Ok. I'm not too familiar with the inner workings of RGW, but i would assume that for a bucket with these parameters: "id": "default.8873277.32", "marker": "default.8873277.32", Tha it would be the only bucket using the files that start with "default.8873277.32"

Re: [ceph-users] Still have orphaned rgw shadow files, ceph 0.94.3

Make sure you use the underscore also, e.g., "default.8873277.32_". Otherwise you could potentially erase objects you did't intend to, like ones who start with "default.8873277.320" and such. On Mon, Aug 31, 2015 at 3:20 PM, Ben Hines wrote: > Ok. I'm not too familiar with the

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

I would say you are probably simply IO starved because you're running too many VMs. To follow on from Warren's response, if you spread those 160 available iops across 15 VMs, you are talking about roughly 10 iops per vm, assuming they have similar workloads. That's almost certainly too little. I

Re: [ceph-users] Still have orphaned rgw shadow files, ceph 0.94.3

Good call, thanks! Is there any risk of also deleting parts of the bucket index? I'm not sure what the objects for the index itself look like, or if they are in the .rgw.buckets pool. On Mon, Aug 31, 2015 at 3:23 PM, Yehuda Sadeh-Weinraub wrote: > Make sure you use the

Re: [ceph-users] Rados: Undefined symbol error

- Original Message - > From: "Aakanksha Pudipeddi-SSI" > To: "Brad Hubbard" > Cc: "ceph-users" > Sent: Tuesday, 1 September, 2015 7:58:33 AM > Subject: RE: [ceph-users] Rados: Undefined symbol error > > Brad, > >

Re: [ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

We have the following in our ceph.conf to bring in new OSDs with a weight of 0: [osd] osd_crush_initial_weight = 0 We then set 'nobackfill' and bring in each OSD at full weight one at a time (letting things settle down before bring in the next OSD). Once all the OSDs are brought in we unset

Re: [ceph-users] Rados: Undefined symbol error

- Original Message - > From: "Brad Hubbard" > To: "Aakanksha Pudipeddi-SSI" > Cc: "ceph-users" > Sent: Tuesday, 1 September, 2015 8:36:33 AM > Subject: Re: [ceph-users] Rados: Undefined symbol error > > -

Re: [ceph-users] Still have orphaned rgw shadow files, ceph 0.94.3

The bucket index objects are most likely in the .rgw.buckets.index pool. Yehuda On Mon, Aug 31, 2015 at 3:27 PM, Ben Hines wrote: > Good call, thanks! > > Is there any risk of also deleting parts of the bucket index? I'm not > sure what the objects for the index itself look

Re: [ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

On Mon, 31 Aug 2015 08:57:23 +0200 Udo Lembke wrote: > Hi Christian, > for my setup "b" takes too long - too much data movement and stress to > all nodes. I have simply (with replica 3) "set noout", reinstall one > node (with new filesystem on the OSDs, but leave them in the crushmap) > and start

Re: [ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

Hello, On Mon, 31 Aug 2015 22:44:05 + Stillwell, Bryan wrote: > We have the following in our ceph.conf to bring in new OSDs with a weight > of 0: > > [osd] > osd_crush_initial_weight = 0 > > > We then set 'nobackfill' and bring in each OSD at full weight one at a > time (letting things

[ceph-users] librados stripper

Hi, Lets assume, I have an 40MB object stripped using librados stripper to ten 4MB sub objects. If I add data to the middle of the object, then how does RADOS handle it ? Concretely, say object A is stripped to A0 - A9. I add 2MB of data starting at 8MB. This means, I now have A2 increased from

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Hello, On Mon, 31 Aug 2015 12:28:15 -0500 Kenneth Van Alstyne wrote: In addition to the spot on comments by Warren and Quentin, verify this by watching your nodes with atop, iostat, etc. The culprit (HDDs) should be plainly visible. More inline: > Christian, et al: > > Sorry for the lack of

[ceph-users] How objects are reshuffled on addition of new OSD

I had a question regarding how OSD locations are determined by CRUSH. >From the CRUSH paper I gather that the replica locations of an object (A) is a vector (v) that is got by the function *c(r,x) = (hash (x) + rp) mod m)*. Now when new OSDs are added, objects are shuffled to maintain uniform

Re: [ceph-users] any recommendation of using EnhanceIO?

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Nick Fisk > Sent: Wednesday, August 19, 2015 5:25 AM > To: 'Samuel Just' > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] any recommendation of using EnhanceIO? > > Hi Sam, > > >

Re: [ceph-users] Is Ceph appropriate for small installations?

On 08/31/2015 08:06 AM, Alexandre DERUMIER wrote: >>> True, true. But I personally think that Ceph doesn't perform well on >>> small <10 node clusters. > > Hi, I can reach 60 iops 4k read with 3 nodes (6ssd each). > True, but your performance is greatly impacted during recovery. So a three

Re: [ceph-users] Monitor segfault

On Mon, Aug 31, 2015 at 9:33 AM, Eino Tuominen wrote: > Hello, > > I'm getting a segmentation fault error from the monitor of our test cluster. > The cluster was in a bad state because I have recently removed three hosts > from it. Now I started cleaning it up and first marked the

Re: [ceph-users] Troubleshooting rgw bucket list

This generally shouldn't be a problem at your bucket sizes. Have you checked that the cluster is actually in a healthy state? The sleeping locks are normal but should be getting woken up; if they aren't it means the object access isn't working for some reason. A down PG or something would be the

Re: [ceph-users] Storage node refurbishing, a "freeze" OSD feature would be nice

On Mon, Aug 31, 2015 at 5:07 AM, Christian Balzer wrote: > > Hello, > > I'm about to add another storage node to small firefly cluster here and > refurbish 2 existing nodes (more RAM, different OSD disks). > > Insert rant about not going to start using ceph-deploy as I would have

Re: [ceph-users] OSD won't go up after node reboot

On Sat, Aug 29, 2015 at 3:32 PM, Евгений Д. wrote: > I'm running 3-node cluster with Ceph (it's Deis cluster, so Ceph daemons are > containerized). There are 3 OSDs and 3 mons. After rebooting all nodes one > by one all monitors are up, but only two OSDs of three are up.

Re: [ceph-users] Monitor segfault

Oh whoops, can you install the ceph-debug packages as well? That will provide line numbers on the call sites. :) -Greg On Mon, Aug 31, 2015 at 10:25 AM, Eino Tuominen wrote: > Hi Greg, > > (gdb) bt > #0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0 > #1

Re: [ceph-users] Is Ceph appropriate for small installations?

Can you share your SSD products and ceph.conf? In a test 3node cluster, 2ssds each intel s3500 i see very disappointing numbers. I maintain a 6 node cluster mixed ssd and sata pools. IOPS are not enough for a kvm hosting company unless you have really low values at disk io throttling. A big

[ceph-users] НА: Is Ceph appropriate for small installations?

Hi! >Hi, I can reach 60 iops 4k read with 3 nodes (6ssd each). It is very interesting! Can you give any details about your config? We can't get more than ~40kiops 4k random reads from 2node x 2ssd pool. :( Under load our SSDs give ~8kiops each, and that is far too low for Intel DC S3700

[ceph-users] How to disable object-map and exclusive features ?

Hi there, I have a ceph-cluster (0.94-2) with >100 rbd kvm images. Most vms are running rock-solid but 7 vms are hanging about once a week. I found out the hanging machines have features: layering, exclusive, object map while all other vms do not have exclusive and object map set. Now I want

Re: [ceph-users] Firefly to Hammer Upgrade -- HEALTH_WARN; too many PGs per OSD (480 > max 300)

On Mon, Aug 31, 2015 at 8:30 AM, 10 minus wrote: > Hi , > > I 'm in the process of upgrading my ceph cluster from Firefly to Hammer. > > The ceph cluster has 12 OSD spread across 4 nodes. > > Mons have been upgraded to hammer, since I have created pools with value > 512 and

Re: [ceph-users] Question about reliability model result

Maybe it's just a precision problem? I calculate the durability from PL(*) columns with the formula: 1-PL(site)-PL(copy)-PL(NRE). Result: 2-cp is 0.99896562 3-cp is 0.99900049 Both of them are approximates to 99.9% Actually the model result is 99.900%. Maybe the author wants us to ignore the

Re: [ceph-users] Monitor segfault

Hi Greg, (gdb) bt #0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x009a996d in ?? () #2 #3 0x0085ada2 in crush_do_rule () #4 0x007a85cb in OSDMap::_pg_to_osds(pg_pool_t const&, pg_t, std::vector*, int*, unsigned

[ceph-users] Monitor segfault

Hello, I'm getting a segmentation fault error from the monitor of our test cluster. The cluster was in a bad state because I have recently removed three hosts from it. Now I started cleaning it up and first marked the removed osd's as lost (ceph osd lost), and then I tried to remove the osd's

Re: [ceph-users] PGs stuck stale during data migration and OSD restart

On Sat, Aug 29, 2015 at 11:50 AM, Gerd Jakobovitsch wrote: > Dear all, > > During a cluster reconfiguration (change of crush tunables from legacy to > TUNABLES2) with large data replacement, several OSDs get overloaded and had > to be restarted; when OSDs stabilize, I got a

[ceph-users] Append data via librados C API in erasure coded pool

Hello, I use librados C API rados_append() to append object data in erasure coded pool, it always return -95 (Operation not supported). Buf if i use the same code to append object data in replicated pool, it works fine. Does erasure coded pool not support append write? Below is my erasure coded

[ceph-users] .rgw.root and .rgw pools

Hi, I have been trying to understand the default pools used by the default zone in a radosgw only configuration and have got a confusion regarding the same. The documentation here ( http://ceph.com/docs/master/radosgw/config-ref/#pools) says that the set of default pools include .rgw, while the

Re: [ceph-users] Testing CephFS

On Mon, Aug 31, 2015 at 12:16 PM, Yan, Zheng wrote: > On Mon, Aug 24, 2015 at 6:38 PM, Gregory Farnum wrote: >> On Mon, Aug 24, 2015 at 11:35 AM, Simon Hallam wrote: >>> Hi Greg, >>> >>> The MDS' detect that the other one went down and

Re: [ceph-users] Monitor segfault

On 08/31/2015 10:37 AM, Eino Tuominen wrote: > Hi Greg, > > Sure, should have gathered that myself... > > (gdb) bt > #0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x009a996d in reraise_fatal (signum=11) at > global/signal_handler.cc:59 > #2

Re: [ceph-users] How to disable object-map and exclusive features ?

Unfortunately, the tool the dynamically enable/disable image features (rbd feature disable ) was added during the Infernalis development cycle. Therefore, in the short-term you would need to recreate the images via export/import or clone/flatten. There are several object map / exclusive

[ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Sorry about the repost from the cbt list, but it was suggested I post here as well: I am attempting to track down some performance issues in a Ceph cluster recently deployed. Our configuration is as follows: 3 storage nodes, each with: - 8 Cores - 64GB