Re: monitor not starting

2012-07-04 Thread Gregory Farnum
On Wednesday, July 4, 2012 at 4:45 AM, Smart Weblications GmbH - Florian Wiessner wrote: Hi List, i today upgraded from 0.43 to 0.48 and now i have one monitor which does not want to start up anymore: ceph version 0.48argonaut-125-g4e774fb

Re: [PATCH] librados: Bump the version to 0.48

2012-07-04 Thread Gregory Farnum
Hmmm — we generally try to modify these versions when the API changes, not on every sprint. It looks to me like Sage added one function in 0.45 where we maybe should have bumped it, but that was a long time ago and at this point we should maybe just eat it? -Greg On Wednesday, July 4, 2012 at

Re: monitor not starting

2012-07-04 Thread Gregory Farnum
On Wednesday, July 4, 2012 at 10:02 AM, Smart Weblications GmbH - Florian Wiessner wrote: Am 04.07.2012 18:25, schrieb Gregory Farnum: On Wednesday, July 4, 2012 at 4:45 AM, Smart Weblications GmbH - Florian Wiessner wrote: Hi List, i today upgraded from 0.43

Re: Ceph for email storage

2012-07-04 Thread Gregory Farnum
On Wednesday, July 4, 2012 at 11:29 AM, Mitsue Acosta Murakami wrote: Hello, We are examining Ceph to use as email storage. In our current system, several clients servers with different services (imap, smtp, etc) access a NFS storage server. The mailboxes are stored in Maildir format, with

Re: Upgrade from 0.47.2 to 0.48 - osd crashes

2012-07-03 Thread Gregory Farnum
That particular assert means augh it's taking too long for the filesystem to handle my requests! The disk format upgrade in particular seems to hit it, so you may have some luck just increasing the timeout (increase filestore_op_thread_suicide_timeout from its default 180 seconds) — but it is

Re: Throughput increase by LACP bonding

2012-07-03 Thread Gregory Farnum
On Tuesday, July 3, 2012 at 12:42 AM, Madhusudhana U wrote: Hi all, I am trying to increase the throughput in the cluster by enabling LACP on both in clients and all ceph cluster nodes. Each client and ceph nodes has two 1G Ethernet interface which I want to aggregate and make it 2G. LACP

Re: Unable to restart Mon after reboot

2012-07-03 Thread Gregory Farnum
On Tue, Jul 3, 2012 at 9:53 AM, Christoph Hellwig h...@infradead.org wrote: On Tue, Jul 03, 2012 at 09:44:38AM -0700, Tommi Virtanen wrote: We've seen similar issues with btrfs, and others have reported that the large metadata btrfs option helps. We're still compiling information, but as of

Re: rbd rm allows removal of mapped device, nukes data, then returns -EBUSY

2012-07-02 Thread Gregory Farnum
On Mon, Jul 2, 2012 at 9:08 AM, Josh Durgin josh.dur...@inktank.com wrote: On 07/01/2012 11:58 PM, Florian Haas wrote: Hi everyone, just wanted to check if this was the expected behavior -- it doesn't look like it would be, to me. What I do is create a 1G RBD, and just for the heck of it,

Re: Does radosgw really need to talk to an MDS?

2012-07-02 Thread Gregory Farnum
On Mon, Jul 2, 2012 at 4:44 AM, Wido den Hollander w...@widodh.nl wrote: Hi, On 02-07-12 13:41, Florian Haas wrote: Hi everyone, radosgw(8) states that the following capabilities must be granted to the user that radosgw uses to connect to RADOS. ceph-authtool -n client.radosgw.gateway

Re: btrfs big metadata

2012-07-02 Thread Gregory Farnum
On Sun, Jul 1, 2012 at 11:56 PM, Stefan Priebe - Profihost AG s.pri...@profihost.ag wrote: Hello list, i found several people who use big metadata -n 64k -l 64k for ceph. But i haven't found any ceph doc or info why to use this? What's the reason to use big metadata feature with ceph? One

Re: speedup ceph / scaling / find the bottleneck

2012-07-02 Thread Gregory Farnum
On Sun, Jul 1, 2012 at 11:12 PM, Stefan Priebe - Profihost AG s.pri...@profihost.ag wrote: Am 02.07.2012 07:02, schrieb Alexandre DERUMIER: Hi, my 2cent, maybe with lower range (like 100MB) of random io, you have more chance to aggregate them in 4MB block ? Yes maybe. If you have just a

Re: Bug with ceph_mount and non-existent directory

2012-07-02 Thread Gregory Farnum
On Tue, Jun 26, 2012 at 8:20 PM, Noah Watkins jayh...@cs.ucsc.edu wrote: I get the following assert failure during cleanup if ceph_mount() is passed a non-existent directory, while ceph_mount() returns success. Nothing critical, but it got triggered with Java unit test framework.    

Re: MDS spinning wild after restart on all nodes

2012-07-02 Thread Gregory Farnum
Amon, I've been going through my backlog of flagged emails and came across this one. Did you ever get that information for the bug that you were going to try and find? -Greg On Fri, Jun 15, 2012 at 9:44 AM, Sage Weil s...@inktank.com wrote: On Fri, 15 Jun 2012, Amon Ott wrote: Hello all, I

Re: Ceph and KVM live migration

2012-07-02 Thread Gregory Farnum
On Sat, Jun 30, 2012 at 8:21 PM, Vladimir Bashkirtsev vladi...@bashkirtsev.com wrote: On 01/07/12 11:59, Josh Durgin wrote: On 06/30/2012 07:15 PM, Vladimir Bashkirtsev wrote: On 01/07/12 10:47, Josh Durgin wrote: On 06/30/2012 05:42 PM, Vladimir Bashkirtsev wrote: Dear all, Currently I

Re: Rados faster than KVM block device?

2012-07-02 Thread Gregory Farnum
On Thu, Jun 28, 2012 at 2:17 PM, Stefan Priebe s.pri...@profihost.ag wrote: Am 28.06.2012 18:12, schrieb Josh Durgin: On 06/28/2012 06:10 AM, Stefan Priebe - Profihost AG wrote: Hello list, my cluster is now pretty stable i'm just wondering about the sequential write values. With rados

Re: Should an OSD crash when journal device is out of space?

2012-07-02 Thread Gregory Farnum
Hey guys, Thanks for the problem report. I've created an issue to track it at http://tracker.newdream.net/issues/2687. It looks like we just assume that if you're using a file, you've got enough space for it. It shouldn't be a big deal to at least do some startup checks which will fail gracefully.

Re: bad performance fio random write - rados bench random write to compare?

2012-07-02 Thread Gregory Farnum
On Tue, Jun 19, 2012 at 7:05 AM, Alexandre DERUMIER aderum...@odiso.com wrote: Hi, Is it possible to do random write bench with rados bench command ? I have very base random write performance with 4K block size inside qemu-kvm, 1000 iops/s max with 3 nodes with 3x 5 disk 15k (Maybe it's

Re: iostat show constants write to osd disk with writeahead journal, normal behaviour ?

2012-07-02 Thread Gregory Farnum
On Tue, Jun 19, 2012 at 12:09 AM, Alexandre DERUMIER aderum...@odiso.com wrote: Hi, more infos, I have active filestore debug = 20, min interval 29 and max interval 30. I see sync_entry each 30s, so it seem work as expected. cat ceph-osd.0.log |grep sync_entry 2012-06-19 07:56:00.084622

Re: Ceph status for Wheezy

2012-07-01 Thread Gregory Farnum
On Saturday, June 30, 2012 at 11:12 PM, Sage Weil wrote: Hi Laszlo, On Sun, 1 Jul 2012, Laszlo Boszormenyi (GCS) wrote: Hi Sage, As previously noted, using leveldb caused some trouble with Ceph could be included in Wheezy or not. I've proposed that the supported architectures should

Re: Designing a cluster guide

2012-06-29 Thread Gregory Farnum
On Thu, May 17, 2012 at 2:27 PM, Gregory Farnum g...@inktank.com wrote: Sorry this got left for so long... On Thu, May 10, 2012 at 6:23 AM, Stefan Priebe - Profihost AG s.pri...@profihost.ag wrote: Hi, the Designing a cluster guide http://wiki.ceph.com/wiki/Designing_a_cluster is pretty

Re: ceph performance under xen?

2012-06-29 Thread Gregory Farnum
On Thu, Jun 28, 2012 at 7:27 AM, Brian Edmonds mor...@gmail.com wrote: I've installed a little, four node Ceph (0.47.2) cluster using Xen virtual machines for testing, and when I run bonnie against a (kernel driver) mount of it, it seems to be somewhat flaky (disturbing log messages,

Re: ceph performance under xen?

2012-06-29 Thread Gregory Farnum
On Fri, Jun 29, 2012 at 1:54 PM, Brian Edmonds mor...@gmail.com wrote: On Fri, Jun 29, 2012 at 11:55 AM, Gregory Farnum g...@inktank.com wrote: So right now you're using the Ceph filesystem, rather than RBD, right? Right, CephFS.  I'm actually not even very clear on what RBD is, and how one

Re: Designing a cluster guide

2012-06-29 Thread Gregory Farnum
On Fri, Jun 29, 2012 at 1:59 PM, Brian Edmonds mor...@gmail.com wrote: On Fri, Jun 29, 2012 at 11:50 AM, Gregory Farnum g...@inktank.com wrote: If you lose a journal, you lose the OSD. Really?  Everything?  Not just recent commits?  I would have hoped it would just come back up in an old

Re: Designing a cluster guide

2012-06-29 Thread Gregory Farnum
On Fri, Jun 29, 2012 at 2:18 PM, Brian Edmonds mor...@gmail.com wrote: On Fri, Jun 29, 2012 at 2:11 PM, Gregory Farnum g...@inktank.com wrote: Well, actually this depends on the filesystem you're using. With btrfs, the OSD will roll back to a consistent state, but you don't know how out

Re: POSIX file lock doesn't get release on process termination

2012-06-28 Thread Gregory Farnum
What versions are you using? This sound familiar, although I think we previously fixed it for fcntl locks, which are different than flock locks in some annoying ways. Anyway, I imagine this is a Ceph bug, but have you run this on a non-Ceph mount on the same system and checked that the behavior

Re: POSIX file lock doesn't get release on process termination

2012-06-28 Thread Gregory Farnum
, kernel 3.2.2 (kernel client)  Anyway, I imagine this is a Ceph bug, but have you run this on a non-Ceph mount on the same system and checked that the behavior is as you expect? Yes, I ran it on an ext4 mount and works as expected 2012/6/28 Gregory Farnum g...@inktank.com: What versions

Re: ceph osd crush add - uknown command crush

2012-06-23 Thread Gregory Farnum
On Thursday, June 21, 2012 at 11:12 AM, Barry Pederson wrote: On Jun 20, 2012, at 4:59 PM, Gregory Farnum wrote: On Wed, Jun 20, 2012 at 2:53 PM, Travis Rhoden trho...@gmail.com (mailto:trho...@gmail.com) wrote: This incorrect syntax is still published in the docs at http://ceph.com

Re: ceph osd crush add - uknown command crush

2012-06-20 Thread Gregory Farnum
On Wed, Jun 20, 2012 at 2:53 PM, Travis Rhoden trho...@gmail.com wrote: This incorrect syntax is still published in the docs at http://ceph.com/docs/master/ops/manage/crush/#adjusting-crush Are the docs that end up on that page in GitHub? I'd be happy to start making fixes and issuing

Re: Building a small Ceph development environment

2012-06-19 Thread Gregory Farnum
You don't need to virtualize anything — I'd recommend running Ubuntu 12.04 on it (you don't need to, but a lot of things will be more performant), building from source, and then setting up the daemons so everybody gets a separate disk. Check out http://ceph.com/docs/master/source/ and the other

Re: Building a small Ceph development environment

2012-06-19 Thread Gregory Farnum
ceph under RHEL 6/Oracle Linux 6? I'm more of an RPM guy. -Terrance On Tuesday, June 19, 2012, Gregory Farnum wrote: You don't need to virtualize anything — I'd recommend running Ubuntu 12.04 on it (you don't need to, but a lot of things will be more performant), building from source

Re: Ceph performance on Ubuntu Oneiric vs Ubuntu Precise

2012-06-18 Thread Gregory Farnum
Do I correctly assume that these nodes hosted only the OSDs, and the monitors were on a separate node? On Mon, Jun 18, 2012 at 10:56 AM, Mark Nelson mark.nel...@inktank.com wrote: Hi Guys, I've been tracking down some performance issues over the past month with our internal test nodes and

Re: RBD layering design draft

2012-06-18 Thread Gregory Farnum
Locking is a separate mechanism we're already working on, which will lock images so that they can't accidentally be mounted at more than one location. :) -Greg On Sun, Jun 17, 2012 at 6:42 AM, Martin Mailand mar...@tuxadero.com wrote: Hi, what's up locked, unlocked, unlocking? -martin Am

rbd locking and handling broken clients

2012-06-13 Thread Gregory Farnum
We've had some user reports lately on rbd images being broken by misbehaving clients — namely, rbd image I is mounted on computer A, computer A starts misbehaving, and so I is mounted on computer B. But because A is misbehaving it keeps writing to the image, corrupting it horribly. To handle this,

Re: mds dump

2012-06-08 Thread Gregory Farnum
On Thursday, June 7, 2012 at 9:53 PM, Martin Wilderoth wrote: Hello, Now my mds are all crashing after a while one by one. Is it possible to recover without removing my rbd images ? This is a pretty familiar MDS crash that we haven't tracked down yet. Sorry. :( However, it has absolutely

Re: Infiniband 40GB

2012-06-04 Thread Gregory Farnum
On Monday, June 4, 2012 at 6:28 AM, Mark Nelson wrote: Hi Alexandre, A lot of our testing is on Ubuntu right now. I'm using the ceph and kernel debs from ceph.gitbuilder.com (http://ceph.gitbuilder.com) for my tests. Post some results to the list once you get your cluster setup! I

Re: Problems when benchmarking Ceph

2012-05-30 Thread Gregory Farnum
The library you're looking for is libceph. It does exist and it's fairly full-featured, but it's not nearly as well documented as the librados C api is. However, you'll probably get more use out of one of the Hadoop bindings. If you check out the git repository you'll find one set in

Re: ceph.spec does not list libuuid as build time dependency

2012-05-30 Thread Gregory Farnum
Thanks for the bug report. I created a tracker entry for it: http://tracker.newdream.net/issues/2484 I don't imagine it will take long for somebody who knows how to handle a .spec to fix. ;) -Greg On Friday, May 25, 2012 at 6:33 AM, Guido Winkelmann wrote: Hi, Building ceph 0.47.2 from

Re: I have some problem to mount ceph file system

2012-05-23 Thread Gregory Farnum
On Wed, May 23, 2012 at 1:51 AM, Frank frankwoo@gmail.com wrote: Hello I have a question about ceph. When I mount ceph, I do the command as follow : # mount -t ceph -o name=admin,secret=XX 10.1.0.1:6789/ /mnt/ceph -vv now I create an user foo and make a secretkey by ceph-authtool

Re: Designing a cluster guide

2012-05-23 Thread Gregory Farnum
On Wed, May 23, 2012 at 12:47 PM, Jerker Nyberg jer...@update.uu.se wrote: On Tue, 22 May 2012, Gregory Farnum wrote: Direct users of the RADOS object store (i.e., librados) can do all kinds of things with the integrity guarantee options. But I don't believe there's currently a way to make

Re: how to mount a specific pool in cephs

2012-05-22 Thread Gregory Farnum
On Tuesday, May 22, 2012 at 2:51 PM, Grant Ashman wrote: Awesome, that seemed to work! However, I feel a bit silly - what I'm after is: /mnt/ceph-data - mounted to pool 0 (data) /mnt/ceph-backup - mounted to pool 3 (backup) but this seemed to change both to mount to pool 3? Am I simply

Re: Designing a cluster guide

2012-05-21 Thread Gregory Farnum
On Sat, May 19, 2012 at 1:37 AM, Stefan Priebe s.pri...@profihost.ag wrote: Hi Greg, Am 17.05.2012 23:27, schrieb Gregory Farnum: It mentions for example Fast CPU for the mds system. What does fast mean? Just the speed of one core? Or is ceph designed to use multi core? Is multi core

Re: Possible memory leak in mon?

2012-05-21 Thread Gregory Farnum
On Fri, May 18, 2012 at 3:07 AM, Vladimir Bashkirtsev vladi...@bashkirtsev.com wrote: On 16/05/12 02:43, Gregory Farnum wrote: On Sun, May 6, 2012 at 5:53 PM, Vladimir Bashkirtsev vladi...@bashkirtsev.com  wrote: On 03/05/12 16:23, Greg Farnum wrote: On Wednesday, May 2, 2012 at 11:24 PM

Re: Replication at file/folder filen

2012-05-21 Thread Gregory Farnum
On Mon, May 21, 2012 at 10:28 AM, Madhusudhana U madhusudhana.u.acha...@gmail.com wrote: Hi all, I assume in CEPH, by default, replications are set for both data and metadata. Is it possible for setting replication for individual file/folders ? I find this very usefule. In most of the cases,

Re: MDS crash, wont startup again

2012-05-21 Thread Gregory Farnum
On Mon, May 21, 2012 at 5:38 AM, Felix Feinhals f...@turtle-entertainment.de wrote: Hi Josh, i quoted the trace and some other stats in my first email, maybe it got stuck in the spam filters. Well next try: snip -3 2012-05-10 14:52:29.509940 7fb1c9351700 1 mds.0.40 handle_mds_map  i am

Re: mkfs on osd - failed in 0.47

2012-05-21 Thread Gregory Farnum
Are you actually running as root? (ie, right perms?) What version of Ceph are you using? If you pulled and built binaries off of master over the weekend, I believe it was broken for a few hours in a way that will manifest somewhat like this. On Mon, May 21, 2012 at 1:49 PM, Stefan Priebe

Re: KVM/RBD Block device hangs

2012-05-21 Thread Gregory Farnum
On Mon, May 21, 2012 at 1:51 PM, Stefan Priebe s.pri...@profihost.ag wrote: Am 21.05.2012 16:59, schrieb Wido den Hollander: Probably after, but both is fine. I just want to know how your cluster is doing and what the PG states are. will do so. dmesg says at least: [ 4975.850765]

Re: Designing a cluster guide

2012-05-21 Thread Gregory Farnum
://www.ddrdrive.com/ - Mail original - De: Stefan Priebe s.pri...@profihost.ag À: Gregory Farnum g...@inktank.com Cc: ceph-devel@vger.kernel.org Envoyé: Samedi 19 Mai 2012 10:37:01 Objet: Re: Designing a cluster guide Hi Greg, Am 17.05.2012 23:27, schrieb Gregory Farnum

Re: Designing a cluster guide

2012-05-17 Thread Gregory Farnum
Sorry this got left for so long... On Thu, May 10, 2012 at 6:23 AM, Stefan Priebe - Profihost AG s.pri...@profihost.ag wrote: Hi, the Designing a cluster guide http://wiki.ceph.com/wiki/Designing_a_cluster is pretty good but it still leaves some questions unanswered. It mentions for

Re: Possible memory leak in mon?

2012-05-15 Thread Gregory Farnum
On Sun, May 6, 2012 at 5:53 PM, Vladimir Bashkirtsev vladi...@bashkirtsev.com wrote: On 03/05/12 16:23, Greg Farnum wrote: On Wednesday, May 2, 2012 at 11:24 PM, Vladimir Bashkirtsev wrote: Greg, Apologies for multiple emails: my mail server is backed by ceph now and it struggled this

Re: Disabling logs in the mon data directory?

2012-05-14 Thread Gregory Farnum
Sorry for the long delay; we've all been really busy here... On Tue, May 1, 2012 at 4:24 PM, Nick Bartos n...@pistoncloud.com wrote: I'm trying to get all logs going to syslog, and disable all ceph log files.  I added the following to [global]:    log file =    log to syslog = true Which

Re: [PATCH] libcephfs: return error when not mounted

2012-05-14 Thread Gregory Farnum
On Mon, May 14, 2012 at 10:49 AM, Gregory Farnum g...@inktank.com wrote: I haven't had the chance to go over everything thoroughly (I notice a few other users of EDOM that I want to check out), but those definitely sound better to me. More better review comments later today. -Greg After

Re: Possible memory leak in mon?

2012-05-14 Thread Gregory Farnum
On Sun, May 6, 2012 at 5:53 PM, Vladimir Bashkirtsev vladi...@bashkirtsev.com wrote: On 03/05/12 16:23, Greg Farnum wrote: On Wednesday, May 2, 2012 at 11:24 PM, Vladimir Bashkirtsev wrote: Greg, Apologies for multiple emails: my mail server is backed by ceph now and it struggled this

Re: Still inconsistant pg's, ceph-osd crashes reliably after trying to repair

2012-03-01 Thread Gregory Farnum
On Thu, Mar 1, 2012 at 10:07 AM, Oliver Francke oliver.fran...@filoo.de wrote: Well, Am 01.03.2012 um 18:15 schrieb Oliver Francke: Hi *, after some crashes we still had to care for some remaining inconsistancies reported via    ceph -w and friends. Well, we traced one of them down via

Re: [WRN] map e### wrongly marked me down or wrong addr

2012-02-28 Thread Gregory Farnum
2012/2/28 Székelyi Szabolcs szeke...@niif.hu: On 2012. February 27. 09:03:11 Sage Weil wrote: On Mon, 27 Feb 2012, Székelyi Szabolcs wrote: whenever I restart osd.0 I see a pair of messages like 2012-02-27 17:26:00.132666 mon.0 osd_1_ip:6789/0 106 : [INF] osd.0 osd_0_ip:6801/29931 failed

Re: Implication of using rados_ioctx_locator_set_key

2012-02-28 Thread Gregory Farnum
On Tue, Feb 28, 2012 at 10:07 AM, Noah Watkins jayh...@cs.ucsc.edu wrote: I'm curious about what performance implications there may be when using rados_ioctx_locator_set_key. If a large number of objects are forced into a single PG using a fixed locator key, are there performance

Re: Problems after crash yesterday

2012-02-27 Thread Gregory Farnum
On Thu, Feb 23, 2012 at 9:14 PM, Gregory Farnum gregory.far...@dreamhost.com wrote: On Wed, Feb 22, 2012 at 12:25 PM, Jens Rehpöhler jens.rehpoeh...@filoo.de wrote: Hi Gregory, On 22.02.2012 18:12, Gregory Farnum wrote: On Feb 22, 2012, at 1:53 AM, Jens Rehpöhler jens.rehpoeh...@filoo.de

Re: [WRN] map e### wrongly marked me down or wrong addr

2012-02-27 Thread Gregory Farnum
2012/2/27 Székelyi Szabolcs szeke...@niif.hu: Hello, whenever I restart osd.0 I see a pair of messages like 2012-02-27 17:26:00.132666 mon.0 osd_1_ip:6789/0 106 : [INF] osd.0 osd_0_ip:6801/29931 failed (by osd.1 osd_1_ip:6806/20125) 2012-02-27 17:26:21.074926 osd.0 osd_0_ip:6801/29931 1 :

Re: ceph does not work

2012-02-24 Thread Gregory Farnum
: On Thu, Feb 23, 2012 at 11:07, Gregory Farnum gregory.far...@dreamhost.com wrote: 3 nodes, each running mon, mds osd with replication level 3 for data met adata pools. ... Actually the OSDs will happily (well, not happily; the will complain. But they will run) run in degraded mode

Re: ceph does not work

2012-02-23 Thread Gregory Farnum
On Thu, Feb 23, 2012 at 11:00 AM, Tommi Virtanen tommi.virta...@dreamhost.com wrote: On Thu, Feb 23, 2012 at 01:15, Дениска-редиска s...@inbox.lv wrote: ehllo here, i have tried to setup ceph .41 in simple configuration: 3 nodes, each running mon, mds osd with replication level 3 for data

Re: Problems after crash yesterday

2012-02-23 Thread Gregory Farnum
On Wed, Feb 22, 2012 at 12:25 PM, Jens Rehpöhler jens.rehpoeh...@filoo.de wrote: Hi Gregory, On 22.02.2012 18:12, Gregory Farnum wrote: On Feb 22, 2012, at 1:53 AM, Jens Rehpöhler jens.rehpoeh...@filoo.de wrote: Some Additios: meanwhile we are at the state: 2012-02-22 10:38:49.587403

Re: Problems after crash yesterday

2012-02-22 Thread Gregory Farnum
On Feb 22, 2012, at 1:53 AM, Jens Rehpöhler jens.rehpoeh...@filoo.de wrote: Some Additios: meanwhile we are at the state: 2012-02-22 10:38:49.587403pg v1044553: 2046 pgs: 2036 active+clean, 10 active+clean+inconsistent; 2110 GB data, 4061 GB used, 25732 GB / 29794 GB avail The

Re: Performance test on Ceph cluster

2012-02-22 Thread Gregory Farnum
On Wed, Feb 22, 2012 at 1:39 AM, madhusudhana madhusudhana.u.acha...@gmail.com wrote: Hi I have finally configured a ceph cluster with 8 nodes. I have 2 MDS servers and 3 monitors and rest of 3 nodes are OSD. Each system has 2T SATA drives. I have 3 partitions created, one for root file

Re: [PATCH] ceph.spec.in: Move libcls_*.so from -devel to base package

2012-02-22 Thread Gregory Farnum
I didn't see him say so, but Sage pulled this in last week...thanks! :) -Greg On Wed, Feb 15, 2012 at 8:29 AM, Holger Macht hma...@suse.de wrote: OSDs (src/osd/ClassHandler.cc) specifically look for libcls_*.so in /usr/$libdir/rados-classes, so libcls_rbd.so and libcls_rgw.so need to be

Re: Repeated messages of heartbeat_check: no heartbeat from

2012-02-22 Thread Gregory Farnum
Wido, Sorry we lost track of this last week — we were all distracted by FAST 12! :) So it looks like they're both on the same map and osd.4 is sending pings to osd.19, but osd.19 is just ignoring them? Or do you really have on debug_os and not debug_osd? :) -Greg On Wed, Feb 15, 2012 at 6:12 AM,

Re: Ceph 0.42: Cannot make RPM from spec-file (unpackaged file ceph-dencoder)

2012-02-21 Thread Gregory Farnum
On Tue, Feb 21, 2012 at 11:07 AM, Sage Weil s...@newdream.net wrote: On Tue, 21 Feb 2012, Guido Winkelmann wrote: Hi, With Ceph 0.42, I am unable to create an RPM from supplied spec-file. When running rpmbuild -ba ceph-0.42/ceph.spec, the command fails at the end with this: Checking for

Re: rbd benchmark details

2012-02-21 Thread Gregory Farnum
On Tue, Feb 21, 2012 at 12:25 PM, Noah Watkins jayh...@cs.ucsc.edu wrote: Hi, we are trying to track down a performance problem with rados that is leading no where. We are using 23 OSDs. Running ceph osd tell osd bench we are seeing decent performance on each osd. However, running rados

Re: rbd benchmark details

2012-02-21 Thread Gregory Farnum
On Tue, Feb 21, 2012 at 1:50 PM, Noah Watkins jayh...@cs.ucsc.edu wrote: On Feb 21, 2012, at 12:40 PM, Gregory Farnum wrote: That's pretty much expected. -t is not actually controlling the number of threads, but the number of requests the rados tool will dispatch at a time. With -t 1 you're

Re: How to easily determine what version of Ceph is included in a kernel?

2012-02-21 Thread Gregory Farnum
On Tue, Feb 21, 2012 at 5:29 PM, Paul Pettigrew paul.pettig...@mach.com.au wrote: G'day all We wish to document a simple way to determine what version of ceph/rbd is included with a given kernel on a Client machine, i.e. not the Ceph Cluster nodes themselves (which we are running v0.42 on).

Re: Which SSD method is better for performance?

2012-02-20 Thread Gregory Farnum
On Mon, Feb 20, 2012 at 4:44 PM, Paul Pettigrew paul.pettig...@mach.com.au wrote: Thanks Sage So following through by two examples, to confirm my understanding HDD SPECS: 8x 2TB SATA HDD's able to do sustained read/write speed of 138MB/s each 1x SSD able to do sustained read/write

Re: Need some help in setting up rbd image with kvm

2012-02-14 Thread Gregory Farnum
On Tue, Feb 14, 2012 at 9:24 AM, Tommi Virtanen tommi.virta...@dreamhost.com wrote: On Tue, Feb 14, 2012 at 04:47, Upendra Moturi upendr...@hexagrid.com wrote: Can we create a rbd device without filesystem mounted and can it be a shared device. Yes. Yes. Note that if you do this, it's not

Re: SimpleMessenger testing plan

2012-02-14 Thread Gregory Farnum
at 1:17 PM, Gregory Farnum gregory.far...@dreamhost.com wrote: We on the team decided a while ago that it's past time to start looking seriously at how we can do proper testing of more of our core components without spinning up a full Ceph instance. We've been trying to sneak it in as we can

Re: [RFC PATCH 0/6] Understanding delays due to throttling under very heavy write load

2012-02-06 Thread Gregory Farnum
On Mon, Feb 6, 2012 at 10:20 AM, Jim Schutt jasc...@sandia.gov wrote: On 02/06/2012 10:22 AM, Yehuda Sadeh Weinraub wrote: On Mon, Feb 6, 2012 at 8:20 AM, Jim Schuttjasc...@sandia.gov  wrote: The above suggests to me that the slowdown is a result of requests not getting submitted at the

Re: [RFC PATCH 0/6] Understanding delays due to throttling under very heavy write load

2012-02-03 Thread Gregory Farnum
On Feb 3, 2012, at 8:18 AM, Jim Schutt jasc...@sandia.gov wrote: On 02/02/2012 05:28 PM, Gregory Farnum wrote: On Thu, Feb 2, 2012 at 12:22 PM, Jim Schuttjasc...@sandia.gov wrote: I found 0 instances of waiting for commit in all my OSD logs for my last run. So I never waited on the journal

Re: how to remove laggy mds ?

2012-02-03 Thread Gregory Farnum
On Fri, Feb 3, 2012 at 1:19 PM, Tommi Virtanen tommi.virta...@dreamhost.com wrote: On Fri, Feb 3, 2012 at 11:48, Jens Rehpöhler j...@shadow.gt.owl.de wrote: one of my test mds servers died a few days ago. (hardware crash) I will not buy a new one. Is there any chance to remove this laggy mds

Re: [RFC PATCH 0/6] Understanding delays due to throttling under very heavy write load

2012-02-02 Thread Gregory Farnum
On Thu, Feb 2, 2012 at 7:29 AM, Jim Schutt jasc...@sandia.gov wrote: I'm currently running 24 OSDs/server, one 1TB 7200 RPM SAS drive per OSD.  During a test I watch both OSD servers with both vmstat and iostat. During a good period, vmstat says the server is sustaining 2 GB/s for multiple

Re: [RFC PATCH 3/6] common/Throttle: throttle in FIFO order

2012-02-02 Thread Gregory Farnum
I went to merge this but then had a question on part of it (below). On Wed, Feb 1, 2012 at 7:54 AM, Jim Schutt jasc...@sandia.gov wrote: Under heavy write load from many clients, many reader threads will be waiting in the policy throttler, all on a single condition variable. When a wakeup is

Re: [RFC PATCH 3/6] common/Throttle: throttle in FIFO order

2012-02-02 Thread Gregory Farnum
On Thu, Feb 2, 2012 at 10:31 AM, Jim Schutt jasc...@sandia.gov wrote: On 02/02/2012 10:53 AM, Gregory Farnum wrote: I went to merge this but then had a question on part of it (below). On Wed, Feb 1, 2012 at 7:54 AM, Jim Schuttjasc...@sandia.gov  wrote: Under heavy write load from many

Re: [EXTERNAL] Re: [RFC PATCH 0/6] Understanding delays due to throttling under very heavy write load

2012-02-02 Thread Gregory Farnum
On Thu, Feb 2, 2012 at 11:06 AM, Jim Schutt jasc...@sandia.gov wrote: On 02/02/2012 10:52 AM, Gregory Farnum wrote: On Thu, Feb 2, 2012 at 7:29 AM, Jim Schuttjasc...@sandia.gov  wrote: The typical pattern I see is that a run starts with tens of seconds of aggregate throughput  2 GB/s

Re: Cannot delete some empty dirs and weird sizes

2012-02-02 Thread Gregory Farnum
On Wed, Feb 1, 2012 at 9:02 AM, Amon Ott a@m-privacy.de wrote: On Tuesday 31 January 2012 wrote Gregory Farnum: On Tue, Jan 31, 2012 at 4:00 AM, Amon Ott a@m-privacy.de wrote: Hi again! We are running Ceph 0.41 and kernel 3.2.2 with current for-linus code (commit

Re: [EXTERNAL] Re: [RFC PATCH 0/6] Understanding delays due to throttling under very heavy write load

2012-02-02 Thread Gregory Farnum
On Thu, Feb 2, 2012 at 12:22 PM, Jim Schutt jasc...@sandia.gov wrote: I found 0 instances of waiting for commit in all my OSD logs for my last run. So I never waited on the journal? Looks like it. Interesting. So far I'm looking at two behaviours I've noticed that seem anomalous to me.

Re: More questions about the thesis.

2012-02-02 Thread Gregory Farnum
if there are physical failure correlations that aren't reflected in your CRUSH model. Does that make sense? On Fri, Feb 3, 2012 at 3:43 AM, Gregory Farnum gregory.far...@dreamhost.com wrote: On Thu, Feb 2, 2012 at 12:35 AM, 胡瀚森 softrank@gmail.com wrote: Hi everyone. I've received a nice and welcome

Re: efficient removal of old objects

2012-02-01 Thread Gregory Farnum
On Wed, Feb 1, 2012 at 12:04 AM, Yehuda Sadeh Weinraub yehud...@gmail.com wrote: (resending to list, sorry tv) On Tue, Jan 31, 2012 at 5:02 PM, Tommi Virtanen tommi.virta...@dreamhost.com wrote: On Tue, Jan 31, 2012 at 16:33, Sage Weil s...@newdream.net wrote: Currently rgw logs objects it

Re: efficient removal of old objects

2012-02-01 Thread Gregory Farnum
On Wed, Feb 1, 2012 at 10:53 AM, Yehuda Sadeh Weinraub yehud...@gmail.com wrote: On Wed, Feb 1, 2012 at 9:39 AM, Gregory Farnum gregory.far...@dreamhost.com wrote: You are dramatically overstating the impact of latency on an inherently parallelizable and non-interactive operation. A couple

wip-osd-op-tracking

2012-02-01 Thread Gregory Farnum
[ This branch follows from the osd op tracking thread: http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/4940] I pushed this branch yesterday and just updated it with some suggestions from Sage. It implements basic op tracking for the OSD, essentially as described in the previous

Re: [RFC PATCH 0/6] Understanding delays due to throttling under very heavy write load

2012-02-01 Thread Gregory Farnum
On Wed, Feb 1, 2012 at 7:54 AM, Jim Schutt jasc...@sandia.gov wrote: Hi, FWIW, I've been trying to understand op delays under very heavy write load, and have been working a little with the policy throttler in hopes of using throttling delays to help track down which ops were backing up.

Re: Cannot delete some empty dirs and weird sizes

2012-01-31 Thread Gregory Farnum
On Tue, Jan 31, 2012 at 4:00 AM, Amon Ott a@m-privacy.de wrote: Hi again! We are running Ceph 0.41 and kernel 3.2.2 with current for-linus code (commit 3d882ce47de80e0294a536bec771b5651885b4d3) now. After some heavy workloads we see quite a few directories that cannot be deleted,

Re: Installing_on_RedHat_or_CentOS

2012-01-30 Thread Gregory Farnum
On Mon, Jan 30, 2012 at 7:46 AM, JJ jjo...@cirrhus9.com wrote: I am trying to follow the direction at http://ceph.newdream.net/wiki/Installing_on_RedHat_or_CentOS This is my 3rd or 4th attempt trying to install ceph on 2 different OSs, (CentOS 5.[57] and Ubuntu 11.10 Lately, on CentOS 5.5,

Re: Ceph behavior in case of network failure

2012-01-30 Thread Gregory Farnum
On Sat, Jan 28, 2012 at 8:52 PM, Madhusudhan madhusudhana.u.acha...@gmail.com wrote: I have configured ceph in centos5.6 after a very long fight. Now, i am in the way to evaluate the Ceph. Forget me if my question looks amateur. If we consider a situation where my core switch fails,

Re: ceph on XFS

2012-01-27 Thread Gregory Farnum
On Fri, Jan 27, 2012 at 12:48 PM, Christian Brunner c...@muc.de wrote: Hi, reading the list archives, I get the impression that XFS is the second best alternative to btrfs. But when I start an ceph-osd on an XFS volume, there is still a big warning: WARNING: not btrfs or ext3.  We don't

Re: wip-pg-stale

2012-01-27 Thread Gregory Farnum
On Fri, Jan 27, 2012 at 1:32 PM, Sage Weil s...@newdream.net wrote: Please review. If the monitor sees an osdmap go by where nodes go down (or up) it will scan its pg_map and mark any pg whose primary is down as 'stale'.  If/when the pg recovers, that will get refreshed.  If not, the admin

Re: osd crash during resync

2012-01-24 Thread Gregory Farnum
On Tue, Jan 24, 2012 at 10:48 AM, Martin Mailand mar...@tuxadero.com wrote: Hi, today I tried the btrfs patch mentioned on the btrfs ml. Therefore I rebooted osd.0 with a new kernel and created a new btrfs on the osd.0, than I took the osd.0 into the cluster. During the the resync of osd.0

Re: osd crash during resync

2012-01-24 Thread Gregory Farnum
On Tue, Jan 24, 2012 at 1:22 PM, Martin Mailand mar...@tuxadero.com wrote: Hi Greg, ok, do you guys still need the core files, or could I delete them? Sam thinks probably not since we have the backtraces and the logs...thanks for asking, though! :) -Greg -- To unsubscribe from this list: send

Re: upgrade from 0.39 to 0.40 failed...

2012-01-23 Thread Gregory Farnum
On Sun, Jan 22, 2012 at 4:25 AM, Smart Weblications GmbH - Florian Wiessner f.wiess...@smart-weblications.de wrote: Am 22.01.2012 02:19, schrieb Yehuda Sadeh Weinraub: On Sat, Jan 21, 2012 at 9:43 AM, Smart Weblications GmbH - Florian Wiessner f.wiess...@smart-weblications.de wrote: 2) v1 --

Re: How to remove lost objects.

2012-01-23 Thread Gregory Farnum
On Thu, Jan 19, 2012 at 12:36 PM, Andrey Stepachev oct...@gmail.com wrote: 2012/1/19 Gregory Farnum gregory.far...@dreamhost.com: On Thu, Jan 19, 2012 at 12:53 AM, Andrey Stepachev oct...@gmail.com wrote: 2012/1/19 Gregory Farnum gregory.far...@dreamhost.com: On Wednesday, January 18, 2012

Re: wip-encoding

2012-01-20 Thread Gregory Farnum
On Fri, Jan 20, 2012 at 10:58 AM, Sage Weil s...@newdream.net wrote: The current pattern is void Foo::encode(bufferlist bl, unsigned features) const {  if (features NEW_THING) {     // encode normally  } else {     // encode old way (wahtever was in this func)  } } I'm not sure what

Re: How to remove lost objects.

2012-01-19 Thread Gregory Farnum
On Thu, Jan 19, 2012 at 12:53 AM, Andrey Stepachev oct...@gmail.com wrote: 2012/1/19 Gregory Farnum gregory.far...@dreamhost.com: On Wednesday, January 18, 2012, Andrey Stepachev oct...@gmail.com wrote: 2012/1/19 Gregory Farnum gregory.far...@dreamhost.com: On Wed, Jan 18, 2012 at 12:48 PM

Re: How to remove lost objects.

2012-01-18 Thread Gregory Farnum
On Wed, Jan 18, 2012 at 12:48 PM, Andrey Stepachev oct...@gmail.com wrote: But still don't know what happens with ceph, so it can't respond and hang. It is not a good behavior, because such situation leads to unresponsible cluster in case of temporal network failure. I'm a little concerned

Re: how can I achieve HA with ceph?

2012-01-18 Thread Gregory Farnum
version? Everything I checked looked good for the rest of the system; did you check if new clients could do things appropriately? -Greg (note: I removed a huge chunk of the alpha mds log covering approx 1 minute) -- Karoly Horvath On Wed, Dec 21, 2011 at 4:13 PM, Gregory Farnum gregory.far

Re: Slow ceph io. High iops. Compared to hadoop.

2012-01-18 Thread Gregory Farnum
On Tue, Jan 17, 2012 at 1:19 PM, Andrey Stepachev oct...@gmail.com wrote: 2012/1/17 Gregory Farnum gregory.far...@dreamhost.com: On Tue, Jan 17, 2012 at 11:37 AM, Andrey Stepachev oct...@gmail.com wrote: 2012/1/17 Sage Weil s...@newdream.net: On Mon, 16 Jan 2012, Andrey Stepachev wrote: Ops

Searching for a Distributed Filesystem

2012-01-18 Thread Gregory Farnum
[http://roland.entierement.nu/blog/2012/01/15/looking-for-the-ultimate-distributed-filesystem.html] Roland, I had just a few comments on your characterization of Ceph that I thought I should share. :) Availability/redundancy 1: Saying Ceph works on the net is a bit of a stretch. It will probably

<    4   5   6   7   8   9   10   11   12   >