Re: KeyFileStore ?

2014-07-31 Thread Haomai Wang
Awesome job! On Thu, Jul 31, 2014 at 1:49 PM, Mark Kirkwood mark.kirkw...@catalyst.net.nz wrote: On 31/07/14 17:25, Sage Weil wrote: After the latest set of bug fixes to the FileStore file naming code I am newly inspired to replace it with something less complex. Right now I'm mostly

RE: Pull Request for ISA EC plug-in

2014-07-31 Thread Andreas Joachim Peters
Ok, thanks for looking into that. I am fixing the 'make dist' target and will try to make it work also with yasm 1.1 ... Will let you know when to retry ... Andreas. From: ceph-devel-ow...@vger.kernel.org [ceph-devel-ow...@vger.kernel.org] on behalf

RE: First attempt at rocksdb monitor store stress testing

2014-07-31 Thread Shu, Xinxin
Hi sage , This maybe due to $(shell) is a feature of GNU make , I think there are two solutions: 1) run the script at configure time rather than at run time. 2) $(shell (./build_tools/build_detect_version)) will generated util/build_version.cc , the file only contain some version info

RE: First attempt at rocksdb monitor store stress testing

2014-07-31 Thread Shu, Xinxin
Hi sage , I create a pull request https://github.com/ceph/rocksdb/pull/3 , please help review. Cheers, xinxin -Original Message- From: Shu, Xinxin Sent: Thursday, July 31, 2014 4:42 PM To: 'Sage Weil' Cc: Mark Nelson; ceph-devel@vger.kernel.org Subject: RE: First attempt at rocksdb

RE: Pull Request for ISA EC plug-in

2014-07-31 Thread Andreas Joachim Peters
Ok, I fixed the 'make dist' problem and hopefully also the YASM include problem. The CEPH yasm-wrapper was removing '-I' from the options passed to yasm, so '-I' didn't work without patching the wrapper itself. I looked into YASM 1.1 and I found that at a certain patch version they fixed the

[PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Ilya Dryomov
This reverts commit 34c6bc2c919a55e5ad4e698510a2f35ee13ab900. This commit can lead to deadlocks by way of what at a high level appears to look like a missing wakeup on mutex_unlock() when CONFIG_MUTEX_SPIN_ON_OWNER is set, which is how most distributions ship their kernels. In particular, it

RE: Cache tiering read-proxy mode

2014-07-31 Thread Wang, Zhiqiang
I've added the testcases and made the style changes. The testcases ran without problem. Pls merge the pull request https://github.com/ceph/ceph/pull/2177, thx. -Original Message- From: Sage Weil [mailto:sw...@redhat.com] Sent: Tuesday, July 29, 2014 11:44 PM To: Wang, Zhiqiang Cc:

About erasure code optimizations for ARM

2014-07-31 Thread Loic Dachary
Hi, When using the jerasure plugin for erasure coded placement groups, it takes advantage of http://en.wikipedia.org/wiki/SIMD when running on an Intel processor. It would be nice to also use http://en.wikipedia.org/wiki/NEON_%28instruction_set%29 when running on an ARM processor. The

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Peter Zijlstra
On Thu, Jul 31, 2014 at 02:16:37PM +0400, Ilya Dryomov wrote: This reverts commit 34c6bc2c919a55e5ad4e698510a2f35ee13ab900. This commit can lead to deadlocks by way of what at a high level appears to look like a missing wakeup on mutex_unlock() when CONFIG_MUTEX_SPIN_ON_OWNER is set, which

Re: First attempt at rocksdb monitor store stress testing

2014-07-31 Thread Mark Nelson
On 07/30/2014 08:46 PM, Shu, Xinxin wrote: Hi mark, Which way do you used to set a higher limitation? use 'ulimit' command or enlarge rocksdb_max_open_files config option? ulimit command, though you might be able to limit the max open files in rocksdb to be smaller than the default too.

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Ilya Dryomov
On Thu, Jul 31, 2014 at 3:57 PM, Peter Zijlstra pet...@infradead.org wrote: On Thu, Jul 31, 2014 at 02:16:37PM +0400, Ilya Dryomov wrote: This reverts commit 34c6bc2c919a55e5ad4e698510a2f35ee13ab900. This commit can lead to deadlocks by way of what at a high level appears to look like a

Re: First attempt at rocksdb monitor store stress testing

2014-07-31 Thread Mark Nelson
Hi Xinxin, On the first page, the first three tables are for latency, and the 4th is for total times. It's all the same workload, 30% read / 70% write with a random distribution of object sizes, but there are 6 different tests: leveldb on spinning disk leveldb on ssd rocksdb with leveled

Re: First attempt at rocksdb monitor store stress testing

2014-07-31 Thread Mark Nelson
FWIW this was the problem I ran into and mentioned in #ceph-devel the other day. The way I solved it was to add -Wno-portability to the configure.ac file in the rocksdb distribution. Perhaps this is a better solution though... Mark On 07/31/2014 03:58 AM, Shu, Xinxin wrote: Hi sage , I

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Peter Zijlstra
On Thu, Jul 31, 2014 at 04:37:29PM +0400, Ilya Dryomov wrote: This didn't make sense to me at first too, and I'll be happy to be proven wrong, but we can reproduce this with rbd very reliably under higher than usual load, and the revert makes it go away. What we are seeing in the rbd

Re: KeyFileStore ?

2014-07-31 Thread Gregory Farnum
On Thu, Jul 31, 2014 at 1:25 AM, Sage Weil sw...@redhat.com wrote: After the latest set of bug fixes to the FileStore file naming code I am newly inspired to replace it with something less complex. Right now I'm mostly thinking about HDDs, although some of this may map well onto hybrid

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Ingo Molnar
* Peter Zijlstra pet...@infradead.org wrote: On Thu, Jul 31, 2014 at 04:37:29PM +0400, Ilya Dryomov wrote: This didn't make sense to me at first too, and I'll be happy to be proven wrong, but we can reproduce this with rbd very reliably under higher than usual load, and the revert makes

Re: KeyFileStore ?

2014-07-31 Thread Matt W. Benjamin
+1 - Sage Weil sw...@redhat.com wrote: Anyway, at the end of the day, small writes or overwrites would still be journaled, but large writes or large new objects would not, which would (I think) be a pretty big improvement. Yes. Overall, I think the design will be much simpler

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Peter Zijlstra
On Thu, Jul 31, 2014 at 03:44:11PM +0200, Ingo Molnar wrote: diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index ae712b25e492..3d726fdaa764 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -473,8 +473,12 @@ __mutex_lock_common(struct mutex *lock, long

Re: KeyFileStore ?

2014-07-31 Thread Mark Nelson
On 07/31/2014 08:18 AM, Gregory Farnum wrote: On Thu, Jul 31, 2014 at 1:25 AM, Sage Weil sw...@redhat.com wrote: After the latest set of bug fixes to the FileStore file naming code I am newly inspired to replace it with something less complex. Right now I'm mostly thinking about HDDs, although

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Mike Galbraith
On Thu, 2014-07-31 at 15:13 +0200, Peter Zijlstra wrote: Smells like maybe current-state != TASK_RUNNING Bingo [ 1200.851004] kjournald D 0002 0 4398 2 0x [ 1200.858283] 8803537bb788 0046 8803537bb7a8 [ 1200.865914]

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Ilya Dryomov
On Thu, Jul 31, 2014 at 6:30 PM, Mike Galbraith umgwanakikb...@gmail.com wrote: On Thu, 2014-07-31 at 15:13 +0200, Peter Zijlstra wrote: Smells like maybe current-state != TASK_RUNNING It just triggered for me too, took longer than usual. Sorry for the churn Peter, this was really confusing.

Re: [PATCH] locking/mutexes: Revert locking/mutexes: Add extra reschedule point

2014-07-31 Thread Peter Zijlstra
On Thu, Jul 31, 2014 at 04:30:52PM +0200, Mike Galbraith wrote: On Thu, 2014-07-31 at 15:13 +0200, Peter Zijlstra wrote: Smells like maybe current-state != TASK_RUNNING Bingo [ 1200.851004] kjournald D 0002 0 4398 2 0x [ 1200.858283]

Re: bucket index sharding - IO throttle

2014-07-31 Thread Guang Yang
Thanks Yehuda. I will do that (sorry I was occupied by some other stuff recently but I will try my best to provide a patch as soon as possible). Thanks, Guang 在 2014年7月31日,上午1:00,Yehuda Sadeh yeh...@inktank.com 写道: Can you send this code through a github pull request (or at least as a

Re: KeyFileStore ?

2014-07-31 Thread Yehuda Sadeh
On Thu, Jul 31, 2014 at 6:18 AM, Gregory Farnum g...@inktank.com wrote: On Thu, Jul 31, 2014 at 1:25 AM, Sage Weil sw...@redhat.com wrote: After the latest set of bug fixes to the FileStore file naming code I am newly inspired to replace it with something less complex. Right now I'm mostly

Re: [ceph-users] question about ApplyManager, SubmitManager and FileJournal

2014-07-31 Thread Gregory Farnum
These sorts of questions are good for ceph-devel@vger.kernel.org, which I've added. :) On Thu, Jul 31, 2014 at 12:24 PM, yuelongguang fasts...@163.com wrote: hi,all recently i dive into the source code, i am a little confused about them, maybe because of many threads,wait,seq. 1. what does

RE: Pull Request for ISA EC plug-in

2014-07-31 Thread Sage Weil
Okay, I almost have it! I've added a configure rule to detect whether yasm is happy, and fixed a few things in the wrapper and makefile. https://github.com/ceph/ceph/commits/wip-ec-isa https://github.com/ceph/ceph/pull/2180 The remaining issue is that yasm 1.1 can't seem to

Re: [PATCH]ECBackend: Using ROUND_UP_TO to refactor get_recovery_chunk_size()

2014-07-31 Thread Samuel Just
Any chance you can submit this as a pull request? On Tue, Jul 29, 2014 at 11:12 PM, Ma, Jianpeng jianpeng...@intel.com wrote: Signed-off-by: Ma Jianpeng jianpeng...@intel.com --- src/osd/ECBackend.h | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) diff --git

Re: [PATCH]ECBackend: Don't directyly use get_recovery_chunk_size() in RecoveryOp::WRITING state.

2014-07-31 Thread Samuel Just
A pull request for this one as well. -Sam On Tue, Jul 29, 2014 at 11:09 PM, Ma, Jianpeng jianpeng...@intel.com wrote: We cannot guarantee that conf-osd_recovery_max_chunk don't change when recoverying a erasure object. If change between RecoveryOp::READING and RecoveryOp::WRITING, it can cause

bug of bucket instance log list?

2014-07-31 Thread star fan
Hi Yehuda, I think there are logic error about max_entries betweenRGWOp_BILog_List::execute() and store-list_bi_log_entries. Look at the codes: in RGWOp_BILog_List::execute() do { listrgw_bi_log_entry entries; int ret = store-list_bi_log_entries(bucket_info.bucket,

Re: bug of bucket instance log list?

2014-07-31 Thread Yehuda Sadeh
On Thu, Jul 31, 2014 at 8:34 PM, star fan jfa...@gmail.com wrote: Hi Yehuda, I think there are logic error about max_entries betweenRGWOp_BILog_List::execute() and store-list_bi_log_entries. Look at the codes: in RGWOp_BILog_List::execute() do { listrgw_bi_log_entry entries;