Re: Long peering - throttle at FileStore::queue_transactions

On Mon, Jan 4, 2016 at 7:21 PM, Sage Weil <s...@newdream.net> wrote: > On Mon, 4 Jan 2016, Guang Yang wrote: >> Hi Cephers, >> Happy New Year! I got question regards to the long PG peering.. >> >> Over the last several days I have been looking into the *long peerin

Long peering - throttle at FileStore::queue_transactions

Hi Cephers, Happy New Year! I got question regards to the long PG peering.. Over the last several days I have been looking into the *long peering* problem when we start a OSD / OSD host, what I observed was that the two peering working threads were throttled (stuck) when trying to queue new

Re: OSD data file are OSD logs

f0c773df759653d, firefly backport is > b8e3f6e190809febf80af66415862e7c7e415214. > -Sam > > On Mon, Jan 4, 2016 at 3:37 PM, Guang Yang <guan...@gmail.com> wrote: >> Hi Cephers, >> Before I open a tracker, I would like check if it is a known issue or not.. >> >

OSD data file are OSD logs

Hi Cephers, Before I open a tracker, I would like check if it is a known issue or not.. One one of our clusters, there was OSD crash during repairing, the crash happened after we issued a PG repair for inconsistent PGs, which failed because the recorded file size (within xattr) mismatched with

Re: Newly added monitor infinitely sync store

for this setting (e.g. 20 minutes)? Or should I add the advise for monitor trouble-shooting? Thanks, Guang On Fri, Nov 13, 2015 at 9:07 PM, Guang Yang <guan...@gmail.com> wrote: > Thanks Sage! I will definitely try those patches. > > For this one, I finally managed to bring

Re: Newly added monitor infinitely sync store

On Mon, Nov 16, 2015 at 5:42 PM, Sage Weil <s...@newdream.net> wrote: > On Mon, 16 Nov 2015, Guang Yang wrote: >> I spoke to a leveldb expert, it looks like this is a known pattern on >> LSM tree data structure - the tail latency for range scan could be far >> longer tha

[no subject]

Hi Joao, We have a problem when trying to add new monitors to the cluster on an unhealthy cluster, which I would like ask for your suggestion. After adding the new monitor, it started syncing the store and went into an infinite loop: 2015-11-12 21:02:23.499510 7f1e8030e700 10

Re: Newly added monitor infinitely sync store

t;s...@newdream.net> wrote: > On Fri, 13 Nov 2015, Guang Yang wrote: >> Thanks Sage! >> >> On Fri, Nov 13, 2015 at 4:15 PM, Sage Weil <s...@newdream.net> wrote: >> > On Fri, 13 Nov 2015, Guang Yang wrote: >> >> I was wrong the previous analysis, it w

Newly added monitor infinitely sync store

Sage and Joao, Is there a way to freeze the election by some tunable to let the sync finish? Thanks, Guang On Fri, Nov 13, 2015 at 9:00 AM, Guang Yang <guan...@gmail.com> wrote: > Hi Joao, > We have a problem when trying to add new monitors to the cluster on an > unhealthy cluster

Re: Newly added monitor infinitely sync store

Thanks Sage! On Fri, Nov 13, 2015 at 4:15 PM, Sage Weil <s...@newdream.net> wrote: > On Fri, 13 Nov 2015, Guang Yang wrote: >> I was wrong the previous analysis, it was not the iterator got reset, >> the problem I can see now, is that during the syncing, a new round of &

Re: Symbolic links like feature on radosgw

Hi Yehuda, We have a user requirement that needs symbolic link like feature on radosgw - two object ids pointing to the same object (ideally it could cross bucket, but same bucket is fine). The closest feature on Amazon S3 I could find is [1], but not exact the same, the one from Amazon S3 API

Re: Weekly performance meeting

On Sep 26, 2014, at 9:12 PM, Mark Nelson mark.nel...@inktank.com wrote: On 09/25/2014 09:47 PM, Guang Yang wrote: Hi Sage, We are very interested to join (and contribute effort) as well. Following are a list of issues we have particular interests: 1 Large number of small files bring

Re: Weekly performance meeting

Hi Sage, We are very interested to join (and contribute effort) as well. Following are a list of issues we have particular interests: 1 Large number of small files bring performance degradation most due to file system lookup (even worst with EC). 2 Messenger uses too many threads which bring

RGW threads hung - more logs

Hi Sage, Sam and Greg, With the radosgw hung issue we discussed this today, I finally got some more logs showing that the reply message has been received by ragosgw, but failed to be dispatched as dispatcher thread was hung. I put all the logs into the tracker -

Issue - 8907

Hi Loic, Can you help to take a quick look over this issue - http://tracker.ceph.com/issues/8907, was it a design choice due to consistency concern? Thanks, Guang-- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More

Re: OSD suicide after being down/in for one day as it needs to search large amount of objects

Thanks Greg. On Aug 20, 2014, at 6:09 AM, Gregory Farnum g...@inktank.com wrote: On Mon, Aug 18, 2014 at 11:30 PM, Guang Yang yguan...@outlook.com wrote: Hi ceph-devel, David (cc’ed) reported a bug (http://tracker.ceph.com/issues/9128) which we came across in our test cluster during our

Re: OSD suicide after being down/in for one day as it needs to search large amount of objects

Thanks Sage. We will provide a patch based on this. Thanks, Guang On Aug 20, 2014, at 11:19 PM, Sage Weil sw...@redhat.com wrote: On Wed, 20 Aug 2014, Guang Yang wrote: Thanks Greg. On Aug 20, 2014, at 6:09 AM, Gregory Farnum g...@inktank.com wrote: On Mon, Aug 18, 2014 at 11:30 PM, Guang

OSD suicide after being down/in for one day as it needs to search large amount of objects

Hi ceph-devel, David (cc’ed) reported a bug (http://tracker.ceph.com/issues/9128) which we came across in our test cluster during our failure testing, basically the way to reproduce it was to leave one OSD daemon down and in for a day, at the same time, keep giving write traffic. When the OSD

Re: assert failure

- From: Guang Yang yguan...@outlook.com To: Huamin Chen hc...@redhat.com Cc: Ceph-devel ceph-devel@vger.kernel.org Sent: Friday, August 15, 2014 2:23:12 PM Subject: Re: assert failure + ceph-devel. Hi Huamin, Did you upgrade the entire cluster to v0.80.5? If I remember correctly, if its peer

Re: assert failure

- Original Message - From: Guang Yang yguan...@outlook.com To: Huamin Chen hc...@redhat.com Cc: ceph-devel@vger.kernel.org Sent: Wednesday, August 13, 2014 10:39:10 PM Subject: Re: assert failure Hi Huamin, At least one known issue in 0.80.1 with the same failing pattern has

OSD disk replacement best practise

Hi cephers, Most recently I am drafting the run books for OSD disk replacement, I think the rule of thumb is to reduce data migration (recover/backfill), and I thought the following procedure should achieve the purpose: 1. ceph osd out osd.XXX (mark it out to trigger data migration) 2. ceph

Re: assert failure

Hi Huamin, At least one known issue in 0.80.1 with the same failing pattern has been fixed in the latest 0.80.4 release of firefly. Here is the tracking ticket - http://tracker.ceph.com/issues/8232. Can you compare the log snippets from within the bug and see if they are the same issue?

Re: bucket index sharding - IO throttle

Hi Yehuda, Can you help to review the latest patch with throttle mechanism you suggested. Thanks! Thanks, Guang On Aug 4, 2014, at 3:20 PM, Guang Yang yguan...@outlook.com wrote: Hi Yehuda, Here is the new pull request - https://github.com/ceph/ceph/pull/2187 Thanks, Guang On Jul 31

Re: bucket index sharding - IO throttle

On Aug 6, 2014, at 12:38 PM, Osier Yang os...@yunify.com wrote: On 2014年08月04日 15:20, Guang Yang wrote: Hi Yehuda, Here is the new pull request - https://github.com/ceph/ceph/pull/2187 I simply applied the patch on git top, and the testing shows rest-bench is completely broken with the 2

Re: bucket index sharding - IO throttle

Hi Yehuda, Here is the new pull request - https://github.com/ceph/ceph/pull/2187 Thanks, Guang On Jul 31, 2014, at 10:40 PM, Guang Yang yguan...@outlook.com wrote: Thanks Yehuda. I will do that (sorry I was occupied by some other stuff recently but I will try my best to provide a patch

Re: KeyFileStore ?

. -Sam On Fri, Aug 1, 2014 at 8:08 AM, Guang Yang yguan...@outlook.com wrote: I really like the idea, one scenario keeps bothering us is that there are too many small files which make the file system indexing slow (so that a single read request could take more than 10 disk IOs for path lookup

Re: KeyFileStore ?

I really like the idea, one scenario keeps bothering us is that there are too many small files which make the file system indexing slow (so that a single read request could take more than 10 disk IOs for path lookup). If we pursuit this proposal, is there a chance we can take one step further,

Re: bucket index sharding - IO throttle

)? It'lll be easier to review and comment. Thanks, Yehuda On Wed, Jul 30, 2014 at 7:58 AM, Guang Yang yguan...@outlook.com wrote: +ceph-devel. Thanks, Guang On Jul 29, 2014, at 10:20 PM, Guang Yang yguan...@outlook.com wrote: Hi Yehuda, Per you review comment in terms of IO

Re: bucket index sharding - IO throttle

+ceph-devel. Thanks, Guang On Jul 29, 2014, at 10:20 PM, Guang Yang yguan...@outlook.com wrote: Hi Yehuda, Per you review comment in terms of IO throttling for bucket index operation, I prototyped the below code (details still need to polish), can you take a look if that is right way

row geo-replication to another data store?

Hi cephers, We are investigating a backup solution for Ceph, in short, we would like a solution to backup a Ceph cluster to another data store (not Ceph cluster, assume it has SWIFT API). We would like to have both full backup and incremental backup on top of the full backup. After going

Re: EC pool - empty files in OSD from radosgw

Hi Yehuda and Sam, Any suggestion on top of the below issue? Thanks, Guang On Jul 12, 2014, at 12:43 AM, Guang Yang yguan...@outlook.com wrote: Hi Loic, I opened an issue in terms of a change brought along with EC pool plus radosgw (http://tracker.ceph.com/issues/8625), in our test cluster

EC pool - empty files in OSD from radosgw

Hi Loic, I opened an issue in terms of a change brought along with EC pool plus radosgw (http://tracker.ceph.com/issues/8625), in our test cluster, we observed a large number of empty files in OSD and the root cause is that for head object from radosgw, there are a couple of transactions coming

Re: v0.80.2?

Hi Sage, Is it possible to include a fix for this bug - http://tracker.ceph.com/issues/8733 considering the scope of the change and regression risk for the next release? We are finalizing our production launch version and this one is a blocker as we use EC pool. Thanks, Guang On Jul 11, 2014,

radosgw - bucket index sharding merge back

Hi Yehuda, I am trying to find a way to merge back the bucket index sharding effort, and with more experience working at Ceph, I realized that the original commit was too huge which introduced trouble for review. I am thinking to break it down into multiple small commits and commit back with a

radosgw - bucket index sharding merge back

Hi Yehuda, I am trying to find a way to merge back the bucket index sharding effort, and with more experience working at Ceph, I realized that the original commit was too huge which introduced trouble for review. I am thinking to break it down into multiple small commits and commit back with a

XFS - number of files in a directory

Hello Cephers, We used to have a Ceph cluster and setup our data pool as 3 replicas, we estimated the number of files (given disk size and object size) for each PG was around 8K, we disabled folder splitting which mean all files located at the root PG folder. Our testing showed a good

Re: CDS G/H - bucket index sharding

Thanks Yehuda, my comments inline... On Jun 23, 2014, at 10:44 PM, Yehuda Sadeh yeh...@inktank.com wrote: On Mon, Jun 23, 2014 at 4:11 AM, Guang Yang yguan...@outlook.com wrote: Hello Yehuda, I drafted a brief summary for the status of the bucket index sharding blueprint and put it here

Re: Changes of scrubbing?

On Jun 11, 2014, at 10:02 PM, Gregory Farnum g...@inktank.com wrote: On Wed, Jun 11, 2014 at 12:54 AM, Guang Yang yguan...@outlook.com wrote: On Jun 11, 2014, at 6:33 AM, Gregory Farnum g...@inktank.com wrote: On Tue, May 20, 2014 at 6:44 PM, Guang Yang yguan...@outlook.com wrote: Hi ceph

Re: Changes of scrubbing?

On Jun 11, 2014, at 6:33 AM, Gregory Farnum g...@inktank.com wrote: On Tue, May 20, 2014 at 6:44 PM, Guang Yang yguan...@outlook.com wrote: Hi ceph-devel, Like some users of Ceph, we are using Ceph for a latency sensitive project, and scrubbing (especially deep-scrubbing) impacts the SLA

Re: Radosgw - bucket index

, Guang Yang yguan...@outlook.com wrote: Hi Yehuda and Sage, Can you help to comment on the ticket, I would like to send out a pull request some time this week for you to review, but before that, it would be nice to see your comments in terms of the interface and any other concerns you may have

Re: Radosgw - bucket index

On May 30, 2014, at 8:35 AM, Guang Yang yguan...@outlook.com wrote: Hi Yehuda, I opened an issue here: http://tracker.ceph.com/issues/8473, please help to review and comment. Thanks, Guang On May 19, 2014, at 2:47 PM, Yehuda Sadeh yeh...@inktank.com wrote: On Sun, May 18, 2014 at 11:18 PM

Re: Radosgw - bucket index

Hi Yehuda, I opened an issue here: http://tracker.ceph.com/issues/8473, please help to review and comment. Thanks, Guang On May 19, 2014, at 2:47 PM, Yehuda Sadeh yeh...@inktank.com wrote: On Sun, May 18, 2014 at 11:18 PM, Guang Yang yguan...@outlook.com wrote: On May 19, 2014, at 7:05 AM

Re: erasure code reliability model

Hello Loic and Koleos, Do we have any wiki page documenting the progress and report of this effort? We are very interested in such as well. Thanks, Guang On May 8, 2014, at 1:19 AM, Loic Dachary l...@dachary.org wrote: Hi, On 07/05/2014 18:43, Koleos Fuskus wrote: Hi Loic, What do you

Re: erasure code reliability model

Thanks Koleos! Guang On May 26, 2014, at 7:02 PM, Koleos Fuskus koleosfus...@yahoo.com wrote: Hello Guang, Here is the wiki: https://wiki.ceph.com/Development/Add_erasure_coding_to_the_durability_modeling Koleos On Monday, May 26, 2014 10:05 AM, Guang Yang yguan...@outlook.com

Questions of KeyValueStore (leveldb) backend

Hello Haomai, We are evaluating the key-value store backend which comes along with Firefly release (thanks for implementing it in Ceph), it is very promising for a couple of our use cases, after going through the related code change, I have a couple of questions which needs your help: 1. One

Re: Questions of KeyValueStore (leveldb) backend

Thanks! On May 26, 2014, at 12:55 PM, Haomai Wang haomaiw...@gmail.com wrote: On Mon, May 26, 2014 at 9:46 AM, Guang Yang yguan...@outlook.com wrote: Hello Haomai, We are evaluating the key-value store backend which comes along with Firefly release (thanks for implementing it in Ceph

Re: Radosgw - bucket index

On May 19, 2014, at 7:05 AM, Sage Weil s...@inktank.com wrote: On Sun, 18 May 2014, Guang wrote: radosgw is using the omap key/value API for objects, which is more or less equivalent to what swift is doing with sqlite. This data passes straight into leveldb on the backend (or whatever other