Do we have any proposed solutions for this problem? Copy the needed info from
base tier to cache tier during promotion? I see it has been there for over 6
months.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Tuesday, January 20, 2015 12:51 AM
To: Wang, Zhiqiang
M
To: Wang, Zhiqiang
Cc: sj...@redhat.com; ceph-devel@vger.kernel.org
Subject: RE: wip-proxy-write and (non-idempotent) client ops
On Tue, 20 Jan 2015, Wang, Zhiqiang wrote:
> Do we have any proposed solutions for this problem? Copy the needed
> info from base tier to cache tier during promoti
This method puts the reqid list in the pg_log instead of the object_info_t, so
that it's preserved even in the delete case, which sounds more reasonable.
-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Sage Weil
Sent: Satur
. Otherwise the
object exists, we check the reqid list in the object_info_t for dup op.
-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Wang, Zhiqiang
Sent: Monday, January 26, 2015 10:35 AM
To: Sage Weil; Gregory Farnum
Cc
lf Of Sage Weil
Sent: Tuesday, January 27, 2015 1:21 AM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org; Gregory Farnum
Subject: RE: idempotent op (esp delete)
On Mon, 26 Jan 2015, Wang, Zhiqiang wrote:
> The downside of this approach is that we may need to search the pg_log
> for a specifi
Oh, sorry, I didn't see this extra_caller_ops change. Now it makes sense.
-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Sage Weil
Sent: Tuesday, January 27, 2015 10:31 PM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kerne
How about handling the DELETE op in the cache tier like this:
1) If the object is in the cache tier, we delete it in cache tier, replace it
with a whiteout, and later flush and evict it.
2) If the object is not in the cache tier, we always proxy the delete op. This
can be done after the proxy wri
ck to.
-Original Message-
From: Ning Yao [mailto:zay11...@gmail.com]
Sent: Monday, March 30, 2015 10:29 PM
To: Wang, Zhiqiang
Cc: Sage Weil; ceph-devel
Subject: Re: RBD Discard issue for Cache_tier
Sounds much easier, if we use this. Maybe snapshot also got this problem, I
may not wa
Hi all,
I ran into a problem during the teuthology test of proxy write. It is like this:
- Client sends 3 writes and a read on the same object to base tier
- Set up cache tiering
- Client retries ops and sends the 3 writes and 1 read to the cache tier
- The 3 writes finished on the base tier, say
We have a bunch of teuthology tests which build cache pool on top of an ec base
pool, and do partial object write. This is ok with the current cache tiering
implementation. But with proxy write, this won't work. In my testing, the error
message is something like below:
2015-05-20 05:51:42.89682
lto:sw...@redhat.com]
Sent: Monday, May 25, 2015 11:04 PM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: Re: Cache pool on top of ec base pool teuthology test
On Mon, 25 May 2015, Wang, Zhiqiang wrote:
> We have a bunch of teuthology tests which build cache pool on top of
&g
Yes, we can do the force promotion check in init_op_flags, as we did before.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Tuesday, May 26, 2015 10:19 AM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: RE: Cache pool on top of ec base pool teuthology test
Hi Sage and all,
Another bug discovered during the proxy write teuthology testing is when
rolling back to a degraded object. This doesn't seem specific to proxy write. I
see a scenario like below from the log file:
- A rollback op comes in, and is enqueued.
- Several other ops on the same objec
That's great we already have such a field. I'll make use of it to fix this.
Thanks for pointing it out.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Tuesday, June 2, 2015 7:42 AM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: Re: A bug in ro
Hi all,
Several teuthology jobs fail with the error 'exception on parallel execution'
in my testing. I don't see any failures/errors/assertions in the mon/osd logs.
And these failed jobs always have 'sentry event' log. However, I'm not able to
open the sentry link.
Here is an example:
http://p
this case.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Tuesday, May 26, 2015 12:27 AM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: Re: 'Racing read got wrong version' during proxy write testing
On Mon, 25 May 2015, Wang, Zhiqiang wrote:
> Hi a
Making the 'copy get' op to be a cache op seems like a good idea.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Thursday, June 4, 2015 9:14 AM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: RE: 'Racing read got wrong version' during
Hi David,
Proxy write hasn't been merge into master yet. It's not likely this is causing
#11511.
-Original Message-
From: David Zafman [mailto:dzaf...@redhat.com]
Sent: Thursday, June 4, 2015 9:46 AM
To: Wang, Zhiqiang; Sage Weil
Cc: ceph-devel@vger.kernel.org
Subject: Re: &#
o:sj...@redhat.com]
Sent: Friday, June 5, 2015 2:20 AM
To: Wang, Zhiqiang
Cc: David Zafman; Sage Weil; ceph-devel@vger.kernel.org
Subject: Re: 'Racing read got wrong version' during proxy write testing
Yeah, I think the right answer is to remove that check and restructure all of
the exis
return;
}
We can't simply remove this check as with the obs not existing case because the
obc is used later...
-Original Message-
From: Wang, Zhiqiang
Sent: Friday, June 5, 2015 8:29 AM
To: Samuel Just
Cc: David Zafman; Sage Weil; ceph-devel@vger.kernel.org
Subject: RE: 'R
I'm also seeing a failure of this in my teuthology run.
-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of David Zafman
Sent: Friday, June 19, 2015 7:13 AM
To: Gregory Farnum; ceph-devel@vger.kernel.org
Subject: rsyslogd
Greg
Hi all,
This is a follow-up of one of the CDS session at
http://tracker.ceph.com/projects/ceph/wiki/Improvement_on_the_cache_tiering_eviction.
We discussed the drawbacks of the current eviction algorithm and several ways
to improve it. Seems like the LRU variants is the right way to go. I come
Hi Nick,
> -Original Message-
> From: Nick Fisk [mailto:n...@fisk.me.uk]
> Sent: Monday, July 20, 2015 5:28 PM
> To: Wang, Zhiqiang; 'Sage Weil'; sj...@redhat.com;
> ceph-devel@vger.kernel.org
> Subject: RE: The design of the eviction improvement
>
>
> -Original Message-
> From: ceph-devel-ow...@vger.kernel.org
> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Tuesday, July 21, 2015 6:38 AM
> To: Wang, Zhiqiang
> Cc: sj...@redhat.com; ceph-devel@vger.kernel.org
> Subject: Re: The d
> -Original Message-
> From: Sage Weil [mailto:sw...@redhat.com]
> Sent: Tuesday, July 21, 2015 9:29 PM
> To: Wang, Zhiqiang
> Cc: sj...@redhat.com; ceph-devel@vger.kernel.org
> Subject: RE: The design of the eviction improvement
>
> On Tue, 21 Jul 20
Hi Matt,
> -Original Message-
> From: Matt W. Benjamin [mailto:m...@cohortfs.com]
> Sent: Tuesday, July 21, 2015 10:16 PM
> To: Wang, Zhiqiang
> Cc: sj...@redhat.com; ceph-devel@vger.kernel.org; Sage Weil
> Subject: Re: The design of the eviction improvement
>
> H
> -Original Message-
> From: Sage Weil [mailto:sw...@redhat.com]
> Sent: Thursday, July 23, 2015 2:51 AM
> To: Allen Samuels
> Cc: Wang, Zhiqiang; sj...@redhat.com; ceph-devel@vger.kernel.org
> Subject: RE: The design of the eviction improvement
>
> On Wed, 22
Hi Allen,
> -Original Message-
> From: Allen Samuels [mailto:allen.samu...@sandisk.com]
> Sent: Thursday, July 23, 2015 2:41 AM
> To: Sage Weil; Wang, Zhiqiang
> Cc: sj...@redhat.com; ceph-devel@vger.kernel.org
> Subject: RE: The design of the eviction improvement
>
Hi all,
I'm experiencing a problem when running ceph with cache tiering. Can someone
help to take a look? Any comments are appreciated, thx!
===Configuration===
I have 2 pools, one is 'rbd', the other is 'cache'. Pool 'cache' is the cache
tier of pool 'rbd'. The 'rbd' pool has 16 HDD OSDs. And
o see.
-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Monday, May 26, 2014 9:18 PM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: Re: Slow request on cache tiering
On 05/26/2014 07:05 AM, Wang, Zhiqiang wrote:
> Hi all,
Hi Zhigiang!
The problem you descri
with the 'slow request' issue,
I suspect they are actually caused by the same problem.
-Original Message-
From: Wang, Zhiqiang
Sent: Tuesday, May 27, 2014 11:06 AM
To: 'Mark Nelson'
Cc: ceph-devel@vger.kernel.org
Subject: RE: Slow request on cache tiering
Thanks Mark! I a
4 IO patterns: 64k sequential read/write and
4k random read/write
Also tried to run 10 VMs, have the same issue
-Original Message-
From: Mark Nelson [mailto:mark.nel...@inktank.com]
Sent: Tuesday, May 27, 2014 7:56 PM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: RE: Slow
them, or just load all the hit sets and check for object
existence before the read? In another word, when min_read_recency_for_promote
is greater than 'hit set interval', we always keep all the hit sets in RAM?
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Mo
n't quite get what you said on the normal hit set rotation part.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Monday, July 21, 2014 11:55 AM
To: Wang, Zhiqiang
Cc: Zhang, Jian; ico...@redhat.com; sam.j...@inktank.com;
ceph-devel@vger.kernel.org
Subject: RE: Cache
t: Monday, July 21, 2014 10:20 PM
To: Wang, Zhiqiang
Cc: Zhang, Jian; ico...@redhat.com; sam.j...@inktank.com;
ceph-devel@vger.kernel.org
Subject: RE: Cache tiering read-proxy mode
On Mon, 21 Jul 2014, Wang, Zhiqiang wrote:
> In the current code, when the evict mode is idle, we just keep the
Hi Sage,
I made this change in
https://github.com/wonzhq/ceph/commit/924e418abb831338e2df7f4a4ec9409b02ee5524
and unit tested it. Could you take a review and give comments? Thanks.
-Original Message-
From: Wang, Zhiqiang
Sent: Tuesday, July 22, 2014 9:38 AM
To: Sage Weil
Cc: Zhang
eph-devel-ow...@vger.kernel.org] On Behalf Of Sage Weil
Sent: Tuesday, July 29, 2014 4:00 AM
To: Wang, Zhiqiang
Cc: Zhang, Jian; 'ico...@redhat.com'; 'sam.j...@inktank.com';
'ceph-devel@vger.kernel.org'
Subject: RE: Cache tiering read-proxy mode
On Mon, 28 Jul 2014, Wan
This fixes a bug when the time of the OSDs and clients are not synchronized
(especially when client is ahead of OSD), and the cache tier dirty ratio
reaches the threshold, the agent skips the flush work because it thinks
the object is too young.
The skipping flush code is as following:
if (obc-
ly 30, 2014 2:55 AM
To: Gregory Farnum
Cc: Sage Weil; Wang, Zhiqiang; ceph-devel@vger.kernel.org
Subject: Re: [PATCH] ceph: use the OSD time as the object mtime instead of the
client time
Off the top of my head... the fscache code (read only) in the CephFS kclient
uses the mtime to see if the it
I've added the testcases and made the style changes. The testcases ran without
problem. Pls merge the pull request https://github.com/ceph/ceph/pull/2177, thx.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Tuesday, July 29, 2014 11:44 PM
To: Wang, Zhiqiang
Cc:
As we discussed before, adding a new field in struct object_info_t to solve the
skipping flush problem. This patch is also available as a pull request at
https://github.com/ceph/ceph/pull/2188
This fixes a bug when the time of the OSDs and clients are not
synchronized (especially when client is
Hi all,
I ran into a problem when restarting an OSD.
Here is my OSD tree before restarting the OSD:
# idweight type name up/down reweight
-6 8 root ssd
-4 4 host zqw-s1-ssd
16 1 osd.16 up 1
17 1
Sent: Thursday, August 21, 2014 11:28 PM
To: Wang, Zhiqiang
Cc: 'ceph-devel@vger.kernel.org'
Subject: Re: A problem when restarting OSD
On Thu, 21 Aug 2014, Wang, Zhiqiang wrote:
> Hi all,
>
> I ran into a problem when restarting an OSD.
>
> Here is my OSD tree before
-
From: David Moreau Simard [mailto:dmsim...@iweb.com]
Sent: Friday, August 22, 2014 10:34 AM
To: Wang, Zhiqiang; Sage Weil
Cc: 'ceph-devel@vger.kernel.org'
Subject: Re: A problem when restarting OSD
I¹m glad you mention this because I¹ve also been running into the same issue
and thi
Hi all,
I've ran into this slow request issue some time ago. The problem is like this:
when running with cache tieing, there are 'slow request' warning messages in
the log file like below.
2014-08-29 10:18:24.669763 7f9b20f1b700 0 log [WRN] : 1 slow requests, 1
included below; oldest blocked
.kernel.org] On Behalf Of Sage Weil
Sent: Saturday, August 30, 2014 10:29 AM
To: Wang, Zhiqiang
Cc: 'ceph-devel@vger.kernel.org'
Subject: Re: Cache tiering slow request issue: currently waiting for rw locks
Hi,
Can you take a look at https://github.com/ceph/ceph/pull/2363 and see if tha
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Wang, Zhiqiang
Sent: Monday, September 1, 2014 9:33 AM
To: Sage Weil
Cc: 'ceph-devel@vger.kernel.org'
Subject: RE: Cache tiering slow request issue: currently waiting for rw locks
I don'
I made some comments based on your comments of the pull request
https://github.com/ceph/ceph/pull/2374. Can you take a look? Thx.
-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Wang, Zhiqiang
Sent: Tuesday, September 2
, but much less than before. Should we go ahead with this
solution?
Sage/Sam, what do you think?
-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Wang, Zhiqiang
Sent: Friday, September 5, 2014 3:21 PM
To: Sage Weil; 'sam.
> -Original Message-
> From: Sage Weil [mailto:sw...@redhat.com]
> Sent: Wednesday, September 10, 2014 12:35 AM
> To: Wang, Zhiqiang
> Cc: 'sam.j...@inktank.com'; 'ceph-devel@vger.kernel.org'
> Subject: RE: Cache tiering slow request issue: currently w
shed to the original pull request
https://github.com/ceph/ceph/pull/2374. Pls take a look, thx.
-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Wednesday, September 10, 2014 12:35 AM
To: Wang, Zhiqiang
Cc: 'sam.j...@inktank.com'; 'ceph-devel@vger.kernel.org&
Hi,
I agree with you this is a bug.
I think we should change the following code:
// KISS: if [lower,upper] spans our target effort, evict it.
if (atime_lower >= agent_state->evict_effort)
return false;
to something like this:
if (100 - atime_upper >= agent_state->evict_ef
Sage,
The pull request is created at: https://github.com/ceph/ceph/pull/2815
-Original Message-
From: Sage Weil [mailto:s...@newdream.net]
Sent: Tuesday, October 28, 2014 12:04 AM
To: Wang, Zhiqiang
Cc: xmdx...@gmail.com; 'ceph-devel@vger.kernel.org'; won...@hotmail.com
S
Good catch :) Looks correct to me.
From: 池信泽 [mailto:xmdx...@gmail.com]
Sent: Tuesday, October 28, 2014 9:24 AM
To: Sage Weil
Cc: Wang, Zhiqiang; ceph-devel@vger.kernel.org; won...@hotmail.com
Subject: Re: Cache pool bug?
I think there is also bug in ReplicatedPG::agent_estimate_atime_temp
I
Yes, if we change atime to be a timestamp, we also need to change the code for
the case when the object is not in any hit sets.
-Original Message-
From: 池信泽 [mailto:xmdx...@gmail.com]
Sent: Tuesday, October 28, 2014 12:46 PM
To: Sage Weil; Wang, Zhiqiang; ceph-devel@vger.kernel.org
Sage,
Thank you for summarize this. I still have some questions and want to clarify
with you.
For the wip-promote-forward, you said that it can't guarantee ordering. Does
this problem exists in the proxy and promote approach? If not, how does the
cache tier osd preserve the ordering? According
Hi all,
I wrote some proxy write code and is doing testing now. I use 'rados put' to
write a full object. I notice that every time when the cache tier OSD sends the
object to the base tier OSD through the Objecter::mutate interface, it retries
3 times. Looks like the 3rd try is a success. I ver
Sage Weil
Sent: Monday, December 8, 2014 10:32 PM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: Re: Three times retries on write
On Mon, 8 Dec 2014, Wang, Zhiqiang wrote:
> Hi all,
>
> I wrote some proxy write code and is doing testing now. I use 'rados put' to
&g
h-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Sage Weil
Sent: Wednesday, December 10, 2014 8:43 AM
To: Cook, Nigel
Cc: Yehuda Sadeh; Josh Durgin; Samuel Just; Wang, Zhiqiang; Jason Dillaman;
ceph-devel
Subject: Re: rados read ordering
On Wed, 10 Dec 2014, C
, 2014 10:26 AM
To: Wang, Zhiqiang
Cc: Sage Weil; Cook, Nigel; Yehuda Sadeh; Josh Durgin; Samuel Just; Jason
Dillaman; ceph-devel
Subject: Re: rados read ordering
On Wed, Dec 10, 2014 at 9:10 AM, Wang, Zhiqiang wrote:
> For multiple clients accessing the same volume, same object, I guess
dnesday, December 10, 2014 12:37 PM
To: Wang, Zhiqiang
Cc: Josh Durgin; Yehuda Sadeh; Samuel Just; Sage Weil; Jason Dillaman;
ceph-devel
Subject: RE: rados read ordering
On thinking of the following use cases and the rbd client..
The ordering scenario is that with client a and b, assuming a and b ar
problem is fixed now.
-Original Message-
From: ceph-devel-ow...@vger.kernel.org
[mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Wang, Zhiqiang
Sent: Tuesday, December 9, 2014 11:38 AM
To: Sage Weil
Cc: ceph-devel@vger.kernel.org
Subject: RE: Three times retries on write
Hi Sage,
I as
Hi Sage,
Can you pls create a new branch based on wip-promote, so that I can push my
proxy write code?
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
ember 11, 2014 1:08 PM
To: Wang, Zhiqiang
Cc: ceph-devel@vger.kernel.org
Subject: Re: wip-proxy-write branch
On Thu, 11 Dec 2014, Wang, Zhiqiang wrote:
> Hi Sage,
>
> Can you pls create a new branch based on wip-promote, so that I can
> push my proxy write code?
Not sure what you mea
64 matches
Mail list logo
