[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Resolution: Fixed Status: Resolved (was: Patch Available) Marking fixed. Thanks for the reviews andrew! And stack and Ted for helping with the addendums. Glad to see this guy finally closed out. Backport work to be done over on HBASE-12702 > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 2.0.0, 1.1.0 > > Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Fix Version/s: (was: 1.1.0) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 2.0.0 > > Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Fix Version/s: (was: 1.0.0) 1.1.0 > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 2.0.0, 1.1.0 > > Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-5162: -- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-5162:
--
Attachment: 5162-addendum2.txt
>From
>https://builds.apache.org/job/HBase-TRUNK/5937/testReport/org.apache.hadoop.hbase.client/TestClientPushback/testClientTracksServerPushback/
> :
{code}
java.lang.AssertionError: expected:<15> but was:<11>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.client.TestClientPushback.testClientTracksServerPushback(TestClientPushback.java:92)
{code}
The attached addendum relaxes the range for memstore load checking.
> Basic client pushback mechanism
> ---
>
> Key: HBASE-5162
> URL: https://issues.apache.org/jira/browse/HBASE-5162
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jesse Yates
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch,
> hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch,
> hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch,
> hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch,
> hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch,
> hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch,
> hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch,
> hbase-5162-trunk-v8.patch, java_HBASE-5162.patch
>
>
> The current blocking we do when we are close to some limits (memstores over
> the multiplier factor, too many store files, global memstore memory) is bad,
> too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
> that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
> - Send some exception to the client, like OverloadedException, that's thrown
> when some situation happens like getting past the low memory barrier. It
> would be thrown when the client gets a handler and does some check while
> putting or deleting. The client would treat this a retryable exception but
> ideally wouldn't check .META. for a new location. It could be fancy and have
> multiple levels of pushback, like send the exception to 25% of the clients,
> and then go up if the situation persists. Should be "easy" to implement but
> we'll be using a lot more IO to send the payload over and over again (but at
> least it wouldn't sit in the RS's memory).
> - Send a message alongside a successful put or delete to tell the client to
> slow down a little, this way we don't have to do back and forth with the
> payload between the client and the server. It's a cleaner (I think) but more
> involved solution.
> In every case the RS should do very obvious things to notify the operators of
> this situation, through logs, web UI, metrics, etc.
> Other ideas?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-5162: -- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-addendum.patch Attaching addedum for trunk. A couple lines of actual fix, push a little clean that made it easier to understand while I was debugging (so necessary if anyone else looks at it later). > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-branch-1-v0.patch Attaching patch for branch-1, [~enis] - thoughts on getting this in? Also, [~apurtell], would you like a patch for 0.98 as well? I think it should be pretty close to 1.0/2.0 version. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch, > hbase-5162-trunk-v12-committed.patch, hbase-5162-trunk-v2.patch, > hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch, > hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch, > hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v12-committed.patch Attaching patch for actual code committed to trunk. Thanks for the copious reviews [~apurtell] and hanging in there. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch, > hbase-5162-trunk-v12-committed.patch, hbase-5162-trunk-v2.patch, > hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch, > hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch, > hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v11.patch Updating to fix annotations. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-5162: -- Fix Version/s: 2.0.0 bq. TestInterfaceAudienceAnnotations Looks like minor issues with annotation and coverage checking needs attention, then I'm +1 for this change. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v2.patch, > hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch, > hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch, > hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v10.patch Updating patch as per Andy's comments and rebased on current master > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v2.patch, > hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch, > hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch, > hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v10.patch, hbase-5162-trunk-v2.patch, > hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch, > hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch, > hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v8.patch Same basic idea as last patch, but StatsTrackingRpcRetryingCaller just uses a simple delegate pattern and RpcRetryingCaller is now an interface implemented by RpcRetryingCallerImpl (to which the stats just delegates). > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v7.patch Updated patch with changes from Andrew's review. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v6.patch Updated patch on latest master (we were getting behind a little bit) and hopefully fixing the checkstyle and findbugs issues. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v5.patch Updating diff to address reviewer's excellent comments (thanks [~apurtell]!). > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v4.patch Attaching a slightly cleaned up version (double checked the checkstyle issues - error count matches master, on my box). Hopefully QA picks this up. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v3.patch Lets go around again. Pretty pretty sure this should be happy on nits. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v2.patch Tests passed locally. Fixed the javadoc/findbugs/checkstyle warnings (I think - there's a lot to parse there :-/). Lets give it another spin. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > hbase-5162-trunk-v2.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Open (was: Patch Available) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v1.patch Updated patch on current master. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch, > java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Fix Version/s: 1.0.0 Assignee: Jesse Yates Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0 > > Attachments: hbase-5162-trunk-v0.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-v0.patch Attaching patch for basic pushback mechanism (disabled by default) on the client. Basically, when the client attempts to make a put, it checks its stats cache to see what the memstore load is for each region it is attempting to talk to. The load is then converted into a wait time (the slowdown part of the patch). If the wait time > 0, we use a special Runnable that sleeps for the specified amount of time before calling the usual runnable. When the region responds to the request it also tacks on the currently load for the region to successful results. Right now its just the memstore load, but could easily be expanded to include other statistics as well (I know [~apurtell] wanted RS CPU for updating CPs). When the client gets that result back it unwraps the result's stats and updates its cache and then hands back the result to go back up the usual call stack. Currently it only supports updating calls to #multi but can be expanded to other operations on the HRegion pretty easily (plumbing is there for on the client side, just needs a couple hooks for the server). > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans > Attachments: hbase-5162-trunk-v0.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5162: - Fix Version/s: (was: 0.95.1) Important issue but won't be done for 0.95 > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans > Attachments: java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5162: - Fix Version/s: (was: 0.94.1) 0.96.0 Is this ready? Moving to 0.96. Feel free to pull back. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans > Fix For: 0.96.0 > > Attachments: java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5162: - Fix Version/s: (was: 0.94.0) 0.94.1 Let's aim for 0.94.1 with this. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans > Fix For: 0.94.1 > > Attachments: java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: java_HBASE-5162.patch Worked up an initial implementation that doesn't actually break the RPC (I think, even if client or the server has monitoring turned on), but avoids doing the exception passing back to the client, which IMHO is the least clean way to handle this as exceptions are for exceptional cases, not the common case. Right now the implementation only covers Puts, but could definitely be extended to cover other writes; I wanted to get some feedback on general style, etc. before going for the full blown implementation (and all the general cleanup associated with a 'real' patch). As far as how it is put together... Client has a backoff policy to take into consideration what the server is saying. This policy can take into account the current, max, and growth size of the write buffer for figuring out how long to sleep. The server pressure is unwrapped from the server in the Result as a MonitoredResult and then updated on the client. The next put will then take into account that pressure when attempting a put. The problem here is that the server can't tell all clients that the pressure has gone down, but that trade-off is common given traditional collision/backoff situations. The client has also been given permission to grow to a multiplier of its writeBufferSize, similar to the memstoremultiplier, allowing it to buffer more writes. If a write is within the expansion range, we want to allow the client to accept more writes while waiting/backing-off, so we launch a flusher thread that after waiting the backoff time will flush the writes (singleton). This gives us back-off as well as some flexiblilty on the client as to how much we buffer. To disable the backoff behavior, its as simple as setting the multiplier to 1, so the expansion = max. Similarly, on the RS, we make the policy for monitoring pluggable and optional to enable the blocking for a given threshold. The policy here can take into account the current number of store files, the number where we will get blocking and the current % full of the memstore (total size and max size). This allows us to ensure that we only slow down based on a given policy (yet to be implemented) but do get protection for the RS from malicious/over-eager clients. This impl may be a little over the top, but it does cover both sides of the on-the-wire story. Only things left to do are to extend this to other writes as well as creating at least one (if not two or three) tunable policies for figuring out blocking. Thoughts? > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans > Fix For: 0.94.0 > > Attachments: java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
