[jira] [Updated] (HBASE-12071) Separate out thread pool for Master <-> RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12071: --- Fix Version/s: 2.0.0 Affects Version/s: 2.0.0 Status: Patch Available (was: In Progress) > Separate out thread pool for Master <-> RegionServer communication > -- > > Key: HBASE-12071 > URL: https://issues.apache.org/jira/browse/HBASE-12071 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-12071.v1-master.patch > > > Over in HBASE-12028, there is a discussion about the case of a RegionServer > still being alive despite all its handler threads being dead. One outcome of > this is that the Master is left hanging on the RS for completion of various > operations - such as region un-assignment when a table is disabled. Does it > make sense to create a separate thread pool for communication between the > Master and the RS? This addresses not just the case of the RPC handler > threads terminating but also long-running queries or co-processor executions > holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12071) Separate out thread pool for Master <-> RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12071: --- Attachment: HBASE-12071.v1-master.patch Comments from the V1 patch against master branch: " Currently we have 3 RpcExecutor in SimpleRpcScheduler for different request types: - priorityExecutor for meta table requests and some admin region operation requests (HConstants.HIGH_QOS) - replicationExecutor for replication requests (HConstants.REPLICATION_QOS) - callExecutor for normal requests (HConstants.REPLAY_QOS and HConstants.NORMAL_QOS) The proposed changes are (1). the comments about only meta table requests uses priorityExecutor is not true - it should be for all system tables - so change the check meta table to check system tables. (2). Split HConstants.HIGH_QOS to two: HConstants.ADMIN_QOS for admin requests and HConstants.SYSTEMTABLE_QOS => in the future, it is flexible that we could extend the code to use different executors for more reliability and scalability. (It is the case in the existing code that HConstants.REPLAY_QOS just use the callExecutor, in the future, it could move to other executor) (3). Add the requests in Admin.proto that involves Master<->RS communication to use priorityExecutor (via HConstants.ADMIN_QOS) (4). Currently, priorityExecutor only uses 1 LinkedBlockingQueue (default queue size=10 & default handler count=10). I changed to use 2 LinkedBlockingQueue. Two alternatives are: (1). increase the handler count to 15-20 to handle more load; or (2). make the number of queues configurable. " > Separate out thread pool for Master <-> RegionServer communication > -- > > Key: HBASE-12071 > URL: https://issues.apache.org/jira/browse/HBASE-12071 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-12071.v1-master.patch > > > Over in HBASE-12028, there is a discussion about the case of a RegionServer > still being alive despite all its handler threads being dead. One outcome of > this is that the Master is left hanging on the RS for completion of various > operations - such as region un-assignment when a table is disabled. Does it > make sense to create a separate thread pool for communication between the > Master and the RS? This addresses not just the case of the RPC handler > threads terminating but also long-running queries or co-processor executions > holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HBASE-12071) Separate out thread pool for Master <-> RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-12071 started by Stephen Yuan Jiang. -- > Separate out thread pool for Master <-> RegionServer communication > -- > > Key: HBASE-12071 > URL: https://issues.apache.org/jira/browse/HBASE-12071 > Project: HBase > Issue Type: Improvement >Reporter: Sudarshan Kadambi >Assignee: Stephen Yuan Jiang > > Over in HBASE-12028, there is a discussion about the case of a RegionServer > still being alive despite all its handler threads being dead. One outcome of > this is that the Master is left hanging on the RS for completion of various > operations - such as region un-assignment when a table is disabled. Does it > make sense to create a separate thread pool for communication between the > Master and the RS? This addresses not just the case of the RPC handler > threads terminating but also long-running queries or co-processor executions > holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249540#comment-14249540 ] Andrew Purtell commented on HBASE-5162: --- I reverted the previous addendum as 6a042c3 and reapplied hbase-5162-trunk-addendum.patch as 99a1139. The build is clean here and TestClientPushback is passing (5 out of 5 trials). > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249541#comment-14249541 ] Dima Spivak commented on HBASE-5162: Well done, Andy. With your latest push, TestClientPushback passes for me. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249538#comment-14249538 ] Andrew Purtell commented on HBASE-5162: --- POM changes were committed that were not part of the addendum patch. I will fix that now. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12590: --- Attachment: HBASE-12590-v4.patch [~j...@cloudera.com] Thank you for your review yesterday! Please take a look at this new patch, I have made some changes based on the review comments. Review board: https://reviews.apache.org/r/28494/diff/# > A solution for data skew in HBase-Mapreduce Job > --- > > Key: HBASE-12590 > URL: https://issues.apache.org/jira/browse/HBASE-12590 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Weichen Ye > Attachments: A Solution for Data Skew in HBase-MapReduce Job > (Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job > (Version3).pdf, HBASE-12590-v3.patch, HBASE-12590-v4.patch, > HBase-12590-v1.patch, HBase-12590-v2.patch > > > 1, Motivation > In production environment, data skew is a very common case. A HBase table may > contains a lot of small regions and several large regions. Small regions > waste a lot of computing resources. If we use a job to scan a table with 3000 > small regions, we need a job with 3000 mappers. Large regions always block > the job. If in a 100-region table, one region is far large then the other 99 > regions. When we run a job with the table as input, 99 mappers will be > completed very quickly, and then we need to wait for the last mapper for a > long time. > 2, Configuration > Add three new configuration > hbase.mapreduce.input.autobalance = true means enabling the “auto balance” in > HBase-MapReduce jobs. The default value is false. > hbase.mapreduce.input.autobalance.maxskewratio= 3 (default is 3). If a region > size is larger than 3x average region size, treat the region as > “proportionately too large”. > hbase.table.row.textkey = true means the row key is text. False means binary > row key. It is used to find the mid row key in large region. The default > value is true. > If (region size >= average size*ratio) : cut the region into two MR input > splits > If (average size <= region size < average size*ratio) : one region as one MR > input split > If (sum of several continuous regions size < average size): combine these > regions into one MR input split. > Example: > In attachment > Welcome to the Review Board. > https://reviews.apache.org/r/28494/diff/# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12231) REST gateway authentication should be independent of Hadoop/HBase security configuration.
[ https://issues.apache.org/jira/browse/HBASE-12231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249533#comment-14249533 ] Jerry He commented on HBASE-12231: -- Hi, [~adityakishore], [~apurtell] Any chance of continuing on this JIRA? We have use case that we expose Rest gateway on the edge node and it is secured by something separate from HBase security. > REST gateway authentication should be independent of Hadoop/HBase security > configuration. > - > > Key: HBASE-12231 > URL: https://issues.apache.org/jira/browse/HBASE-12231 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 0.98.6.1 >Reporter: Aditya Kishore >Assignee: Aditya Kishore > Labels: authentication, security > Attachments: > HBASE-12231-REST-gateway-authentication-should-be-in.patch > > > It is reasonable that a user would want to secure the HBase REST interface > without having to incur the cost of securing the entire HBase/Hadoop > ecosystem. > The current implementation does not allow that and needs to be modified. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249531#comment-14249531 ] Andrew Purtell commented on HBASE-5162: --- The precommit with addendum passed here so what was committed ? > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249526#comment-14249526 ] Hudson commented on HBASE-5162: --- FAILURE: Integrated in HBase-TRUNK #5936 (See [https://builds.apache.org/job/HBase-TRUNK/5936/]) HBASE-5162 Basic client pushback mechanism (Jesse Yates); ADDENDUM (stack: rev f7154e1ac9e6c0bd2dd3c39078a52d49d3485504) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java * hbase-client/pom.xml > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12590: --- Description: 1, Motivation In production environment, data skew is a very common case. A HBase table may contains a lot of small regions and several large regions. Small regions waste a lot of computing resources. If we use a job to scan a table with 3000 small regions, we need a job with 3000 mappers. Large regions always block the job. If in a 100-region table, one region is far large then the other 99 regions. When we run a job with the table as input, 99 mappers will be completed very quickly, and then we need to wait for the last mapper for a long time. 2, Configuration Add three new configuration hbase.mapreduce.input.autobalance = true means enabling the “auto balance” in HBase-MapReduce jobs. The default value is false. hbase.mapreduce.input.autobalance.maxskewratio= 3 (default is 3). If a region size is larger than 3x average region size, treat the region as “proportionately too large”. hbase.table.row.textkey = true means the row key is text. False means binary row key. It is used to find the mid row key in large region. The default value is true. If (region size >= average size*ratio) : cut the region into two MR input splits If (average size <= region size < average size*ratio) : one region as one MR input split If (sum of several continuous regions size < average size): combine these regions into one MR input split. Example: In attachment Welcome to the Review Board. https://reviews.apache.org/r/28494/diff/# was: 1, Motivation In production environment, data skew is a very common case. A HBase table always contains a lot of small regions and several large regions. Small regions waste a lot of computing resources. If we use a job to scan a table with 3000 small regions, we need a job with 3000 mappers. Large regions always block the job. If in a 100-region table, one region is far larger then the other 99 regions. When we run a job with the table as input, 99 mappers will be completed very quickly, and we need to wait for the last mapper for a long time. 2, Configuration Add two new configuration. hbase.mapreduce.split.autobalance = true means enabling the “auto balance” in HBase-MapReduce jobs. The default value is false. hbase.mapreduce.split.targetsize = 1073741824 (default 1GB). The target size of mapreduce splits. If a region size is large than the target size, cut the region into two split.If the sum of several small continuous region size less than the target size, combine these regions into one split. Example: In attachment Welcome to the Review Board. https://reviews.apache.org/r/28494/diff/# > A solution for data skew in HBase-Mapreduce Job > --- > > Key: HBASE-12590 > URL: https://issues.apache.org/jira/browse/HBASE-12590 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Weichen Ye > Attachments: A Solution for Data Skew in HBase-MapReduce Job > (Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job > (Version3).pdf, HBASE-12590-v3.patch, HBase-12590-v1.patch, > HBase-12590-v2.patch > > > 1, Motivation > In production environment, data skew is a very common case. A HBase table may > contains a lot of small regions and several large regions. Small regions > waste a lot of computing resources. If we use a job to scan a table with 3000 > small regions, we need a job with 3000 mappers. Large regions always block > the job. If in a 100-region table, one region is far large then the other 99 > regions. When we run a job with the table as input, 99 mappers will be > completed very quickly, and then we need to wait for the last mapper for a > long time. > 2, Configuration > Add three new configuration > hbase.mapreduce.input.autobalance = true means enabling the “auto balance” in > HBase-MapReduce jobs. The default value is false. > hbase.mapreduce.input.autobalance.maxskewratio= 3 (default is 3). If a region > size is larger than 3x average region size, treat the region as > “proportionately too large”. > hbase.table.row.textkey = true means the row key is text. False means binary > row key. It is used to find the mid row key in large region. The default > value is true. > If (region size >= average size*ratio) : cut the region into two MR input > splits > If (average size <= region size < average size*ratio) : one region as one MR > input split > If (sum of several continuous regions size < average size): combine these > regions into one MR input split. > Example: > In attachment > Welcome to the Review Board. > https://reviews.apache.org/r/28494/diff/# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249523#comment-14249523 ] Lars Hofhansl edited comment on HBASE-5954 at 12/17/14 5:59 AM: [~stack] since you mention it's more mess. Mind having a look? :) I'll get to some actual testing hopefully this week. was (Author: lhofhansl): [~stack] since you mention it's more mess. Mind having a look? I'll get to some actual testing hopefully this week. > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement > Components: HFile, wal >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 2.0.0 > > Attachments: 5954-WIP-trunk.txt, 5954-WIP-v2-trunk.txt, > 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, > 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, > 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > > At least get recommendation into 0.96 doc and some numbers running w/ this > hdfs feature enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249523#comment-14249523 ] Lars Hofhansl commented on HBASE-5954: -- [~stack] since you mention it's more mess. Mind having a look? I'll get to some actual testing hopefully this week. > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement > Components: HFile, wal >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 2.0.0 > > Attachments: 5954-WIP-trunk.txt, 5954-WIP-v2-trunk.txt, > 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, > 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, > 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > > At least get recommendation into 0.96 doc and some numbers running w/ this > hdfs feature enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5954: - Attachment: 5954-WIP-v2-trunk.txt rebased patch... Doesn't compile because of HBASE-5162. Just want to park it. > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement > Components: HFile, wal >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 2.0.0 > > Attachments: 5954-WIP-trunk.txt, 5954-WIP-v2-trunk.txt, > 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, > 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, > 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > > At least get recommendation into 0.96 doc and some numbers running w/ this > hdfs feature enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249517#comment-14249517
]
Hudson commented on HBASE-12645:
FAILURE: Integrated in HBase-1.0 #592 (See
[https://builds.apache.org/job/HBase-1.0/592/])
HBASE-12645 HBaseTestingUtility is using ${$HOME} for rootDir (Varun Saxena)
(stack: rev e82f048d7b17b871f813bf6f4aa7ecda226d1b19)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
> {noformat}
> Either the testing utility has a regression or there's a config regression in
> this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249516#comment-14249516
]
Lars Hofhansl commented on HBASE-5162:
--
Looks like this break the compile now...?
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-client: Compilation failure: Compilation failure:
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[44,32]
package org.apache.hadoop.mapred does not exist
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[45,35]
package org.apache.hadoop.mapreduce does not exist
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[191,67]
cannot find symbol
[ERROR] symbol: class Job
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[212,18]
cannot find symbol
[ERROR] symbol: class Job
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[248,46]
cannot find symbol
[ERROR] symbol: class JobConf
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[269,69]
cannot find symbol
[ERROR] symbol: class JobConf
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[305,66]
cannot find symbol
[ERROR] symbol: class JobConf
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/lars/dev/hbase-master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[325,71]
cannot find symbol
[ERROR] symbol: class Job
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
{noformat}
> Basic client pushback mechanism
> ---
>
> Key: HBASE-5162
> URL: https://issues.apache.org/jira/browse/HBASE-5162
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jesse Yates
> Fix For: 1.0.0, 2.0.0
>
> Attachments: hbase-5162-branch-1-v0.patch,
> hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch,
> hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch,
> hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch,
> hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch,
> hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch,
> hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch,
> hbase-5162-trunk-v8.patch, java_HBASE-5162.patch
>
>
> The current blocking we do when we are close to some limits (memstores over
> the multiplier factor, too many store files, global memstore memory) is bad,
> too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
> that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
> - Send some exception to the client, like OverloadedException, that's thrown
> when some situation happens like getting past the low memory barrier. It
> would be thrown when the client gets a handler and does some check while
> putting or deleting. The client would treat this a retryable exception but
> ideally wouldn't check .META. for a new location. It could be fancy and have
> multiple levels of pushback, like send the exception to 25% of the clients,
> and then go up if the situation persists. Should be "easy" to implement but
> we'll be using a lot more IO to send the payload over and over again (but at
> least it wouldn't sit in the RS's memory).
> - Send a message alongside a successful put or delete to tell the client to
> slow down a little, this way we don't have to do back and forth with the
> payload between the client and the server. It's a cleaner (I think) but more
> involved solution.
> In every case the RS should do very obvious things to notify the operators of
> this situation, through logs, web UI, metrics, etc.
> Other ideas?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249507#comment-14249507 ] Misty Stanley-Jones commented on HBASE-12688: - Live. Looks gorgeous!! > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch, > HBASE-12688.01.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11861) Native MOB Compaction mechanisms.
[ https://issues.apache.org/jira/browse/HBASE-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11861: - Attachment: mob compaction-out-of-region.pdf The high level design for the mob compaction out-of-regions, and the cons and pros are listed in the document as well. > Native MOB Compaction mechanisms. > - > > Key: HBASE-11861 > URL: https://issues.apache.org/jira/browse/HBASE-11861 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh > Attachments: 141030-mob-compaction.pdf, mob > compaction-out-of-region.pdf, mob compaction.pdf > > > Currently, the first cut of mob will have external processes to age off old > mob data (the ttl cleaner), and to compact away deleted or over written data > (the sweep tool). > From an operational point of view, having two external tools, especially one > that relies on MapReduce is undesirable. In this issue we'll tackle > integrating these into hbase without requiring external processes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249496#comment-14249496 ] stack commented on HBASE-12688: --- I'm asking [~misty] to do it Fancy new suit... need to get it up there (smile) > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch, > HBASE-12688.01.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11861) Native MOB Compaction mechanisms.
[ https://issues.apache.org/jira/browse/HBASE-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11861: - Attachment: (was: mob compaction-out-of-region.pdf) > Native MOB Compaction mechanisms. > - > > Key: HBASE-11861 > URL: https://issues.apache.org/jira/browse/HBASE-11861 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh > Attachments: 141030-mob-compaction.pdf, mob compaction.pdf > > > Currently, the first cut of mob will have external processes to age off old > mob data (the ttl cleaner), and to compact away deleted or over written data > (the sweep tool). > From an operational point of view, having two external tools, especially one > that relies on MapReduce is undesirable. In this issue we'll tackle > integrating these into hbase without requiring external processes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12645:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Ok. 0.98 is different in minihbasecluster. Skipping it. If you are up for a
backport, would be cool [~varun_saxena] Thanks for the patch and again if you
want changes, just say.
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
> {noformat}
> Either the testing utility has a regression or there's a config regression in
> this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12688) Update site with a bootstrap-based UI
[
https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249491#comment-14249491
]
Nick Dimiduk commented on HBASE-12688:
--
I ran the script in dev-support but my terminal was flooded with
{noformat}
ls:
/Users/ndimiduk/repos/hbase/target/staging/./.svn/pristine/6d/6d435702f464e668a6768cd9abb3c0b8299d2c83.svn-base:
No such file or directory
ls:
/Users/ndimiduk/repos/hbase/target/staging/./.svn/pristine/6d/6d4691c6ff3e0f0b195bb89a1b0a941148822901.svn-base:
No such file or directory
ls:
/Users/ndimiduk/repos/hbase/target/staging/./.svn/pristine/6d/6d472c26c5b32c1ba47e32a85daf8a8da06080f0.svn-base:
No such file or directory
ls:
/Users/ndimiduk/repos/hbase/target/staging/./.svn/pristine/6d/6d4830298e118447e3694ec6868443be44a6a85c.svn-base:
No such file or directory
ls:
/Users/ndimiduk/repos/hbase/target/staging/./.svn/pristine/6d/6d4a3de6c0906b001738fdfc81c22f30d3e83d48.svn-base:
No such file or directory
{noformat}
Will spin it by hand tomorrow unless you're itching to push it.
> Update site with a bootstrap-based UI
> -
>
> Key: HBASE-12688
> URL: https://issues.apache.org/jira/browse/HBASE-12688
> Project: HBase
> Issue Type: Bug
> Components: site
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Fix For: 2.0.0
>
> Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch,
> HBASE-12688.01.patch
>
>
> Looks like we can "upgrade" our look pretty cheaply by just swapping to a
> different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x
> version (upstream has moved onto 3.x), but it's a start. There's some
> out-of-the-box configuration choices regarding menu bar location. We can also
> look into some of our own custom CSS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249488#comment-14249488
]
stack commented on HBASE-12645:
---
Looks like I committed this to master mistakenly here:
{code}
commit 89006f76f7f8b206c3575c146f76a3509a398190
Author: stack
Date: Tue Dec 16 16:41:27 2014 -0800
HBASE-12645 HBaseTestingUtility is using ${$HOME} for rootDir (Varun Saxena)
{code}
I'm going to leave it in. If there is anything you want me to change
[~varun_saxena], just say and I'll add as an addendum. Sorry about that. Let
me get this on other branches too since it an important fix.
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
> {noformat}
> Either the testing utility has a regression or there's a config regression in
> this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249477#comment-14249477 ] stack commented on HBASE-5162: -- [~jesse_yates] Done sir. BTW, thanks for working on this oldie. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249476#comment-14249476 ] Hudson commented on HBASE-12688: FAILURE: Integrated in HBase-TRUNK #5935 (See [https://builds.apache.org/job/HBase-TRUNK/5935/]) HBASE-12688 Update site with a bootstrap-based UI (ndimiduk: rev 13970bfa5a2cf4ed1f9ff8edfd5160adfba7fb2b) * src/main/site/site.vm * src/main/site/site.xml * pom.xml > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch, > HBASE-12688.01.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249474#comment-14249474 ] stack commented on HBASE-5162: -- I can do it. No worries [~jesse_yates] > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249470#comment-14249470 ] Jesse Yates commented on HBASE-5162: Was just waiting on a +1 from someone else. Im away from my computer till late tonight, but can do it tomorrow am, unless someone wants to grab it first > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249468#comment-14249468 ] Andrew Purtell commented on HBASE-5162: --- Let's commit the addendum > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249462#comment-14249462 ] stack commented on HBASE-5162: -- The test this patch adds has been failing. You lot on it (i'm sure you are!) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249459#comment-14249459
]
stack commented on HBASE-12645:
---
I edited your patch and attached a new version and wondered if you were ok w/
it [~varun_saxena]
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
> {noformat}
> Either the testing utility has a regression or there's a config regression in
> this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249457#comment-14249457 ] stack commented on HBASE-12688: --- You know how to push site or you want me to do it [~ndimiduk]? > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch, > HBASE-12688.01.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12690) list_quotas command is failing with not able to load Java class
[
https://issues.apache.org/jira/browse/HBASE-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-12690:
--
Affects Version/s: (was: 0.99.2)
> list_quotas command is failing with not able to load Java class
> ---
>
> Key: HBASE-12690
> URL: https://issues.apache.org/jira/browse/HBASE-12690
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-12690.patch
>
>
> {noformat}
> hbase(main):004:0> list_quotas
> OWNERQUOTAS
> ERROR: cannot load Java class org.apache.hadoop.hbase.client.ConnectionFactor
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12701) Document how to set the split policy on a given table
Misty Stanley-Jones created HBASE-12701:
---
Summary: Document how to set the split policy on a given table
Key: HBASE-12701
URL: https://issues.apache.org/jira/browse/HBASE-12701
Project: HBase
Issue Type: Task
Components: documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Need to document in the ref guide how to set/change the region split policy for
a single table user the API and the HBase shell as noted below as an example.
Using Java:
HTableDescriptor tableDesc = new HTableDescriptor("test");
tableDesc.setValue(HTableDescriptor.SPLIT_POLICY,
ConstantSizeRegionSplitPolicy.class.getName());
tableDesc.addFamily(new HColumnDescriptor(Bytes.toBytes("cf1")));
admin.createTable(tableDesc);
Using HBase Shell:
create 'test', {METHOD => 'table_att', CONFIG => {'SPLIT_POLICY' =>
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}},
{NAME => 'cf1'}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12121) maven release plugin does not allow for customized goals
[
https://issues.apache.org/jira/browse/HBASE-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249421#comment-14249421
]
Hudson commented on HBASE-12121:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #714 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/714/])
HBASE-12121 maven release plugin does not allow for customized goals (Enoch
Hsu) (stack: rev 5ba06da9fb64b1508e6a9a5cec8da1ce1175a193)
* pom.xml
> maven release plugin does not allow for customized goals
>
>
> Key: HBASE-12121
> URL: https://issues.apache.org/jira/browse/HBASE-12121
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.98.6
>Reporter: Enoch Hsu
>Assignee: Enoch Hsu
>Priority: Minor
> Fix For: 1.0.0, 2.0.0, 0.98.10
>
> Attachments: HBASE-12121.patch
>
>
> Inside the pom under the maven-release-plugin there is a configuration that
> defines what the release-plugin uses like so
>
>
> apache-release
>
> -Dmaven.test.skip.exec
> pom.xml
>
> There is no property for goals so if the user passes in a goal from the
> command line it will not get executed and the default behavior will be used
> instead.
> I propose to add in the following
> ${goals}
> This will allow custom release goal options
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12121) maven release plugin does not allow for customized goals
[
https://issues.apache.org/jira/browse/HBASE-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249393#comment-14249393
]
Hudson commented on HBASE-12121:
SUCCESS: Integrated in HBase-0.98 #748 (See
[https://builds.apache.org/job/HBase-0.98/748/])
HBASE-12121 maven release plugin does not allow for customized goals (Enoch
Hsu) (stack: rev 5ba06da9fb64b1508e6a9a5cec8da1ce1175a193)
* pom.xml
> maven release plugin does not allow for customized goals
>
>
> Key: HBASE-12121
> URL: https://issues.apache.org/jira/browse/HBASE-12121
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.98.6
>Reporter: Enoch Hsu
>Assignee: Enoch Hsu
>Priority: Minor
> Fix For: 1.0.0, 2.0.0, 0.98.10
>
> Attachments: HBASE-12121.patch
>
>
> Inside the pom under the maven-release-plugin there is a configuration that
> defines what the release-plugin uses like so
>
>
> apache-release
>
> -Dmaven.test.skip.exec
> pom.xml
>
> There is no property for goals so if the user passes in a goal from the
> command line it will not get executed and the default behavior will be used
> instead.
> I propose to add in the following
> ${goals}
> This will allow custom release goal options
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-12688: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the opinions everyone. > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch, > HBASE-12688.01.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-12688: - Attachment: HBASE-12688.01.patch Uses reflow, takes [~stack]'s comments into account, uses 'spacelab' theme. Brings back the old 25px logo because I think it looks better. Updated what's here: http://people.apache.org/~ndimiduk/site/ > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch, > HBASE-12688.01.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249362#comment-14249362
]
Varun Saxena commented on HBASE-12645:
--
bq. Something like this Varun Saxena?
Sorry. Couldnt get you. Something like what ?
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
> {noformat}
> Either the testing utility has a regression or there's a config regression in
> this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249348#comment-14249348
]
Hudson commented on HBASE-12645:
FAILURE: Integrated in HBase-TRUNK #5934 (See
[https://builds.apache.org/job/HBase-TRUNK/5934/])
HBASE-12645 HBaseTestingUtility is using ${$HOME} for rootDir (Varun Saxena)
(stack: rev 89006f76f7f8b206c3575c146f76a3509a398190)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
> {noformat}
> Either the testing utility has a regression or there's a config regression in
> this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12121) maven release plugin does not allow for customized goals
[
https://issues.apache.org/jira/browse/HBASE-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249347#comment-14249347
]
Hudson commented on HBASE-12121:
FAILURE: Integrated in HBase-TRUNK #5934 (See
[https://builds.apache.org/job/HBase-TRUNK/5934/])
HBASE-12121 maven release plugin does not allow for customized goals (Enoch
Hsu) (stack: rev 5b5c981d954e0e7769e486039371c4603d2ef915)
* pom.xml
> maven release plugin does not allow for customized goals
>
>
> Key: HBASE-12121
> URL: https://issues.apache.org/jira/browse/HBASE-12121
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.98.6
>Reporter: Enoch Hsu
>Assignee: Enoch Hsu
>Priority: Minor
> Fix For: 1.0.0, 2.0.0, 0.98.10
>
> Attachments: HBASE-12121.patch
>
>
> Inside the pom under the maven-release-plugin there is a configuration that
> defines what the release-plugin uses like so
>
>
> apache-release
>
> -Dmaven.test.skip.exec
> pom.xml
>
> There is no property for goals so if the user passes in a goal from the
> command line it will not get executed and the default behavior will be used
> instead.
> I propose to add in the following
> ${goals}
> This will allow custom release goal options
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11861) Native MOB Compaction mechanisms.
[ https://issues.apache.org/jira/browse/HBASE-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11861: - Attachment: mob compaction-out-of-region.pdf Upload the high-level design for the out-of-region compaction. > Native MOB Compaction mechanisms. > - > > Key: HBASE-11861 > URL: https://issues.apache.org/jira/browse/HBASE-11861 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh > Attachments: 141030-mob-compaction.pdf, mob > compaction-out-of-region.pdf, mob compaction.pdf > > > Currently, the first cut of mob will have external processes to age off old > mob data (the ttl cleaner), and to compact away deleted or over written data > (the sweep tool). > From an operational point of view, having two external tools, especially one > that relies on MapReduce is undesirable. In this issue we'll tackle > integrating these into hbase without requiring external processes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249339#comment-14249339
]
Hadoop QA commented on HBASE-12645:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687624/12645.v7.txt
against master branch at commit d845a92fc8007085999c8beebca7e7b00ca5322e.
ATTACHMENT ID: 12687624
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestClientPushback
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12102//console
This message is automatically generated.
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hado
[jira] [Commented] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249319#comment-14249319 ] stack commented on HBASE-12688: --- Suggest choosing a style that is more plain, 'spacelab', or 'flatly'. I think if you let version and published go left, the gray bar under them will get fatter and fit them better (least I had same problem over on http://htrace.incubator.apache.org/ where I followed your lead here and letting them go left seemed to 'fix' the background thing). We can figure how to get the search back later. Looks great. > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12688) Update site with a bootstrap-based UI
[ https://issues.apache.org/jira/browse/HBASE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249301#comment-14249301 ] Nick Dimiduk commented on HBASE-12688: -- For those interested, here's the results: http://people.apache.org/~ndimiduk/site/ > Update site with a bootstrap-based UI > - > > Key: HBASE-12688 > URL: https://issues.apache.org/jira/browse/HBASE-12688 > Project: HBase > Issue Type: Bug > Components: site >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 2.0.0 > > Attachments: HBASE-12688.00-reflow.patch, HBASE-12688.00.patch > > > Looks like we can "upgrade" our look pretty cheaply by just swapping to a > different skin. This fluido-skin uses Twitter Bootstrap. It's an older 2.x > version (upstream has moved onto 3.x), but it's a start. There's some > out-of-the-box configuration choices regarding menu bar location. We can also > look into some of our own custom CSS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12121) maven release plugin does not allow for customized goals
[
https://issues.apache.org/jira/browse/HBASE-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249284#comment-14249284
]
Hudson commented on HBASE-12121:
FAILURE: Integrated in HBase-1.0 #591 (See
[https://builds.apache.org/job/HBase-1.0/591/])
HBASE-12121 maven release plugin does not allow for customized goals (Enoch
Hsu) (stack: rev 0635705351b896c88f6ac29be06428fe184843c9)
* pom.xml
> maven release plugin does not allow for customized goals
>
>
> Key: HBASE-12121
> URL: https://issues.apache.org/jira/browse/HBASE-12121
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.98.6
>Reporter: Enoch Hsu
>Assignee: Enoch Hsu
>Priority: Minor
> Fix For: 1.0.0, 2.0.0, 0.98.10
>
> Attachments: HBASE-12121.patch
>
>
> Inside the pom under the maven-release-plugin there is a configuration that
> defines what the release-plugin uses like so
>
>
> apache-release
>
> -Dmaven.test.skip.exec
> pom.xml
>
> There is no property for goals so if the user passes in a goal from the
> command line it will not get executed and the default behavior will be used
> instead.
> I propose to add in the following
> ${goals}
> This will allow custom release goal options
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12652) Allow unmanaged connections in MetaTableAccessor
[ https://issues.apache.org/jira/browse/HBASE-12652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249283#comment-14249283 ] Hudson commented on HBASE-12652: FAILURE: Integrated in HBase-1.0 #591 (See [https://builds.apache.org/job/HBase-1.0/591/]) Revert "HBASE-12652 Allow unmanaged connections in MetaTableAccessor (Solomon Duskis)" (stack: rev 4587a693b0e61531821e716a4e2292b163c59770) * hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaTableAccessor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWithTags.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java * hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java > Allow unmanaged connections in MetaTableAccessor > > > Key: HBASE-12652 > URL: https://issues.apache.org/jira/browse/HBASE-12652 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Solomon Duskis >Assignee: Solomon Duskis > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12652.patch, HBASE-12652.patch, > HBASE-12652B.patch, HBASE-12652C-0.99.patch, HBASE-12652C.patch > > > Passing in an unmanaged connnection to MetaTableAccessor causes an exception. > MetaTableAccessor should be able to use both unmanaged and managed > connections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11412) Minimize a number of hbase-client transitive dependencies
[
https://issues.apache.org/jira/browse/HBASE-11412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249286#comment-14249286
]
Hadoop QA commented on HBASE-11412:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12682868/hbase11412.txt
against master branch at commit 5b5c981d954e0e7769e486039371c4603d2ef915.
ATTACHMENT ID: 12682868
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[44,32]
package org.apache.hadoop.mapred does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[45,35]
package org.apache.hadoop.mapreduce does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[191,67]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[212,18]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[248,46]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[269,69]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[305,66]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[325,71]
cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-client: Compilation failure: Compilation failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[44,32]
package org.apache.hadoop.mapred does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[45,35]
package org.apache.hadoop.mapreduce does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[191,67]
cannot find symbol
[ERROR] symbol: class Job
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[212,18]
cannot find symbol
[ERROR] symbol: class Job
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[248,46]
cannot find symbol
[ERROR] symbol: class JobConf
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[269,69]
cannot find symbol
[ERROR] symbol: class JobConf
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[305,66]
cannot find symbol
[ERROR] symbol: class JobConf
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java:[325,71]
cannot find symbol
[ERROR] symbol: class Job
[ERROR] location: class org.apache.hadoop.hbase.security.token.TokenUtil
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249275#comment-14249275 ] Hudson commented on HBASE-10201: FAILURE: Integrated in HBase-TRUNK #5933 (See [https://builds.apache.org/job/HBase-TRUNK/5933/]) HBASE-10201 Addendum changes TestPerColumnFamilyFlush to LargeTest (tedyu: rev 885b065683499540f467cb54086a3f60e64b9c8a) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPerColumnFamilyFlush.java > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo > Fix For: 2.0.0 > > Attachments: 10201-addendum.txt, 3149-trunk-v1.txt, > HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, > HBASE-10201-0.99.patch, HBASE-10201.patch, HBASE-10201_1.patch, > HBASE-10201_10.patch, HBASE-10201_11.patch, HBASE-10201_12.patch, > HBASE-10201_13.patch, HBASE-10201_13.patch, HBASE-10201_14.patch, > HBASE-10201_15.patch, HBASE-10201_16.patch, HBASE-10201_17.patch, > HBASE-10201_18.patch, HBASE-10201_19.patch, HBASE-10201_2.patch, > HBASE-10201_3.patch, HBASE-10201_4.patch, HBASE-10201_5.patch, > HBASE-10201_6.patch, HBASE-10201_7.patch, HBASE-10201_8.patch, > HBASE-10201_9.patch, compactions.png, count.png, io.png, memstore.png > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11412) Minimize a number of hbase-client transitive dependencies
[
https://issues.apache.org/jira/browse/HBASE-11412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-11412:
--
Status: Patch Available (was: Open)
Just ran into this over in htrace. Our list of transitive includes is a little
on the obscene side.
> Minimize a number of hbase-client transitive dependencies
> -
>
> Key: HBASE-11412
> URL: https://issues.apache.org/jira/browse/HBASE-11412
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 0.98.3
>Reporter: Sergey Beryozkin
>Priority: Minor
> Fix For: 1.0.0, 2.0.0
>
> Attachments: hbase11412.txt
>
>
> hbase-client has a number of transitive dependencies not needed for a client
> mode execution. In my test I've added the following exclusions:
> {code:xml}
>
>
> com.sun.jersey
> jersey-server
>
>
> com.sun.jersey
> jersey-core
>
>
> com.sun.jersey
> jersey-json
>
>
> com.sun.jersey.contribs
> jersey-guice
>
>
> com.google.inject
> guice
>
>
> com.google.inject.extensions
> guice-servlet
>
>
> org.mortbay.jetty
> jetty
>
>
> org.mortbay.jetty
> jetty-util
>
>
> commons-httpclient
> commons-httpclient
>
>
> {code}
> Proposal: add related exclusions to some of the dependencies hbase-client
> depends upon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12682) compaction thread throttle value is not correct in hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249274#comment-14249274
]
Hudson commented on HBASE-12682:
FAILURE: Integrated in HBase-TRUNK #5933 (See
[https://builds.apache.org/job/HBase-TRUNK/5933/])
HBASE-12682 compaction thread throttle value is not correct in
hbase-default.xml (Jerry He) (stack: rev
d845a92fc8007085999c8beebca7e7b00ca5322e)
* hbase-common/src/main/resources/hbase-default.xml
> compaction thread throttle value is not correct in hbase-default.xml
>
>
> Key: HBASE-12682
> URL: https://issues.apache.org/jira/browse/HBASE-12682
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-12682-master.patch
>
>
> While doing some compaction tuning and looking up some of the parameters, I
> noticed that hbase.regionserver.thread.compaction.throttle default value in
> the documentation and in hbase-site.xml seems to be off the mark.
> {code}
>
> hbase.regionserver.thread.compaction.throttle
> 2560
> There are two different thread pools for compactions, one
> for large compactions and
> the other for small compactions. This helps to keep compaction of lean
> tables (such as
> hbase:meta) fast. If a compaction is larger
> than this threshold, it
> goes into the large compaction pool. In most cases, the default value
> is appropriate. Default:
> 2 x hbase.hstore.compaction.max x hbase.hregion.memstore.flush.size
> (which defaults to 128).
> The value field assumes that the value of
> hbase.hregion.memstore.flush.size is unchanged from
> the default.
>
> {code}
> It should be in bytes. Or am I missing anything?
> It is not a problem in 0.98 since the property is not in hbase-site.xml over
> there. It is obtained dynamically:
> {code}
> throttlePoint = conf.getLong("hbase.regionserver.thread.compaction.throttle",
> 2 * maxFilesToCompact * storeConfigInfo.getMemstoreFlushSize());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12121) maven release plugin does not allow for customized goals
[
https://issues.apache.org/jira/browse/HBASE-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12121:
--
Resolution: Fixed
Fix Version/s: 0.98.10
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to 0.98+
> maven release plugin does not allow for customized goals
>
>
> Key: HBASE-12121
> URL: https://issues.apache.org/jira/browse/HBASE-12121
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.98.6
>Reporter: Enoch Hsu
>Assignee: Enoch Hsu
>Priority: Minor
> Fix For: 1.0.0, 2.0.0, 0.98.10
>
> Attachments: HBASE-12121.patch
>
>
> Inside the pom under the maven-release-plugin there is a configuration that
> defines what the release-plugin uses like so
>
>
> apache-release
>
> -Dmaven.test.skip.exec
> pom.xml
>
> There is no property for goals so if the user passes in a goal from the
> command line it will not get executed and the default behavior will be used
> instead.
> I propose to add in the following
> ${goals}
> This will allow custom release goal options
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249258#comment-14249258 ] Ted Yu commented on HBASE-5162: --- +1 > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249254#comment-14249254
]
Hadoop QA commented on HBASE-5162:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687610/hbase-5162-trunk-addendum.patch
against master branch at commit 9e7f7211b95ef4e9c64f5e54411c7c9fa7eeb235.
ATTACHMENT ID: 12687610
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12101//console
This message is automatically generated.
> Basic client pushback mechanism
> ---
>
> Key: HBASE-5162
> URL: https://issues.apache.org/jira/browse/HBASE-5162
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.92.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jesse Yates
> Fix For: 1.0.0, 2.0.0
>
> Attachments: hbase-5162-branch-1-v0.patch,
> hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch,
> hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch,
> hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch,
> hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch,
> hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch,
> hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch,
> hbase-5162-trunk-v8.patch, java_HBASE-5162.patch
>
>
> The current blocking we do when we are close to some limits (memstores over
> the multiplier factor, too many store files, global memstore memory) is bad,
> too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
> that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
> - Send some exception to the client, like OverloadedException, that's thrown
> when some situation happens like getting past the low memory barrier. It
> would be thrown when the client gets a handler
[jira] [Commented] (HBASE-12641) Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster
[ https://issues.apache.org/jira/browse/HBASE-12641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249240#comment-14249240 ] stack commented on HBASE-12641: --- [~apurtell] You good w/ the above? > Grant all permissions of hbase zookeeper node to hbase superuser in a secure > cluster > > > Key: HBASE-12641 > URL: https://issues.apache.org/jira/browse/HBASE-12641 > Project: HBase > Issue Type: Improvement > Components: Zookeeper >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: 1.0.0 > > Attachments: HBASE-12641-v1.diff > > > Currently in a secure cluster, only the master/regionserver kerberos user can > manage the znode of hbase. But he master/regionserver kerberos user is for > rpc connection and we usually use another super user to manage the cluster. > In some special scenarios, we need to manage the data of znode with the > supper user. > eg: > a, To get the data of the znode for debugging. > b, HBASE-8253: We need to delete the znode for the corrupted hlog to avoid it > block the replication. > So we grant all permissions of hbase zookeeper node to hbase superuser during > creating these znodes. > Suggestions are welcomed. > [~apurtell] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12664) TestDefaultLoadBalancer.testBalanceCluster fails in CachedDNSToSwitchMapping
[
https://issues.apache.org/jira/browse/HBASE-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-12664.
---
Resolution: Fixed
Resolving again since we seem to be past this disaster commit. Thanks for the
bail out [~apurtell] and [~ndimiduk]
> TestDefaultLoadBalancer.testBalanceCluster fails in CachedDNSToSwitchMapping
>
>
> Key: HBASE-12664
> URL: https://issues.apache.org/jira/browse/HBASE-12664
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12664.txt, HBASE-12664-0.99-addendum.patch
>
>
> https://builds.apache.org/job/HBase-TRUNK/5893/testReport/org.apache.hadoop.hbase.master.balancer/TestDefaultLoadBalancer/testBalanceCluster/
> {code}
> java.lang.Exception: test timed out after 6 milliseconds
> at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
> at
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
> at java.net.InetAddress.getAllByName(InetAddress.java:1155)
> at java.net.InetAddress.getAllByName(InetAddress.java:1091)
> at java.net.InetAddress.getByName(InetAddress.java:1041)
> at org.apache.hadoop.net.NetUtils.normalizeHostName(NetUtils.java:561)
> at org.apache.hadoop.net.NetUtils.normalizeHostNames(NetUtils.java:578)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:109)
> at
> org.apache.hadoop.hbase.master.RackManager.getRack(RackManager.java:66)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:274)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:152)
> at
> org.apache.hadoop.hbase.master.balancer.SimpleLoadBalancer.balanceCluster(SimpleLoadBalancer.java:201)
> at
> org.apache.hadoop.hbase.master.balancer.TestDefaultLoadBalancer.testBalanceCluster(TestDefaultLoadBalancer.java:119)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249234#comment-14249234 ] Hudson commented on HBASE-12699: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #713 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/713/]) HBASE-12699 undefined method 'setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true (Stephen Jiang) (apurtell: rev 71c0e5b9de244e0bb96ca4c5b010f981acbda1ee) * hbase-shell/src/main/ruby/hbase/admin.rb > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch, HBASE-12699.v1addendum.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249230#comment-14249230 ] Hudson commented on HBASE-12699: FAILURE: Integrated in HBase-1.0 #590 (See [https://builds.apache.org/job/HBase-1.0/590/]) HBASE-12699 Addendum modifies shell help (Stephen Jiang) (tedyu: rev bb1b2f4eebc683a2a7aa80b341664ba8129f5080) * hbase-shell/src/main/ruby/shell/commands/alter.rb > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch, HBASE-12699.v1addendum.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12645:
--
Attachment: 12645.v7.txt
Something like this [~varun_saxena]?
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: 12645.v7.txt, HBASE-12645.002.patch,
> HBASE-12645.003.patch, HBASE-12645.004.patch, HBASE-12645.004.patch,
> HBASE-12645.005.patch, HBASE-12645.006.patch, HBASE-12645.006.patch,
> HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
> at
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits.setupBeforeClass(TestReadOldRootAndMetaEdits.java:70)
> {noformat}
> Either the testing utility has a regression or there's a config regression in
> this test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249215#comment-14249215 ] Hudson commented on HBASE-12699: FAILURE: Integrated in HBase-TRUNK #5932 (See [https://builds.apache.org/job/HBase-TRUNK/5932/]) HBASE-12699 Addendum modifies shell help (Stephen Jiang) (tedyu: rev 889675fb9c83e85edebc31f309c8e03dccfff5ab) * 12699.v1addendum.patch HBASE-12699 Addendum modifies shell help (Stephen Jiang) (tedyu: rev 9e7f7211b95ef4e9c64f5e54411c7c9fa7eeb235) * hbase-shell/src/main/ruby/shell/commands/alter.rb * 12699.v1addendum.patch > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch, HBASE-12699.v1addendum.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12682) compaction thread throttle value is not correct in hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12682:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks for the patch [~jerryhe]
> compaction thread throttle value is not correct in hbase-default.xml
>
>
> Key: HBASE-12682
> URL: https://issues.apache.org/jira/browse/HBASE-12682
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-12682-master.patch
>
>
> While doing some compaction tuning and looking up some of the parameters, I
> noticed that hbase.regionserver.thread.compaction.throttle default value in
> the documentation and in hbase-site.xml seems to be off the mark.
> {code}
>
> hbase.regionserver.thread.compaction.throttle
> 2560
> There are two different thread pools for compactions, one
> for large compactions and
> the other for small compactions. This helps to keep compaction of lean
> tables (such as
> hbase:meta) fast. If a compaction is larger
> than this threshold, it
> goes into the large compaction pool. In most cases, the default value
> is appropriate. Default:
> 2 x hbase.hstore.compaction.max x hbase.hregion.memstore.flush.size
> (which defaults to 128).
> The value field assumes that the value of
> hbase.hregion.memstore.flush.size is unchanged from
> the default.
>
> {code}
> It should be in bytes. Or am I missing anything?
> It is not a problem in 0.98 since the property is not in hbase-site.xml over
> there. It is obtained dynamically:
> {code}
> throttlePoint = conf.getLong("hbase.regionserver.thread.compaction.throttle",
> 2 * maxFilesToCompact * storeConfigInfo.getMemstoreFlushSize());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249199#comment-14249199
]
zhangduo commented on HBASE-10201:
--
{quote}
I forgot to say thank you zhangduo for your persistence on getting this in.
{quote}
It's my pleasure to contribute code to a famous project:)
Thanks
> Port 'Make flush decisions per column family' to trunk
> --
>
> Key: HBASE-10201
> URL: https://issues.apache.org/jira/browse/HBASE-10201
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Ted Yu
>Assignee: zhangduo
> Fix For: 2.0.0
>
> Attachments: 10201-addendum.txt, 3149-trunk-v1.txt,
> HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch,
> HBASE-10201-0.99.patch, HBASE-10201.patch, HBASE-10201_1.patch,
> HBASE-10201_10.patch, HBASE-10201_11.patch, HBASE-10201_12.patch,
> HBASE-10201_13.patch, HBASE-10201_13.patch, HBASE-10201_14.patch,
> HBASE-10201_15.patch, HBASE-10201_16.patch, HBASE-10201_17.patch,
> HBASE-10201_18.patch, HBASE-10201_19.patch, HBASE-10201_2.patch,
> HBASE-10201_3.patch, HBASE-10201_4.patch, HBASE-10201_5.patch,
> HBASE-10201_6.patch, HBASE-10201_7.patch, HBASE-10201_8.patch,
> HBASE-10201_9.patch, compactions.png, count.png, io.png, memstore.png
>
>
> Currently the flush decision is made using the aggregate size of all column
> families. When large and small column families co-exist, this causes many
> small flushes of the smaller CF. We need to make per-CF flush decisions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249191#comment-14249191 ] Ted Yu commented on HBASE-10201: Addendum pushed to master branch. > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo > Fix For: 2.0.0 > > Attachments: 10201-addendum.txt, 3149-trunk-v1.txt, > HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, > HBASE-10201-0.99.patch, HBASE-10201.patch, HBASE-10201_1.patch, > HBASE-10201_10.patch, HBASE-10201_11.patch, HBASE-10201_12.patch, > HBASE-10201_13.patch, HBASE-10201_13.patch, HBASE-10201_14.patch, > HBASE-10201_15.patch, HBASE-10201_16.patch, HBASE-10201_17.patch, > HBASE-10201_18.patch, HBASE-10201_19.patch, HBASE-10201_2.patch, > HBASE-10201_3.patch, HBASE-10201_4.patch, HBASE-10201_5.patch, > HBASE-10201_6.patch, HBASE-10201_7.patch, HBASE-10201_8.patch, > HBASE-10201_9.patch, compactions.png, count.png, io.png, memstore.png > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249188#comment-14249188 ] stack commented on HBASE-10201: --- That'll work. Thanks. > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo > Fix For: 2.0.0 > > Attachments: 10201-addendum.txt, 3149-trunk-v1.txt, > HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, > HBASE-10201-0.99.patch, HBASE-10201.patch, HBASE-10201_1.patch, > HBASE-10201_10.patch, HBASE-10201_11.patch, HBASE-10201_12.patch, > HBASE-10201_13.patch, HBASE-10201_13.patch, HBASE-10201_14.patch, > HBASE-10201_15.patch, HBASE-10201_16.patch, HBASE-10201_17.patch, > HBASE-10201_18.patch, HBASE-10201_19.patch, HBASE-10201_2.patch, > HBASE-10201_3.patch, HBASE-10201_4.patch, HBASE-10201_5.patch, > HBASE-10201_6.patch, HBASE-10201_7.patch, HBASE-10201_8.patch, > HBASE-10201_9.patch, compactions.png, count.png, io.png, memstore.png > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10278) Provide better write predictability
[ https://issues.apache.org/jira/browse/HBASE-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249181#comment-14249181 ] stack commented on HBASE-10278: --- [~busbey] Could do the 0.89fb tack first, before this. How would you implement this in new regime [~busbey]? It changes FSHLog. You'd do a derivative or decorated FSHLog, SwitchingFSHLog? Can get rid of stuff like the 'enabled' flag and checks. Looking at last patch, pity we couldn't switch to the other writer when rolling log on current writer (given rolling takes a while). Looks like this was a consideration: "2241 * NOTE: Don't switch if there is a ongoing log roll. Most likely, this could be a redundant 2242 * step." Patch is worth a study. Pity has to be a syncmonitor but not sure how else you'd do it. On keeping around edits, one implementation, rather than append the WAL directly as we do now, instead, kept the edits in a single list and then did bulk appends (IIRC, no advantage doing bulk append over single appends). Edits stayed in List until syncs came back to say it was ok let them go. IIRC, it was not that much slower. It was a little more involved (was easier just doing the WAL append immediately since then we were done) but it might be worth considering having a single list of all outstanding edits on other side of the ring buffer as store for edits in flight (downside would be extra thread coordination) > Provide better write predictability > --- > > Key: HBASE-10278 > URL: https://issues.apache.org/jira/browse/HBASE-10278 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: 10278-trunk-v2.1.patch, 10278-trunk-v2.1.patch, > 10278-wip-1.1.patch, Multiwaldesigndoc.pdf, SwitchWriterFlow.pptx > > > Currently, HBase has one WAL per region server. > Whenever there is any latency in the write pipeline (due to whatever reasons > such as n/w blip, a node in the pipeline having a bad disk, etc), the overall > write latency suffers. > Jonathan Hsieh and I analyzed various approaches to tackle this issue. We > also looked at HBASE-5699, which talks about adding concurrent multi WALs. > Along with performance numbers, we also focussed on design simplicity, > minimum impact on MTTR & Replication, and compatibility with 0.96 and 0.98. > Considering all these parameters, we propose a new HLog implementation with > WAL Switching functionality. > Please find attached the design doc for the same. It introduces the WAL > Switching feature, and experiments/results of a prototype implementation, > showing the benefits of this feature. > The second goal of this work is to serve as a building block for concurrent > multiple WALs feature. > Please review the doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10201:
---
Attachment: 10201-addendum.txt
>From
>https://builds.apache.org/job/HBase-TRUNK/5931/testReport/org.apache.hadoop.hbase.regionserver/TestPerColumnFamilyFlush/testLogReplayWithDistributedReplay/
> :
{code}
java.lang.IllegalStateException: A mini-cluster is already running
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:865)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:799)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:770)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:757)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplay(TestPerColumnFamilyFlush.java:349)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplayWithDistributedReplay(TestPerColumnFamilyFlush.java:430)
{code}
Attached addendum changes TestPerColumnFamilyFlush to large test such that
there would be no collision as shown above in Jenkins build.
TestPerColumnFamilyFlush passed with the addendum.
> Port 'Make flush decisions per column family' to trunk
> --
>
> Key: HBASE-10201
> URL: https://issues.apache.org/jira/browse/HBASE-10201
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Ted Yu
>Assignee: zhangduo
> Fix For: 2.0.0
>
> Attachments: 10201-addendum.txt, 3149-trunk-v1.txt,
> HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch,
> HBASE-10201-0.99.patch, HBASE-10201.patch, HBASE-10201_1.patch,
> HBASE-10201_10.patch, HBASE-10201_11.patch, HBASE-10201_12.patch,
> HBASE-10201_13.patch, HBASE-10201_13.patch, HBASE-10201_14.patch,
> HBASE-10201_15.patch, HBASE-10201_16.patch, HBASE-10201_17.patch,
> HBASE-10201_18.patch, HBASE-10201_19.patch, HBASE-10201_2.patch,
> HBASE-10201_3.patch, HBASE-10201_4.patch, HBASE-10201_5.patch,
> HBASE-10201_6.patch, HBASE-10201_7.patch, HBASE-10201_8.patch,
> HBASE-10201_9.patch, compactions.png, count.png, io.png, memstore.png
>
>
> Currently the flush decision is made using the aggregate size of all column
> families. When large and small column families co-exist, this causes many
> small flushes of the smaller CF. We need to make per-CF flush decisions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12690) list_quotas command is failing with not able to load Java class
[
https://issues.apache.org/jira/browse/HBASE-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249155#comment-14249155
]
Hudson commented on HBASE-12690:
FAILURE: Integrated in HBase-TRUNK #5931 (See
[https://builds.apache.org/job/HBase-TRUNK/5931/])
HBASE-12690 list_quotas command is failing with not able to load Java class
(Ashish) (tedyu: rev 92bc36b762a1e3f8976262169ffdaafcac60a7e7)
* hbase-shell/src/main/ruby/hbase/quotas.rb
> list_quotas command is failing with not able to load Java class
> ---
>
> Key: HBASE-12690
> URL: https://issues.apache.org/jira/browse/HBASE-12690
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0, 0.99.2
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-12690.patch
>
>
> {noformat}
> hbase(main):004:0> list_quotas
> OWNERQUOTAS
> ERROR: cannot load Java class org.apache.hadoop.hbase.client.ConnectionFactor
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Status: Patch Available (was: Open) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249145#comment-14249145 ] Hudson commented on HBASE-12699: FAILURE: Integrated in HBase-0.98 #747 (See [https://builds.apache.org/job/HBase-0.98/747/]) HBASE-12699 undefined method 'setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true (Stephen Jiang) (apurtell: rev 71c0e5b9de244e0bb96ca4c5b010f981acbda1ee) * hbase-shell/src/main/ruby/hbase/admin.rb > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch, HBASE-12699.v1addendum.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-trunk-addendum.patch Attaching addedum for trunk. A couple lines of actual fix, push a little clean that made it easier to understand while I was debugging (so necessary if anyone else looks at it later). > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, > hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249118#comment-14249118 ] Ted Yu commented on HBASE-12699: Addendum integrated to branch-1 and master. > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch, HBASE-12699.v1addendum.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249104#comment-14249104 ] Stephen Yuan Jiang commented on HBASE-12699: add an addendum to fix the help comments. > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch, HBASE-12699.v1addendum.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12699: --- Attachment: HBASE-12699.v1addendum.patch > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch, HBASE-12699.v1addendum.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[ https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vandana Ayyalasomayajula updated HBASE-12694: - Attachment: HBASE-12694-branch-1.patch Patch for branch-1. > testTableExistsIfTheSpecifiedTableRegionIsSplitParent in > TestSplitTransactionOnCluster class leaves regions in transition > - > > Key: HBASE-12694 > URL: https://issues.apache.org/jira/browse/HBASE-12694 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0, 0.98.9 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > Attachments: HBASE-12694-branch-1.patch > > > There seems to a clean up issue with the unit test > "testTableExistsIfTheSpecifiedTableRegionIsSplitParent" in > TestSplitTransactionOnCluster class. It always has a region in transition. So > any unit test that executes after that test, will cause balance method to > return false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[ https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vandana Ayyalasomayajula reassigned HBASE-12694: Assignee: Vandana Ayyalasomayajula > testTableExistsIfTheSpecifiedTableRegionIsSplitParent in > TestSplitTransactionOnCluster class leaves regions in transition > - > > Key: HBASE-12694 > URL: https://issues.apache.org/jira/browse/HBASE-12694 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0, 0.98.9 >Reporter: Vandana Ayyalasomayajula >Assignee: Vandana Ayyalasomayajula >Priority: Minor > > There seems to a clean up issue with the unit test > "testTableExistsIfTheSpecifiedTableRegionIsSplitParent" in > TestSplitTransactionOnCluster class. It always has a region in transition. So > any unit test that executes after that test, will cause balance method to > return false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249086#comment-14249086 ] Andrew Purtell commented on HBASE-11290: The new LockCache in the v2 patch is reminiscent of HFile's IdLock. Can this be refactored so those places share common code? (There's RowLock in HRegion also, but that would be significantly more work, maybe a future time.) > Unlock RegionStates > --- > > Key: HBASE-11290 > URL: https://issues.apache.org/jira/browse/HBASE-11290 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Virag Kothari > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, > HBASE-11290.draft.patch > > > Even though RegionStates is a highly accessed data structure in HMaster. Most > of it's methods are synchronized. Which limits concurrency. Even simply > making some of the getters non-synchronized by using concurrent data > structures has helped with region assignments. We can go as simple as this > approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249079#comment-14249079 ] Andrew Purtell commented on HBASE-5162: --- Either option sounds good IMHO :-) > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249066#comment-14249066 ] Nick Dimiduk commented on HBASE-5162: - bq. What if we open a branch (soon) for 1.1. Why wait for 1.1, any reason this couldn't go into 1.0.1? I mean, if it's okay for 0.98.x, why not 1.0.x? > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12690) list_quotas command is failing with not able to load Java class
[
https://issues.apache.org/jira/browse/HBASE-12690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12690:
---
Resolution: Fixed
Fix Version/s: (was: 1.0.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Integrated to master branch.
Thanks for the patch, Ashish.
> list_quotas command is failing with not able to load Java class
> ---
>
> Key: HBASE-12690
> URL: https://issues.apache.org/jira/browse/HBASE-12690
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0, 0.99.2
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-12690.patch
>
>
> {noformat}
> hbase(main):004:0> list_quotas
> OWNERQUOTAS
> ERROR: cannot load Java class org.apache.hadoop.hbase.client.ConnectionFactor
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248998#comment-14248998 ] Sean Busbey commented on HBASE-5954: Oh certainly. I expect there's going to be a follow on ticket to add docs to the ref guide explaining the basic trade off and I want to make sure the code bits are in place so that whoever has to write that section can focus on tests rather than code gaps. > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement > Components: HFile, wal >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 2.0.0 > > Attachments: 5954-WIP-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > > At least get recommendation into 0.96 doc and some numbers running w/ this > hdfs feature enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248994#comment-14248994 ] Lars Hofhansl commented on HBASE-5954: -- That's a good idea. Will do. (In the end... We know there will be a non-trivial impact on rotating disks, we're doing this for to guard against power outages). > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement > Components: HFile, wal >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 2.0.0 > > Attachments: 5954-WIP-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > > At least get recommendation into 0.96 doc and some numbers running w/ this > hdfs feature enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248968#comment-14248968 ] Andrew Purtell commented on HBASE-5162: --- Thanks. If it's a minor fix please commit an addendum. If it's more involved, we can revert and commit again later. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248961#comment-14248961 ] Jesse Yates commented on HBASE-5162: Hmmm, well that's an issue. Looking into the NPE now. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12699: --- Fix Version/s: 0.98.10 Thanks. Cherry picked back and pushed. TestShell passes and I manually tested altering a table using both DEFERRED_LOG flush and DURABILITY (separately). > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0, 0.98.10 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248946#comment-14248946 ] Hudson commented on HBASE-5162: --- FAILURE: Integrated in HBase-TRUNK #5930 (See [https://builds.apache.org/job/HBase-TRUNK/5930/]) HBASE-5162 Basic client pushback mechanism (jyates: rev a411227b0ebf78b4ee8ae7179e162b54734e77de) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientExponentialBackoff.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelayingRunner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java * hbase-protocol/src/main/protobuf/Client.proto * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicyFactory.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerStatisticTracker.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultStatsUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestFastFailWithoutTestUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicasClient.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ServerStatistics.java * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exceptio
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248944#comment-14248944 ] Hudson commented on HBASE-8450: --- FAILURE: Integrated in HBase-TRUNK #5930 (See [https://builds.apache.org/job/HBase-TRUNK/5930/]) HBASE-9431 Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims (stack: rev e5d813c46b41ab4fb48d72731eb34422f260b81a) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java > Update hbase-default.xml and general recommendations to better suit current > hw, h2, experience, etc. > > > Key: HBASE-8450 > URL: https://issues.apache.org/jira/browse/HBASE-8450 > Project: HBase > Issue Type: Task > Components: Usability >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.95.1 > > Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt, 8450v5.txt > > > This is a critical task we need to do before we release; review our defaults. > On cursory review, there are configs in hbase-default.xml that no longer have > matching code; there are some that should be changed because we know better > now and others that should be amended because hardware and deploys are bigger > than they used to be. > We could also move stuff around so that the must-edit stuff is near the top > (zk quorum config. is mid-way down the page) and beef up the descriptions -- > especially since these descriptions shine through in the refguide. > Lastly, I notice that our tgz does not "include" an hbase-default.xml other > than the one bundled up in the jar. Maybe we should make it more accessible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248943#comment-14248943 ] Hudson commented on HBASE-10201: FAILURE: Integrated in HBase-TRUNK #5930 (See [https://builds.apache.org/job/HBase-TRUNK/5930/]) HBASE-10201 Port 'Make flush decisions per column family' to trunk (stack: rev c7fad665f34fd3c17999d5cc60b04d3faff6a7f5) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFlushRegionEntry.java * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushLargeStoresPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPerColumnFamilyFlush.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DisabledWALProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushAllStoresPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestDefaultWALProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushPolicyFactory.java > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo > Fix For: 2.0.0 > > Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, > HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, > HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, > HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, > HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, > HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, > HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, > HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, > HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, > compactions.png, count.png, io.png, memstore.png > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9431) Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims
[ https://issues.apache.org/jira/browse/HBASE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248945#comment-14248945 ] Hudson commented on HBASE-9431: --- FAILURE: Integrated in HBase-TRUNK #5930 (See [https://builds.apache.org/job/HBase-TRUNK/5930/]) HBASE-9431 Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims (stack: rev e5d813c46b41ab4fb48d72731eb34422f260b81a) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java > Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims > --- > > Key: HBASE-9431 > URL: https://issues.apache.org/jira/browse/HBASE-9431 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 1.0.0, 2.0.0 > > Attachments: 9431.txt, hbase-12072_v1.patch, hbase-9431.patch > > > HBASE-8450 claimes 'hbase.bulkload.retries.number' is set to 10 when its > still 0 ([~jeffreyz] noticed). Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9431) Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims
[ https://issues.apache.org/jira/browse/HBASE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248921#comment-14248921 ] Hudson commented on HBASE-9431: --- SUCCESS: Integrated in HBase-1.0 #589 (See [https://builds.apache.org/job/HBase-1.0/589/]) HBASE-9431 Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims (stack: rev 36269b64038c6fcf076a1d8df90ea087559eb208) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java > Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims > --- > > Key: HBASE-9431 > URL: https://issues.apache.org/jira/browse/HBASE-9431 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 1.0.0, 2.0.0 > > Attachments: 9431.txt, hbase-12072_v1.patch, hbase-9431.patch > > > HBASE-8450 claimes 'hbase.bulkload.retries.number' is set to 10 when its > still 0 ([~jeffreyz] noticed). Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248920#comment-14248920 ] Hudson commented on HBASE-8450: --- SUCCESS: Integrated in HBase-1.0 #589 (See [https://builds.apache.org/job/HBase-1.0/589/]) HBASE-9431 Set 'hbase.bulkload.retries.number' to 10 as HBASE-8450 claims (stack: rev 36269b64038c6fcf076a1d8df90ea087559eb208) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java > Update hbase-default.xml and general recommendations to better suit current > hw, h2, experience, etc. > > > Key: HBASE-8450 > URL: https://issues.apache.org/jira/browse/HBASE-8450 > Project: HBase > Issue Type: Task > Components: Usability >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.95.1 > > Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt, 8450v5.txt > > > This is a critical task we need to do before we release; review our defaults. > On cursory review, there are configs in hbase-default.xml that no longer have > matching code; there are some that should be changed because we know better > now and others that should be amended because hardware and deploys are bigger > than they used to be. > We could also move stuff around so that the must-edit stuff is near the top > (zk quorum config. is mid-way down the page) and beef up the descriptions -- > especially since these descriptions shine through in the refguide. > Lastly, I notice that our tgz does not "include" an hbase-default.xml other > than the one bundled up in the jar. Maybe we should make it more accessible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248907#comment-14248907 ] Sean Busbey commented on HBASE-5954: Can you update the patch to include changes to WALPerformanceEvaluation to allow a mix of sync / fsync calls so we can see what the perf impact will look like? > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement > Components: HFile, wal >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 2.0.0 > > Attachments: 5954-WIP-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > > At least get recommendation into 0.96 doc and some numbers running w/ this > hdfs feature enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12699) undefined method `setAsyncLogFlush' exception thrown when setting DEFERRED_LOG_FLUSH=>true
[ https://issues.apache.org/jira/browse/HBASE-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248893#comment-14248893 ] Enis Soztutar commented on HBASE-12699: --- Commit to 0.98 as well ? Ping [~apurtell]. > undefined method `setAsyncLogFlush' exception thrown when setting > DEFERRED_LOG_FLUSH=>true > -- > > Key: HBASE-12699 > URL: https://issues.apache.org/jira/browse/HBASE-12699 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 2.0.0, 0.99.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 1.0.0, 2.0.0 > > Attachments: HBASE-12699.v1.branch-1.patch, > HBASE-12699.v1.master.patch > > Original Estimate: 24h > Time Spent: 4h > Remaining Estimate: 1h > > In hbase shell, when trying to set DEFERRED_LOG_FLUSH during create or alter, > an undefined method `setAsyncLogFlush' exception was thrown. > This is due to that DEFERRED_LOG_FLUSH was deprecated and the > setAsyncLogFlush method was removed. It was replaced by DURABILITY. > DEFERRED_LOG_FLUSH=>true is the same as DURABILITY=>'ASYNC_WAL' > The default is DURABILITY=>'SYNC_WAL', which is the same as the default > DEFERRED_LOG_FLUSH=>false > We should ask user to use the DURABILITY setting. In the meantime, for > backward compatibility, the hbase shell should still allow setting > DEFERRED_LOG_FLUSH. Internally, instead of calling setAsyncLogFlush, it > should call setDurability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248882#comment-14248882 ] Lars Hofhansl commented on HBASE-5954: -- After thinking about this for a little bit, I think we need to do this by edit and column family, the way my patch has it. Please have a look at this patch, I think we should get this in. Along with this we need to document that you *have* to run HDFS with sync-on-close and sync-behind-writes. (In fact everybody should do that in any case!) > Allow proper fsync support for HBase > > > Key: HBASE-5954 > URL: https://issues.apache.org/jira/browse/HBASE-5954 > Project: HBase > Issue Type: Improvement > Components: HFile, wal >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 2.0.0 > > Attachments: 5954-WIP-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, > 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, > 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, > 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt > > > At least get recommendation into 0.96 doc and some numbers running w/ this > hdfs feature enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12645) HBaseTestingUtility is using ${$HOME} for rootDir
[
https://issues.apache.org/jira/browse/HBASE-12645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248847#comment-14248847
]
Hadoop QA commented on HBASE-12645:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687527/HBASE-12645.006.patch
against master branch at commit 1359e87b1757dd32d81fa2039af55156de1b3298.
ATTACHMENT ID: 12687527
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12099//console
This message is automatically generated.
> HBaseTestingUtility is using ${$HOME} for rootDir
> -
>
> Key: HBASE-12645
> URL: https://issues.apache.org/jira/browse/HBASE-12645
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 1.0.0
>Reporter: Nick Dimiduk
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 1.0.0, 2.0.0
>
> Attachments: HBASE-12645.002.patch, HBASE-12645.003.patch,
> HBASE-12645.004.patch, HBASE-12645.004.patch, HBASE-12645.005.patch,
> HBASE-12645.006.patch, HBASE-12645.006.patch, HBASE-12645.patch
>
>
> I noticed this while running tests on branch-1
> {noformat}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 sec <<<
> FAILURE! - in
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits
> org.apache.hadoop.hbase.regionserver.wal.TestReadOldRootAndMetaEdits Time
> elapsed: 0.009 sec <<< ERROR!
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /homes/hortonnd/hbase
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:423)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:588)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1053)
>
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248816#comment-14248816 ] stack commented on HBASE-10201: --- I forgot to say thank you [~Apache9] for your persistence on getting this in. > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo > Fix For: 2.0.0 > > Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, > HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, > HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, > HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, > HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, > HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, > HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, > HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, > HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, > compactions.png, count.png, io.png, memstore.png > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12700) Backport 'Make flush decisions per column family' from master
stack created HBASE-12700: - Summary: Backport 'Make flush decisions per column family' from master Key: HBASE-12700 URL: https://issues.apache.org/jira/browse/HBASE-12700 Project: HBase Issue Type: Task Reporter: stack Fix For: 1.1.0 Task to backport this nice feature to 1.1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-10201: -- Resolution: Fixed Fix Version/s: (was: 1.0.0) Release Note: Adds new flushing policy mechanism. Default, org.apache.hadoop.hbase.regionserver.FlushLargeStoresPolicy, will try to avoid flushing out the small column families in a region, those whose memstores are < hbase.hregion.percolumnfamilyflush.size.lower.bound. To restore the old behavior of flushes writing out all column families, set hbase.regionserver.flush.policy to org.apache.hadoop.hbase.regionserver.FlushAllStoresPolicy either in hbase-default.xml or on a per-table basis by setting the policy to use with HTableDescriptor.getFlushPolicyClassName(). Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master branch > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo > Fix For: 2.0.0 > > Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, > HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, > HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, > HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, > HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, > HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, > HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, > HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, > HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, > compactions.png, count.png, io.png, memstore.png > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248793#comment-14248793 ] stack commented on HBASE-10201: --- bq. Can we target 1.1 instead? Sure. > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo > Fix For: 1.0.0, 2.0.0 > > Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, > HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, > HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_10.patch, > HBASE-10201_11.patch, HBASE-10201_12.patch, HBASE-10201_13.patch, > HBASE-10201_13.patch, HBASE-10201_14.patch, HBASE-10201_15.patch, > HBASE-10201_16.patch, HBASE-10201_17.patch, HBASE-10201_18.patch, > HBASE-10201_19.patch, HBASE-10201_2.patch, HBASE-10201_3.patch, > HBASE-10201_4.patch, HBASE-10201_5.patch, HBASE-10201_6.patch, > HBASE-10201_7.patch, HBASE-10201_8.patch, HBASE-10201_9.patch, > compactions.png, count.png, io.png, memstore.png > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248780#comment-14248780 ] Andrew Purtell commented on HBASE-5162: --- I have a feeling [~enis] won't take this at this late time for 1.0. What if we open a branch (soon) for 1.1. I think we can look at this for 0.98 if we don't break non-private interfaces and we can get it back into 0.98 from master via any 1.x. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5162) Basic client pushback mechanism
[ https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5162: --- Attachment: hbase-5162-branch-1-v0.patch Attaching patch for branch-1, [~enis] - thoughts on getting this in? Also, [~apurtell], would you like a patch for 0.98 as well? I think it should be pretty close to 1.0/2.0 version. > Basic client pushback mechanism > --- > > Key: HBASE-5162 > URL: https://issues.apache.org/jira/browse/HBASE-5162 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Jesse Yates > Fix For: 1.0.0, 2.0.0 > > Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, > hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, > hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, > hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, > hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, > hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, > hbase-5162-trunk-v8.patch, java_HBASE-5162.patch > > > The current blocking we do when we are close to some limits (memstores over > the multiplier factor, too many store files, global memstore memory) is bad, > too coarse and confusing. After hitting HBASE-5161, it really becomes obvious > that we need something better. > I did a little brainstorm with Stack, we came up quickly with two solutions: > - Send some exception to the client, like OverloadedException, that's thrown > when some situation happens like getting past the low memory barrier. It > would be thrown when the client gets a handler and does some check while > putting or deleting. The client would treat this a retryable exception but > ideally wouldn't check .META. for a new location. It could be fancy and have > multiple levels of pushback, like send the exception to 25% of the clients, > and then go up if the situation persists. Should be "easy" to implement but > we'll be using a lot more IO to send the payload over and over again (but at > least it wouldn't sit in the RS's memory). > - Send a message alongside a successful put or delete to tell the client to > slow down a little, this way we don't have to do back and forth with the > payload between the client and the server. It's a cleaner (I think) but more > involved solution. > In every case the RS should do very obvious things to notify the operators of > this situation, through logs, web UI, metrics, etc. > Other ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12644) Visibility Labels: issue with storing super users in labels table
[ https://issues.apache.org/jira/browse/HBASE-12644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248757#comment-14248757 ] Jerry He commented on HBASE-12644: -- Got it. Thanks, Ted. > Visibility Labels: issue with storing super users in labels table > - > > Key: HBASE-12644 > URL: https://issues.apache.org/jira/browse/HBASE-12644 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.98.8, 0.99.2 >Reporter: Jerry He >Assignee: Jerry He > Fix For: 1.0.0, 2.0.0 > > Attachments: 12644-0.98.patch, HBASE-12644-master-v2.patch, > HBASE-12644-master-v3.patch, HBASE-12644-master.patch > > > Super users have all the permissions for ACL and Visibility labels. > They are defined in hbase-site.xml. > Currently in VisibilityController, we persist super user with their system > permission in hbase:labels. > This makes change in super user difficult. > There are two issues: > In the current DefaultVisibilityLabelServiceImpl.addSystemLabel, we only add > super user when we initially create the 'system' label. > No additional update after that even if super user changed. See code for > details. > > Additionally, there is no mechanism to remove any super user from the labels > table. > > We probably should not persist super users in the labels table. > They are in hbase-site.xml and can just stay in labelsCache and used from > labelsCache after retrieval by Visibility Controller. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
