[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Attachment: (was: HBase-4095-V9-branch.patch)

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.5
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HBase-4095-V7-branch.patch, 
> HBase-4095-V7-trunk.patch, HBase-4095-V8-branch.patch, 
> HBase-4095-V8-trunk.patch, HBase-4095-V9-branch.patch, 
> HBase-4095-V9-trunk.patch, HlogFileIsVeryLarge.gif, 
> LatestLLTResults-20110810.rar, RelatedLogs2011-07-28.txt, 
> TestResultForPatch-V4.rar, flowChart-IntroductionToThePatch.gif, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,130

[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Attachment: HBase-4095-V9-trunk.patch
HBase-4095-V9-branch.patch

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.5
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HBase-4095-V7-branch.patch, 
> HBase-4095-V7-trunk.patch, HBase-4095-V8-branch.patch, 
> HBase-4095-V8-trunk.patch, HBase-4095-V9-branch.patch, 
> HBase-4095-V9-trunk.patch, HlogFileIsVeryLarge.gif, 
> LatestLLTResults-20110810.rar, RelatedLogs2011-07-28.txt, 
> TestResultForPatch-V4.rar, flowChart-IntroductionToThePatch.gif, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbas

[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Attachment: (was: HBase-4095-V9-trunk.patch)

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.5
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HBase-4095-V7-branch.patch, 
> HBase-4095-V7-trunk.patch, HBase-4095-V8-branch.patch, 
> HBase-4095-V8-trunk.patch, HBase-4095-V9-branch.patch, 
> HBase-4095-V9-trunk.patch, HlogFileIsVeryLarge.gif, 
> LatestLLTResults-20110810.rar, RelatedLogs2011-07-28.txt, 
> TestResultForPatch-V4.rar, flowChart-IntroductionToThePatch.gif, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309

[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with batches per region sent from a client(?)

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086155#comment-13086155
 ] 

Andrew Purtell commented on HBASE-4210:
---

@Lars: Can you talk a bit about the use case you are considering here? What is 
the motivation?



> Allow coprocessor to interact with batches per region sent from a client(?)
> ---
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
>  Issue Type: New Feature
>Reporter: Lars Hofhansl
>Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly 
> one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts 
> and deletes as they arrive from the client.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4175) Fix FSUtils.createTableDescriptor()

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086151#comment-13086151
 ] 

ramkrishna.s.vasudevan commented on HBASE-4175:
---

Thanks for your review Stack and Ted.
@Stack
Yes i could have done that.  Next time will not repeat this. :)
Will resubmit with the changes as you have suggested.

> Fix FSUtils.createTableDescriptor()
> ---
>
> Key: HBASE-4175
> URL: https://issues.apache.org/jira/browse/HBASE-4175
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4175.patch, HBASE-4175_1.patch
>
>
> Currently createTableDescriptor() doesn't return anything.
> The caller wouldn't know whether the descriptor is created or not. See 
> exception handling:
> {code}
>} catch(IOException ioe) {
>  LOG.info("IOException while trying to create tableInfo in HDFS", ioe);
>}
> {code}
> We should return a boolean.
> If the table descriptor exists already, maybe we should deserialize from hdfs 
> and compare with htableDescriptor argument. If they differ, I am not sure 
> what the proper action would be.
> Maybe we can add a boolean argument, force, to createTableDescriptor(). When 
> force is true, existing table descriptor would be overwritten.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086146#comment-13086146
 ] 

stack commented on HBASE-4176:
--

bq. I can try and integrate it into the book. Just wondering if there's any 
tools that I can use or that anyone recommends? Or should I go at it with a 
text-editor?

I just hack on it w/ texteditor.  ooo does docbook 4.x or something.  old.  our 
book is docbook 5.  Let me hack on it.  Not need to have you suffer the horrors 
of our doc system, especially after your having just passed via thrift and the 
shell.

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language (3).xml, Filter Language(2).docx, Filter 
> Language(2).xml, Filter Language(3).docx, Filter Language.docx, 
> HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086145#comment-13086145
 ] 

jirapos...@reviews.apache.org commented on HBASE-4176:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1326/#review1490
---

Ship it!


On 'How would I make createFilterFromArgument in FilterBase abstract? ', you 
can't.  Thats my fault for not seeing static qualifier.

On other criticisms, your responses are good by me.

Will commit in morning.


/src/main/ruby/shell/commands/scan.rb


Good.


- Michael


On 2011-08-17 04:53:07, Anirudh Todi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1326/
bq.  ---
bq.  
bq.  (Updated 2011-08-17 04:53:07)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  https://issues.apache.org/jira/browse/HBASE-4176: Exposing HBase Filters 
to the Thrift API
bq.  
bq.  Currently, to use any of the filters, one has to explicitly add a scanner 
for the filter in the Thrift API making it messy and long. 
bq.  With this patch, I am trying to add support for all the filters in a clean 
way. 
bq.  The user specifies a filter via a string. The string is parsed on the 
server to construct the filter. More information can be found in the attached 
document named Filter Language
bq.  
bq.  This patch is trying to extend and further the progress made by the 
patches in HBASE-1744
bq.  
bq.  There is document attached to the HBASE-4176 JIRA that describes this 
patch in further detail
bq.  
bq.  
bq.  This addresses bug HBASE-4176.
bq.  https://issues.apache.org/jira/browse/HBASE-4176
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java 
1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FamilyFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/QualifierFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/RowFilter.java 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueExcludeFilter.java
 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ValueFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/thrift/generated/TScan.java 
1158073 
bq./src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift 1158073 
bq./src/main/ruby/hbase/table.rb 1158073 
bq./src/main/ruby/shell/commands/scan.rb 1158073 
bq./src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java 
1158073 
bq.  
bq.  Diff: https://reviews.apache.org/r/1326/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  patch includes one test: TestParseFilter.java
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Anirudh
bq.  
bq.



> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project:

[jira] [Resolved] (HBASE-4211) Do init-sizing of the StringBuilder making a ServerName.

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4211.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Assignee: Benoit Sigoure
 Hadoop Flags: [Reviewed]

Did this remove of synchronized also reported by B while I was at it this 
commit:

{code}
 .../java/org/apache/hadoop/hbase/ServerName.java   |3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/src/main/java/org/apache/hadoop/hbase/ServerName.java 
b/src/main/java/org/apache/hadoop/hbase/ServerName.java
index 6b03832..4ddb5b7 100644
--- a/src/main/java/org/apache/hadoop/hbase/ServerName.java
+++ b/src/main/java/org/apache/hadoop/hbase/ServerName.java
@@ -128,7 +128,8 @@ public class ServerName implements Comparable {
   * startcode formatted as  ','  ',' 

   */
  public static String getServerName(String hostName, int port, long startcode) 
{
-StringBuilder name = new StringBuilder(hostName);
+final StringBuilder name = new StringBuilder(hostName.length() + 1 + 5 + 1 
+ 13);
+name.append(hostName);
name.append(SERVERNAME_SEPARATOR);
name.append(port);
name.append(SERVERNAME_SEPARATOR);
{code}

> Do init-sizing of the StringBuilder making a ServerName.
> 
>
> Key: HBASE-4211
> URL: https://issues.apache.org/jira/browse/HBASE-4211
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: Benoit Sigoure
>Priority: Minor
> Fix For: 0.92.0
>
>
> Simple patch from Benoît.
> ---
>  .../java/org/apache/hadoop/hbase/ServerName.java   |3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> diff --git a/src/main/java/org/apache/hadoop/hbase/ServerName.java 
> b/src/main/java/org/apache/hadoop/hbase/ServerName.java
> index 6b03832..4ddb5b7 100644
> --- a/src/main/java/org/apache/hadoop/hbase/ServerName.java
> +++ b/src/main/java/org/apache/hadoop/hbase/ServerName.java
> @@ -128,7 +128,8 @@ public class ServerName implements Comparable 
> {
>* startcode formatted as  ','  ',' 
> 
>*/
>   public static String getServerName(String hostName, int port, long 
> startcode) {
> -StringBuilder name = new StringBuilder(hostName);
> +final StringBuilder name = new StringBuilder(hostName.length() + 1 + 5 + 
> 1 + 13);
> +name.append(hostName);
> name.append(SERVERNAME_SEPARATOR);
> name.append(port);
> name.append(SERVERNAME_SEPARATOR);
> --
> 1.7.6.434.g1d2b3

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4211) Do init-sizing of the StringBuilder making a ServerName.

[stack (JIRA)]

Do init-sizing of the StringBuilder making a ServerName.


 Key: HBASE-4211
 URL: https://issues.apache.org/jira/browse/HBASE-4211
 Project: HBase
  Issue Type: Bug
Reporter: stack
Priority: Minor


Simple patch from Benoît.

---
 .../java/org/apache/hadoop/hbase/ServerName.java   |3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/src/main/java/org/apache/hadoop/hbase/ServerName.java 
b/src/main/java/org/apache/hadoop/hbase/ServerName.java
index 6b03832..4ddb5b7 100644
--- a/src/main/java/org/apache/hadoop/hbase/ServerName.java
+++ b/src/main/java/org/apache/hadoop/hbase/ServerName.java
@@ -128,7 +128,8 @@ public class ServerName implements Comparable {
   * startcode formatted as  ','  ',' 

   */
  public static String getServerName(String hostName, int port, long startcode) 
{
-StringBuilder name = new StringBuilder(hostName);
+final StringBuilder name = new StringBuilder(hostName.length() + 1 + 5 + 1 
+ 13);
+name.append(hostName);
name.append(SERVERNAME_SEPARATOR);
name.append(port);
name.append(SERVERNAME_SEPARATOR);
--
1.7.6.434.g1d2b3

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4176) Exposing HBase Filters to the Thrift API

[Anirudh Todi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anirudh Todi updated HBASE-4176:


Attachment: Filter Language (3).xml
Filter Language(3).docx

Attaching updated Filter Language document along with its corresponding docbook 
article (saved in ooo).

I can try and integrate it into the book. Just wondering if there's any tools 
that I can use or that anyone recommends? Or should I go at it with a 
text-editor?

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language (3).xml, Filter Language(2).docx, Filter 
> Language(2).xml, Filter Language(3).docx, Filter Language.docx, 
> HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086135#comment-13086135
 ] 

jirapos...@reviews.apache.org commented on HBASE-4176:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1326/
---

(Updated 2011-08-17 04:53:07.890603)


Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and Jonathan Gray.


Changes
---

- Minor change: swapped the order of examples in the scan help menu from the 
shell so that it agrees with the explanation

@Stack:
- How would I make createFilterFromArgument in FilterBase abstract? An abstract 
method be static too. And if I just make it abstract - the compiler complains 
saying "cannot override - overriding method is static." Am I missing something 
here?
- I implemented it such that the operators would all be upper case. I can 
change it if you want. Have inserted a line into the document to that effect
- I have defined 'S', 'K', etc as ints and I have them as a byte array b/c I 
have different uses for these in different places. In checkForSkip() - I 
compare character by character and use the ints. When I'm pushing something 
onto the operator stack - I push the byte array


Summary
---

https://issues.apache.org/jira/browse/HBASE-4176: Exposing HBase Filters to the 
Thrift API

Currently, to use any of the filters, one has to explicitly add a scanner for 
the filter in the Thrift API making it messy and long. 
With this patch, I am trying to add support for all the filters in a clean way. 
The user specifies a filter via a string. The string is parsed on the server to 
construct the filter. More information can be found in the attached document 
named Filter Language

This patch is trying to extend and further the progress made by the patches in 
HBASE-1744

There is document attached to the HBASE-4176 JIRA that describes this patch in 
further detail


This addresses bug HBASE-4176.
https://issues.apache.org/jira/browse/HBASE-4176


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java 
1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java 
1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java 
1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/FamilyFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java 
1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java 
1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/QualifierFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/RowFilter.java 1158073 
  
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueExcludeFilter.java
 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java 
1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/filter/ValueFilter.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java 1158073 
  /src/main/java/org/apache/hadoop/hbase/thrift/generated/TScan.java 1158073 
  /src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift 1158073 
  /src/main/ruby/hbase/table.rb 1158073 
  /src/main/ruby/shell/commands/scan.rb 1158073 
  /src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java 1158073 

Diff: https://reviews.apache.org/r/1326/diff


Testing
---

patch includes one test: TestParseFilter.java


Thanks,

Anirudh



> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority

[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086134#comment-13086134
 ] 

jirapos...@reviews.apache.org commented on HBASE-4027:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1489
---



src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java


Please add javadoc for these methods.



src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java


This shows that Cacheable can reside in on-heap cache.
The description for Cacheable should be refined.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java


HeapSize is covered by Cacheable so is not needed here.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java


Indentation is incorrect here.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java


Since HFileBlock implements Cacheable, people may get confused by what 
'selfWithoutByteBuffer' means.
Please add javadoc.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java


Please add javadoc for what this method does.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java


Whitespace.



src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java


Indentation.


- Ted


On 2011-08-17 04:16:06, Li Pi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1214/
bq.  ---
bq.  
bq.  (Updated 2011-08-17 04:16:06)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan 
Gray, and Li Pi.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Review request - I apparently can't edit tlipcon's earlier posting of my 
diff, so creating a new one.
bq.  
bq.  
bq.  This addresses bug HBase-4027.
bq.  https://issues.apache.org/jira/browse/HBase-4027
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.conf/hbase-env.sh 2d55d27 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 
1338453 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 
886c31d 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java 
PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
PRE-CREATION 
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 PRE-CREATION 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
e2c6c93 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 
7b7bf73 
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 
1ad2ece 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java 
f0a9832 
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
PRE-CREATION 
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 
4387170 
bq.  
bq.  Diff: https://reviews.apache.org/r/1214/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran benchmarks against it in HBase standalone mode. Wrote test cases for 
all classes, multithreaded test cases exist for the cache.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Li
bq.  
bq.



> Enable direct byte buffers LruBlockCache
> 
>
> 

[jira] [Commented] (HBASE-4117) Slow Query Log

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086126#comment-13086126
 ] 

stack commented on HBASE-4117:
--

@Andrew I appreciate your showing up in your architect hat flashing the red 
card.  Please stay on the prowl and flag the violations.

In this case, on review, I think we are talking honest logging (logging already 
has a system that facilitates redirection and feeding events to other than 
files that live under HBASE_LOG_DIR -- which reminds me, I need to go back to 
Riley and make sure that these events CAN be directed to other than the .log 
file cleanly).  As Riley says, these logs at an extreme will likely make most 
sense if shown in context amidst the logging of what else was going on on the 
server.

The patch also includes nice refactoring of base classes -- Scan, Get, etc. -- 
to make it so they can dump out a self-description (either a 
fingerprint/synopsis, or a full catalog of the payload being carried) in text 
or json that looks like it can be passed to whatever asks for the info (log or 
metrics or...)

We are still missing the general theory on how to pass hbase emissions but I 
think this patch helps move us closer to a general system much more than it 
adds distraction.

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4027) Enable direct byte buffers LruBlockCache

[Li Pi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Pi updated HBASE-4027:
-

Attachment: hbase4027v12.diff

a few more changes, substantially re-factored.

> Enable direct byte buffers LruBlockCache
> 
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jason Rutherglen
>Assignee: Li Pi
>Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, 
> HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, 
> hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, 
> hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, 
> hbase4027v11.diff, hbase4027v12.diff, slabcachepatch.diff, 
> slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, 
> slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside 
> of the heap.
> They need to be manually free'd, which can be accomplished using an 
> documented {{clean}} method.
> The feature will be optional.  After implementing, we can benchmark for 
> differences in speed and garbage collection observances.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086117#comment-13086117
 ] 

jirapos...@reviews.apache.org commented on HBASE-4027:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---

(Updated 2011-08-17 04:16:06.047294)


Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, 
and Li Pi.


Changes
---

refactored, no more casting, cacheable added. multithreaded test that hammers 
single key added. general code cleanup stuff as well. heapsize not implemented 
completely yet.


Summary
---

Review request - I apparently can't edit tlipcon's earlier posting of my diff, 
so creating a new one.


This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027


Diffs (updated)
-

  conf/hbase-env.sh 2d55d27 
  src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c 
  src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 
  src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 
  src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 
  src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d 
  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 
PRE-CREATION 
  
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
 PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 
  src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 
  src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 
1ad2ece 
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 
PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 

Diff: https://reviews.apache.org/r/1214/diff


Testing
---

Ran benchmarks against it in HBase standalone mode. Wrote test cases for all 
classes, multithreaded test cases exist for the cache.


Thanks,

Li



> Enable direct byte buffers LruBlockCache
> 
>
> Key: HBASE-4027
> URL: https://issues.apache.org/jira/browse/HBASE-4027
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jason Rutherglen
>Assignee: Li Pi
>Priority: Minor
> Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, 
> HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, 
> hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, 
> hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, 
> hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff, 
> slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, 
> slabcachepatchv4.5.diff, slabcachepatchv4.diff
>
>
> Java offers the creation of direct byte buffers which are allocated outside 
> of the heap.
> They need to be manually free'd, which can be accomplished using an 
> documented {{clean}} method.
> The feature will be optional.  After implementing, we can benchmark for 
> differences in speed and garbage collection observances.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with batches per region sent from a client(?)

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086112#comment-13086112
 ] 

Lars Hofhansl commented on HBASE-4210:
--

The options for that might be {pre|post}Batch hook, or an RpcObserver and 
start- and end-type hooks.

> Allow coprocessor to interact with batches per region sent from a client(?)
> ---
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
>  Issue Type: New Feature
>Reporter: Lars Hofhansl
>Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly 
> one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts 
> and deletes as they arrive from the client.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4210) Allow coprocessor to interact with batches per region sent from a client(?)

[Lars Hofhansl (JIRA)]

Allow coprocessor to interact with batches per region sent from a client(?)
---

 Key: HBASE-4210
 URL: https://issues.apache.org/jira/browse/HBASE-4210
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Priority: Minor


Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly 
one row|cell operations.
It might be a good idea to allow a coprocessor to deal with batches of puts and 
deletes as they arrive from the client.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4195) Possible inconsistency in a memstore read after a reseek, possible performance improvement

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086111#comment-13086111
 ] 

Ted Yu commented on HBASE-4195:
---

TestHRegion.testWritesWhileGetting failed in build 266:
https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/lastCompletedBuild/testReport/org.apache.hadoop.hbase.regionserver/TestHRegion/testWritesWhileGetting/

> Possible inconsistency in a memstore read after a reseek, possible 
> performance improvement
> --
>
> Key: HBASE-4195
> URL: https://issues.apache.org/jira/browse/HBASE-4195
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Priority: Critical
>
> This follows the dicussion around HBASE-3855, and the random errors (20% 
> failure on trunk) on the unit test 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting
> I saw some points related to numIterReseek, used in the 
> MemStoreScanner#getNext (line 690):
> {noformat}679 protected KeyValue getNext(Iterator it) {
> 680 KeyValue ret = null;
> 681 long readPoint = ReadWriteConsistencyControl.getThreadReadPoint();
> 682 //DebugPrint.println( " MS@" + hashCode() + ": threadpoint = " + 
> readPoint);
> 683
> 684 while (ret == null && it.hasNext()) {
> 685   KeyValue v = it.next();
> 686   if (v.getMemstoreTS() <= readPoint) {
> 687 // keep it.
> 688 ret = v;
> 689   }
> 690   numIterReseek--;
> 691   if (numIterReseek == 0) {
> 692 break;
> 693}
> 694 }
> 695 return ret;
> 696   }{noformat}
> This function is called by seek, reseek, and next. The numIterReseek is only 
> usefull for reseek.
> There are some issues, I am not totally sure it's the root cause of the test 
> case error, but it could explain partly the randomness of the error, and one 
> point is for sure a bug.
> 1) In getNext, numIterReseek is decreased, then compared to zero. The seek 
> function sets numIterReseek to zero before calling getNext. It means that the 
> value will be actually negative, hence the test will always fail, and the 
> loop will continue. It is the expected behaviour, but it's quite smart.
> 2) In "reseek", numIterReseek is not set between the loops on the two 
> iterators. If the numIterReseek is equals to zero after the loop on the first 
> one, the loop on the second one will never call seek, as numIterReseek will 
> be negative.
> 3) Still in "reseek", the test to call "seek" is (kvsetNextRow == null && 
> numIterReseek == 0). In other words, if kvsetNextRow is not null when 
> numIterReseek equals zero, numIterReseek will start to be negative at the 
> next iteration and seek will never be called.
> 4) You can have side effects if reseek ends with a numIterReseek > 0: the 
> following calls to the "next" function will decrease numIterReseek to zero, 
> and getNext will break instead of continuing the loop. As a result, later 
> calls to next() may return null or not depending on how is configured the 
> default value for numIterReseek.
> To check if the issue comes from point 4, you can set the numIterReseek to 
> zero before returning in reseek:
> {noformat}  numIterReseek = 0;
>   return (kvsetNextRow != null || snapshotNextRow != null);
> }{noformat}
> On my env, on trunk, it seems to work, but as it's random I am not really 
> sure. I also had to modify the test (I added a loop) to make it fails more 
> often, the original test was working quite well here.
> It has to be confirmed that this totally fix (it could be partial or 
> unrelated) 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting 
> before implementing a complete solution.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086082#comment-13086082
 ] 

Gary Helmling commented on HBASE-4117:
--

Riley, would you mind putting this up on reviews.apache.org?  This is a pretty 
big change to read through as a patch.  Thanks!

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4190) Coprocessors: pull up some cp constants from cp package to o.a.h.h.HConstants

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086076#comment-13086076
 ] 

Hudson commented on HBASE-4190:
---

Integrated in HBase-TRUNK #2121 (See 
[https://builds.apache.org/job/HBase-TRUNK/2121/])
HBASE-4190 Coprocessors: pull up some cp constants from cp package to 
o.a.h.h.HConstants

stack : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/ObserverContext.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterCoprocessorEnvironment.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/WALObserver.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorEnvironment.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/WALCoprocessorEnvironment.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/Coprocessor.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionCoprocessorEnvironment.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java


> Coprocessors: pull up some cp constants from cp package to o.a.h.h.HConstants
> -
>
> Key: HBASE-4190
> URL: https://issues.apache.org/jira/browse/HBASE-4190
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Affects Versions: 0.90.4
>Reporter: Mingjie Lai
>Assignee: Mingjie Lai
>Priority: Minor
> Fix For: 0.92.0
>
>
> At HBase-3810, stack gave a comment after patch committed:
> > This is a bit odd where a class in the parent package has references to a 
> > sub package.
> > Should these classes or at least their constants be pulled up to be at same 
> > level as HTableD?
> Create a new jira where the constants will be pulled from 
> o.a.h.h.regionserver.RegionCoprocessorHost to o.a.h.h.HConstants. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086071#comment-13086071
 ] 

Andrew Purtell commented on HBASE-4117:
---

bq. This information resides in the TaskMonitor singleton, which can then be 
exposed any way you like. I think this design makes a lot of sense; historical 
information goes in the main log, while current status information can be 
queried from a clean interface and exposed any way you like.

Fair enough. I stand corrected / mollified.

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Attachment: flowChart-IntroductionToThePatch.gif
HBase-4095-V9-trunk.patch
HBase-4095-V9-branch.patch

Thanks, Ted. I've changed them.

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.5
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HBase-4095-V7-branch.patch, 
> HBase-4095-V7-trunk.patch, HBase-4095-V8-branch.patch, 
> HBase-4095-V8-trunk.patch, HBase-4095-V9-branch.patch, 
> HBase-4095-V9-trunk.patch, HlogFileIsVeryLarge.gif, 
> LatestLLTResults-20110810.rar, RelatedLogs2011-07-28.txt, 
> TestResultForPatch-V4.rar, flowChart-IntroductionToThePatch.gif, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-0

[jira] [Assigned] (HBASE-3229) Table creation, though using "async" call to master, can actually run for a while and cause RPC timeout

[Ming Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming Ma reassigned HBASE-3229:
--

Assignee: Ming Ma

> Table creation, though using "async" call to master, can actually run for a 
> while and cause RPC timeout
> ---
>
> Key: HBASE-3229
> URL: https://issues.apache.org/jira/browse/HBASE-3229
> Project: HBase
>  Issue Type: Bug
>  Components: client, master
>Affects Versions: 0.90.0
>Reporter: Jonathan Gray
>Assignee: Ming Ma
>Priority: Critical
> Fix For: 0.92.0
>
>
> Our create table methods in HBaseAdmin are synchronous from client POV.  
> However, underneath, we're using an "async" create and then looping waiting 
> for table availability.  Because the create is async and we loop instead of 
> block on RPC, we don't expect RPC timeouts.
> However, when creating a table with lots of initial regions, the "async" 
> create can actually take a long time (more than 30 seconds in this case) 
> which causes the client to timeout and gives impression something failed.
> We should make the create truly async so that this can't happen.  And rather 
> than doing one-off, inline assignment as it is today, we should reuse the 
> fancy enable/disable code stack just added to make this faster and more 
> optimal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4208) hbase book - several updates (8-16-2011)

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086051#comment-13086051
 ] 

Hudson commented on HBASE-4208:
---

Integrated in HBase-TRUNK #2120 (See 
[https://builds.apache.org/job/HBase-TRUNK/2120/])
HBASE-4208   updates to book.xml, performance.xml

dmeil : 
Files : 
* /hbase/trunk/src/docbkx/performance.xml
* /hbase/trunk/src/docbkx/book.xml


> hbase book - several updates (8-16-2011)
> 
>
> Key: HBASE-4208
> URL: https://issues.apache.org/jira/browse/HBASE-4208
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: docbkx_HBASE_4208.patch
>
>
> performance.xml
> * adding a 'deleting from hbase' section
> ** table-as-queue example.
> ** htable.delete(delete) executes RPC each invocation
> book.xml
> * updating schema design...
> ** TTL
> ** more explicit examples of attributes/cf's for keysize section.
> ** nit capitalization of 'storefile' to 'StoreFile'
> * mapreduce
> ** formatted code example (indenting was honked)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4197) RegionServer expects all scanner to be subclasses of HRegion.RegionScanner

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086050#comment-13086050
 ] 

Lars Hofhansl commented on HBASE-4197:
--

@Andrew: The change generally is neutral in terms of behavior (mostly interface 
changes). The only case that I missed was that instanceof also guards again 
null.
Or are you referring to Todd's request (recently on the dev mailing list) to 
have changes bake for a few days before they are committed?


> RegionServer expects all scanner to be subclasses of HRegion.RegionScanner
> --
>
> Key: HBASE-4197
> URL: https://issues.apache.org/jira/browse/HBASE-4197
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 4197-bigger.txt, 4197-test.diff, 4197-v2.txt, 4197.txt, 
> ScannerTest.java
>
>
> Returning just an InternalScanner from RegionObsever.{pre|post}OpenScanner 
> leads to the following exception when using the scanner.
> java.io.IOException: InternalScanner implementation is expected to be 
> HRegion.RegionScanner.
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2023)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:314)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1225)
> The problem is in HRegionServer.next(...):
> {code} 
> InternalScanner s = this.scanners.get(scannerName);
> ...
>   // Call coprocessor. Get region info from scanner.
>   HRegion region = null;
>   if (s instanceof HRegion.RegionScanner) {
> HRegion.RegionScanner rs = (HRegion.RegionScanner) s;
> region = getRegion(rs.getRegionName().getRegionName());
>   } else {
> throw new IOException("InternalScanner implementation is expected " +
> "to be HRegion.RegionScanner.");
>   }
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[Riley Patterson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086045#comment-13086045
 ] 

Riley Patterson commented on HBASE-4117:


This isn't really meant to enable gathering of status information. It is a 
bunch of historical information which is highly related to other information in 
the main log. Without the context of log events regarding compactions, region 
reassignments, etc. that we gain by placing it in the main log, we lose most of 
the utility. It can also provide context to other log events that may have been 
caused by RPC handlers getting locked up on slow queries.

We could additionally log this information somewhere in a monitor in memory or 
in its own separate log and open this up to RPC access, but it would duplicate 
a lot of historical information which would likely see infrequent use. I don't 
think there's any more reason to do this for this particular feature than for 
most other events logged in our main log file.

Note that I'm also finishing up work on an initial implementation of 
HBASE-4057, which deals directly with status information for the calls 
currently being serviced by RPC handlers and enables exposure of the contents 
of getFingerprint and getDetails in a more general way. This information 
resides in the TaskMonitor singleton, which can then be exposed any way you 
like. I think this design makes a lot of sense; historical information goes in 
the main log, while current status information can be queried from a clean 
interface and exposed any way you like.

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4197) RegionServer expects all scanner to be subclasses of HRegion.RegionScanner

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086031#comment-13086031
 ] 

Ted Yu commented on HBASE-4197:
---

@Andrew:
You meant this: Added check of scanner against null, right ?
If you have a better way of handling null pointer in 
HRegionServer$QosFunction.apply(), please share.

> RegionServer expects all scanner to be subclasses of HRegion.RegionScanner
> --
>
> Key: HBASE-4197
> URL: https://issues.apache.org/jira/browse/HBASE-4197
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 4197-bigger.txt, 4197-test.diff, 4197-v2.txt, 4197.txt, 
> ScannerTest.java
>
>
> Returning just an InternalScanner from RegionObsever.{pre|post}OpenScanner 
> leads to the following exception when using the scanner.
> java.io.IOException: InternalScanner implementation is expected to be 
> HRegion.RegionScanner.
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2023)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:314)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1225)
> The problem is in HRegionServer.next(...):
> {code} 
> InternalScanner s = this.scanners.get(scannerName);
> ...
>   // Call coprocessor. Get region info from scanner.
>   HRegion region = null;
>   if (s instanceof HRegion.RegionScanner) {
> HRegion.RegionScanner rs = (HRegion.RegionScanner) s;
> region = getRegion(rs.getRegionName().getRegionName());
>   } else {
> throw new IOException("InternalScanner implementation is expected " +
> "to be HRegion.RegionScanner.");
>   }
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4195) Possible inconsistency in a memstore read after a reseek, possible performance improvement

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086030#comment-13086030
 ] 

stack commented on HBASE-4195:
--

bq. Can someone confirm that we should not see partial writes in this case?

If we are seeing partial writes then we are not paying attention to RWCC 
sequence numbers properly.  The refactored reseek is probably how the code was 
before hbase-3855 (I didn't check) and that code was around when RWCC was being 
worked out is my guess... so it probably 'worked' (not seeing parital writes).  
The new patch adding seeks may not be watching RWCC properly.

You've done some nice analysis above N.  Any chance of your seeing it home?  I 
will run my dumb TestHRegion test if you have any patch you'd like me to try.



> Possible inconsistency in a memstore read after a reseek, possible 
> performance improvement
> --
>
> Key: HBASE-4195
> URL: https://issues.apache.org/jira/browse/HBASE-4195
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Priority: Critical
>
> This follows the dicussion around HBASE-3855, and the random errors (20% 
> failure on trunk) on the unit test 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting
> I saw some points related to numIterReseek, used in the 
> MemStoreScanner#getNext (line 690):
> {noformat}679 protected KeyValue getNext(Iterator it) {
> 680 KeyValue ret = null;
> 681 long readPoint = ReadWriteConsistencyControl.getThreadReadPoint();
> 682 //DebugPrint.println( " MS@" + hashCode() + ": threadpoint = " + 
> readPoint);
> 683
> 684 while (ret == null && it.hasNext()) {
> 685   KeyValue v = it.next();
> 686   if (v.getMemstoreTS() <= readPoint) {
> 687 // keep it.
> 688 ret = v;
> 689   }
> 690   numIterReseek--;
> 691   if (numIterReseek == 0) {
> 692 break;
> 693}
> 694 }
> 695 return ret;
> 696   }{noformat}
> This function is called by seek, reseek, and next. The numIterReseek is only 
> usefull for reseek.
> There are some issues, I am not totally sure it's the root cause of the test 
> case error, but it could explain partly the randomness of the error, and one 
> point is for sure a bug.
> 1) In getNext, numIterReseek is decreased, then compared to zero. The seek 
> function sets numIterReseek to zero before calling getNext. It means that the 
> value will be actually negative, hence the test will always fail, and the 
> loop will continue. It is the expected behaviour, but it's quite smart.
> 2) In "reseek", numIterReseek is not set between the loops on the two 
> iterators. If the numIterReseek is equals to zero after the loop on the first 
> one, the loop on the second one will never call seek, as numIterReseek will 
> be negative.
> 3) Still in "reseek", the test to call "seek" is (kvsetNextRow == null && 
> numIterReseek == 0). In other words, if kvsetNextRow is not null when 
> numIterReseek equals zero, numIterReseek will start to be negative at the 
> next iteration and seek will never be called.
> 4) You can have side effects if reseek ends with a numIterReseek > 0: the 
> following calls to the "next" function will decrease numIterReseek to zero, 
> and getNext will break instead of continuing the loop. As a result, later 
> calls to next() may return null or not depending on how is configured the 
> default value for numIterReseek.
> To check if the issue comes from point 4, you can set the numIterReseek to 
> zero before returning in reseek:
> {noformat}  numIterReseek = 0;
>   return (kvsetNextRow != null || snapshotNextRow != null);
> }{noformat}
> On my env, on trunk, it seems to work, but as it's random I am not really 
> sure. I also had to modify the test (I added a loop) to make it fails more 
> often, the original test was working quite well here.
> It has to be confirmed that this totally fix (it could be partial or 
> unrelated) 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting 
> before implementing a complete solution.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4197) RegionServer expects all scanner to be subclasses of HRegion.RegionScanner

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086029#comment-13086029
 ] 

Andrew Purtell commented on HBASE-4197:
---

This was committed too quickly.

> RegionServer expects all scanner to be subclasses of HRegion.RegionScanner
> --
>
> Key: HBASE-4197
> URL: https://issues.apache.org/jira/browse/HBASE-4197
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 4197-bigger.txt, 4197-test.diff, 4197-v2.txt, 4197.txt, 
> ScannerTest.java
>
>
> Returning just an InternalScanner from RegionObsever.{pre|post}OpenScanner 
> leads to the following exception when using the scanner.
> java.io.IOException: InternalScanner implementation is expected to be 
> HRegion.RegionScanner.
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2023)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:314)
> at 
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1225)
> The problem is in HRegionServer.next(...):
> {code} 
> InternalScanner s = this.scanners.get(scannerName);
> ...
>   // Call coprocessor. Get region info from scanner.
>   HRegion region = null;
>   if (s instanceof HRegion.RegionScanner) {
> HRegion.RegionScanner rs = (HRegion.RegionScanner) s;
> region = getRegion(rs.getRegionName().getRegionName());
>   } else {
> throw new IOException("InternalScanner implementation is expected " +
> "to be HRegion.RegionScanner.");
>   }
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086028#comment-13086028
 ] 

Andrew Purtell commented on HBASE-4117:
---

I'm not suggesting turning down good work.

However I renew my objection to the result of adding this feature as is. We end 
up with a hodgepodge of methods for gathering status information or taking 
administrative action. Do I use the shell? Do I grab some JSON at some path on 
the server? (And at more than one path?) Do I grep the logs? This is not good 
design. Why can't this information be made retrievable via the Java admin API 
via RPC (and therefore the shell can manipulate it)? Why can't we have a common 
interface that lets us expose this type of information by RPC or by servlet or 
by logging?

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4120) isolation and allocation

[Jean-Daniel Cryans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Daniel Cryans updated HBASE-4120:
--

Fix Version/s: (was: 0.90.5)
   0.94.0

Bumping to 0.94, for sure it shouldn't be in 0.90 and I'd argue that 0.92 is 
choke full of new features.

> isolation and allocation
> 
>
> Key: HBASE-4120
> URL: https://issues.apache.org/jira/browse/HBASE-4120
> Project: HBase
>  Issue Type: New Feature
>  Components: master, regionserver
>Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0
>Reporter: Liu Jia
> Fix For: 0.94.0
>
> Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, 
> Design_document_for_HBase_isolation_and_allocation_Revised.pdf, 
> HBase_isolation_and_allocation_user_guide.pdf, 
> Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch
>
>
> The HBase isolation and allocation tool is designed to help users manage 
> cluster resource among different application and tables.
> When we have a large scale of HBase cluster with many applications running on 
> it, there will be lots of problems. In Taobao there is a cluster for many 
> departments to test their applications performance, these applications are 
> based on HBase. With one cluster which has 12 servers, there will be only one 
> application running exclusively on this server, and many other applications 
> must wait until the previous test finished.
> After we add allocation manage function to the cluster, applications can 
> share the cluster and run concurrently. Also if the Test Engineer wants to 
> make sure there is no interference, he/she can move out other tables from 
> this group.
> In groups we use table priority to allocate resource, when system is busy; we 
> can make sure high-priority tables are not affected lower-priority tables
> Different groups can have different region server configurations, some groups 
> optimized for reading can have large block cache size, and others optimized 
> for writing can have large memstore size. 
> Tables and region servers can be moved easily between groups; after changing 
> the configuration, a group can be restarted alone instead of restarting the 
> whole cluster.
> git entry : https://github.com/ICT-Ope/HBase_allocation .
> We hope our work is helpful.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4152) Rename o.a.h.h.regionserver.wal.WALObserver to o.a.h.h.regionserver.wal.WALActionsListener

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4152:
--

  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 
> ---
>
> Key: HBASE-4152
> URL: https://issues.apache.org/jira/browse/HBASE-4152
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Andrew Purtell
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4152.txt
>
>
> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3873) Mavenize Hadoop Snappy JAR/SOs project dependencies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086018#comment-13086018
 ] 

stack commented on HBASE-3873:
--

@Chul  Can you make a new issue with a patch?  I'll do doc changes...  Thanks.

> Mavenize Hadoop Snappy JAR/SOs project dependencies
> ---
>
> Key: HBASE-3873
> URL: https://issues.apache.org/jira/browse/HBASE-3873
> Project: HBase
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.90.2
> Environment: Linux
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>  Labels: build
> Fix For: 0.92.0
>
> Attachments: HBASE-3873.patch, HBASE-3873.patch
>
>
> (This JIRA builds on HBASE-3691)
> I'm working on simplifying how to use Hadoop Snappy from other based maven 
> projects. The idea is that hadoop-snappy JAR and the SOs (snappy and 
> hadoop-snappy) would be picked up from a Maven repository (like any other 
> dependencies). SO files will be picked up based on the architecture where the 
> build is running (32 or 64 bits).
> For Hbase this would remove the need to manually copy snappy JAR and SOs 
> (snappy and hadoop-snappy) into HADOOP_HOME/lib or HBASE_HOME/lib and 
> hadoop-snappy would be handled as a regular maven dependency (with a trick 
> for the SOs file).
> The changes would affect only the pom.xml and the would be in a 'snappy' 
> profile, thus requiring '-Dsnappy' option in Maven invocations to trigger the 
> including of snappy JAR and SOs.
> Because hadoop-snappy (JAR and SOs) are not currently avail in public Maven 
> repos, until that happens, Hbase developer would have to checkout and 'mvn 
> install' hadoop-snappy. Which is (IMO) simpler than what will have to be done 
> in once HBASE-3691 is committed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4190) Coprocessors: pull up some cp constants from cp package to o.a.h.h.HConstants

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4190.
--

   Resolution: Fixed
Fix Version/s: (was: 0.90.5)
   0.92.0
 Hadoop Flags: [Reviewed]

Lets go with this for now Mingjie.  Thanks for the patch.  Applied to TRUNK.

> Coprocessors: pull up some cp constants from cp package to o.a.h.h.HConstants
> -
>
> Key: HBASE-4190
> URL: https://issues.apache.org/jira/browse/HBASE-4190
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Affects Versions: 0.90.4
>Reporter: Mingjie Lai
>Assignee: Mingjie Lai
>Priority: Minor
> Fix For: 0.92.0
>
>
> At HBase-3810, stack gave a comment after patch committed:
> > This is a bit odd where a class in the parent package has references to a 
> > sub package.
> > Should these classes or at least their constants be pulled up to be at same 
> > level as HTableD?
> Create a new jira where the constants will be pulled from 
> o.a.h.h.regionserver.RegionCoprocessorHost to o.a.h.h.HConstants. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3873) Mavenize Hadoop Snappy JAR/SOs project dependencies

[Chul Kwon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086015#comment-13086015
 ] 

Chul Kwon commented on HBASE-3873:
--

I figured out the problem. 
When you give -Dsnappy.prefix=$SNAPPY_BUILD_DIR option, the builder tries to 
find libjvm.so in the Snappy directory, not at 
$JAVA_HOME/jre/lib/{$OS_ARCH}/server.
Therefore, you need to either fix the source code or simply make a symlink in 
the $SNAPPY_BUILD_DIR to your libjvm.so

> Mavenize Hadoop Snappy JAR/SOs project dependencies
> ---
>
> Key: HBASE-3873
> URL: https://issues.apache.org/jira/browse/HBASE-3873
> Project: HBase
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 0.90.2
> Environment: Linux
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
>  Labels: build
> Fix For: 0.92.0
>
> Attachments: HBASE-3873.patch, HBASE-3873.patch
>
>
> (This JIRA builds on HBASE-3691)
> I'm working on simplifying how to use Hadoop Snappy from other based maven 
> projects. The idea is that hadoop-snappy JAR and the SOs (snappy and 
> hadoop-snappy) would be picked up from a Maven repository (like any other 
> dependencies). SO files will be picked up based on the architecture where the 
> build is running (32 or 64 bits).
> For Hbase this would remove the need to manually copy snappy JAR and SOs 
> (snappy and hadoop-snappy) into HADOOP_HOME/lib or HBASE_HOME/lib and 
> hadoop-snappy would be handled as a regular maven dependency (with a trick 
> for the SOs file).
> The changes would affect only the pom.xml and the would be in a 'snappy' 
> profile, thus requiring '-Dsnappy' option in Maven invocations to trigger the 
> including of snappy JAR and SOs.
> Because hadoop-snappy (JAR and SOs) are not currently avail in public Maven 
> repos, until that happens, Hbase developer would have to checkout and 'mvn 
> install' hadoop-snappy. Which is (IMO) simpler than what will have to be done 
> in once HBASE-3691 is committed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4161) Incorrect use of listStatus() in HBase region initialization.

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4161.
--

   Resolution: Fixed
Fix Version/s: 0.90.5
 Hadoop Flags: [Reviewed]

Committed to the branch.  Thanks for the patch Pritam.

> Incorrect use of listStatus() in HBase region initialization.
> -
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Pritam Damania
> Fix For: 0.90.5
>
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children 
> in a "recovered.edits" directory. This directory may not exist and depending 
> on the version of HDFS listStatus() might return null or an exception. If it 
> does throw an exception the entire process of opening the region is aborted, 
> just because the recovered.edits directory is not present.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in HBase region initialization.

[Pritam Damania (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086007#comment-13086007
 ] 

Pritam Damania commented on HBASE-4161:
---

Yes if it does apply to 90.

> Incorrect use of listStatus() in HBase region initialization.
> -
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children 
> in a "recovered.edits" directory. This directory may not exist and depending 
> on the version of HDFS listStatus() might return null or an exception. If it 
> does throw an exception the entire process of opening the region is aborted, 
> just because the recovered.edits directory is not present.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086004#comment-13086004
 ] 

stack commented on HBASE-4209:
--

I do not not remember why it SIGKILLs (I was going to blame hadoop saying we 
copied all from there but I just checked and don't see it there so it must be 
an addition of ours). If plain kill works for you, make a patch and we'll add 
it.  Thanks Roman.

> The HBase hbase-daemon.sh SIGKILLs master when stopping it
> --
>
> Key: HBASE-4209
> URL: https://issues.apache.org/jira/browse/HBASE-4209
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: Roman Shaposhnik
>
> There's a bit of code in hbase-daemon.sh that makes HBase master being 
> SIGKILLed when stopping it rather than trying SIGTERM (like it does for other 
> daemons). When HBase is executed in a standalone mode (and the only daemon 
> you need to run is master) that causes newly created tables to go missing as 
> unflushed data is thrown out. If there was not a good reason to kill master 
> with SIGKILL perhaps we can take that special case out and rely on SIGTERM.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085998#comment-13085998
 ] 

stack commented on HBASE-4117:
--

Thanks Riley.

Over in hbase-880 when we were talking refactoring, base class there was 
Operation -- e.g. 
https://issues.apache.org/jira/secure/attachment/12391309/NewCilentAPIProposoal4.gif
 -- in a few proposals and RowOperation in others.  You already have a Row 
interface, so maybe Operation is ok... (I don't see row specifics going on in 
your Operation instance currently).  Good stuff.

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4209) The HBase hbase-daemon.sh SIGKILLs master when stopping it

[Roman Shaposhnik (JIRA)]

The HBase hbase-daemon.sh SIGKILLs master when stopping it
--

 Key: HBASE-4209
 URL: https://issues.apache.org/jira/browse/HBASE-4209
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: Roman Shaposhnik


There's a bit of code in hbase-daemon.sh that makes HBase master being 
SIGKILLed when stopping it rather than trying SIGTERM (like it does for other 
daemons). When HBase is executed in a standalone mode (and the only daemon you 
need to run is master) that causes newly created tables to go missing as 
unflushed data is thrown out. If there was not a good reason to kill master 
with SIGKILL perhaps we can take that special case out and rely on SIGTERM.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4201) Fix delayed RPC crash

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reassigned HBASE-4201:


Assignee: Vlad Dogaru

Assigning Vlad.

> Fix delayed RPC crash
> -
>
> Key: HBASE-4201
> URL: https://issues.apache.org/jira/browse/HBASE-4201
> Project: HBase
>  Issue Type: Bug
>  Components: ipc
>Reporter: Vlad Dogaru
>Assignee: Vlad Dogaru
>Priority: Minor
>
> Delayed RPC crashes if return value is not delayed and endDelay is called 
> before the actual function returns a value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4201) Fix delayed RPC crash

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085994#comment-13085994
 ] 

jirapos...@reviews.apache.org commented on HBASE-4201:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1519/#review1486
---

Ship it!


LGTM

- Michael


On 2011-08-16 01:49:32, Vlad Dogaru wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1519/
bq.  ---
bq.  
bq.  (Updated 2011-08-16 01:49:32)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  This was initially part of the larger patch for HBASE-4192, but it is 
semantically independent and I split it to a new patch and added a unit test at 
Todd's suggestion.
bq.  
bq.  
bq.  This addresses bug HBASE-4201.
bq.  https://issues.apache.org/jira/browse/HBASE-4201
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 7117bce 
bq.src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java 0b21c6c 
bq.  
bq.  Diff: https://reviews.apache.org/r/1519/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Ran TestDelayedRpc, added a new test case for this situation.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Vlad
bq.  
bq.



> Fix delayed RPC crash
> -
>
> Key: HBASE-4201
> URL: https://issues.apache.org/jira/browse/HBASE-4201
> Project: HBase
>  Issue Type: Bug
>  Components: ipc
>Reporter: Vlad Dogaru
>Priority: Minor
>
> Delayed RPC crashes if return value is not delayed and endDelay is called 
> before the actual function returns a value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4201) Fix delayed RPC crash

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4201:
-

Hadoop Flags: [Reviewed]
  Status: Patch Available  (was: Open)

Marking patch available.

> Fix delayed RPC crash
> -
>
> Key: HBASE-4201
> URL: https://issues.apache.org/jira/browse/HBASE-4201
> Project: HBase
>  Issue Type: Bug
>  Components: ipc
>Reporter: Vlad Dogaru
>Priority: Minor
>
> Delayed RPC crashes if return value is not delayed and endDelay is called 
> before the actual function returns a value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[Riley Patterson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085990#comment-13085990
 ] 

Riley Patterson commented on HBASE-4117:


Jackson does look like it could be a better choice for JSON encoding; I used 
jettison for consistency with my HLog Pretty Printer (HBASE-3968). It seems 
like jackson is more actively maintained, and we should probably be consistent 
throughout the project, so I'll go ahead and change this patch to use jackson 
as well as patching the HLog Pretty Printer to also use jackson.

Separating the logging into its own method sounds like a good idea as well. On 
it.

The slow logging can be turned off by setting hbase.ipc.warn.response.time very 
large. I could add an additional check for nonnegativity which would make 
disabling slow logging a bit more intuitive (set to -1). I do think that this 
feature should be enabled even when DEBUG logging is not, as it could be useful 
for DBA's as well as core developers.

I'm open for suggestions for the name of the super class. Operation sounds fine.

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4202) Check filesystem permissions on startup

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4202:
-

Labels: noob  (was: )

Marking as noob.  Would be sweet if someone tried this on 0.90.x hbase and if 
still an issue, make a change so we exit rather than do the pathological dance 
Matthias describes above.

> Check filesystem permissions on startup
> ---
>
> Key: HBASE-4202
> URL: https://issues.apache.org/jira/browse/HBASE-4202
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.20.4
> Environment: debian squeeze
>Reporter: Matthias Hofschen
>  Labels: noob
>
> We added a new node to a 44 node cluster starting the datanode, mapred and 
> regionserver processes on it. The Unix filesystem was configured incorrectly, 
> i.e. /tmp was not writable to processes. All three processes had issues with 
> this. Datanode and mapred shutdown on exception.
> Regionserver did not stop, in fact reported to master that its up without 
> regions. So master assigned regions to it. Regionserver would not accept 
> them, resulting in a constant assign, reject, reassign cycle, that put many 
> regions into a state of not being available. There are no logs about this, 
> but we could observer the regioncount fluctuate by hundredths of regions and 
> the application throwing many NotServingRegion exceptions.  
> In fact to the master process the regionserver looked fine, so it was trying 
> to send regions its way. Regionserver rejected them. So the master/balancer 
> was going into a assign/reassign cycle destabilizing the cluster. Many puts 
> and gets simply failed with NotServingRegionExceptions and took a long time 
> to complete.
> Exception from regionserver:
> 2011-08-06 23:57:13,953 INFO 
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, 
> state: SyncConnected, type: NodeCreated, path: /hbase/master
> 2011-08-06 23:57:13,957 INFO 
> org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at 
> 17.1.0.1:6 that we are up
> 2011-08-06 23:57:13,957 INFO 
> org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at 
> 17.1.0.1:6 that we are up
> 2011-08-07 00:07:39.648::INFO:  Logging to STDERR via 
> org.mortbay.log.StdErrLog
> 2011-08-07 00:07:39.712::INFO:  jetty-6.1.14
> 2011-08-07 00:07:39.742::WARN:  tmpdir
> java.io.IOException: Permission denied
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.checkAndCreate(File.java:1704)
> at java.io.File.createTempFile(File.java:1792)
> at java.io.File.createTempFile(File.java:1828)
> at 
> org.mortbay.jetty.webapp.WebAppContext.getTempDirectory(WebAppContext.java:745)
> at 
> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:458)
> at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at 
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at 
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at 
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.apache.hadoop.http.HttpServer.start(HttpServer.java:461)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.startServiceThreads(HRegionServer.java:1168)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.init(HRegionServer.java:792)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
> at java.lang.Thread.run(Thread.java:619)
> Exception from datanode:
> 2011-08-06 23:37:20,444 INFO org.apache.hadoop.http.HttpServer: Jetty bound 
> to port 50075
> 2011-08-06 23:37:20,444 INFO org.mortbay.log: jetty-6.1.14
> 2011-08-06 23:37:20,469 WARN org.mortbay.log: tmpdir
> java.io.IOException: Permission denied
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.checkAndCreate(File.java:1704)
> at java.io.File.createTempFile(File.java:1792)
> at java.io.File.createTempFile(File.java:1828)
> at 
> org.mortbay.jetty.webapp.WebAppContext.getTempDirectory(WebAppContext.java:745)
> at 
> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:458)
> at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at 
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at 
> org.mortbay.jetty.handler.Con

[jira] [Commented] (HBASE-4207) Run test suite in parallel, multiple concurrent test instances.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085948#comment-13085948
 ] 

stack commented on HBASE-4207:
--

Its doing some kinda weird pause every so often.  What I have attached seemed 
to run longer than serial running of tests.  Let me mess some more.

> Run test suite in parallel, multiple concurrent test instances.
> ---
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests 
> in parallel.   I'm trying it.  I'll attach the patch to do classes in 
> parallel (as opposed to methods) with four threads per core max.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4208) hbase book - several updates (8-16-2011)

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085946#comment-13085946
 ] 

stack commented on HBASE-4208:
--

+1  Nice.  Terse.  Useful.

> hbase book - several updates (8-16-2011)
> 
>
> Key: HBASE-4208
> URL: https://issues.apache.org/jira/browse/HBASE-4208
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: docbkx_HBASE_4208.patch
>
>
> performance.xml
> * adding a 'deleting from hbase' section
> ** table-as-queue example.
> ** htable.delete(delete) executes RPC each invocation
> book.xml
> * updating schema design...
> ** TTL
> ** more explicit examples of attributes/cf's for keysize section.
> ** nit capitalization of 'storefile' to 'StoreFile'
> * mapreduce
> ** formatted code example (indenting was honked)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in HBase region initialization.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085943#comment-13085943
 ] 

stack commented on HBASE-4161:
--

No worries Pritam.   You want this applied to 0.90 then?

> Incorrect use of listStatus() in HBase region initialization.
> -
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children 
> in a "recovered.edits" directory. This directory may not exist and depending 
> on the version of HDFS listStatus() might return null or an exception. If it 
> does throw an exception the entire process of opening the region is aborted, 
> just because the recovered.edits directory is not present.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4176) Exposing HBase Filters to the Thrift API

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4176:
-

Attachment: Filter Language(2).xml

Here is the word doc as a docbook article (save as in ooo).  I can integrate 
this as part of the commit (will futz w/ formatting that time too).

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language(2).docx, Filter Language(2).xml, Filter 
> Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[Anirudh Todi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085940#comment-13085940
 ] 

Anirudh Todi commented on HBASE-4176:
-

Hi Stack,

- I can work on converting the doc into the book
- Will also update this JIRA soon with a patch including your comments

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language(2).docx, Filter Language.docx, 
> HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085935#comment-13085935
 ] 

jirapos...@reviews.apache.org commented on HBASE-4176:
--



bq.  On 2011-08-16 19:58:03, Michael Stack wrote:
bq.  > I'm +1 on commit.  Will commit in next day or so unless objection.  Let 
me go work on converting the doc. so can be included in book.   Anirudh, answer 
the below questions if you don't mind.  I don't think any necessitate you 
making new patch.

I mean, I can correct the below on commit.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1326/#review1483
---


On 2011-08-16 02:42:36, Anirudh Todi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1326/
bq.  ---
bq.  
bq.  (Updated 2011-08-16 02:42:36)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  https://issues.apache.org/jira/browse/HBASE-4176: Exposing HBase Filters 
to the Thrift API
bq.  
bq.  Currently, to use any of the filters, one has to explicitly add a scanner 
for the filter in the Thrift API making it messy and long. 
bq.  With this patch, I am trying to add support for all the filters in a clean 
way. 
bq.  The user specifies a filter via a string. The string is parsed on the 
server to construct the filter. More information can be found in the attached 
document named Filter Language
bq.  
bq.  This patch is trying to extend and further the progress made by the 
patches in HBASE-1744
bq.  
bq.  There is document attached to the HBASE-4176 JIRA that describes this 
patch in further detail
bq.  
bq.  
bq.  This addresses bug HBASE-4176.
bq.  https://issues.apache.org/jira/browse/HBASE-4176
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java 
1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FamilyFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/QualifierFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/RowFilter.java 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueExcludeFilter.java
 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ValueFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/thrift/generated/TScan.java 
1158073 
bq./src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift 1158073 
bq./src/main/ruby/hbase/table.rb 1158073 
bq./src/main/ruby/shell/commands/scan.rb 1158073 
bq./src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java 
1158073 
bq.  
bq.  Diff: https://reviews.apache.org/r/1326/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  patch includes one test: TestParseFilter.java
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Anirudh
bq.  
bq.



> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085932#comment-13085932
 ] 

jirapos...@reviews.apache.org commented on HBASE-4176:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1326/#review1483
---

Ship it!


I'm +1 on commit.  Will commit in next day or so unless objection.  Let me go 
work on converting the doc. so can be included in book.   Anirudh, answer the 
below questions if you don't mind.  I don't think any necessitate you making 
new patch.


/src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java


Nice.



/src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java


Should this be abstract?



/src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java


Why define these as ints when you have it below in byte array?  Same for 
'SKIP'.   We'll only work if all is upper-case?  Thats OK?  Same for AND and 
OR.  I'm ok w/ that long as its called out in doc.



/src/main/ruby/shell/commands/scan.rb


Now I understand what Todd was going on about -- he was suggesting you 
point at the hbase doc rather than at the issue.  That we can fix easy.



/src/main/ruby/shell/commands/scan.rb


What you mean by #2 here?  You mean stating filters explicitly rather than 
via your new string mechanism here?

We should swap the order here so it agrees with the example below.


- Michael


On 2011-08-16 02:42:36, Anirudh Todi wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1326/
bq.  ---
bq.  
bq.  (Updated 2011-08-16 02:42:36)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  https://issues.apache.org/jira/browse/HBASE-4176: Exposing HBase Filters 
to the Thrift API
bq.  
bq.  Currently, to use any of the filters, one has to explicitly add a scanner 
for the filter in the Thrift API making it messy and long. 
bq.  With this patch, I am trying to add support for all the filters in a clean 
way. 
bq.  The user specifies a filter via a string. The string is parsed on the 
server to construct the filter. More information can be found in the attached 
document named Filter Language
bq.  
bq.  This patch is trying to extend and further the progress made by the 
patches in HBASE-1744
bq.  
bq.  There is document attached to the HBASE-4176 JIRA that describes this 
patch in further detail
bq.  
bq.  
bq.  This addresses bug HBASE-4176.
bq.  https://issues.apache.org/jira/browse/HBASE-4176
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java 
1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FamilyFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java 1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/QualifierFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop/hbase/filter/RowFilter.java 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueExcludeFilter.java
 1158073 
bq.
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java 
1158073 
bq./src/main/java/org/apache/hadoop

[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in HBase region initialization.

[Pritam Damania (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085931#comment-13085931
 ] 

Pritam Damania commented on HBASE-4161:
---

@Stack : Yes you are right, sorry I didn't look at the right branch.

> Incorrect use of listStatus() in HBase region initialization.
> -
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children 
> in a "recovered.edits" directory. This directory may not exist and depending 
> on the version of HDFS listStatus() might return null or an exception. If it 
> does throw an exception the entire process of opening the region is aborted, 
> just because the recovered.edits directory is not present.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4208) hbase book - several updates (8-16-2011)

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4208:
-

Status: Patch Available  (was: Open)

> hbase book - several updates (8-16-2011)
> 
>
> Key: HBASE-4208
> URL: https://issues.apache.org/jira/browse/HBASE-4208
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: docbkx_HBASE_4208.patch
>
>
> performance.xml
> * adding a 'deleting from hbase' section
> ** table-as-queue example.
> ** htable.delete(delete) executes RPC each invocation
> book.xml
> * updating schema design...
> ** TTL
> ** more explicit examples of attributes/cf's for keysize section.
> ** nit capitalization of 'storefile' to 'StoreFile'
> * mapreduce
> ** formatted code example (indenting was honked)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4208) hbase book - several updates (8-16-2011)

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4208:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> hbase book - several updates (8-16-2011)
> 
>
> Key: HBASE-4208
> URL: https://issues.apache.org/jira/browse/HBASE-4208
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: docbkx_HBASE_4208.patch
>
>
> performance.xml
> * adding a 'deleting from hbase' section
> ** table-as-queue example.
> ** htable.delete(delete) executes RPC each invocation
> book.xml
> * updating schema design...
> ** TTL
> ** more explicit examples of attributes/cf's for keysize section.
> ** nit capitalization of 'storefile' to 'StoreFile'
> * mapreduce
> ** formatted code example (indenting was honked)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4208) hbase book - several updates (8-16-2011)

[Doug Meil (JIRA)]

hbase book - several updates (8-16-2011)


 Key: HBASE-4208
 URL: https://issues.apache.org/jira/browse/HBASE-4208
 Project: HBase
  Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
 Attachments: docbkx_HBASE_4208.patch

performance.xml
* adding a 'deleting from hbase' section
** table-as-queue example.
** htable.delete(delete) executes RPC each invocation

book.xml
* updating schema design...
** TTL
** more explicit examples of attributes/cf's for keysize section.
** nit capitalization of 'storefile' to 'StoreFile'
* mapreduce
** formatted code example (indenting was honked)



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4208) hbase book - several updates (8-16-2011)

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4208:
-

Attachment: docbkx_HBASE_4208.patch

> hbase book - several updates (8-16-2011)
> 
>
> Key: HBASE-4208
> URL: https://issues.apache.org/jira/browse/HBASE-4208
> Project: HBase
>  Issue Type: Improvement
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: docbkx_HBASE_4208.patch
>
>
> performance.xml
> * adding a 'deleting from hbase' section
> ** table-as-queue example.
> ** htable.delete(delete) executes RPC each invocation
> book.xml
> * updating schema design...
> ** TTL
> ** more explicit examples of attributes/cf's for keysize section.
> ** nit capitalization of 'storefile' to 'StoreFile'
> * mapreduce
> ** formatted code example (indenting was honked)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4207) Run test suite in parallel, multiple concurrent test instances.

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4207:
-

Attachment: parallel.build.txt

Some pom machinations to add parallel test

> Run test suite in parallel, multiple concurrent test instances.
> ---
>
> Key: HBASE-4207
> URL: https://issues.apache.org/jira/browse/HBASE-4207
> Project: HBase
>  Issue Type: Task
>  Components: test
>Reporter: stack
> Attachments: parallel.build.txt
>
>
> From a suggestion by Lohit up on the list, surefire allows running unit tests 
> in parallel.   I'm trying it.  I'll attach the patch to do classes in 
> parallel (as opposed to methods) with four threads per core max.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4207) Run test suite in parallel, multiple concurrent test instances.

[stack (JIRA)]

Run test suite in parallel, multiple concurrent test instances.
---

 Key: HBASE-4207
 URL: https://issues.apache.org/jira/browse/HBASE-4207
 Project: HBase
  Issue Type: Task
  Components: test
Reporter: stack
 Attachments: parallel.build.txt

>From a suggestion by Lohit up on the list, surefire allows running unit tests 
>in parallel.   I'm trying it.  I'll attach the patch to do classes in parallel 
>(as opposed to methods) with four threads per core max.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4203) While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085900#comment-13085900
 ] 

stack commented on HBASE-4203:
--

Your explanation above helps a lot.  Now its clear what you are going on about. 
 So,  how do you distingush between the case where a master comes on line and 
the RIT is OPENING but NO exception has happened processing .META... its just a 
bit slow and will move to OPENED in a little while... how you distinguish this 
case form the pathological case you describe above where we have to wait 30 
mins?

Otherwise, yes, this is important one to fix.  Good on you Ram.

> While master restarts and if the META region's state is OPENING then master 
> cannot assign META until timeout monitor deducts
> 
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode 
> will be OPENING.
> 3. If at this point the master restarts then the master will start processing 
> the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout 
> monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(180 sec/30 min) then it will 
> take 30 mins to open the META region itself.
> Soln:
> 
> Better not to wait for the Timeout monitor period to open catalog tables on 
> Master restart

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4205) Enhance HTable javadoc

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085897#comment-13085897
 ] 

Hudson commented on HBASE-4205:
---

Integrated in HBase-TRUNK #2119 (See 
[https://builds.apache.org/job/HBase-TRUNK/2119/])
HBASE-4205 Enhance HTable javadoc

stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* /hbase/trunk/CHANGES.txt


> Enhance HTable javadoc
> --
>
> Key: HBASE-4205
> URL: https://issues.apache.org/jira/browse/HBASE-4205
> Project: HBase
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 0.92.0
>Reporter: Eric Charles
>Assignee: Eric Charles
>Priority: Minor
> Fix For: 0.90.5
>
> Attachments: HBASE-4205_01.patch
>
>
> Better document HTable is not thread-safe.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4206) jenkins hash implementation uses longs unnecessarily

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085894#comment-13085894
 ] 

stack commented on HBASE-4206:
--

Any chance of a patch Ron that fixes the above?  Thanks.

> jenkins hash implementation uses longs unnecessarily
> 
>
> Key: HBASE-4206
> URL: https://issues.apache.org/jira/browse/HBASE-4206
> Project: HBase
>  Issue Type: Improvement
>  Components: util
>Reporter: Ron Yang
>Priority: Minor
>
> I don't believe you need to use long for a,b,c and as a result no longer need 
> to & against INT_MASK.
> At a minimum the private static longs should be made final, and the "main" 
> method should not print the absolute value of the hash but instead use 
> something like Integer.toHexString

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4175) Fix FSUtils.createTableDescriptor()

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085892#comment-13085892
 ] 

stack commented on HBASE-4175:
--

Patch looks generally good to me.  You might fix some javadoc if you make 
another patch version and add doc'ing of the new boolean return.  In future, 
you might add override that doesn't take the boolean which defaults to 'false' 
then you would have had less code to change.  Good stuff Ram.

> Fix FSUtils.createTableDescriptor()
> ---
>
> Key: HBASE-4175
> URL: https://issues.apache.org/jira/browse/HBASE-4175
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4175.patch, HBASE-4175_1.patch
>
>
> Currently createTableDescriptor() doesn't return anything.
> The caller wouldn't know whether the descriptor is created or not. See 
> exception handling:
> {code}
>} catch(IOException ioe) {
>  LOG.info("IOException while trying to create tableInfo in HDFS", ioe);
>}
> {code}
> We should return a boolean.
> If the table descriptor exists already, maybe we should deserialize from hdfs 
> and compare with htableDescriptor argument. If they differ, I am not sure 
> what the proper action would be.
> Maybe we can add a boolean argument, force, to createTableDescriptor(). When 
> force is true, existing table descriptor would be overwritten.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4071) Data GC: Remove all versions > TTL EXCEPT the last written version

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085889#comment-13085889
 ] 

stack commented on HBASE-4071:
--

bq. In both cases I think only at compaction time do we have enough information 
to remove expired cells (if minversions is >0).

Fair enough.

bq. So you think test coverage of the existing functionality is sufficient? 
That is very good to know.

IIRC, there are some decent tests for this stuff.  Also, the operation is so 
fundamental that if broke, it'd bubble up as a broken test somewhere (the 
connection back down into this code might be an indirection on top of an 
indirection, but a test will break I'd say).

bq. What's the general feeling? Should I aim for minimal intrusion or attempt 
to do a bit refactoring to abstract these policies into an interface? Leaning 
towards the latter, but on the other hand the change would be more risky.

I'm for the latter.  We're trying to push out next major revision of hbase.  
It'll get some scrutiny and testing so if you've broken something it should 
show (or if not, our coverage needs improving).

Good stuff LarsH.

> Data GC: Remove all versions > TTL EXCEPT the last written version
> --
>
> Key: HBASE-4071
> URL: https://issues.apache.org/jira/browse/HBASE-4071
> Project: HBase
>  Issue Type: New Feature
>Reporter: stack
> Attachments: MinVersions.diff
>
>
> We were chatting today about our backup cluster.  What we want is to be able 
> to restore the dataset from any point of time but only within a limited 
> timeframe -- say one week.  Thereafter, if the versions are older than one 
> week, rather than as we do with TTL where we let go of all versions older 
> than TTL, instead, let go of all versions EXCEPT the last one written.  So, 
> its like versions==1 when TTL > one week.  We want to allow that if an error 
> is caught within a week of its happening -- user mistakenly removes a 
> critical table -- then we'll be able to restore up the the moment just before 
> catastrophe hit otherwise, we keep one version only.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085855#comment-13085855
 ] 

stack commented on HBASE-4117:
--

How is jettison related to jackson, the json lib we use elsewhere in hbase?  It 
throws events as it trips over the json.  Can jackson do this?

The clause under  'if (processingTime > warnResponseTime) {' could be broken 
out into a method?

Should the above test include if logging is DEBUG or more or is thought that 
this slow logging will be always on?

You don't want to log something here:

{code}
+} catch (JSONException e) {
+  // ignore
+}
{code}

Could be confusing if I'm expecting logging and its not coming out -- would 
appreciate the failure being logged.  There is another instance later in this 
code block.

Would 'Operation' be better than 'DatabaseCommand'?  Currently this superclass 
factors into a superclass various kinds of representations of hbase operations 
-- tostring, fingerprint as map, json -- etc.  Thats fine for now.  It could do 
more with time.

I like this getFingerprint.

getDetails seems too generic a name for the method.  If you are going to go 
generic, toMap to go with toString might be better?  (Oh, nvm, I see you went 
one better and made the toString member of the superclass -- good).

Are you removing toString from Delete?  Thats probably not good.  The toString 
could use your fingerprinting.

Shouldn't the methods in Row be added to your new base class?  Row Interface 
should only have 'row' things in it, not getDetails and getFamilyMap.

A couple of tests would be good where you do some toStringing and toJSON'ing 
and verify its doing right thing.

Nice patch Riley.  Nice functionality.






> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4071) Data GC: Remove all versions > TTL EXCEPT the last written version

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085853#comment-13085853
 ] 

Lars Hofhansl commented on HBASE-4071:
--

Thanks Stack.

re if (currentCount <= minVersions): in this case the count was already 
incremented by the maxversions check (which is just outside of the context 
provided by the diff).

re expunging expired rows on flush: Currently it seems this is done as an 
optimization.
The reasons why believe this cannot be done if minversions>0 are: (1) The cache 
might not have all version, so I cannot count the versions to determine the 
cut-off point. (2) Even if we have minversions=1, there is no guarantee that 
the versions in the cache include the latest one (puts could have been 
backdated).
In both cases I think only at compaction time do we have enough information to 
remove expired cells (if minversions is >0).

re comments: Of course. That's why there is version control :)
I just left these in as comments to have a place where I can put a comment as 
to why I believe these are not necessary.

So you think test coverage of the existing functionality is sufficient? That is 
very good to know.
I'll add tests for the new functionality.

What's the general feeling? Should I aim for minimal intrusion or attempt to do 
a bit refactoring to abstract these policies into an interface? Leaning towards 
the latter, but on the other hand the change would be more risky.

> Data GC: Remove all versions > TTL EXCEPT the last written version
> --
>
> Key: HBASE-4071
> URL: https://issues.apache.org/jira/browse/HBASE-4071
> Project: HBase
>  Issue Type: New Feature
>Reporter: stack
> Attachments: MinVersions.diff
>
>
> We were chatting today about our backup cluster.  What we want is to be able 
> to restore the dataset from any point of time but only within a limited 
> timeframe -- say one week.  Thereafter, if the versions are older than one 
> week, rather than as we do with TTL where we let go of all versions older 
> than TTL, instead, let go of all versions EXCEPT the last one written.  So, 
> its like versions==1 when TTL > one week.  We want to allow that if an error 
> is caught within a week of its happening -- user mistakenly removes a 
> critical table -- then we'll be able to restore up the the moment just before 
> catastrophe hit otherwise, we keep one version only.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4175) Fix FSUtils.createTableDescriptor()

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085846#comment-13085846
 ] 

ramkrishna.s.vasudevan commented on HBASE-4175:
---

@Ted,
Actually there were 2 catch blocks with the same content.
I removed the first one that comes in 
createTableDescriptor(HTableDescriptor htableDescriptor, Configuration conf,
  boolean forceCreation)
but did not remove it from 
createTableDescriptor(FileSystem fs, Path rootdir,HTableDescriptor 
htableDescriptor, boolean forceCreation)
-> This is because there are scenarios where we can directly use the second api 
for createTableDescriptor(In testcases it is getting used directly).  So we 
need some mechanism to know what exception has come.

If we use the first createTableDescriptor() it is any way calling the second 
one so the catch block is not needed.
Is it ok Ted?

> Fix FSUtils.createTableDescriptor()
> ---
>
> Key: HBASE-4175
> URL: https://issues.apache.org/jira/browse/HBASE-4175
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4175.patch, HBASE-4175_1.patch
>
>
> Currently createTableDescriptor() doesn't return anything.
> The caller wouldn't know whether the descriptor is created or not. See 
> exception handling:
> {code}
>} catch(IOException ioe) {
>  LOG.info("IOException while trying to create tableInfo in HDFS", ioe);
>}
> {code}
> We should return a boolean.
> If the table descriptor exists already, maybe we should deserialize from hdfs 
> and compare with htableDescriptor argument. If they differ, I am not sure 
> what the proper action would be.
> Maybe we can add a boolean argument, force, to createTableDescriptor(). When 
> force is true, existing table descriptor would be overwritten.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085841#comment-13085841
 ] 

ramkrishna.s.vasudevan commented on HBASE-4015:
---

@Stack,
Have made the basic changes as discussed.  Will verify scenarios and will 
upload the patch sooner.

> Refactor the TimeoutMonitor to make it less racy
> 
>
> Key: HBASE-4015
> URL: https://issues.apache.org/jira/browse/HBASE-4015
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 0.90.3
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBASE-4015_1_trunk.patch, Timeoutmonitor with state 
> diagrams.pdf
>
>
> The current implementation of the TimeoutMonitor acts like a race condition 
> generator, mostly making things worse rather than better. It does it's own 
> thing for a while without caring for what's happening in the rest of the 
> master.
> The first thing that needs to happen is that the regions should not be 
> processed in one big batch, because that sometimes can take minutes to 
> process (meanwhile a region that timed out opening might have opened, then 
> what happens is it will be reassigned by the TimeoutMonitor generating the 
> never ending PENDING_OPEN situation).
> Those operations should also be done more atomically, although I'm not sure 
> how to do it in a scalable way in this case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4206) jenkins hash implementation uses longs unnecessarily

[Ron Yang (JIRA)]

jenkins hash implementation uses longs unnecessarily


 Key: HBASE-4206
 URL: https://issues.apache.org/jira/browse/HBASE-4206
 Project: HBase
  Issue Type: Improvement
  Components: util
Reporter: Ron Yang
Priority: Minor


I don't believe you need to use long for a,b,c and as a result no longer need 
to & against INT_MASK.

At a minimum the private static longs should be made final, and the "main" 
method should not print the absolute value of the hash but instead use 
something like Integer.toHexString

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4205) Enhance HTable javadoc

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4205.
--

   Resolution: Fixed
Fix Version/s: 0.90.5
 Assignee: Eric Charles
 Hadoop Flags: [Reviewed]

Applied to branch and trunk.  This is an improvement.  Regards close throwing 
IOE, what a caller would do on IOE, I'm not sure... I'm of the mind that we 
shouldn't be throwing exceptions on a close but then, what if exception happens 
because we are flushing buffered edits.  Caller would want to know their edits 
didn't make it across.

Thanks for the patch Eric.

> Enhance HTable javadoc
> --
>
> Key: HBASE-4205
> URL: https://issues.apache.org/jira/browse/HBASE-4205
> Project: HBase
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 0.92.0
>Reporter: Eric Charles
>Assignee: Eric Charles
>Priority: Minor
> Fix For: 0.90.5
>
> Attachments: HBASE-4205_01.patch
>
>
> Better document HTable is not thread-safe.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4204) book.xml - typo in rolling restart

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085815#comment-13085815
 ] 

stack commented on HBASE-4204:
--

+1

> book.xml - typo in rolling restart
> --
>
> Key: HBASE-4204
> URL: https://issues.apache.org/jira/browse/HBASE-4204
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4204.patch
>
>
> Found by Shrijeet Paliwal
> Following line under 'A.5.1. Rolling Restart' should be :
> $ echo "balance_switch false" | ./bin/hbase
> $ echo "balance_switch false" | ./bin/hbase shell
> -Shrijeet

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-3968) HLog Pretty Printer

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-3968:
--

Fix Version/s: 0.90.4

> HLog Pretty Printer
> ---
>
> Key: HBASE-3968
> URL: https://issues.apache.org/jira/browse/HBASE-3968
> Project: HBase
>  Issue Type: New Feature
>  Components: io, regionserver, util
>Reporter: Nicolas Spiegelberg
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: hbase
> Fix For: 0.90.4
>
> Attachments: HBASE-3968.patch
>
>
> We currently have a rudimentary way to print HLog data, but it is limited and 
> currently prints key-only information. We need extend this functionality, 
> similar to how we developed HFile's pretty printer. Ideas for functionality:
> - filter by sequence_id
> - filter by row / region
> - option to print values in addition to key info
> - option to print output in JSON format (so scripts can easily parse for 
> analysis)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4175) Fix FSUtils.createTableDescriptor()

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085765#comment-13085765
 ] 

Ted Yu commented on HBASE-4175:
---

HBASE-4175_1.patch looks good overall.
In createTableDescriptor(FileSystem fs, ...) the catch block seems unnecessary:
{code}
+} catch (IOException ioe) {
   LOG.info("IOException while trying to create tableInfo in HDFS", ioe);
+  throw ioe;
{code}


> Fix FSUtils.createTableDescriptor()
> ---
>
> Key: HBASE-4175
> URL: https://issues.apache.org/jira/browse/HBASE-4175
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4175.patch, HBASE-4175_1.patch
>
>
> Currently createTableDescriptor() doesn't return anything.
> The caller wouldn't know whether the descriptor is created or not. See 
> exception handling:
> {code}
>} catch(IOException ioe) {
>  LOG.info("IOException while trying to create tableInfo in HDFS", ioe);
>}
> {code}
> We should return a boolean.
> If the table descriptor exists already, maybe we should deserialize from hdfs 
> and compare with htableDescriptor argument. If they differ, I am not sure 
> what the proper action would be.
> Maybe we can add a boolean argument, force, to createTableDescriptor(). When 
> force is true, existing table descriptor would be overwritten.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4205) Enhance HTable javadoc

[Eric Charles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Charles updated HBASE-4205:


Attachment: HBASE-4205_01.patch

Attached HBASE-4205_01.patch proposes the changes related for the non 
thread-safety.

It also points on the close() method that throws IOException. I wonder what we 
should say about that: should a IOException on close() be considered as a 
failed operation? If yes, we need to cleary state this in javadoc.

> Enhance HTable javadoc
> --
>
> Key: HBASE-4205
> URL: https://issues.apache.org/jira/browse/HBASE-4205
> Project: HBase
>  Issue Type: Improvement
>  Components: client
>Affects Versions: 0.92.0
>Reporter: Eric Charles
>Priority: Minor
> Attachments: HBASE-4205_01.patch
>
>
> Better document HTable is not thread-safe.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4205) Enhance HTable javadoc

[Eric Charles (JIRA)]

Enhance HTable javadoc
--

 Key: HBASE-4205
 URL: https://issues.apache.org/jira/browse/HBASE-4205
 Project: HBase
  Issue Type: Improvement
  Components: client
Affects Versions: 0.92.0
Reporter: Eric Charles
Priority: Minor


Better document HTable is not thread-safe.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4204) book.xml - typo in rolling restart

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085751#comment-13085751
 ] 

Hudson commented on HBASE-4204:
---

Integrated in HBase-TRUNK #2118 (See 
[https://builds.apache.org/job/HBase-TRUNK/2118/])
HBASE-4204 small typo in rolling restart

dmeil : 
Files : 
* /hbase/trunk/src/docbkx/book.xml


> book.xml - typo in rolling restart
> --
>
> Key: HBASE-4204
> URL: https://issues.apache.org/jira/browse/HBASE-4204
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4204.patch
>
>
> Found by Shrijeet Paliwal
> Following line under 'A.5.1. Rolling Restart' should be :
> $ echo "balance_switch false" | ./bin/hbase
> $ echo "balance_switch false" | ./bin/hbase shell
> -Shrijeet

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4175) Fix FSUtils.createTableDescriptor()

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4175:
--

Attachment: HBASE-4175_1.patch

> Fix FSUtils.createTableDescriptor()
> ---
>
> Key: HBASE-4175
> URL: https://issues.apache.org/jira/browse/HBASE-4175
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4175.patch, HBASE-4175_1.patch
>
>
> Currently createTableDescriptor() doesn't return anything.
> The caller wouldn't know whether the descriptor is created or not. See 
> exception handling:
> {code}
>} catch(IOException ioe) {
>  LOG.info("IOException while trying to create tableInfo in HDFS", ioe);
>}
> {code}
> We should return a boolean.
> If the table descriptor exists already, maybe we should deserialize from hdfs 
> and compare with htableDescriptor argument. If they differ, I am not sure 
> what the proper action would be.
> Maybe we can add a boolean argument, force, to createTableDescriptor(). When 
> force is true, existing table descriptor would be overwritten.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4175) Fix FSUtils.createTableDescriptor()

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4175:
--

Status: Patch Available  (was: Open)

> Fix FSUtils.createTableDescriptor()
> ---
>
> Key: HBASE-4175
> URL: https://issues.apache.org/jira/browse/HBASE-4175
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Ted Yu
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-4175.patch, HBASE-4175_1.patch
>
>
> Currently createTableDescriptor() doesn't return anything.
> The caller wouldn't know whether the descriptor is created or not. See 
> exception handling:
> {code}
>} catch(IOException ioe) {
>  LOG.info("IOException while trying to create tableInfo in HDFS", ioe);
>}
> {code}
> We should return a boolean.
> If the table descriptor exists already, maybe we should deserialize from hdfs 
> and compare with htableDescriptor argument. If they differ, I am not sure 
> what the proper action would be.
> Maybe we can add a boolean argument, force, to createTableDescriptor(). When 
> force is true, existing table descriptor would be overwritten.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4117) Slow Query Log

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085710#comment-13085710
 ] 

Ted Yu commented on HBASE-4117:
---

Please combine javadoc in one line:
{code}
+/** Default value for above param */
+private static final int DEFAULT_WARN_RESPONSE_TIME = 1000; // milliseconds
{code}
{code}
+public abstract class DatabaseCommand {
{code}
Maybe we can find a better name for the above class whose purpose is for 
logging and debugging ?

> Slow Query Log
> --
>
> Key: HBASE-4117
> URL: https://issues.apache.org/jira/browse/HBASE-4117
> Project: HBase
>  Issue Type: New Feature
>  Components: ipc
>Reporter: Riley Patterson
>Assignee: Riley Patterson
>Priority: Minor
>  Labels: client, ipc
> Attachments: HBASE-4117.patch
>
>
> Produce log messages for slow queries. The RPC server will decide what is 
> slow based on a configurable "warn response time" parameter. Queries 
> designated as slow will then output a "response too slow" message followed by 
> a fingerprint of the query, and a summary limited in size by another 
> configurable parameter (to limit log spamming).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085701#comment-13085701
 ] 

Ted Yu commented on HBASE-4095:
---

Config parameters are renamed in HLog but not in the test, please make them 
consistent.
{code}
+"hbase.regionserver.hlog.lowreplication.tolerable"
{code}
A better name would be hbase.regionserver.hlog.tolerable.lowreplication
{code}
+"hbase.regionserver.hlog.lowreplication.rolls", 5);
{code}
Would hbase.regionserver.hlog.lowreplication.rolllimit be more informative ?

In batchWriteAndWait() remaining always decrements by 200. This can be made 
more accurate through calling System.currentTimeMillis()

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.5
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HBase-4095-V7-branch.patch, 
> HBase-4095-V7-trunk.patch, HBase-4095-V8-branch.patch, 
> HBase-4095-V8-trunk.patch, HlogFileIsVeryLarge.gif, 
> LatestLLTResults-20110810.rar, RelatedLogs2011-07-28.txt, 
> TestResultForPatch-V4.rar, hbase-root-regionserver-193-195-5-111.rar, 
> surefire-report-V5-trunk.html, surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:

[jira] [Commented] (HBASE-4204) book.xml - typo in rolling restart

[Doug Meil (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085653#comment-13085653
 ] 

Doug Meil commented on HBASE-4204:
--

The patch didn't auto-apply, so I made the change manually.  It's a one-liner, 
though.  
Note:  I had Stack review this patch last night since he did the rolling 
restart description originally.

> book.xml - typo in rolling restart
> --
>
> Key: HBASE-4204
> URL: https://issues.apache.org/jira/browse/HBASE-4204
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4204.patch
>
>
> Found by Shrijeet Paliwal
> Following line under 'A.5.1. Rolling Restart' should be :
> $ echo "balance_switch false" | ./bin/hbase
> $ echo "balance_switch false" | ./bin/hbase shell
> -Shrijeet

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4204) book.xml - typo in rolling restart

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4204:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> book.xml - typo in rolling restart
> --
>
> Key: HBASE-4204
> URL: https://issues.apache.org/jira/browse/HBASE-4204
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4204.patch
>
>
> Found by Shrijeet Paliwal
> Following line under 'A.5.1. Rolling Restart' should be :
> $ echo "balance_switch false" | ./bin/hbase
> $ echo "balance_switch false" | ./bin/hbase shell
> -Shrijeet

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4204) book.xml - typo in rolling restart

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4204:
-

Status: Patch Available  (was: Open)

> book.xml - typo in rolling restart
> --
>
> Key: HBASE-4204
> URL: https://issues.apache.org/jira/browse/HBASE-4204
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4204.patch
>
>
> Found by Shrijeet Paliwal
> Following line under 'A.5.1. Rolling Restart' should be :
> $ echo "balance_switch false" | ./bin/hbase
> $ echo "balance_switch false" | ./bin/hbase shell
> -Shrijeet

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4204) book.xml - typo in rolling restart

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-4204:
-

Attachment: book_HBASE_4204.patch

> book.xml - typo in rolling restart
> --
>
> Key: HBASE-4204
> URL: https://issues.apache.org/jira/browse/HBASE-4204
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: book_HBASE_4204.patch
>
>
> Found by Shrijeet Paliwal
> Following line under 'A.5.1. Rolling Restart' should be :
> $ echo "balance_switch false" | ./bin/hbase
> $ echo "balance_switch false" | ./bin/hbase shell
> -Shrijeet

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4204) book.xml - typo in rolling restart

[Doug Meil (JIRA)]

book.xml - typo in rolling restart
--

 Key: HBASE-4204
 URL: https://issues.apache.org/jira/browse/HBASE-4204
 Project: HBase
  Issue Type: Bug
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor


Found by Shrijeet Paliwal


Following line under 'A.5.1. Rolling Restart' should be :

$ echo "balance_switch false" | ./bin/hbase

$ echo "balance_switch false" | ./bin/hbase shell

-Shrijeet



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Attachment: HBase-4095-V8-trunk.patch
HBase-4095-V8-branch.patch

I fixed some spellings and sentences acording to J-d's suggestion. About the 
sleepings, I have made some changes. Added a timeout mechanism to decrease the 
sleeping time.
But some sleepings is also required. Like the following:
{noformat}
 assertTrue(dfsCluster.stopDataNode(pipeline[1].getName()) != null);
 Thread.sleep(1);
{noformat}
Maybe there's some differece according to different env. But some code like 
this is already there. 1 maybe a empirical value.
I tried to use mockito. But it makes things more difficult in this scenario.
{quote}
This is not how we do brackets:
+  public boolean isFewerReplicasRollEnabled()
+  {
+return fewerReplicasRollEnabled;
+  }
{quote}
I just got the Eclipse-based Apache Formatter xml and applied it in my editor. 
Before this, I format the code mannually:(. So this problem will not happen 
again.
{quote}
Where you remove an assertion. Can you leave that in?
{quote} 
I've putted them back.

stack, J-d, expecting your review again. Thanks a lot.

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.5
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HBase-4095-V7-branch.patch, 
> HBase-4095-V7-trunk.patch, HBase-4095-V8-branch.patch, 
> HBase-4095-V8-trunk.patch, HlogFileIsVeryLarge.gif, 
> LatestLLTResults-20110810.rar, RelatedLogs2011-07-28.txt, 
> TestResultForPatch-V4.rar, hbase-root-regionserver-193-195-5-111.rar, 
> surefire-report-V5-trunk.html, surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:2

[jira] [Commented] (HBASE-4203) While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085610#comment-13085610
 ] 

ramkrishna.s.vasudevan commented on HBASE-4203:
---

In the above logs I had reduced the timeout monitor period to a lesser value.
Hence you can see that the META region assignment started in about 18mins

> While master restarts and if the META region's state is OPENING then master 
> cannot assign META until timeout monitor deducts
> 
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode 
> will be OPENING.
> 3. If at this point the master restarts then the master will start processing 
> the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout 
> monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(180 sec/30 min) then it will 
> take 30 mins to open the META region itself.
> Soln:
> 
> Better not to wait for the Timeout monitor period to open catalog tables on 
> Master restart

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4203) While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-4203:
--

Description: 
1. Start Master and 2 RS.
2. If any exception happens while opening the META region the state in znode 
will be OPENING.
3. If at this point the master restarts then the master will start processing 
the regions in RIT.
4. If the znode is found to be in OPENING then master waits for timeout monitor 
to deduct and then call opening.
5. If default timeout monitor is configured(180 sec/30 min) then it will 
take 30 mins to open the META region itself.
Soln:

Better not to wait for the Timeout monitor period to open catalog tables on 
Master restart

  was:
1. Start Master and 2 RS.
2. If any exception happens while opening the META region the state in znode 
will be OPENING.
3. If at this point the master restarts then the master will start processing 
the regions in RIT.
4. If the znode is found to be in OPENING then master waits for timeout monitor 
to deduct and then call opening.
5. If default timeout monitor is configured then it may take long time to open 
the META region.

Soln:

Better not to wait for the Timeout monitor period to open catalog tables on 
Master restart


> While master restarts and if the META region's state is OPENING then master 
> cannot assign META until timeout monitor deducts
> 
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode 
> will be OPENING.
> 3. If at this point the master restarts then the master will start processing 
> the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout 
> monitor to deduct and then call opening.
> 5. If default timeout monitor is configured(180 sec/30 min) then it will 
> take 30 mins to open the META region itself.
> Soln:
> 
> Better not to wait for the Timeout monitor period to open catalog tables on 
> Master restart

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4203) While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085606#comment-13085606
 ] 

ramkrishna.s.vasudevan commented on HBASE-4203:
---

Attaching the logs to show the impact of how timetaking it is to wait for 
timeout monitor to deduct and then assign META

{color:red}2011-08-16 14:41:22,653 DEBUG 
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region 
-ROOT-,,0.70236052 on linux146,60020,1313485860002
2011-08-16 14:59:57,907 INFO org.apache.hadoop.hbase.master.AssignmentManager: 
Regions in transition timed out:  .META.,,1.1028785192 state=OPENING, ts=0, 
server=linux76,60020,1313485434045{color}
2011-08-16 14:59:57,907 INFO org.apache.hadoop.hbase.master.AssignmentManager: 
Region has been OPENING for too long, reassigning region=.META.,,1.1028785192
2011-08-16 14:59:57,912 INFO org.apache.hadoop.hbase.master.AssignmentManager: 
Successfully transitioned region=.META.,,1.1028785192 into OFFLINE and forcing 
a new assignment
2011-08-16 14:59:57,912 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Forcing OFFLINE; was=.META.,,1.1028785192 state=OPENING, ts=0, 
server=linux76,60020,1313485434045
2011-08-16 14:59:57,912 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Handling transition=M_ZK_REGION_OFFLINE, server=linux76,6,1313485532331, 
region=1028785192/.META.
2011-08-16 14:59:58,918 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
No previous transition plan was found (or we are ignoring an existing plan) for 
.META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, 
dest=linux146,60020,1313485860002; 1 (online=1, exclude=null) available servers
2011-08-16 14:59:58,918 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Assigning region .META.,,1.1028785192 to linux146,60020,1313485860002
2011-08-16 14:59:58,918 DEBUG org.apache.hadoop.hbase.master.ServerManager: New 
connection to linux146,60020,1313485860002
2011-08-16 15:00:04,653 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Handling transition=RS_ZK_REGION_OPENING, server=linux146,60020,1313485860002, 
region=1028785192/.META.
2011-08-16 15:00:04,681 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Handling transition=RS_ZK_REGION_OPENING, server=linux146,60020,1313485860002, 
region=1028785192/.META.
2011-08-16 15:00:04,760 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Handling transition=RS_ZK_REGION_OPENED, server=linux146,60020,1313485860002, 
region=1028785192/.META.
2011-08-16 15:00:04,762 DEBUG 
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED 
event for 1028785192; deleting unassigned node


> While master restarts and if the META region's state is OPENING then master 
> cannot assign META until timeout monitor deducts
> 
>
> Key: HBASE-4203
> URL: https://issues.apache.org/jira/browse/HBASE-4203
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> 1. Start Master and 2 RS.
> 2. If any exception happens while opening the META region the state in znode 
> will be OPENING.
> 3. If at this point the master restarts then the master will start processing 
> the regions in RIT.
> 4. If the znode is found to be in OPENING then master waits for timeout 
> monitor to deduct and then call opening.
> 5. If default timeout monitor is configured then it may take long time to 
> open the META region.
> Soln:
> 
> Better not to wait for the Timeout monitor period to open catalog tables on 
> Master restart

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4203) While master restarts and if the META region's state is OPENING then master cannot assign META until timeout monitor deducts

[ramkrishna.s.vasudevan (JIRA)]

While master restarts and if the META region's state is OPENING then master 
cannot assign META until timeout monitor deducts


 Key: HBASE-4203
 URL: https://issues.apache.org/jira/browse/HBASE-4203
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor


1. Start Master and 2 RS.
2. If any exception happens while opening the META region the state in znode 
will be OPENING.
3. If at this point the master restarts then the master will start processing 
the regions in RIT.
4. If the znode is found to be in OPENING then master waits for timeout monitor 
to deduct and then call opening.
5. If default timeout monitor is configured then it may take long time to open 
the META region.

Soln:

Better not to wait for the Timeout monitor period to open catalog tables on 
Master restart

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira