[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081312#comment-13081312
 ] 

Hudson commented on HBASE-4169:
---

Integrated in HBase-TRUNK #2094 (See 
[https://builds.apache.org/job/HBase-TRUNK/2094/])
HBASE-4169 FSUtils LeaseRecovery for non HDFS FileSystems

stack : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSNonHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, 
> HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081314#comment-13081314
 ] 

stack commented on HBASE-4169:
--

Reverting to fix class name (please base new patch on mine Lohit)

> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, HBASE-4169.1.patch, HBASE-4169.2.patch, 
> HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4160) HBase shell move and online may be unusable if region name or server includes binary-encoded data

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081322#comment-13081322
 ] 

jirapos...@reviews.apache.org commented on HBASE-4160:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1285/#review1329
---

Ship it!


Let me commit (Tests for shell are a tough to add -- there is a test framework 
but not fair asking non-ruby head to write one I'd say).  The close_region case 
looks fine -- the ruby 'string' gets passed through to the hbaseadmin method 
that takes a byte array.

- Michael


On 2011-08-04 01:54:09, jmhsieh wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1285/
bq.  ---
bq.  
bq.  (Updated 2011-08-04 01:54:09)
bq.  
bq.  
bq.  Review request for hbase, Todd Lipcon and Ted Yu.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Similar to HBASE-4115, this entails a conversion of 
org.apache.hadoop.hbase.utils.Bytes.toBytes to a to_java_bytes call in the 
'move' and 'online' call.
bq.  
bq.  Root cause is due to how Bytes.toBytes assumes UTF-8 whild ruby's 
to_java_bytes assumes straight bytes.
bq.  
bq.  
bq.  This addresses bug hbase-4160.
bq.  https://issues.apache.org/jira/browse/hbase-4160
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/ruby/hbase/admin.rb 1d72424 
bq.  
bq.  Diff: https://reviews.apache.org/r/1285/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  For move:
bq.  in shell: issue
bq.  move "\x7f\x80","\x79\x80"
bq.  in clipse set breakpoint on HMaster.move(...); args are byte arrays [127, 
-128], [127, -128]
bq.  
bq.  I can't figure out how to trigger the 'online' call from the shell.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.



> HBase shell move and online may be unusable if region name or server includes 
> binary-encoded data
> -
>
> Key: HBASE-4160
> URL: https://issues.apache.org/jira/browse/HBASE-4160
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.3
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
>Priority: Minor
> Attachments: 
> 0001-HBASE-4160-HBase-shell-move-and-online-may-be-unusab.patch
>
>
> Similar to HBASE-4115, this entails a conversion of 
> org.apache.hadoop.hbase.utils.Bytes.toBytes to a to_java_bytes call in the 
> 'move' and 'online' call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081321#comment-13081321
 ] 

jirapos...@reviews.apache.org commented on HBASE-4176:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1326/
---

Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and Jonathan Gray.


Summary
---

https://issues.apache.org/jira/browse/HBASE-4176: Exposing HBase Filters to the 
Thrift API

Currently, to use any of the filters, one has to explicitly add a scanner for 
the filter in the Thrift API making it messy and long. 
With this patch, I am trying to add support for all the filters in a clean way. 
The user specifies a filter via a string. The string is parsed on the server to 
construct the filter. More information can be found in the attached document 
named Filter Language

This patch is trying to extend and further the progress made by the patches in 
HBASE-1744

There is document attached to the HBASE-4176 JIRA that describes this patch in 
further detail


This addresses bug HBASE-4176.
https://issues.apache.org/jira/browse/HBASE-4176


Diffs
-

  /src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FamilyFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/Filter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FilterList.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/QualifierFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/RowFilter.java 1155098 
  
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueExcludeFilter.java
 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ValueFilter.java 1155098 
  /src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/1326/diff


Testing
---

patch includes one test: TestParseFilter.java


Thanks,

Anirudh



> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4178) Use of Random.nextLong() in HRegionServer.addScanner(...)

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081323#comment-13081323
 ] 

Gary Helmling commented on HBASE-4178:
--

What happens if a region server restarts?  Would we be resetting and starting 
the AtomicLong over again with numbers that were previously handed out?  If so, 
it's possible this change would vastly reduce the assignable space effectively 
used and increase the probability of collisions.

I'm not sure if there's behavior in the client code that would effectively 
invalidate the existing scanner in the case of a server restart -- would have 
to check.  We could also ensure uniqueness (and side-step the counter 
resetting) by checking the server start code either on the client side or 
changing scanner id from long to byte[] and prepending the start code with a 
separator.

The current random assignment does nicely avoid this potential issue, though.  
Yes there _is_ a possibility of collisions.  But is this really an issue that 
needs fixing?  Personally, I'm open to arguments either way.

> Use of Random.nextLong() in HRegionServer.addScanner(...)
> -
>
> Key: HBASE-4178
> URL: https://issues.apache.org/jira/browse/HBASE-4178
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.3
>Reporter: Lars Hofhansl
>Priority: Minor
>
> ScannerIds are currently assigned by getting a random long. While it would be 
> a rare occurrence that two scanners received the same ids on the same region 
> server the results would seem to be... Bad.
> A client scanner would get results from a different server scanner, and maybe 
> only from some of the region servers.
> A safer approach would be using an AtomicLong. We do not have to worry about 
> running of numbers: If we got 1 scanners per second it'd take > 2.9m 
> years to reach 2^63.
> Then again the same reasoning would imply that this collisions would be 
> happening too rarely to be of concern (assuming a good random number 
> generator). So maybe this is a none-issue.
> AtomicLong would also imply a minor performance hit on multi core machines, 
> as it would force a memory barrier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4160) HBase shell move and online may be unusable if region name or server includes binary-encoded data

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4160:
-

   Resolution: Fixed
Fix Version/s: 0.90.5
 Hadoop Flags: [Reviewed]
   Status: Resolved  (was: Patch Available)

Committed to trunk and branch.  Thanks for the patch Jonathan.

> HBase shell move and online may be unusable if region name or server includes 
> binary-encoded data
> -
>
> Key: HBASE-4160
> URL: https://issues.apache.org/jira/browse/HBASE-4160
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.3
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
>Priority: Minor
> Fix For: 0.90.5
>
> Attachments: 
> 0001-HBASE-4160-HBase-shell-move-and-online-may-be-unusab.patch
>
>
> Similar to HBASE-4115, this entails a conversion of 
> org.apache.hadoop.hbase.utils.Bytes.toBytes to a to_java_bytes call in the 
> 'move' and 'online' call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[Lohit Vijayarenu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lohit Vijayarenu updated HBASE-4169:


Attachment: 4169-v5.txt

Attached is patch based on Stack's changes (4169-v4.txt). Changed name of 
NonHDFS to MapR

> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, 
> HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4078) Silent Data Offlining During HDFS Flakiness

[Pritam Damania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritam Damania updated HBASE-4078:
--

Attachment: 0001-Validate-store-files.patch

This is a patch for HBASE-4078

> Silent Data Offlining During HDFS Flakiness
> ---
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
>  Issue Type: Bug
>  Components: io, regionserver
>Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
>Reporter: Nicolas Spiegelberg
>Assignee: Pritam Damania
>Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 .  The bug fix for this JIRA is a temporary workaround for 
> improperly moving partially-written files from TMP into the region directory 
> when a FS error occurs.  Unfortunately, the fix is to ignore all IO 
> exceptions, which masks off-lining due to FS flakiness.  We need to 
> permanently fix the problem that created HBASE-1436 & then at least have the 
> option to not open a region during times of flakey FS.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4178) Use of Random.nextLong() in HRegionServer.addScanner(...)

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081338#comment-13081338
 ] 

Andrew Purtell commented on HBASE-4178:
---

If we really want to be bullet proof, scanner IDs could be UUIDs with MAC and 
time components. Seems like this issue is about what would be a really rare 
event though. (I guess an experiment to confirm?)

> Use of Random.nextLong() in HRegionServer.addScanner(...)
> -
>
> Key: HBASE-4178
> URL: https://issues.apache.org/jira/browse/HBASE-4178
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.3
>Reporter: Lars Hofhansl
>Priority: Minor
>
> ScannerIds are currently assigned by getting a random long. While it would be 
> a rare occurrence that two scanners received the same ids on the same region 
> server the results would seem to be... Bad.
> A client scanner would get results from a different server scanner, and maybe 
> only from some of the region servers.
> A safer approach would be using an AtomicLong. We do not have to worry about 
> running of numbers: If we got 1 scanners per second it'd take > 2.9m 
> years to reach 2^63.
> Then again the same reasoning would imply that this collisions would be 
> happening too rarely to be of concern (assuming a good random number 
> generator). So maybe this is a none-issue.
> AtomicLong would also imply a minor performance hit on multi core machines, 
> as it would force a memory barrier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS Flakiness

[Pritam Damania (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081341#comment-13081341
 ] 

Pritam Damania commented on HBASE-4078:
---

Here is the review board link for this patch : 
https://reviews.apache.org/r/1327/diff/

> Silent Data Offlining During HDFS Flakiness
> ---
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
>  Issue Type: Bug
>  Components: io, regionserver
>Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
>Reporter: Nicolas Spiegelberg
>Assignee: Pritam Damania
>Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 .  The bug fix for this JIRA is a temporary workaround for 
> improperly moving partially-written files from TMP into the region directory 
> when a FS error occurs.  Unfortunately, the fix is to ignore all IO 
> exceptions, which masks off-lining due to FS flakiness.  We need to 
> permanently fix the problem that created HBASE-1436 & then at least have the 
> option to not open a region during times of flakey FS.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081344#comment-13081344
 ] 

Hudson commented on HBASE-4169:
---

Integrated in HBase-TRUNK #2095 (See 
[https://builds.apache.org/job/HBase-TRUNK/2095/])
HBASE-4169 FSUtils LeaseRecovery for non HDFS FileSystems -- reverted until 
we address issues raised around class name

stack : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSNonHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, 
> HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4014) Coprocessors: Flag the presence of coprocessors in logged exceptions

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081354#comment-13081354
 ] 

jirapos...@reviews.apache.org commented on HBASE-4014:
--



bq.  On 2011-08-08 20:31:00, Gary Helmling wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java, 
line 81
bq.  > 
bq.  >
bq.  > -1.  This doesn't belong in RegionServerServices, it's part of the 
cp framework.

removed.


bq.  On 2011-08-08 20:31:00, Gary Helmling wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java, 
line 134
bq.  > 
bq.  >
bq.  > By moving loaded coprocessor set to CoprocessorHost, you don't need 
this anymore.

removed.


bq.  On 2011-08-08 20:31:00, Gary Helmling wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java, 
line 220
bq.  > 
bq.  >
bq.  > Duplicates same method in MasterCoprocessorHost?  Move to 
CoprocessorHost and then you only need one implementation.

moved up to CoprocessorHost. handleCoprocessorThrowable() calls 
Coprocessor.abortServer() in some cases (depending on the type of the Throwable 
param). MasterCoprocessorHost and RegionCoprocessorHost() override 
abortServer() to pass along their respective services information to the abort 
message.


bq.  On 2011-08-08 20:31:00, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/MasterServices.java, line 67
bq.  > 
bq.  >
bq.  > -1.  This doesn't belong in MasterServices.

removed.


bq.  On 2011-08-08 20:31:00, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 186
bq.  > 
bq.  >
bq.  > Move into CoprocessorHost

moved.


bq.  On 2011-08-08 20:31:00, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java, 
line 71
bq.  > 
bq.  >
bq.  > I think CoprocessorHost should have a static HashSet for the 
loaded coprocessor class names.  Then add a static method to access the 
property:
bq.  > 
bq.  > public static HashSet getLoadedCoprocessors();
bq.  > 
bq.  > Then HMaster.abort() and HRegionServer.abort() just need to call 
CoprocessorHost.getLoadedCoprocessors() when logging.

Added this HashSet to CoprocessorHost and reader method getLoadedCoprocessors() 
and writer method addToLoadedCPs().


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/969/#review1326
---


On 2011-08-06 03:19:56, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/969/
bq.  ---
bq.  
bq.  (Updated 2011-08-06 03:19:56)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  https://issues.apache.org/jira/browse/HBASE-4014 Coprocessors: Flag the 
presence of coprocessors in logged exceptions
bq.  
bq.  The general gist here is to wrap each of 
{Master,RegionServer}CoprocessorHost's coprocessor call inside a 
bq.  
bq.  "try { ... } catch (Throwable e) { handleCoprocessorThrowable(e) }"
bq.  
bq.  block. 
bq.  
bq.  handleCoprocessorThrowable() is responsible for either passing 'e' along 
to the client (if 'e' is an IOException) or, otherwise, aborting the service 
(Regionserver or Master).
bq.  
bq.  The abort message contains a list of the loaded coprocessors for crash 
analysis.
bq.  
bq.  
bq.  This addresses bug HBASE-4014.
bq.  https://issues.apache.org/jira/browse/HBASE-4014
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
18ba6e7 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 8beeb68 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
aa930f5 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterServices.java 7d9fd9d 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
23225d7 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java 
c44da73 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
8ffa086 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/w

[jira] [Commented] (HBASE-4014) Coprocessors: Flag the presence of coprocessors in logged exceptions

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081353#comment-13081353
 ] 

jirapos...@reviews.apache.org commented on HBASE-4014:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/969/
---

(Updated 2011-08-09 01:18:57.289734)


Review request for hbase, Gary Helmling and Mingjie Lai.


Changes
---

Thanks again to Gary for his reviews.

-Moved HashSet of coprocessor names into a static member of 
CoprocessorHost, out of HMaster and HRegionServer.
-add CoprocessorHost.getLoadedCoprocessors() to read this set, and 
addToLoadedCPs(), a synchronized method to add to this set.
-moved handleCoprocessorThrowable() into CoprocessorHost from 
MasterCoprocessorHost and RegionCoprocessorHost, since it's the same in both.


Summary
---

https://issues.apache.org/jira/browse/HBASE-4014 Coprocessors: Flag the 
presence of coprocessors in logged exceptions

The general gist here is to wrap each of {Master,RegionServer}CoprocessorHost's 
coprocessor call inside a 

"try { ... } catch (Throwable e) { handleCoprocessorThrowable(e) }"

block. 

handleCoprocessorThrowable() is responsible for either passing 'e' along to the 
client (if 'e' is an IOException) or, otherwise, aborting the service 
(Regionserver or Master).

The abort message contains a list of the loaded coprocessors for crash analysis.


This addresses bug HBASE-4014.
https://issues.apache.org/jira/browse/HBASE-4014


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
18ba6e7 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 8beeb68 
  src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
aa930f5 
  src/main/java/org/apache/hadoop/hbase/master/MasterServices.java 7d9fd9d 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 23225d7 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java 
c44da73 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 
8ffa086 
  
src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java 
03df574 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorException.java
 PRE-CREATION 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorException.java
 PRE-CREATION 
  src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 78e7d62 
  
src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java
 ab12968 

Diff: https://reviews.apache.org/r/969/diff


Testing
---

patch includes two tests:

TestMasterCoprocessorException.java
TestRegionServerCoprocessorException.java

both tests pass in my build environment.


Thanks,

Eugene



> Coprocessors: Flag the presence of coprocessors in logged exceptions
> 
>
> Key: HBASE-4014
> URL: https://issues.apache.org/jira/browse/HBASE-4014
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Reporter: Andrew Purtell
>Assignee: Eugene Koontz
> Fix For: 0.92.0
>
> Attachments: HBASE-4014.patch, HBASE-4014.patch, HBASE-4014.patch, 
> HBASE-4014.patch
>
>
> For some initial triage of bug reports for core versus for deployments with 
> loaded coprocessors, we need something like the Linux kernel's taint flag, 
> and list of linked in modules that show up in the output of every OOPS, to 
> appear above or below exceptions that appear in the logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4014) Coprocessors: Flag the presence of coprocessors in logged exceptions

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081356#comment-13081356
 ] 

jirapos...@reviews.apache.org commented on HBASE-4014:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/969/
---

(Updated 2011-08-09 01:24:48.983396)


Review request for hbase, Gary Helmling and Mingjie Lai.


Changes
---

Updated patch: please ignore previous diff revision 5.


Summary
---

https://issues.apache.org/jira/browse/HBASE-4014 Coprocessors: Flag the 
presence of coprocessors in logged exceptions

The general gist here is to wrap each of {Master,RegionServer}CoprocessorHost's 
coprocessor call inside a 

"try { ... } catch (Throwable e) { handleCoprocessorThrowable(e) }"

block. 

handleCoprocessorThrowable() is responsible for either passing 'e' along to the 
client (if 'e' is an IOException) or, otherwise, aborting the service 
(Regionserver or Master).

The abort message contains a list of the loaded coprocessors for crash analysis.


This addresses bug HBASE-4014.
https://issues.apache.org/jira/browse/HBASE-4014


Diffs (updated)
-

  src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
18ba6e7 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 8beeb68 
  src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
aa930f5 
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 23225d7 
  src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java 
c44da73 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorException.java
 PRE-CREATION 
  
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorException.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/969/diff


Testing
---

patch includes two tests:

TestMasterCoprocessorException.java
TestRegionServerCoprocessorException.java

both tests pass in my build environment.


Thanks,

Eugene



> Coprocessors: Flag the presence of coprocessors in logged exceptions
> 
>
> Key: HBASE-4014
> URL: https://issues.apache.org/jira/browse/HBASE-4014
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Reporter: Andrew Purtell
>Assignee: Eugene Koontz
> Fix For: 0.92.0
>
> Attachments: HBASE-4014.patch, HBASE-4014.patch, HBASE-4014.patch, 
> HBASE-4014.patch
>
>
> For some initial triage of bug reports for core versus for deployments with 
> loaded coprocessors, we need something like the Linux kernel's taint flag, 
> and list of linked in modules that show up in the output of every OOPS, to 
> appear above or below exceptions that appear in the logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4014) Coprocessors: Flag the presence of coprocessors in logged exceptions

[Eugene Koontz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koontz updated HBASE-4014:
-

Attachment: HBASE-4014.patch

> Coprocessors: Flag the presence of coprocessors in logged exceptions
> 
>
> Key: HBASE-4014
> URL: https://issues.apache.org/jira/browse/HBASE-4014
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Reporter: Andrew Purtell
>Assignee: Eugene Koontz
> Fix For: 0.92.0
>
> Attachments: HBASE-4014.patch, HBASE-4014.patch, HBASE-4014.patch, 
> HBASE-4014.patch, HBASE-4014.patch
>
>
> For some initial triage of bug reports for core versus for deployments with 
> loaded coprocessors, we need something like the Linux kernel's taint flag, 
> and list of linked in modules that show up in the output of every OOPS, to 
> appear above or below exceptions that appear in the logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3327) For increment workloads, retain memstores in memory after flushing them

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081361#comment-13081361
 ] 

Yi Liang commented on HBASE-3327:
-

Is there a patch for 0.90.3?

> For increment workloads, retain memstores in memory after flushing them
> ---
>
> Key: HBASE-3327
> URL: https://issues.apache.org/jira/browse/HBASE-3327
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Karthik Ranganathan
>
> This is an improvement based on our observation of what happens in an 
> increment workload. The working set is typically small and is contained in 
> the memstores. 
> 1. The reason the memstores get flushed is because the number of wal logs 
> limit gets hit. 
> 2. This in turn triggers compactions, which evicts the block cache. 
> 3. Flushing of memstore and eviction of the block cache causes disk reads for 
> increments coming in after this because the data is no longer in memory.
> We could solve this elegantly by retaining the memstores AFTER they are 
> flushed into files. This would mean we can quickly populate the new memstore 
> with the working set of data from memory itself without having to hit disk. 
> We can throttle the number of such memstores we retain, or the memory 
> allocated to it. In fact, allocating a percentage of the block cache to this 
> would give us a huge boost.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[Michael Weng (JIRA)]

Failed to run RowCounter on top of Hadoop branch-0.22
-

 Key: HBASE-4179
 URL: https://issues.apache.org/jira/browse/HBASE-4179
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 0.92.0
 Environment: Running hbase on top of hadoop branch-0.22
Reporter: Michael Weng
 Fix For: 0.92.0


:~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081365#comment-13081365
 ] 

Todd Lipcon commented on HBASE-4179:


HADOOP-5679 changed this function to return int instead of void.

> Failed to run RowCounter on top of Hadoop branch-0.22
> -
>
> Key: HBASE-4179
> URL: https://issues.apache.org/jira/browse/HBASE-4179
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.92.0
> Environment: Running hbase on top of hadoop branch-0.22
>Reporter: Michael Weng
> Fix For: 0.92.0
>
>
> :~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
> ~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  
> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[Michael Weng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081368#comment-13081368
 ] 

Michael Weng commented on HBASE-4179:
-

The root cause is due to the API change between hadoop 0.20-append and hadoop 
0.22.

In 0.20-append:
 public void driver(String[] args) throws Throwable 

In 0.22:
 public int driver(String[] args) throws Throwable 

Proposed fix:
 using getMethod at run time instead of invoke the function directly.

> Failed to run RowCounter on top of Hadoop branch-0.22
> -
>
> Key: HBASE-4179
> URL: https://issues.apache.org/jira/browse/HBASE-4179
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.92.0
> Environment: Running hbase on top of hadoop branch-0.22
>Reporter: Michael Weng
> Fix For: 0.92.0
>
>
> :~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
> ~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  
> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Attachment: HBase-4095-V6-trunk.patch
HBase-4095-V6-branch.patch

Just modified the code comments for the patches of V6.

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-

[jira] [Assigned] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-4179:
-

Assignee: Michael Weng

Please provide a patch as you outlined.

> Failed to run RowCounter on top of Hadoop branch-0.22
> -
>
> Key: HBASE-4179
> URL: https://issues.apache.org/jira/browse/HBASE-4179
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.92.0
> Environment: Running hbase on top of hadoop branch-0.22
>Reporter: Michael Weng
>Assignee: Michael Weng
> Fix For: 0.92.0
>
>
> :~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
> ~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  
> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

HBase should check the isSecurityEnabled flag
-

 Key: HBASE-4180
 URL: https://issues.apache.org/jira/browse/HBASE-4180
 Project: HBase
  Issue Type: Bug
  Components: security
Reporter: Bochun Bai


Hadoop 0.21.0's UserGroupInfomation support the security check flag and always 
returns false.
HBase should check both the method existence and the return value.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3287) Add option to cache blocks on hfile write and evict blocks on hfile close

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081372#comment-13081372
 ] 

Yi Liang commented on HBASE-3287:
-

Is there a patch for 0.90.3? 

> Add option to cache blocks on hfile write and evict blocks on hfile close
> -
>
> Key: HBASE-3287
> URL: https://issues.apache.org/jira/browse/HBASE-3287
> Project: HBase
>  Issue Type: New Feature
>  Components: io, regionserver
>Affects Versions: 0.90.0
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Fix For: 0.92.0
>
> Attachments: HBASE-3287-FINAL-trunk.patch
>
>
> This issue is about adding configuration options to add/remove from the block 
> cache when creating/closing files.  For use cases with lots of flushing and 
> compacting, this might be desirable to prevent cache misses and maximize the 
> effective utilization of total block cache capacity.
> The first option, {{hbase.rs.cacheblocksonwrite}}, will make it so we 
> pre-cache blocks as we are writing out new files.
> The second option, {{hbase.rs.evictblocksonclose}}, will make it so we evict 
> blocks when files are closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bochun Bai updated HBASE-4180:
--

Attachment: HBASE-4180.patch

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bochun Bai updated HBASE-4180:
--

Status: Patch Available  (was: Open)

adds only one line after check the isSecurityCheckEnabled.
Call and assign it.

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[Michael Weng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Weng updated HBASE-4179:


Attachment: HBASE-4179-trunk.patch

Patch attached.

> Failed to run RowCounter on top of Hadoop branch-0.22
> -
>
> Key: HBASE-4179
> URL: https://issues.apache.org/jira/browse/HBASE-4179
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.92.0
> Environment: Running hbase on top of hadoop branch-0.22
>Reporter: Michael Weng
>Assignee: Michael Weng
> Fix For: 0.92.0
>
> Attachments: HBASE-4179-trunk.patch
>
>
> :~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
> ~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  
> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-4180:
-

Assignee: Bochun Bai

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081376#comment-13081376
 ] 

Ted Yu commented on HBASE-4180:
---

+1 on patch.

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4168) A client continues to try and connect to a powered down regionserver

[Anirudh Todi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anirudh Todi updated HBASE-4168:


Attachment: HBASE-4168(2).patch

@Ted - the patch I submitted is from the open-source trunk which I checked out 
here - https://svn.apache.org/repos/asf/hbase/trunk/

I see your source of confusion now. In trunk's CatalogTracker, line 469 is:

{noformat}
} else if (cause != null && cause.getMessage() != null
{noformat}

the internal branch had:

{noformat}
} else if (cause.getMessage() != null)
{noformat}

and when I conducted Experiment-4 using the internal branch, cause turned out 
to be null - and I received a NullPointerException at that line

However, would it still be better to return false and retry connecting to the 
META instead of throwing an exception there?
I have uploaded a new patch in which I am handling the IOException unwrapped 
from RemoteException in a similar manner.

> A client continues to try and connect to a powered down regionserver
> 
>
> Key: HBASE-4168
> URL: https://issues.apache.org/jira/browse/HBASE-4168
> Project: HBase
>  Issue Type: Bug
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: HBASE-4168(2).patch, HBASE-4168-revised.patch, 
> HBASE-4168.patch, hbase-hadoop-master-msgstore232.snc4.facebook.com.log
>
>
> Experiment-1
> Started a dev cluster - META is on the same regionserver as my key-value. I 
> kill the regionserver process but donot power down the machine.
> The META is able to migrate to a new regionserver and the regions are also 
> able to reopen elsewhere.
> The client is able to talk to the META and find the new kv location and get 
> it.
> Experiment-2
> Started a dev cluster - META is on a different regionserver as my key-value. 
> I kill the regionserver process but donot power down the machine.
> The META remains where it is and the regions are also able to reopen 
> elsewhere.
> The client is able to talk to the META and find the new kv location and get 
> it.
> Experiment-3
> Started a dev cluster - META is on a different regionserver as my key-value. 
> I power down the machine hosting this regionserver.
> The META remains where it is and the regions are also able to reopen 
> elsewhere.
> The client is able to talk to the META and find the new kv location and get 
> it.
> Experiment-4 (This is the problematic one)
> Started a dev cluster - META is on the same regionserver as my key-value. I 
> power down the machine hosting this regionserver.
> The META is able to migrate to a new regionserver - however - it takes a 
> really long time (~30 minutes)
> The regions on that regionserver DONOT reopen (I waited for 1 hour)
> The client is able to find the new location of the META, however, the META 
> keeps redirecting the client to powered down
> regionserver as the location of the key-value it is trying to get. Thus the 
> client's get is unsuccessful.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Fix Version/s: 0.90.4
   Status: Patch Available  (was: Reopened)

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 16:29:58,775 DEBUG org.apach

[jira] [Commented] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081379#comment-13081379
 ] 

Ted Yu commented on HBASE-4179:
---

Please wrap the line so that it is no longer than 80 characters.

> Failed to run RowCounter on top of Hadoop branch-0.22
> -
>
> Key: HBASE-4179
> URL: https://issues.apache.org/jira/browse/HBASE-4179
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.92.0
> Environment: Running hbase on top of hadoop branch-0.22
>Reporter: Michael Weng
>Assignee: Michael Weng
> Fix For: 0.92.0
>
> Attachments: HBASE-4179-trunk.patch
>
>
> :~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
> ~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  
> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4114) Metrics for HFile HDFS block locality

[Ming Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming Ma updated HBASE-4114:
---

Attachment: HBASE-4114-trunk.patch

Thanks, Ted. Here is the fix.

> Metrics for HFile HDFS block locality
> -
>
> Key: HBASE-4114
> URL: https://issues.apache.org/jira/browse/HBASE-4114
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics, regionserver
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, 
> HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, HBASE-4114-trunk.patch
>
>
> Normally, when we put hbase and HDFS in the same cluster ( e.g., region 
> server runs on the datenode ), we have a reasonably good data locality, as 
> explained by Lars. Also Work has been done by Jonathan to address the startup 
> situation.
> There are scenarios where regions can be on a different machine from the 
> machines that hold the underlying HFile blocks, at least for some period of 
> time. This will have performance impact on whole table scan operation and map 
> reduce job during that time.
> 1.After load balancer moves the region and before compaction (thus 
> generate HFile on the new region server ) on that region, HDFS block can be 
> remote.
> 2.When a new machine is added, or removed, Hbase's region assignment 
> policy is different from HDFS's block reassignment policy.
> 3.Even if there is no much hbase activity, HDFS can load balance HFile 
> blocks as other non-hbase applications push other data to HDFS.
> Lots has been or will be done in load balancer, as summarized by Ted. I am 
> curious if HFile HDFS block locality should be used as another factor here.
> I have done some experiments on how HDFS block locality can impact map reduce 
> latency. First we need to define a metrics to measure HFile data locality.
> Metrics defintion:
> For a given table, or a region server, or a region, we can define the 
> following. The higher the value, the more local HFile is from region server's 
> point of view.
> HFile locality index = ( Total number of HDFS blocks that can be retrieved 
> locally by the region server ) / ( Total number of HDFS blocks for all HFiles 
> )
> Test Results:
> This is to show how HFile locality can impact the latency. It is based on a 
> table with 1M rows, 36KB per row; regions are distributed in balance. The map 
> job is RowCounter.
> HFile Locality Index  Map job latency ( in sec )
> 28%   157
> 36%   150
> 47%   142
> 61%   133
> 73%   122
> 89%   103
> 99%   95
> So the first suggestion is to expose HFile locality index as a new region 
> server metrics. It will be ideal if we can somehow measure HFile locality 
> index on a per map job level.
> Regarding if/when we should include that as another factor for load balancer, 
> that will be a different work item. It is unclear how load balancer can take 
> various factors into account to come up with the best load balancer strategy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4168) A client continues to try and connect to a powered down regionserver

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081383#comment-13081383
 ] 

Ted Yu commented on HBASE-4168:
---

I think reconnecting to the META makes sense.

Can you add 'cause != null && ' to the above line and show us what IOException 
was thrown ?

Thanks

> A client continues to try and connect to a powered down regionserver
> 
>
> Key: HBASE-4168
> URL: https://issues.apache.org/jira/browse/HBASE-4168
> Project: HBase
>  Issue Type: Bug
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: HBASE-4168(2).patch, HBASE-4168-revised.patch, 
> HBASE-4168.patch, hbase-hadoop-master-msgstore232.snc4.facebook.com.log
>
>
> Experiment-1
> Started a dev cluster - META is on the same regionserver as my key-value. I 
> kill the regionserver process but donot power down the machine.
> The META is able to migrate to a new regionserver and the regions are also 
> able to reopen elsewhere.
> The client is able to talk to the META and find the new kv location and get 
> it.
> Experiment-2
> Started a dev cluster - META is on a different regionserver as my key-value. 
> I kill the regionserver process but donot power down the machine.
> The META remains where it is and the regions are also able to reopen 
> elsewhere.
> The client is able to talk to the META and find the new kv location and get 
> it.
> Experiment-3
> Started a dev cluster - META is on a different regionserver as my key-value. 
> I power down the machine hosting this regionserver.
> The META remains where it is and the regions are also able to reopen 
> elsewhere.
> The client is able to talk to the META and find the new kv location and get 
> it.
> Experiment-4 (This is the problematic one)
> Started a dev cluster - META is on the same regionserver as my key-value. I 
> power down the machine hosting this regionserver.
> The META is able to migrate to a new regionserver - however - it takes a 
> really long time (~30 minutes)
> The regions on that regionserver DONOT reopen (I waited for 1 hour)
> The client is able to find the new location of the META, however, the META 
> keeps redirecting the client to powered down
> regionserver as the location of the key-value it is trying to get. Thus the 
> client's get is unsuccessful.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4114) Metrics for HFile HDFS block locality

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081384#comment-13081384
 ] 

Ted Yu commented on HBASE-4114:
---

+1 on patch.

> Metrics for HFile HDFS block locality
> -
>
> Key: HBASE-4114
> URL: https://issues.apache.org/jira/browse/HBASE-4114
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics, regionserver
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, 
> HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, HBASE-4114-trunk.patch
>
>
> Normally, when we put hbase and HDFS in the same cluster ( e.g., region 
> server runs on the datenode ), we have a reasonably good data locality, as 
> explained by Lars. Also Work has been done by Jonathan to address the startup 
> situation.
> There are scenarios where regions can be on a different machine from the 
> machines that hold the underlying HFile blocks, at least for some period of 
> time. This will have performance impact on whole table scan operation and map 
> reduce job during that time.
> 1.After load balancer moves the region and before compaction (thus 
> generate HFile on the new region server ) on that region, HDFS block can be 
> remote.
> 2.When a new machine is added, or removed, Hbase's region assignment 
> policy is different from HDFS's block reassignment policy.
> 3.Even if there is no much hbase activity, HDFS can load balance HFile 
> blocks as other non-hbase applications push other data to HDFS.
> Lots has been or will be done in load balancer, as summarized by Ted. I am 
> curious if HFile HDFS block locality should be used as another factor here.
> I have done some experiments on how HDFS block locality can impact map reduce 
> latency. First we need to define a metrics to measure HFile data locality.
> Metrics defintion:
> For a given table, or a region server, or a region, we can define the 
> following. The higher the value, the more local HFile is from region server's 
> point of view.
> HFile locality index = ( Total number of HDFS blocks that can be retrieved 
> locally by the region server ) / ( Total number of HDFS blocks for all HFiles 
> )
> Test Results:
> This is to show how HFile locality can impact the latency. It is based on a 
> table with 1M rows, 36KB per row; regions are distributed in balance. The map 
> job is RowCounter.
> HFile Locality Index  Map job latency ( in sec )
> 28%   157
> 36%   150
> 47%   142
> 61%   133
> 73%   122
> 89%   103
> 99%   95
> So the first suggestion is to expose HFile locality index as a new region 
> server metrics. It will be ideal if we can somehow measure HFile locality 
> index on a per map job level.
> Regarding if/when we should include that as another factor for load balancer, 
> that will be a different work item. It is unclear how load balancer can take 
> various factors into account to come up with the best load balancer strategy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4114) Metrics for HFile HDFS block locality

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4114:
--

Status: Patch Available  (was: Open)

> Metrics for HFile HDFS block locality
> -
>
> Key: HBASE-4114
> URL: https://issues.apache.org/jira/browse/HBASE-4114
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics, regionserver
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, 
> HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, HBASE-4114-trunk.patch
>
>
> Normally, when we put hbase and HDFS in the same cluster ( e.g., region 
> server runs on the datenode ), we have a reasonably good data locality, as 
> explained by Lars. Also Work has been done by Jonathan to address the startup 
> situation.
> There are scenarios where regions can be on a different machine from the 
> machines that hold the underlying HFile blocks, at least for some period of 
> time. This will have performance impact on whole table scan operation and map 
> reduce job during that time.
> 1.After load balancer moves the region and before compaction (thus 
> generate HFile on the new region server ) on that region, HDFS block can be 
> remote.
> 2.When a new machine is added, or removed, Hbase's region assignment 
> policy is different from HDFS's block reassignment policy.
> 3.Even if there is no much hbase activity, HDFS can load balance HFile 
> blocks as other non-hbase applications push other data to HDFS.
> Lots has been or will be done in load balancer, as summarized by Ted. I am 
> curious if HFile HDFS block locality should be used as another factor here.
> I have done some experiments on how HDFS block locality can impact map reduce 
> latency. First we need to define a metrics to measure HFile data locality.
> Metrics defintion:
> For a given table, or a region server, or a region, we can define the 
> following. The higher the value, the more local HFile is from region server's 
> point of view.
> HFile locality index = ( Total number of HDFS blocks that can be retrieved 
> locally by the region server ) / ( Total number of HDFS blocks for all HFiles 
> )
> Test Results:
> This is to show how HFile locality can impact the latency. It is based on a 
> table with 1M rows, 36KB per row; regions are distributed in balance. The map 
> job is RowCounter.
> HFile Locality Index  Map job latency ( in sec )
> 28%   157
> 36%   150
> 47%   142
> 61%   133
> 73%   122
> 89%   103
> 99%   95
> So the first suggestion is to expose HFile locality index as a new region 
> server metrics. It will be ideal if we can somehow measure HFile locality 
> index on a per map job level.
> Regarding if/when we should include that as another factor for load balancer, 
> that will be a different work item. It is unclear how load balancer can take 
> various factors into account to come up with the best load balancer strategy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[Liu Jia (JIRA)]

HConnectionManager can't find cached HRegionInterface makes client very slow


 Key: HBASE-4181
 URL: https://issues.apache.org/jira/browse/HBASE-4181
 Project: HBase
  Issue Type: Bug
  Components: client
Affects Versions: 0.90.4, 0.92.0
Reporter: Liu Jia
Priority: Critical


HRegionInterface getHRegionConnection(final String hostname,
final int port, final InetSocketAddress isa, final boolean master)
throws IOException 


/
String rsName = isa != null ? isa.toString() : Addressing
  .createHostAndPortStr(hostname, port);  
here,if isa is null, the Addressing created a address like "node41:60010"
  
isa.toString():new InetSocketAddress(hostname, port).toString(); 
  
instead of Addressing.createHostAndPortStr(hostname, port);
server = this.servers.get(rsName);  
  if (server == null) {
// create a unique lock for this RS (if necessary)
this.connectionLock.putIfAbsent(rsName, rsName);
// get the RS lock
synchronized (this.connectionLock.get(rsName)) {
  // do one more lookup in case we were stalled above
  server = this.servers.get(rsName);
  if (server == null) {
try {
  if (clusterId.hasId()) {
conf.set(HConstants.CLUSTER_ID, clusterId.getId());
  }
  // Only create isa when we need to.
  InetSocketAddress address = isa != null ? isa
  : new InetSocketAddress(hostname, port);
  // definitely a cache miss. establish an RPC for this RS
  server = (HRegionInterface) HBaseRPC.waitForProxy(
  serverInterfaceClass, HRegionInterface.VERSION, address,
  this.conf, this.maxRPCAttempts, this.rpcTimeout,
  this.rpcTimeout);
  this.servers.put(address.toString(), server);  
but here address.toString() send an address like "node41/10.61.2l.171:60010"
 so 
this method can never get cached address and make client request very slow  
due to it's 
synchronized.

  
} catch (RemoteException e) {
  LOG.warn("RemoteException connecting to RS", e);
  // Throw what the RemoteException was carrying.
  throw RemoteExceptionHandler.decodeRemoteException(e);
}
  }
}
///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[Liu Jia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Jia updated HBASE-4181:
---

Description: 
HRegionInterface getHRegionConnection(final String hostname,
final int port, final InetSocketAddress isa, final boolean master)
throws IOException 


/
String rsName = isa != null ? isa.toString() : Addressing
  .createHostAndPortStr(hostname, port); 


here,if isa is null, the Addressing created a address like "node41:60010"
 should use 
"isa.toString():new InetSocketAddress(hostname, port).toString();" 
 instead of 
"Addressing.createHostAndPortStr(hostname, port);"


server = this.servers.get(rsName);  
  if (server == null) {
// create a unique lock for this RS (if necessary)
this.connectionLock.putIfAbsent(rsName, rsName);
// get the RS lock
synchronized (this.connectionLock.get(rsName)) {
  // do one more lookup in case we were stalled above
  server = this.servers.get(rsName);
  if (server == null) {
try {
  if (clusterId.hasId()) {
conf.set(HConstants.CLUSTER_ID, clusterId.getId());
  }
  // Only create isa when we need to.
  InetSocketAddress address = isa != null ? isa
  : new InetSocketAddress(hostname, port);
  // definitely a cache miss. establish an RPC for this RS
  server = (HRegionInterface) HBaseRPC.waitForProxy(
  serverInterfaceClass, HRegionInterface.VERSION, address,
  this.conf, this.maxRPCAttempts, this.rpcTimeout,
  this.rpcTimeout);
  this.servers.put(address.toString(), server);

  
but here address.toString() send an address like "node41/10.61.2l.171:60010
so this method can never get cached address and make client request very 
slow due to it's synchronized.


  
} catch (RemoteException e) {
  LOG.warn("RemoteException connecting to RS", e);
  // Throw what the RemoteException was carrying.
  throw RemoteExceptionHandler.decodeRemoteException(e);
}
  }
}
///

  was:
HRegionInterface getHRegionConnection(final String hostname,
final int port, final InetSocketAddress isa, final boolean master)
throws IOException 


/
String rsName = isa != null ? isa.toString() : Addressing
  .createHostAndPortStr(hostname, port);  
here,if isa is null, the Addressing created a address like "node41:60010"
  
isa.toString():new InetSocketAddress(hostname, port).toString(); 
  
instead of Addressing.createHostAndPortStr(hostname, port);
server = this.servers.get(rsName);  
  if (server == null) {
// create a unique lock for this RS (if necessary)
this.connectionLock.putIfAbsent(rsName, rsName);
// get the RS lock
synchronized (this.connectionLock.get(rsName)) {
  // do one more lookup in case we were stalled above
  server = this.servers.get(rsName);
  if (server == null) {
try {
  if (clusterId.hasId()) {
conf.set(HConstants.CLUSTER_ID, clusterId.getId());
  }
  // Only create isa when we need to.
  InetSocketAddress address = isa != null ? isa
  : new InetSocketAddress(hostname, port);
  // definitely a cache miss. establish an RPC for this RS
  server = (HRegionInterface) HBaseRPC.waitForProxy(
  serverInterfaceClass, HRegionInterface.VERSION, address,
  this.conf, this.maxRPCAttempts, this.rpcTimeout,
  this.rpcTimeout);
  this.servers.put(address.toString(), server);  
but here address.toString() send an address like "node41/10.61.2l.171:60010"
 so 
this method can never get cached address and make client request very slow  
due to it's 
synchronized.

  
} catch (RemoteException e) {
  LOG.warn("RemoteException connecting to RS", e);
  // Throw what the RemoteException was carrying.
  throw RemoteExceptionHandler.decodeRemoteException(

[jira] [Updated] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[Liu Jia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Jia updated HBASE-4181:
---

Attachment: HConnectionManager.patch

Fix this problem

> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081387#comment-13081387
 ] 

stack commented on HBASE-4169:
--

@Lohit Does the MapR class need to be in hbase?  Could it be in jars of yours 
rather than in hbase core?  If it has to be in hbase core, no problem, will 
commit as is.

> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, 
> HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[sinfox (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081389#comment-13081389
 ] 

sinfox commented on HBASE-4181:
---

I encounter the same problem .
Thanks to this patch, it is  effective.

> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081392#comment-13081392
 ] 

Gary Helmling commented on HBASE-4180:
--

-1 on the patch

{{User.IS_SECURE_HADOOP}} is a private variable used to differentiate between 
the API incompatible changes to {{UserGroupInformation}} between Hadoop 0.20 
and 0.21+.  The API-incompatible methods are accommodated using reflection so 
that we're not compile-time bound to Hadoop 0.21+.  Whether 
hadoop.security.authentication is set to "simple" or "kerberos" (which is what 
{{isSecurityEnabled()}} returns) makes no difference to accounting for the API 
changes.

Can you describe more of the underlying problem that you're observing that this 
patch is intended to fix?



> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081395#comment-13081395
 ] 

stack commented on HBASE-4176:
--

@Anirudh:  Does this patch depend on hbase-1744 going in first?  Should we make 
that happen first? Meantime here are some comments on what you have posted here.

In future, date, add author to docs, and in this case I'd cite the JIRA it 
addresses -- no biggie, just for next time.

This doc looks great.  We should be able to add it to the hbase documentation 
and include it in a chapter on thrift as a subsection on filters.

Specifying a filter, do I have to give full package name too?  (Same for 
comparators?)  It looks like I do not need too... but maybe it should be 
possible?

Doing the compound filters, nested parens will be interpreted properly? (nvm... 
in your example I see they do)

Great examples.

Your doc is excellent.










  

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081396#comment-13081396
 ] 

Ted Yu commented on HBASE-4181:
---

InetSocketAddress ctor involves DNS query. We should try not to call it.
The current code resorts to InetSocketAddress ctor under a cache miss.
Shall we modify the following code so that the key is in the same format as 
Addressing.createHostAndPortStr() ?
{code}
  this.servers.put(address.toString(), server);
{code}

> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081397#comment-13081397
 ] 

Gary Helmling commented on HBASE-4180:
--

So is the issue here that we're not correctly accounting for 
{{UserGroupInformation}} API changes between Hadoop 0.21 and 0.22+?

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[Lohit Vijayarenu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081398#comment-13081398
 ] 

Lohit Vijayarenu commented on HBASE-4169:
-

@stack if the class is in hbase core the users could run different versions of 
hbase without upgrading MapR. 

> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, 
> HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081400#comment-13081400
 ] 

stack commented on HBASE-4176:
--

Could we use this mini-language in the shell building filters?

Are you changing meaning of LESS_OR_EQUAL, GREATER, etc. in compare filter?

Misspelling: getDropDepenedentColumn() {

Some odd formatting issue here:

{code}
246 
  public static final byte [] regexStringType = new byte [] 
{'r','e','g','e','x',
247 

 's','t','r','i','n','g'};
{code}

You probably want to fix this ' 
 * https://our.intern.facebook.com/intern/wiki/index.php/HBase/Filter_Language'

Can we do other than this?

{code}
64  
filterHashMap = new HashMap();
65  
registerFilter("KeyOnlyFilter", new KeyOnlyFilter());
66  
registerFilter("FirstKeyOnlyFilter", new FirstKeyOnlyFilter());
67  
registerFilter("PrefixFilter", new PrefixFilter());
68  
registerFilter("ColumnPrefixFilter", new ColumnPrefixFilter());
69  
registerFilter("MultipleColumnPrefixFilter", new 
MultipleColumnPrefixFilter());
70  
registerFilter("ColumnCountGetFilter", new ColumnCountGetFilter());
71  
registerFilter("PageFilter", new PageFilter());
72  
registerFilter("ColumnPaginationFilter", new ColumnPaginationFilter());
73  
registerFilter("InclusiveStopFilter", new InclusiveStopFilter());
74  
registerFilter("TimestampsFilter", new TimestampsFilter());
75  
registerFilter("RowFilter", new RowFilter());
76  
registerFilter("FamilyFilter", new FamilyFilter());
77  
registerFilter("QualifierFilter", new QualifierFilter());
78  
registerFilter("ValueFilter", new ValueFilter());
79  
registerFilter("ColumnRangeFilter", new ColumnRangeFilter());
80  
registerFilter("SingleColumnValueFilter", new SingleColumnValueFilter());
81  
registerFilter("SingleColumnValueExcludeFilter", new 
SingleColumnValueExcludeFilter());
82  
registerFilter("DependentColumnFilter", new DependentColumnFilter());
{code}

Ask around.  I will too.  IIRC, you can't get a listing of classes under a dir 
on CLASSPATH so maybe there is nothing to do here and you have to explicitly 
list those filters you can build from String.  We can commit as is but would be 
cool if could figure alternative to hardcoding.

I did not review your parser.  A skim says its looking good.

Test looks good too.

Sorry I didn't put my comments up in reviewboard.  I was half way through when 
I realized I was doing this dumb filling comments into the issue.

Good stuff.



> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081403#comment-13081403
 ] 

Bochun Bai commented on HBASE-4180:
---

0.20 doesn't have isSecurityEnabled,
0.20-SEC does have isSecurityEnabled,
0.21 does have isSecurityEnabled but not means it must configured to use it.

If we need HBase work with a non-security hadoop, I think we need find another 
way to identify it.


> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081408#comment-13081408
 ] 

Bochun Bai commented on HBASE-4180:
---

0.21 API incompatible is problem for downstream projects.
Hive extracts a shims components for this. 
Will HBase consider some similar changes?

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081409#comment-13081409
 ] 

Ted Yu commented on HBASE-4176:
---

Can we use the bottom two code snippets from 
http://www.velocityreviews.com/forums/t137693-find-all-implementing-classes-in-classpath.html
 ?
It would allow us to find subclasses of FilterBase.

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4169:
-

   Resolution: Fixed
Fix Version/s: 0.92.0
 Hadoop Flags: [Reviewed]
   Status: Resolved  (was: Patch Available)

Committed to TRUNK.  Thanks for the patch Lohit.

> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, 
> HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081414#comment-13081414
 ] 

stack commented on HBASE-4180:
--

@Bochun

Can you rephrase this '0.21 does have isSecurityEnabled but not means it must 
configured to use it.'?  What is the difference between 0.20-SEC and 0.21?

HBase does not run on raw (non-secure) 0.21.  It needs a few small patches to 
do so.  It will run on 0.22 and 0.23 non-secure.  Until we figure otherwise, 
we'd like to have it so the same hbase binary will run on 0.20-append, CDH3, 
0.22, 0.23, MapR, etc.  I've not looked at whats involved making hbase run 
secure on all these versions (Gary is our man when it comes to that).  
Hopefully we can make it so same hbase binary can run on all secure hadoop 
combos (Will this be possible Gary?)

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081415#comment-13081415
 ] 

Gary Helmling commented on HBASE-4180:
--

As I said before, the presence of {{isSecurityEnabled}} is used to identify 
*API* changes, not whether Hadoop is configured for Kerberos authentication.  
We really don't care about the return value of {{isSecurityEnabled}}.

HBase does work with non-secure Hadoop, both 0.20 (pre-security) and 
0.20-security (with hadoop.security.authentication=simple).

If there are API differences between {{UserGroupInformation}} in Hadoop 0.21 
and 0.22+, we should account for those.  Otherwise I still don't see what 
exactly the issue is here.  Are you encountering problems running on Hadoop 
0.21 (which is not supported) that this is intended to solve?


> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081416#comment-13081416
 ] 

stack commented on HBASE-4176:
--

That snippet would seem to span classloaders so doesn't look to be bad (nice 
find Ted).  I'd say that we punt doing such a lookup to another issue.  There 
is already enough going on in here.

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[Lohit Vijayarenu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081419#comment-13081419
 ] 

Lohit Vijayarenu commented on HBASE-4169:
-

Thanks everyone for the help

> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, 
> HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081417#comment-13081417
 ] 

Gary Helmling commented on HBASE-4180:
--

@Bochun

{quote}
0.21 API incompatible is problem for downstream projects.
Hive extracts a shims components for this.
Will HBase consider some similar changes?
{quote}

This is the purpose of the {{User}} class -- to isolate the UGI changes.  See 
{{User.HadoopUser}} and {{User.SecureHadoopUser}}.  If we need to add a 
separate version for Hadoop 0.21, we can do that, but as Stack mentions there 
are other, non-related problems with running on Hadoop 0.21.

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081420#comment-13081420
 ] 

Bochun Bai commented on HBASE-4180:
---

@Gray, yes.
I am trying run hbase on Hadoop 0.21 and it throws ClassNotFoundException.

org.apache.hadoop.hbase.User.login() thinks it is a secure version and invoke 
SecureHadoopUser.login.

The SecureHadoopUser.login() needs org.apache.hadoop.security.SecurityUtil 
class exist.


> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081422#comment-13081422
 ] 

Bochun Bai commented on HBASE-4180:
---

@stack, It is cleared, Thanks!
0.21 is still not a stable version in Hadoop so it is reasonable not supporting 
it for hbase.
I will close this with "won't fix" later.
Would you please kindly give me some hits finding out "HBase does not run on 
raw (non-secure) 0.21. It needs a few small patches to do so"? How can I find 
these small patches?

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081423#comment-13081423
 ] 

Jieshan Bean commented on HBASE-4095:
-

Review Board related address: https://reviews.apache.org/r/1354/

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.13099373866

[jira] [Commented] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081424#comment-13081424
 ] 

jirapos...@reviews.apache.org commented on HBASE-4095:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1354/
---

(Updated 2011-08-09 04:20:46.103510)


Review request for hbase.


Summary
---

Issue: https://issues.apache.org/jira/browse/HBASE-4095

Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
the reason why they got so huge:

1. The replicas is less than the expect number. So the method of 
checkLowReplication will be called each sync.
2. The method checkLowReplication request log-roll first, and set 
logRollRequested as true: 
3. requestLogRoll() just commit the roll request. It may not execute in time, 
like the following scenario(It returns at the beginning, for the 
numEntries.get() == 0):
public byte [][] rollWriter() throws FailedLogCloseException, IOException {
// Return if nothing to flush.
if (this.writer != null && this.numEntries.get() <= 0) {
  return null;
}
4.logRollRequested was true until the log-roll executed. So during the time, 
each request of log-roll in sync() was skipped.


This addresses bug HBASE-4095.
https://issues.apache.org/jira/browse/HBASE-4095


Diffs
-

  /src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 1155186 

Diff: https://reviews.apache.org/r/1354/diff


Testing
---

Please find the test result from the attachment of 
https://issues.apache.org/jira/browse/HBASE-4095


Thanks,

Jieshan



> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting

[jira] [Commented] (HBASE-4152) Rename o.a.h.h.regionserver.wal.WALObserver to o.a.h.h.regionserver.wal.WALActionsListener

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081425#comment-13081425
 ] 

jirapos...@reviews.apache.org commented on HBASE-4152:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1317/#review1336
---

Ship it!


+1 Be careful when you apply though; seems to be an issue where patch does not 
apply to one of the files

- Michael


On 2011-08-08 19:26:08, Ted Yu wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1317/
bq.  ---
bq.  
bq.  (Updated 2011-08-08 19:26:08)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Rename o.a.h.h.regionserver.wal.WALObserver to 
o.a.h.h.regionserver.wal.WALActionsListener 
bq.  
bq.  
bq.  This addresses bug HBASE-4152.
bq.  https://issues.apache.org/jira/browse/HBASE-4152
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1154588 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 
1154588 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 
1154588 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 
PRE-CREATION 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 
1154588 
bq.
/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
 1154588 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java 
1154588 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 PRE-CREATION 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 
1154588 
bq.
/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1154588 
bq.  
bq.  Diff: https://reviews.apache.org/r/1317/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Unit test suite.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Ted
bq.  
bq.



> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 
> ---
>
> Key: HBASE-4152
> URL: https://issues.apache.org/jira/browse/HBASE-4152
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Andrew Purtell
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4152.txt
>
>
> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081427#comment-13081427
 ] 

Gary Helmling commented on HBASE-4180:
--

@Bochun

I'll add a patch that checks the return value of isSecurityEnabled() in 
User.login().  This should avert the ClassNotFoundException.

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081429#comment-13081429
 ] 

stack commented on HBASE-4180:
--

@Bochun See comment here:

https://issues.apache.org/jira/browse/HBASE-2233?focusedCommentId=13046828&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13046828

You could take the patches from the hadoop issues referenced and they should 
apply to 0.21

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4179.
--

Resolution: Fixed

I committed patch to TRUNK.  I did the line wraps that Ted suggested on commit 
but yeah, for future patches if you could try and respect the <80 chars per 
line that'd be grand.

Thank you for the patch Michael.

> Failed to run RowCounter on top of Hadoop branch-0.22
> -
>
> Key: HBASE-4179
> URL: https://issues.apache.org/jira/browse/HBASE-4179
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.92.0
> Environment: Running hbase on top of hadoop branch-0.22
>Reporter: Michael Weng
>Assignee: Michael Weng
> Fix For: 0.92.0
>
> Attachments: HBASE-4179-trunk.patch
>
>
> :~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
> ~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  
> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Bochun Bai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081433#comment-13081433
 ] 

Bochun Bai commented on HBASE-4180:
---

Thanks @stack.
@Gray, do you mean changes like this?
if (IS_SECURE_HADOOP && UserGroupInfomation.isSecurityEnabed()) {
  SecurityHadoopUser.login..
} else {
  HadoopUser.login..
}



> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3287) Add option to cache blocks on hfile write and evict blocks on hfile close

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081432#comment-13081432
 ] 

stack commented on HBASE-3287:
--

@Yi This issue looks like its old enough for the patch herein to have made 
0.90. Are you not finding it there?  (I believe there were issues found with 
the way we cached blocks on write found subsequent to the application of this 
patch, wrong name was used when file was added to cache -- in trunk its been 
redone)

> Add option to cache blocks on hfile write and evict blocks on hfile close
> -
>
> Key: HBASE-3287
> URL: https://issues.apache.org/jira/browse/HBASE-3287
> Project: HBase
>  Issue Type: New Feature
>  Components: io, regionserver
>Affects Versions: 0.90.0
>Reporter: Jonathan Gray
>Assignee: Jonathan Gray
> Fix For: 0.92.0
>
> Attachments: HBASE-3287-FINAL-trunk.patch
>
>
> This issue is about adding configuration options to add/remove from the block 
> cache when creating/closing files.  For use cases with lots of flushing and 
> compacting, this might be desirable to prevent cache misses and maximize the 
> effective utilization of total block cache capacity.
> The first option, {{hbase.rs.cacheblocksonwrite}}, will make it so we 
> pre-cache blocks as we are writing out new files.
> The second option, {{hbase.rs.evictblocksonclose}}, will make it so we evict 
> blocks when files are closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS Flakiness

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081435#comment-13081435
 ] 

stack commented on HBASE-4078:
--

I added some comments over on reviewboard but then realized that the patch 
looks like hbase-4078.  Is it same patch?  Thanks.

> Silent Data Offlining During HDFS Flakiness
> ---
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
>  Issue Type: Bug
>  Components: io, regionserver
>Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
>Reporter: Nicolas Spiegelberg
>Assignee: Pritam Damania
>Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 .  The bug fix for this JIRA is a temporary workaround for 
> improperly moving partially-written files from TMP into the region directory 
> when a FS error occurs.  Unfortunately, the fix is to ignore all IO 
> exceptions, which masks off-lining due to FS flakiness.  We need to 
> permanently fix the problem that created HBASE-1436 & then at least have the 
> option to not open a region during times of flakey FS.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081438#comment-13081438
 ] 

ramkrishna.s.vasudevan commented on HBASE-4095:
---

Patch looks good. 



> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 16:29:58

[jira] [Commented] (HBASE-4152) Rename o.a.h.h.regionserver.wal.WALObserver to o.a.h.h.regionserver.wal.WALActionsListener

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081436#comment-13081436
 ] 

Ted Yu commented on HBASE-4152:
---

Integrated to TRUNK.

Thanks for the review Jonathan and Michael.

> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 
> ---
>
> Key: HBASE-4152
> URL: https://issues.apache.org/jira/browse/HBASE-4152
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Andrew Purtell
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4152.txt
>
>
> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4078) Silent Data Offlining During HDFS Flakiness

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081439#comment-13081439
 ] 

stack commented on HBASE-4078:
--

Pardon my sillyness above where I am saying that the patch for this issue is 
the same as the patch for this issue.

> Silent Data Offlining During HDFS Flakiness
> ---
>
> Key: HBASE-4078
> URL: https://issues.apache.org/jira/browse/HBASE-4078
> Project: HBase
>  Issue Type: Bug
>  Components: io, regionserver
>Affects Versions: 0.89.20100924, 0.90.3, 0.92.0
>Reporter: Nicolas Spiegelberg
>Assignee: Pritam Damania
>Priority: Blocker
> Attachments: 0001-Validate-store-files.patch
>
>
> See HBASE-1436 .  The bug fix for this JIRA is a temporary workaround for 
> improperly moving partially-written files from TMP into the region directory 
> when a FS error occurs.  Unfortunately, the fix is to ignore all IO 
> exceptions, which masks off-lining due to FS flakiness.  We need to 
> permanently fix the problem that created HBASE-1436 & then at least have the 
> option to not open a region during times of flakey FS.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081440#comment-13081440
 ] 

stack commented on HBASE-4095:
--

I'd clean up the extra spaces the patch adds before commit.  It adds a bunch.

Reviewing I kept scratching my head.  This seems to be a slightly odd case in 
that it happens on a cluster of three nodes or less yet we're changing the way 
a pretty important piece of code works.  Any chance of a unit test Jieshan?  
The HLog has a bunch of tests that do various mock ups.  Shouldn't be too hard 
studying what goes on there to figure how to get a little test in.  If you 
think this too much, that's fine, we can commit since you got two +1s but I 
thought I'd ask anyways.

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,

[jira] [Commented] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081441#comment-13081441
 ] 

jirapos...@reviews.apache.org commented on HBASE-4181:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1402/
---

Review request for hbase.


Summary
---

HConnectionManager can't find cached HRegionInterface makes client very slow
Addressing.createHostAndPortStr(hostname, port); //the Addressing created a 
address like "node41:60010"
..
this.servers.put(address.toString(), server); 

//but here address.toString() send an address like "node41/10.61.2l.171:60010


This addresses bug HBase-4181.
https://issues.apache.org/jira/browse/HBase-4181


Diffs
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
 1155206 

Diff: https://reviews.apache.org/r/1402/diff


Testing
---

Tests passed locally. 


Thanks,

Jia



> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4114) Metrics for HFile HDFS block locality

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081445#comment-13081445
 ] 

stack commented on HBASE-4114:
--

Very nice work Ming.  Patch looks good.  Nice tests (I didn't now you could set 
hostnames in minidfscluster).

To be clear, we're adding a lookup of a storefiles filestatus after each open 
of a storefile (pity open doesn't just return this).  Thats extra loading on 
namenode but should be fine I think in the scheme of things (I see you did 
calc's above -- so yeah, should be fine)

One method is named with a capital: MapHostNameToServerName.  This is 
unorthodox.  Other "heresies" are the new line before a function curly-bracket 
open, sometimes, or on loops.

In HostAndWeight, what if two regionservers running on same host?  Should it be 
HostPortAndWeight?

You should just do a return on the string in the toString:

{code}
+String s = "number of unique hosts in the disribution=" +
+  this.hostAndWeights.size();
+return s;
{code}

... and do we need that preamble 'number of unique hosts in the ...' (is this 
string right?)

Is getTopBlockLocations actually used by the balancer?  I don't see it.

You might want to protect yourself against clusterstatus and master services 
being null in the regionserver.

> Metrics for HFile HDFS block locality
> -
>
> Key: HBASE-4114
> URL: https://issues.apache.org/jira/browse/HBASE-4114
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics, regionserver
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, 
> HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, HBASE-4114-trunk.patch
>
>
> Normally, when we put hbase and HDFS in the same cluster ( e.g., region 
> server runs on the datenode ), we have a reasonably good data locality, as 
> explained by Lars. Also Work has been done by Jonathan to address the startup 
> situation.
> There are scenarios where regions can be on a different machine from the 
> machines that hold the underlying HFile blocks, at least for some period of 
> time. This will have performance impact on whole table scan operation and map 
> reduce job during that time.
> 1.After load balancer moves the region and before compaction (thus 
> generate HFile on the new region server ) on that region, HDFS block can be 
> remote.
> 2.When a new machine is added, or removed, Hbase's region assignment 
> policy is different from HDFS's block reassignment policy.
> 3.Even if there is no much hbase activity, HDFS can load balance HFile 
> blocks as other non-hbase applications push other data to HDFS.
> Lots has been or will be done in load balancer, as summarized by Ted. I am 
> curious if HFile HDFS block locality should be used as another factor here.
> I have done some experiments on how HDFS block locality can impact map reduce 
> latency. First we need to define a metrics to measure HFile data locality.
> Metrics defintion:
> For a given table, or a region server, or a region, we can define the 
> following. The higher the value, the more local HFile is from region server's 
> point of view.
> HFile locality index = ( Total number of HDFS blocks that can be retrieved 
> locally by the region server ) / ( Total number of HDFS blocks for all HFiles 
> )
> Test Results:
> This is to show how HFile locality can impact the latency. It is based on a 
> table with 1M rows, 36KB per row; regions are distributed in balance. The map 
> job is RowCounter.
> HFile Locality Index  Map job latency ( in sec )
> 28%   157
> 36%   150
> 47%   142
> 61%   133
> 73%   122
> 89%   103
> 99%   95
> So the first suggestion is to expose HFile locality index as a new region 
> server metrics. It will be ideal if we can somehow measure HFile locality 
> index on a per map job level.
> Regarding if/when we should include that as another factor for load balancer, 
> that will be a different work item. It is unclear how load balancer can take 
> various factors into account to come up with the best load balancer strategy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081447#comment-13081447
 ] 

Jieshan Bean commented on HBASE-4095:
-

Yes, this problem happens in a special scenario. Though I've taken many times 
of unit tests on it, I'm running more. The result will be attached later.
I also took several days tests on the patch in our real environment(During the 
time, I killed some datanodes and restarted them, and repeated the steps again 
and again),I paid special attention to the HLog paths, it' indeed a good 
solution to this problem.

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HBase-4095-V6-branch.patch, 
> HBase-4095-V6-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,13099

[jira] [Updated] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-4180:
-

Attachment: HBASE-4180_0.90.patch

Patch against 0.90 branch adding check for isSecurityEnabled() within 
User.login()

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch, HBASE-4180_0.90.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4178) Use of Random.nextLong() in HRegionServer.addScanner(...)

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081448#comment-13081448
 ] 

stack commented on HBASE-4178:
--

IIUC, if a regionserver restarted, then it'd start scannerids over at zero 
again.  Any scanners that had been running against the server when it died will 
have to go get new ids for the regions they had been scanning over in their new 
locations (Scanner ids are scoped to a region scan; client sets up new scanner 
id every time it crosses into new region)

> Use of Random.nextLong() in HRegionServer.addScanner(...)
> -
>
> Key: HBASE-4178
> URL: https://issues.apache.org/jira/browse/HBASE-4178
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.3
>Reporter: Lars Hofhansl
>Priority: Minor
>
> ScannerIds are currently assigned by getting a random long. While it would be 
> a rare occurrence that two scanners received the same ids on the same region 
> server the results would seem to be... Bad.
> A client scanner would get results from a different server scanner, and maybe 
> only from some of the region servers.
> A safer approach would be using an AtomicLong. We do not have to worry about 
> running of numbers: If we got 1 scanners per second it'd take > 2.9m 
> years to reach 2^63.
> Then again the same reasoning would imply that this collisions would be 
> happening too rarely to be of concern (assuming a good random number 
> generator). So maybe this is a none-issue.
> AtomicLong would also imply a minor performance hit on multi core machines, 
> as it would force a memory barrier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary Helmling updated HBASE-4180:
-

Attachment: HBASE-4180_trunk.patch

Patch against trunk adding check for isSecurityEnabled() within User.login()

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch, HBASE-4180_0.90.patch, 
> HBASE-4180_trunk.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3855) Performance degradation of memstore because reseek is linear

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081451#comment-13081451
 ] 

stack commented on HBASE-3855:
--

@Andrew You think we are getting back bad answers from MemStore?  Looking at 
patch, I fail to see how it'd return an incorrect answer but maybe I'm missing 
something?

> Performance degradation of memstore because reseek is linear
> 
>
> Key: HBASE-3855
> URL: https://issues.apache.org/jira/browse/HBASE-3855
> Project: HBase
>  Issue Type: Improvement
>Reporter: dhruba borthakur
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: memstoreReseek.txt, memstoreReseek2.txt
>
>
> The scanner use reseek to find the next row (or next column) as part of a 
> scan. The reseek code iterates over a Set to position itself at the right 
> place. If there are many thousands of kvs that need to be skipped over, then 
> the time-cost is very high. In this case, a seek would be far lesser in cost 
> than a reseek.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[Gary Helmling (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081452#comment-13081452
 ] 

Gary Helmling commented on HBASE-4180:
--

@Bochun

No, unfortunately that would break the ability to compile (or run) HBase 
against non-secure Hadoop.

See the two attached patches (one for branch 0.90, one for trunk).  I assume 
you are running a 0.90 release?  Please try applying the 0.90 version of the 
patch and see if that fixes the problem for you.

I will be testing the patch as well against Hadoop 0.20.2, and a 0.20-security 
version.

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch, HBASE-4180_0.90.patch, 
> HBASE-4180_trunk.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4114) Metrics for HFile HDFS block locality

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081454#comment-13081454
 ] 

Ted Yu commented on HBASE-4114:
---

getTopBlockLocations() will be used in another JIRA which enhances load 
balancer.

Clusterstatus and master services are passed from master to balancer. Region 
server isn't involved.

Regards

> Metrics for HFile HDFS block locality
> -
>
> Key: HBASE-4114
> URL: https://issues.apache.org/jira/browse/HBASE-4114
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics, regionserver
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, 
> HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, HBASE-4114-trunk.patch
>
>
> Normally, when we put hbase and HDFS in the same cluster ( e.g., region 
> server runs on the datenode ), we have a reasonably good data locality, as 
> explained by Lars. Also Work has been done by Jonathan to address the startup 
> situation.
> There are scenarios where regions can be on a different machine from the 
> machines that hold the underlying HFile blocks, at least for some period of 
> time. This will have performance impact on whole table scan operation and map 
> reduce job during that time.
> 1.After load balancer moves the region and before compaction (thus 
> generate HFile on the new region server ) on that region, HDFS block can be 
> remote.
> 2.When a new machine is added, or removed, Hbase's region assignment 
> policy is different from HDFS's block reassignment policy.
> 3.Even if there is no much hbase activity, HDFS can load balance HFile 
> blocks as other non-hbase applications push other data to HDFS.
> Lots has been or will be done in load balancer, as summarized by Ted. I am 
> curious if HFile HDFS block locality should be used as another factor here.
> I have done some experiments on how HDFS block locality can impact map reduce 
> latency. First we need to define a metrics to measure HFile data locality.
> Metrics defintion:
> For a given table, or a region server, or a region, we can define the 
> following. The higher the value, the more local HFile is from region server's 
> point of view.
> HFile locality index = ( Total number of HDFS blocks that can be retrieved 
> locally by the region server ) / ( Total number of HDFS blocks for all HFiles 
> )
> Test Results:
> This is to show how HFile locality can impact the latency. It is based on a 
> table with 1M rows, 36KB per row; regions are distributed in balance. The map 
> job is RowCounter.
> HFile Locality Index  Map job latency ( in sec )
> 28%   157
> 36%   150
> 47%   142
> 61%   133
> 73%   122
> 89%   103
> 99%   95
> So the first suggestion is to expose HFile locality index as a new region 
> server metrics. It will be ideal if we can somehow measure HFile locality 
> index on a per map job level.
> Regarding if/when we should include that as another factor for load balancer, 
> that will be a different work item. It is unclear how load balancer can take 
> various factors into account to come up with the best load balancer strategy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4180) HBase should check the isSecurityEnabled flag

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081453#comment-13081453
 ] 

stack commented on HBASE-4180:
--

Patch looks good to me Gary.

> HBase should check the isSecurityEnabled flag
> -
>
> Key: HBASE-4180
> URL: https://issues.apache.org/jira/browse/HBASE-4180
> Project: HBase
>  Issue Type: Bug
>  Components: security
>Reporter: Bochun Bai
>Assignee: Bochun Bai
> Attachments: HBASE-4180.patch, HBASE-4180_0.90.patch, 
> HBASE-4180_trunk.patch
>
>
> Hadoop 0.21.0's UserGroupInfomation support the security check flag and 
> always returns false.
> HBase should check both the method existence and the return value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081456#comment-13081456
 ] 

jirapos...@reviews.apache.org commented on HBASE-4181:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1402/
---

(Updated 2011-08-09 05:53:25.766068)


Review request for hbase.


Summary (updated)
---

HConnectionManager can't find cached HRegionInterface makes client very slow
Addressing.createHostAndPortStr(hostname, port); //the Addressing created a 
address like "node41:60010"
..
this.servers.put(address.toString(), server); 

//but here address.toString() send an address like "node41/10.61.2l.171:60010


This addresses bug HBase-4181.
https://issues.apache.org/jira/browse/HBase-4181


Diffs
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
 1155206 

Diff: https://reviews.apache.org/r/1402/diff


Testing
---

Tests passed locally. 


Thanks,

Jia



> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4114) Metrics for HFile HDFS block locality

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081455#comment-13081455
 ] 

stack commented on HBASE-4114:
--

bq. ...in the regionserver.

Should have been 'in the loadbalancer'

> Metrics for HFile HDFS block locality
> -
>
> Key: HBASE-4114
> URL: https://issues.apache.org/jira/browse/HBASE-4114
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics, regionserver
>Reporter: Ming Ma
>Assignee: Ming Ma
> Attachments: HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, 
> HBASE-4114-trunk.patch, HBASE-4114-trunk.patch, HBASE-4114-trunk.patch
>
>
> Normally, when we put hbase and HDFS in the same cluster ( e.g., region 
> server runs on the datenode ), we have a reasonably good data locality, as 
> explained by Lars. Also Work has been done by Jonathan to address the startup 
> situation.
> There are scenarios where regions can be on a different machine from the 
> machines that hold the underlying HFile blocks, at least for some period of 
> time. This will have performance impact on whole table scan operation and map 
> reduce job during that time.
> 1.After load balancer moves the region and before compaction (thus 
> generate HFile on the new region server ) on that region, HDFS block can be 
> remote.
> 2.When a new machine is added, or removed, Hbase's region assignment 
> policy is different from HDFS's block reassignment policy.
> 3.Even if there is no much hbase activity, HDFS can load balance HFile 
> blocks as other non-hbase applications push other data to HDFS.
> Lots has been or will be done in load balancer, as summarized by Ted. I am 
> curious if HFile HDFS block locality should be used as another factor here.
> I have done some experiments on how HDFS block locality can impact map reduce 
> latency. First we need to define a metrics to measure HFile data locality.
> Metrics defintion:
> For a given table, or a region server, or a region, we can define the 
> following. The higher the value, the more local HFile is from region server's 
> point of view.
> HFile locality index = ( Total number of HDFS blocks that can be retrieved 
> locally by the region server ) / ( Total number of HDFS blocks for all HFiles 
> )
> Test Results:
> This is to show how HFile locality can impact the latency. It is based on a 
> table with 1M rows, 36KB per row; regions are distributed in balance. The map 
> job is RowCounter.
> HFile Locality Index  Map job latency ( in sec )
> 28%   157
> 36%   150
> 47%   142
> 61%   133
> 73%   122
> 89%   103
> 99%   95
> So the first suggestion is to expose HFile locality index as a new region 
> server metrics. It will be ideal if we can somehow measure HFile locality 
> index on a per map job level.
> Regarding if/when we should include that as another factor for load balancer, 
> that will be a different work item. It is unclear how load balancer can take 
> various factors into account to come up with the best load balancer strategy.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4039) Users should be able to choose custom TableInputFormats without modifying TableMapReduceUtil.initTableMapperJob().

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4039:
-

  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

Applied to trunk.  Thanks for patch Brock (and review Ted)

> Users should be able to choose custom TableInputFormats without modifying 
> TableMapReduceUtil.initTableMapperJob().
> --
>
> Key: HBASE-4039
> URL: https://issues.apache.org/jira/browse/HBASE-4039
> Project: HBase
>  Issue Type: New Feature
>  Components: mapreduce
>Affects Versions: 0.90.3
> Environment: HBase-0.90.3. OS, hardware specs not relevant.
>Reporter: Edward Choi
>Assignee: Brock Noland
>Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-4039.1.patch
>
>
> Currently, 
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob() 
> forces any Hbase job to use the default TableInputFormat.class as the job's 
> input format class.
> "job.setInputFormatClass(TableInputFormat.class);" ==> This line is included 
> in initTableMapperJob().
> This restriction causes users to modify initTableMapperJob() in addition to 
> implementing their own TableInputFormat. 
> It would be nicer if users can use custom TableInputFormats without 
> additionally tampering with HBase source code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[Liu Jia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Jia updated HBASE-4181:
---

Attachment: HBASE-4181.patch

some modification

> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HBASE-4181.patch, HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081470#comment-13081470
 ] 

Hudson commented on HBASE-4169:
---

Integrated in HBase-TRUNK #2097 (See 
[https://builds.apache.org/job/HBase-TRUNK/2097/])
HBASE-4169 FSUtils LeaseRecovery for non HDFS FileSystems

stack : 
Files : 
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
>Assignee: Lohit Vijayarenu
> Fix For: 0.92.0
>
> Attachments: 4169-v4.txt, 4169-v5.txt, HBASE-4169.1.patch, 
> HBASE-4169.2.patch, HBASE-4196.3.patch, HBASE-4196.3.v2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4179) Failed to run RowCounter on top of Hadoop branch-0.22

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081469#comment-13081469
 ] 

Hudson commented on HBASE-4179:
---

Integrated in HBase-TRUNK #2097 (See 
[https://builds.apache.org/job/HBase-TRUNK/2097/])
HBASE-4179 Failed to run RowCounter on top of Hadoop branch-0.22

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapred/Driver.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java


> Failed to run RowCounter on top of Hadoop branch-0.22
> -
>
> Key: HBASE-4179
> URL: https://issues.apache.org/jira/browse/HBASE-4179
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.92.0
> Environment: Running hbase on top of hadoop branch-0.22
>Reporter: Michael Weng
>Assignee: Michael Weng
> Fix For: 0.92.0
>
> Attachments: HBASE-4179-trunk.patch
>
>
> :~/hadoop$ HADOOP_CLASSPATH=`~/hbase/bin/hbase classpath` bin/hadoop jar 
> ~/hbase/hbase-0.91.0-SNAPSHOT.jar rowcounter usertable 
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.hadoop.util.ProgramDriver.driver([Ljava/lang/String;)V 
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51) 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  
> at java.lang.reflect.Method.invoke(Method.java:597) 
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192) 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4152) Rename o.a.h.h.regionserver.wal.WALObserver to o.a.h.h.regionserver.wal.WALActionsListener

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081468#comment-13081468
 ] 

Hudson commented on HBASE-4152:
---

Integrated in HBase-TRUNK #2097 (See 
[https://builds.apache.org/job/HBase-TRUNK/2097/])
HBASE-4152 Rename o.a.h.h.regionserver.wal.WALObserver to 
o.a.h.h.regionserver.wal.WALActionsListener

tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java


> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 
> ---
>
> Key: HBASE-4152
> URL: https://issues.apache.org/jira/browse/HBASE-4152
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Andrew Purtell
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4152.txt
>
>
> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[Liu Jia (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081471#comment-13081471
 ] 

Liu Jia commented on HBASE-4181:


@Ted 
With this line
{noformat} 
String rsName = isa != null ? isa.toString() : Addressing
.createHostAndPortStr(hostname, port); 
{noformat} 
if the isa is not null, rsName will be initiated by isa.toString() method,
So the above part and the following part should be changed to 
InetSocketAddress.createUnresolved(host,port) together to avoid DNS query.
{noformat} 
this.servers.put(address.toString(), server);
{noformat} 



> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HBASE-4181.patch, HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081472#comment-13081472
 ] 

jirapos...@reviews.apache.org commented on HBASE-4181:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1402/
---

(Updated 2011-08-09 06:41:43.389550)


Review request for hbase.


Changes
---

to avoid DNS query.


Summary
---

HConnectionManager can't find cached HRegionInterface makes client very slow
Addressing.createHostAndPortStr(hostname, port); //the Addressing created a 
address like "node41:60010"
..
this.servers.put(address.toString(), server); 

//but here address.toString() send an address like "node41/10.61.2l.171:60010


This addresses bug HBase-4181.
https://issues.apache.org/jira/browse/HBase-4181


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
 1155226 

Diff: https://reviews.apache.org/r/1402/diff


Testing
---

Tests passed locally. 


Thanks,

Jia



> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HBASE-4181.patch, HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4181) HConnectionManager can't find cached HRegionInterface makes client very slow

[Liu Jia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Jia updated HBASE-4181:
---

Attachment: HBASE-4181-trunk-v2.patch

> HConnectionManager can't find cached HRegionInterface makes client very slow
> 
>
> Key: HBASE-4181
> URL: https://issues.apache.org/jira/browse/HBASE-4181
> Project: HBase
>  Issue Type: Bug
>  Components: client
>Affects Versions: 0.90.4, 0.92.0
>Reporter: Liu Jia
>Priority: Critical
>  Labels: HConnectionManager
> Attachments: HBASE-4181-trunk-v2.patch, HBASE-4181.patch, 
> HConnectionManager.patch
>
>
> HRegionInterface getHRegionConnection(final String hostname,
> final int port, final InetSocketAddress isa, final boolean master)
> throws IOException 
> /
>   String rsName = isa != null ? isa.toString() : Addressing
>   .createHostAndPortStr(hostname, port); 
> 
> here,if isa is null, the Addressing created a address like "node41:60010"
>  should 
> use "isa.toString():new InetSocketAddress(hostname, port).toString();" 
>  instead 
> of "Addressing.createHostAndPortStr(hostname, port);"
>   server = this.servers.get(rsName);  
>   if (server == null) {
> // create a unique lock for this RS (if necessary)
> this.connectionLock.putIfAbsent(rsName, rsName);
> // get the RS lock
> synchronized (this.connectionLock.get(rsName)) {
>   // do one more lookup in case we were stalled above
>   server = this.servers.get(rsName);
>   if (server == null) {
> try {
>   if (clusterId.hasId()) {
> conf.set(HConstants.CLUSTER_ID, clusterId.getId());
>   }
>   // Only create isa when we need to.
>   InetSocketAddress address = isa != null ? isa
>   : new InetSocketAddress(hostname, port);
>   // definitely a cache miss. establish an RPC for this RS
>   server = (HRegionInterface) HBaseRPC.waitForProxy(
>   serverInterfaceClass, HRegionInterface.VERSION, address,
>   this.conf, this.maxRPCAttempts, this.rpcTimeout,
>   this.rpcTimeout);
>   this.servers.put(address.toString(), server);
>   
> but here address.toString() send an address like 
> "node41/10.61.2l.171:60010
> so this method can never get cached address and make client request very 
> slow due to it's synchronized.
>   
>   
> } catch (RemoteException e) {
>   LOG.warn("RemoteException connecting to RS", e);
>   // Throw what the RemoteException was carrying.
>   throw RemoteExceptionHandler.decodeRemoteException(e);
> }
>   }
> }
> ///

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081476#comment-13081476
 ] 

jirapos...@reviews.apache.org commented on HBASE-4176:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1326/
---

(Updated 2011-08-09 06:46:26.124460)


Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, and Jonathan Gray.


Changes
---

1. Added author and date to the javadocs and put in a reference to the relevant 
JIRA
2. Fixed some minor formatting and spelling issues


Summary
---

https://issues.apache.org/jira/browse/HBASE-4176: Exposing HBase Filters to the 
Thrift API

Currently, to use any of the filters, one has to explicitly add a scanner for 
the filter in the Thrift API making it messy and long. 
With this patch, I am trying to add support for all the filters in a clean way. 
The user specifies a filter via a string. The string is parsed on the server to 
construct the filter. More information can be found in the attached document 
named Filter Language

This patch is trying to extend and further the progress made by the patches in 
HBASE-1744

There is document attached to the HBASE-4176 JIRA that describes this patch in 
further detail


This addresses bug HBASE-4176.
https://issues.apache.org/jira/browse/HBASE-4176


Diffs (updated)
-

  /src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FamilyFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/Filter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FilterList.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/FirstKeyOnlyFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/KeyOnlyFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/filter/ParseFilter.java PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/QualifierFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/RowFilter.java 1155098 
  
/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueExcludeFilter.java
 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java 
1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java 1155098 
  /src/main/java/org/apache/hadoop/hbase/filter/ValueFilter.java 1155098 
  /src/test/java/org/apache/hadoop/hbase/filter/TestParseFilter.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/1326/diff


Testing
---

patch includes one test: TestParseFilter.java


Thanks,

Anirudh



> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4176) Exposing HBase Filters to the Thrift API

[Anirudh Todi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081478#comment-13081478
 ] 

Anirudh Todi commented on HBASE-4176:
-

@Stack and @Ted - thanks for reviewing the patch!

The patch does not depend on hbase-1744.

@Stack - I am not changing the meaning of LESS_OR_EQUAL, GREATER etc - if you 
look at the existing version of CompareFilter, LESS_OR_EQUAL seems to 
correspond to "<" instead of "<=" and vice verse for LESS. The same goes for 
GREATER and GREATER_OR_EQUAL and for EQUAL and NOT_EQUAL

@Stack when you say you should be able to specify a filter using the full 
package name - does that mean the filter string could look like:

{noformat}
"org.apache.hadoop.hbase.filter.KeyOnlyFilter()"
{noformat}

@Stack - I thought a great deal about how I would implement it without 
hard-coding the names of the filters I am supporting. 
However, I was unable to come up with anything. if you or anyone else has an 
idea - I'd love to hear it

@Stack - I would have to look into how we would extend this mini-language to 
allow one to construct Filter objects from the shell. It's an interesting idea 
though!

> Exposing HBase Filters to the Thrift API
> 
>
> Key: HBASE-4176
> URL: https://issues.apache.org/jira/browse/HBASE-4176
> Project: HBase
>  Issue Type: Improvement
>  Components: thrift
>Reporter: Anirudh Todi
>Assignee: Anirudh Todi
>Priority: Minor
> Attachments: Filter Language.docx, HBASE-4176.patch
>
>
> Currently, to use any of the filters, one has to explicitly add a scanner for 
> the filter in the Thrift API making it messy and long. With this patch, I am 
> trying to add support for all the filters in a clean way. The user specifies 
> a filter via a string. The string is parsed on the server to construct the 
> filter. More information can be found in the attached document named Filter 
> Language
> This patch is trying to extend and further the progress made by the patches 
> in the HBASE-1744 JIRA (https://issues.apache.org/jira/browse/HBASE-1744)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Jieshan Bean (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jieshan Bean updated HBASE-4095:


Attachment: surefire-report-V5-trunk.html
HBase-4095-V5-trunk.patch

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 16:29:58,775 DEBUG org.apache.hadoop.hbase.regionserver.LogRoller: 
> Hlog roll period 360ms elapsed
> 2

[jira] [Commented] (HBASE-4095) Hlog may not be rolled in a long time if checkLowReplication's request of LogRoll is blocked

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081027#comment-13081027
 ] 

Ted Yu commented on HBASE-4095:
---

Ran through unit tests.
+1 on patch.

> Hlog may not be rolled in a long time if checkLowReplication's request of 
> LogRoll is blocked
> 
>
> Key: HBASE-4095
> URL: https://issues.apache.org/jira/browse/HBASE-4095
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.3
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Attachments: HBASE-4095-90-v2.patch, HBASE-4095-90.patch, 
> HBASE-4095-trunk-v2.patch, HBASE-4095-trunk.patch, 
> HBase-4095-V4-Branch.patch, HBase-4095-V5-Branch.patch, 
> HBase-4095-V5-trunk.patch, HlogFileIsVeryLarge.gif, 
> RelatedLogs2011-07-28.txt, TestResultForPatch-V4.rar, 
> hbase-root-regionserver-193-195-5-111.rar, surefire-report-V5-trunk.html, 
> surefire-report-branch.html
>
>
> Some large Hlog files(Larger than 10G) appeared in our environment, and I got 
> the reason why they got so huge:
> 1. The replicas is less than the expect number. So the method of 
> checkLowReplication will be called each sync.
> 2. The method checkLowReplication request log-roll first, and set 
> logRollRequested as true: 
> {noformat}
> private void checkLowReplication() {
> // if the number of replicas in HDFS has fallen below the initial
> // value, then roll logs.
> try {
>   int numCurrentReplicas = getLogReplication();
>   if (numCurrentReplicas != 0 &&
> numCurrentReplicas < this.initialReplication) {
>   LOG.warn("HDFS pipeline error detected. " +
>   "Found " + numCurrentReplicas + " replicas but expecting " +
>   this.initialReplication + " replicas. " +
>   " Requesting close of hlog.");
>   requestLogRoll();
>   logRollRequested = true;
>   }
> } catch (Exception e) {
>   LOG.warn("Unable to invoke DFSOutputStream.getNumCurrentReplicas" + e +
> " still proceeding ahead...");
> }
> }
> {noformat}
> 3.requestLogRoll() just commit the roll request. It may not execute in time, 
> for it must got the un-fair lock of cacheFlushLock.
> But the lock may be carried by the cacheflush threads.
> 4.logRollRequested was true until the log-roll executed. So during the time, 
> each request of log-roll in sync() was skipped.
> Here's the logs while the problem happened(Please notice the file size of 
> hlog "193-195-5-111%3A20020.1309937386639" in the last row):
> 2011-07-06 15:28:59,284 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 15:29:56,929 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: 
> HDFS pipeline error detected. Found 2 replicas but expecting 3 replicas.  
> Requesting close of hlog.
> 2011-07-06 15:29:56,933 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Renaming flushed file at 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/.tmp/4656903854447026847
>  to 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983
> 2011-07-06 15:29:57,391 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Added 
> hdfs://193.195.5.112:9000/hbase/Htable_UFDR_034/a3780cf0c909d8cf8f8ed618b290cc95/value/8603005630220380983,
>  entries=445880, sequenceid=248900, memsize=207.5m, filesize=130.1m
> 2011-07-06 15:29:57,478 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Finished memstore flush of ~207.5m for region 
> Htable_UFDR_034,07664,1309936974158.a3780cf0c909d8cf8f8ed618b290cc95. in 
> 10839ms, sequenceid=248900, compaction requested=false
> 2011-07-06 15:28:59,236 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309926531955,
>  entries=216459, filesize=2370387468. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119
> 2011-07-06 15:29:46,714 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: 
> Roll 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937339119,
>  entries=32434, filesize=239589754. New hlog 
> /hbase/.logs/193-195-5-111,20020,1309922880081/193-195-5-111%3A20020.1309937386639
> 2011-07-06 16:29:58,775 DEBUG org.apache.hadoop.hbase.regionserver.LogRoller: 
> Hlog roll period 360ms elapsed
> 2011-07

[jira] [Commented] (HBASE-4152) Rename o.a.h.h.regionserver.wal.WALObserver to o.a.h.h.regionserver.wal.WALActionsListener

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081037#comment-13081037
 ] 

jirapos...@reviews.apache.org commented on HBASE-4152:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1317/
---

Review request for hbase.


Summary
---

Rename o.a.h.h.regionserver.wal.WALObserver to 
o.a.h.h.regionserver.wal.WALActionsListener 


This addresses bug HBASE-4152.
https://issues.apache.org/jira/browse/HBASE-4152


Diffs
-

  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java 1154588 
  
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 PRE-CREATION 
  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 
1154588 
  
/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1154588 
  
/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
 1154588 
  /src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 1154588 
  /src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 1154588 
  
/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 
1154588 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1154588 

Diff: https://reviews.apache.org/r/1317/diff


Testing
---

Unit test suite.


Thanks,

Ted



> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 
> ---
>
> Key: HBASE-4152
> URL: https://issues.apache.org/jira/browse/HBASE-4152
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Andrew Purtell
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4152.txt
>
>
> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4152) Rename o.a.h.h.regionserver.wal.WALObserver to o.a.h.h.regionserver.wal.WALActionsListener

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081040#comment-13081040
 ] 

jirapos...@reviews.apache.org commented on HBASE-4152:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1317/
---

(Updated 2011-08-08 16:39:46.462192)


Review request for hbase.


Summary
---

Rename o.a.h.h.regionserver.wal.WALObserver to 
o.a.h.h.regionserver.wal.WALActionsListener 


This addresses bug HBASE-4152.
https://issues.apache.org/jira/browse/HBASE-4152


Diffs (updated)
-

  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 
1154588 
  
/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1154588 
  /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java 1154588 
  
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1154588 
  /src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 1154588 
  /src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 1154588 
  
/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 
PRE-CREATION 
  /src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 
1154588 
  
/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
 1154588 

Diff: https://reviews.apache.org/r/1317/diff


Testing
---

Unit test suite.


Thanks,

Ted



> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 
> ---
>
> Key: HBASE-4152
> URL: https://issues.apache.org/jira/browse/HBASE-4152
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Andrew Purtell
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4152.txt
>
>
> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4169) FSUtils LeaseRecovery for non HDFS FileSystems.

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081058#comment-13081058
 ] 

Todd Lipcon commented on HBASE-4169:


This patch is a little strange - why use reflection here to access the static 
method in FSHDFSUtils instead of just creating a new interface or abstract 
class, and using non-static methods?

I think a more conventional approach would be:
- Create an abstract class FSUtils, which declares either default 
implementations or abstract methods for all of the things that different file 
systems would want to override
- This class would also have a method getInstance(Filesystem, Configuration), 
which returns a subclass of FSUtils dependent on scheme, using 
ReflectionUtils.newInstance
- If this ever becomes a performance hotspot we could add a cache here, but for 
rare operations like recoverLease it shouldn't matter


> FSUtils LeaseRecovery for non HDFS FileSystems.
> ---
>
> Key: HBASE-4169
> URL: https://issues.apache.org/jira/browse/HBASE-4169
> Project: HBase
>  Issue Type: Bug
>  Components: util
>Affects Versions: 0.90.3, 0.90.4
>Reporter: Lohit Vijayarenu
> Attachments: HBASE-4169.1.patch, HBASE-4169.2.patch
>
>
> FSUtils.recoverFileLease uses HDFS's recoverLease method to get lease before 
> splitting hlog file.
> This might not work for other filesystem implementations. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4177) Handling read failures during recovery‏ - when HMaster calls Namenode recovery, recovery may be a failure leading to read failure while splitting logs

[ramkrishna.s.vasudevan (JIRA)]

Handling read failures during recovery‏ - when HMaster calls Namenode recovery, 
recovery may be a failure leading to read failure while splitting logs
--

 Key: HBASE-4177
 URL: https://issues.apache.org/jira/browse/HBASE-4177
 Project: HBase
  Issue Type: Bug
  Components: master
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan


As per the mailing thread with the heading
'Handling read failures during recovery‏' we found this problem.
As part of split Logs the HMaster calls Namenode recovery.  The recovery is an 
asynchronous process. 
In HDFS
===
Even though client is getting the updated block info from Namenode on first
read failure, client is discarding the new info and using the old info only
to retrieve the data from datanode. So, all the read
retries are failing. [Method parameter reassignment - Not reflected in
caller]. 
In HBASE
===
In HMaster code we tend to wait for  1sec.  But if the recovery had some 
failure then split log may not happen and may lead to dataloss.
So may be we need to decide upon the actual delay that needs to be introduced 
once Hmaster calls NN recovery.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4177) Handling read failures during recovery‏ - when HMaster calls Namenode recovery, recovery may be a failure leading to read failure while splitting logs

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081098#comment-13081098
 ] 

Ted Yu commented on HBASE-4177:
---

Looking at FSUtils.recoverFileLease(), we check the type of fs inside while 
loop. This is unnecessary.

w.r.t. soft limit for the lease, we have:
{code}
  if (waitedFor > FSConstants.LEASE_SOFTLIMIT_PERIOD) {
LOG.warn("Waited " + waitedFor + "ms for lease recovery on " + p +
  ":" + e.getMessage());
  }
{code}
I think we should wait for the remainder of soft limit (which is 60 seconds).


> Handling read failures during recovery‏ - when HMaster calls Namenode 
> recovery, recovery may be a failure leading to read failure while splitting 
> logs
> --
>
> Key: HBASE-4177
> URL: https://issues.apache.org/jira/browse/HBASE-4177
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>
> As per the mailing thread with the heading
> 'Handling read failures during recovery‏' we found this problem.
> As part of split Logs the HMaster calls Namenode recovery.  The recovery is 
> an asynchronous process. 
> In HDFS
> ===
> Even though client is getting the updated block info from Namenode on first
> read failure, client is discarding the new info and using the old info only
> to retrieve the data from datanode. So, all the read
> retries are failing. [Method parameter reassignment - Not reflected in
> caller]. 
> In HBASE
> ===
> In HMaster code we tend to wait for  1sec.  But if the recovery had some 
> failure then split log may not happen and may lead to dataloss.
> So may be we need to decide upon the actual delay that needs to be introduced 
> once Hmaster calls NN recovery.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4152) Rename o.a.h.h.regionserver.wal.WALObserver to o.a.h.h.regionserver.wal.WALActionsListener

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081139#comment-13081139
 ] 

jirapos...@reviews.apache.org commented on HBASE-4152:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1317/#review1324
---



/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


It seems like you've replaced  with   (an 
extra space everywhere including the imports)


- Jonathan


On 2011-08-08 16:39:46, Ted Yu wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1317/
bq.  ---
bq.  
bq.  (Updated 2011-08-08 16:39:46)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Rename o.a.h.h.regionserver.wal.WALObserver to 
o.a.h.h.regionserver.wal.WALActionsListener 
bq.  
bq.  
bq.  This addresses bug HBASE-4152.
bq.  https://issues.apache.org/jira/browse/HBASE-4152
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 
1154588 
bq.
/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1154588 
bq./src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java 
1154588 
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 PRE-CREATION 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
1154588 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 
1154588 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 
1154588 
bq.
/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 
PRE-CREATION 
bq./src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 
1154588 
bq.
/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
 1154588 
bq.  
bq.  Diff: https://reviews.apache.org/r/1317/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Unit test suite.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Ted
bq.  
bq.



> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 
> ---
>
> Key: HBASE-4152
> URL: https://issues.apache.org/jira/browse/HBASE-4152
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Andrew Purtell
>Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4152.txt
>
>
> Rename o.a.h.h.regionserver.wal.WALObserver to 
> o.a.h.h.regionserver.wal.WALActionsListener 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira