date:20120308


[ 
https://issues.apache.org/jira/browse/HBASE-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225909#comment-13225909
 ] 

Lars Hofhansl commented on HBASE-5520:
--

Didn't realize that KeyValueHeap does have a reseek method already. This makes 
this much simpler than I thought (so strike my comment in HBASE-2028).
Do you want to use requestSeek (with forward == true) as opposed to reseek, to 
make use of the lazy-seek optimization?


> Support seek() reseek() at RegionScanner
> 
>
> Key: HBASE-5520
> URL: https://issues.apache.org/jira/browse/HBASE-5520
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Anoop Sam John
> Attachments: HBASE-5520_1.patch
>
>
> seek() reseek() is not supported currently at the RegionScanner level. We can 
> support the same.
> This is created following the discussion under HBASE-2038

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225905#comment-13225905
 ] 

Lars Hofhansl commented on HBASE-5541:
--

That is not to say that there are no holes. :)


> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5512) Add support for INCLUDE_AND_SEEK_USING_HINT


 [ 
https://issues.apache.org/jira/browse/HBASE-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5512:
-

Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

Marking as won't fix, as we now question the usefulnes.

> Add support for INCLUDE_AND_SEEK_USING_HINT
> ---
>
> Key: HBASE-5512
> URL: https://issues.apache.org/jira/browse/HBASE-5512
> Project: HBase
>  Issue Type: Improvement
>Reporter: Zhihong Yu
>Assignee: Lars Hofhansl
> Attachments: 5512-v2.txt, 5512.txt
>
>
> This came up from HBASE-2038
> From Anoop:
> - What we wanted from the filter is include a row and then seek to the next 
> row which we are interested in. I cant see such a facility with our Filter 
> right now. Correct me if I am wrong. So suppose we already seeked to one row 
> and this need to be included in the result, then the Filter should return 
> INCLUDE. Then when the next next() call happens, then only we can return a 
> SEEK_USING_HINT. So one extra row reading is needed. This might create even 
> one unwanted HFileBlock fetch (who knows).
> Can we add reseek() at higher level?
> From Lars:
> Yep, for that we'd need to add INCLUDE_AND_SEEK_USING_HINT (similar to the 
> INCLUDE_AND_SEEK_NEXT_ROW that we already have). Shouldn't be hard to add, 
> I'm happy to do that, if that's the route we want to go with this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225900#comment-13225900
 ] 

stack commented on HBASE-5541:
--

Grand

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks

2012-03-08 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225898#comment-13225898
 ] 

Lars Hofhansl commented on HBASE-5541:
--

When we get to the CP post hooks all locks are released, the WAL is synched, 
and the mvcc is rolled forward. No edit will be removed from the memstore (that 
only happens when the wal sync fails).
What that means is that a post hook cannot undo the operation (the same as 
standalone Put and Delete).

I spent some time trying to poke holes in the design, I think it is tight.


> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5520) Support seek() reseek() at RegionScanner


[ 
https://issues.apache.org/jira/browse/HBASE-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225896#comment-13225896
 ] 

ramkrishna.s.vasudevan commented on HBASE-5520:
---

seek() api if we provide currently the Memstore scanner and store scanner does 
not allow going back from a current position.  
So only reseek() has been provided.  Please provide your comments.

> Support seek() reseek() at RegionScanner
> 
>
> Key: HBASE-5520
> URL: https://issues.apache.org/jira/browse/HBASE-5520
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Anoop Sam John
> Attachments: HBASE-5520_1.patch
>
>
> seek() reseek() is not supported currently at the RegionScanner level. We can 
> support the same.
> This is created following the discussion under HBASE-2038

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5520) Support seek() reseek() at RegionScanner

2012-03-08 Thread ramkrishna.s.vasudevan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5520:
--

Attachment: HBASE-5520_1.patch

Patch for trunk.

> Support seek() reseek() at RegionScanner
> 
>
> Key: HBASE-5520
> URL: https://issues.apache.org/jira/browse/HBASE-5520
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Anoop Sam John
> Attachments: HBASE-5520_1.patch
>
>
> seek() reseek() is not supported currently at the RegionScanner level. We can 
> support the same.
> This is created following the discussion under HBASE-2038

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225892#comment-13225892
 ] 

stack commented on HBASE-5325:
--

HBase and RegionServer should be under org.apache.hbase too I'd think.

> Expose basic information about the master-status through jmx beans 
> ---
>
> Key: HBASE-5325
> URL: https://issues.apache.org/jira/browse/HBASE-5325
> Project: HBase
>  Issue Type: Improvement
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
>Priority: Minor
> Fix For: 0.92.1, 0.94.0
>
> Attachments: HBASE-5325.1.patch, HBASE-5325.2.patch, 
> HBASE-5325.3.branch-0.92.patch, HBASE-5325.3.patch, HBASE-5325.wip.patch
>
>
> Similar to the Namenode and Jobtracker, it would be good if the hbase master 
> could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225886#comment-13225886
 ] 

stack commented on HBASE-5526:
--

FYI, hbase-default.xml makes it up into the reference guide.  You might then 
just beef up the hbase-default.xml description (like default seems to be 000 so 
you might suggest values with description of what the resulting perms will look 
like).  Good stuff lads.

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225882#comment-13225882
 ] 

stack commented on HBASE-5541:
--

Commit is fine.

What about case where the edits are applied, sync succeeds, but the cp 
postdelete and postput fail?  Then mvcc will have been updated but we have 
removed the edits from memstore.

i suppose its ok?  The read point is advanced but if we take this edit in 
isolation, no mutations made it through?

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set


[ 
https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225866#comment-13225866
 ] 

Hudson commented on HBASE-5529:
---

Integrated in HBase-TRUNK-security #132 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/132/])
HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory 
Chanan) (Revision 1298574)

 Result = FAILURE
larsh : 
Files : 
* /hbase/trunk/pom.xml


> MR test failures becuase MALLOC_ARENA_MAX is not set
> 
>
> Key: HBASE-5529
> URL: https://issues.apache.org/jira/browse/HBASE-5529
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce, test
>Affects Versions: 0.92.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.92.1, 0.94.0, 0.96.0
>
> Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch
>
>
> When running unit tests on CentOS 6 I get a bunch of unit test failures in 
> mapreduce-related tests due to:
> 2012-03-03 00:14:18,776 WARN  [Container Monitor]
> monitor.ContainersMonitorImpl$MonitoringThread(436): Container
> [pid=21446,containerID=container_1330762435849_0002_01_01] is
> running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb
> physical memory used; 6.9gb of 4.2gb virtual memory used. Killing
> container.
> Note: this also came up in the mapreduce project. See: 
> https://issues.apache.org/jira/browse/MAPREDUCE-3933
> Patch coming shortly

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225863#comment-13225863
 ] 

Hudson commented on HBASE-5541:
---

Integrated in HBase-TRUNK-security #132 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/132/])
HBASE-5541 Avoid holding the rowlock during HLog sync in 
HRegion.mutateRowWithLocks (Revision 1298678)

 Result = FAILURE
larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5514) Compile against hadoop 0.24-SNAPSHOT


[ 
https://issues.apache.org/jira/browse/HBASE-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225865#comment-13225865
 ] 

Hudson commented on HBASE-5514:
---

Integrated in HBase-TRUNK-security #132 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/132/])
HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1298488)

 Result = FAILURE
tedyu : 
Files : 
* /hbase/trunk/pom.xml
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java


> Compile against hadoop 0.24-SNAPSHOT
> 
>
> Key: HBASE-5514
> URL: https://issues.apache.org/jira/browse/HBASE-5514
> Project: HBase
>  Issue Type: Bug
>  Components: build, test
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Mingjie Lai
>Assignee: Mingjie Lai
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-5514-2.patch, HBASE-5514-3.patch, 
> HBASE-5514-4.patch, HBASE-5514.patch
>
>
> Need to compile hbase against the latest hadoop trunk which just had NN HA 
> merged in. 
> 1) add a hadoop 0.24 profile
> 2) HBASE-5480
> 3) HADOOP-8124 removed deprecated Syncable.sync(). It brings compile errors 
> for hbase against hadoop trunk(0.24). TestHLogSplit and TestHLog still call 
> the deprecated sync(). Need to replace it with hflush() so the compilation 
> can pass. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225864#comment-13225864
 ] 

Hudson commented on HBASE-5526:
---

Integrated in HBase-TRUNK-security #132 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/132/])
HBASE-5526 ^Cnfigurable file and directory based umask (Jesse Yates) 
(Revision 1298657)

 Result = FAILURE
larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/trunk/src/main/resources/hbase-default.xml
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java


> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache

[
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225862#comment-13225862
]

Hudson commented on HBASE-5074:
---

Integrated in HBase-TRUNK-security #132 (See
[https://builds.apache.org/job/HBase-TRUNK-security/132/])
[jira] [HBASE-5074] Support checksums in HBase block cache

Author: Dhruba

Summary:
HFile is enhanced to store a checksum for each block. HDFS checksum verification
is avoided while reading data into the block cache. On a checksum verification
failure, we retry the file system read request with hdfs checksums switched on
(thanks Todd).

I have a benchmark that shows that it reduces iops on the disk by about 40%. In
this experiment, the entire memory on the regionserver is allocated to the
regionserver's jvm and the OS buffer cache size is negligible. I also measured
negligible (<5%) additional cpu usage while using hbase-level checksums.

The salient points of this patch:

1. Each hfile's trailer used to have a 4 byte version number. I enhanced this so
that these 4 bytes can be interpreted as a (major version number, minor
version). Pre-existing hfiles have a minor version of 0. The new hfile format
has a minor version of 1 (thanks Mikhail). The hfile major version remains
unchanged at 2. The reason I did not introduce a new major version number is
because the code changes needed to store/read checksums do not differ much from
existing V2 writers/readers.

2. Introduced a HFileSystem object which is a encapsulates the FileSystem
objects needed to access data from hfiles and hlogs. HDFS FileSystem objects
already had the ability to switch off checksum verifications for reads.

3. The majority of the code changes are located in hbase.io.hfie package. The
retry of a read on an initial checksum failure occurs inside the hbase.io.hfile
package itself. The code changes to hbase.regionserver package are minor.

4. The format of a hfileblock is the header followed by the data followed by the
checksum(s). Each 16 K (configurable) size of data has a 4 byte checksum. The
hfileblock header has two additional fields: a 4 byte value to store the
bytesPerChecksum and a 4 byte value to store the size of the user data
(excluding the checksum data). This is well explained in the associated
javadocs.

5. I added a test to test backward compatibility. I will be writing more unit
tests that triggers checksum verification failures aggressively. I have left a
few redundant log messages in the code (just for easier debugging) and will
remove them in later stage of this patch. I will also be adding metrics on
number of checksum verification failures/success in a later version of this
diff.

6. By default, hbase-level checksums are switched on and hdfs level checksums
are switched off for hfile-reads. No changes to Hlog code path here.

Test Plan: The default setting is to switch on hbase checksums for hfile-reads,
thus all existing tests actually validate the new code pieces. I will be writing
more unit tests for triggering checksum verification failures.

Reviewers: mbautin

Reviewed By: mbautin

CC: JIRA, tedyu, mbautin, dhruba, todd, stack

Differential Revision: https://reviews.facebook.net/D1521 (Revision 1298641)

Result = FAILURE
mbautin :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* /hbase/trunk/src/main/java/o

[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row


[ 
https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225861#comment-13225861
 ] 

Hudson commented on HBASE-5515:
---

Integrated in HBase-TRUNK-security #132 (See 
[https://builds.apache.org/job/HBase-TRUNK-security/132/])
HBASE-5515 Add a processRow API that supports atomic multiple reads and 
writes on a row (Scott Chen) (Revision 1298533)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestProcessRowEndpoint.java


> Add a processRow API that supports atomic multiple reads and writes on a row
> 
>
> Key: HBASE-5515
> URL: https://issues.apache.org/jira/browse/HBASE-5515
> Project: HBase
>  Issue Type: New Feature
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.96.0
>
> Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, 
> HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, 
> HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, 
> HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, 
> HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, 
> HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, 
> HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, 
> HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, 
> HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, 
> HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch
>
>
> We have modified HRegion.java internally to do some atomic row processing. It 
> will be nice to have a plugable API for this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5547) Don't delete HFiles when in "backup mode"

2012-03-08 Thread Lars Hofhansl (Created) (JIRA)

Don't delete HFiles when in "backup mode"
-

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl


This came up in a discussion I had with Stack.
It would be nice if HBase could be notified that a backup is in progress (via a 
znode for example) and in that case either:
1. rename HFiles to be delete to .bck
2. rename the HFiles into a special directory
3. rename them to a general trash directory (which would not need to be tied to 
backup mode).

That way it should be able to get a consistent backup based on HFiles (HDFS 
snapshots or hard links would be better options here, but we do not have those).

#1 makes cleanup a bit harder.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225849#comment-13225849
 ] 

Lars Hofhansl commented on HBASE-5541:
--

@Stack: you OK with the explanation above?

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225821#comment-13225821
 ] 

stack commented on HBASE-5325:
--

This patch adds beans at org.apache.hbase, right?  HBase and RegionServer are 
bad names for mbeans.

> Expose basic information about the master-status through jmx beans 
> ---
>
> Key: HBASE-5325
> URL: https://issues.apache.org/jira/browse/HBASE-5325
> Project: HBase
>  Issue Type: Improvement
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
>Priority: Minor
> Fix For: 0.92.1, 0.94.0
>
> Attachments: HBASE-5325.1.patch, HBASE-5325.2.patch, 
> HBASE-5325.3.branch-0.92.patch, HBASE-5325.3.patch, HBASE-5325.wip.patch
>
>
> Similar to the Namenode and Jobtracker, it would be good if the hbase master 
> could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5543) Add a keepalive option for IPC connections

2012-03-08 Thread ramkrishna.s.vasudevan (Updated) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225818#comment-13225818
 ] 

stack commented on HBASE-5543:
--

bq. Instead of adding to the rpc to make it keep alive longer, maybe be make it 
async, returning some sort of uuid token that the client can poll (or get 
notified) for progress instead?

I like this idea.

> Add a keepalive option for IPC connections
> --
>
> Key: HBASE-5543
> URL: https://issues.apache.org/jira/browse/HBASE-5543
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors, ipc
>Reporter: Andrew Purtell
>
> On the user list someone wrote in with a connection failure due to a long 
> running coprocessor:
> {quote}
> On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
> 2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1, 
> client version=0, methodsFingerPrint=0), rpc version=1, client version=29, 
> methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
> 2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
> {quote}
> I suggested in response we might consider give our RPC a keepalive option for 
> calls that may run for a long time (like execCoprocessor).
> LarsH +1ed the idea:
> {quote}
> +1 on "keepalive". It's a shame (especially for long running server code) to 
> do all the work, just to find out at the end that the client has given up.
> Or maybe there should be a way to cancel an operation if the clients decides 
> it does not want to wait any longer (PostgreSQL does that for example). Here 
> that would mean the server would need to check periodically and coprocessors 
> would need to be written to support that - so maybe that's no-starter.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5545) region can't be opened for a long time. Because the creating File failed.


 [ 
https://issues.apache.org/jira/browse/HBASE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5545:
--

Fix Version/s: 0.92.2

> region can't be opened for a long time. Because the creating File failed.
> -
>
> Key: HBASE-5545
> URL: https://issues.apache.org/jira/browse/HBASE-5545
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.6
>Reporter: gaojinchao
>Assignee: gaojinchao
> Fix For: 0.90.7, 0.92.2
>
>
> Scenario:
> 
> 1. File is created 
> 2. But while writing data, all datanodes might have crashed. So writing data 
> will fail.
> 3. Now even if close is called in finally block, close also will fail and 
> throw the Exception because writing data failed.
> 4. After this if RS try to create the same file again, then 
> AlreadyBeingCreatedException will come.
> Suggestion to handle this scenario.
> ---
> 1. Check for the existence of the file, if exists delete the file and create 
> new file. 
> Here delete call for the file will not check whether the file is open or 
> closed.
> Overwrite Option:
> 
> 1. Overwrite option will be applicable if you are trying to overwrite a 
> closed file.
> 2. If the file is not closed, then even with overwrite option Same 
> AlreadyBeingCreatedException will be thrown.
> This is the expected behaviour to avoid the Multiple clients writing to same 
> file.
> Region server logs:
> org.apache.hadoop.ipc.RemoteException: 
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to 
> create file /hbase/test1/12c01902324218d14b17a5880f24f64b/.tmp/.regioninfo 
> for 
> DFSClient_hb_rs_158-1-131-48,20020,1331107668635_1331107669061_-252463556_25 
> on client 158.1.132.19 because current leaseholder is trying to recreate file.
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1570)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1440)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1382)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:658)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:547)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1137)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1133)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1131)
> at org.apache.hadoop.ipc.Client.call(Client.java:961)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:245)
> at $Proxy6.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at $Proxy6.create(Unknown Source)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:3643)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:778)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:364)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:630)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:611)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:518)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:424)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2672)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2658)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:330)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:116)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> [2012-03-07 20:51:45,858] [WARN ] 
> [RS_OPEN_REGION-158-1-131-48,20020,1331107668635-23] 
> [com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker 131] Retrying the 
> method call: public abstract void 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.create(

[jira] [Commented] (HBASE-5545) region can't be opened for a long time. Because the creating File failed.

2012-03-08 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225815#comment-13225815
 ] 

ramkrishna.s.vasudevan commented on HBASE-5545:
---

@Gao
Thanks for bringing in this. Also we need to see any other place like 
compaction, flush (incase similar scenarios) are there
Because every where we first create a file in tmp .  Then rename.
Also changing the version to 0.92.2

> region can't be opened for a long time. Because the creating File failed.
> -
>
> Key: HBASE-5545
> URL: https://issues.apache.org/jira/browse/HBASE-5545
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.6
>Reporter: gaojinchao
>Assignee: gaojinchao
> Fix For: 0.90.7, 0.92.2
>
>
> Scenario:
> 
> 1. File is created 
> 2. But while writing data, all datanodes might have crashed. So writing data 
> will fail.
> 3. Now even if close is called in finally block, close also will fail and 
> throw the Exception because writing data failed.
> 4. After this if RS try to create the same file again, then 
> AlreadyBeingCreatedException will come.
> Suggestion to handle this scenario.
> ---
> 1. Check for the existence of the file, if exists delete the file and create 
> new file. 
> Here delete call for the file will not check whether the file is open or 
> closed.
> Overwrite Option:
> 
> 1. Overwrite option will be applicable if you are trying to overwrite a 
> closed file.
> 2. If the file is not closed, then even with overwrite option Same 
> AlreadyBeingCreatedException will be thrown.
> This is the expected behaviour to avoid the Multiple clients writing to same 
> file.
> Region server logs:
> org.apache.hadoop.ipc.RemoteException: 
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to 
> create file /hbase/test1/12c01902324218d14b17a5880f24f64b/.tmp/.regioninfo 
> for 
> DFSClient_hb_rs_158-1-131-48,20020,1331107668635_1331107669061_-252463556_25 
> on client 158.1.132.19 because current leaseholder is trying to recreate file.
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1570)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1440)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1382)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:658)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:547)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1137)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1133)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1131)
> at org.apache.hadoop.ipc.Client.call(Client.java:961)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:245)
> at $Proxy6.create(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at $Proxy6.create(Unknown Source)
> at 
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:3643)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:778)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:364)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:630)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:611)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:518)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:424)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2672)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2658)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:330)
> at 
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:116)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thre

[jira] [Commented] (HBASE-5546) Master assigns region in the original region server when opening region failed

2012-03-08 Thread ramkrishna.s.vasudevan (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225816#comment-13225816
 ] 

ramkrishna.s.vasudevan commented on HBASE-5546:
---

+1 on clearing the region plan and updating it.

> Master assigns region in the original region server when opening region 
> failed  
> 
>
> Key: HBASE-5546
> URL: https://issues.apache.org/jira/browse/HBASE-5546
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Priority: Minor
> Fix For: 0.92.1
>
>
> Master assigns region in the original region server when 
> RS_ZK_REGION_FAILED_OPEN envent was coming.
> Maybe we should choose other region server.
> [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5543) Add a keepalive option for IPC connections

2012-03-08 Thread Jonathan Hsieh (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225803#comment-13225803
 ] 

Jonathan Hsieh commented on HBASE-5543:
---

Instead of adding to the rpc to make it keep alive longer, maybe be make it 
async, returning some sort of uuid token that the client can poll (or get 
notified) for progress instead?

> Add a keepalive option for IPC connections
> --
>
> Key: HBASE-5543
> URL: https://issues.apache.org/jira/browse/HBASE-5543
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors, ipc
>Reporter: Andrew Purtell
>
> On the user list someone wrote in with a connection failure due to a long 
> running coprocessor:
> {quote}
> On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
> 2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1, 
> client version=0, methodsFingerPrint=0), rpc version=1, client version=29, 
> methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
> 2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
> {quote}
> I suggested in response we might consider give our RPC a keepalive option for 
> calls that may run for a long time (like execCoprocessor).
> LarsH +1ed the idea:
> {quote}
> +1 on "keepalive". It's a shame (especially for long running server code) to 
> do all the work, just to find out at the end that the client has given up.
> Or maybe there should be a way to cancel an operation if the clients decides 
> it does not want to wait any longer (PostgreSQL does that for example). Here 
> that would mean the server would need to check periodically and coprocessors 
> would need to be written to support that - so maybe that's no-starter.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5545) region can't be opened for a long time. Because the creating File failed.

2012-03-08 Thread gaojinchao (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

gaojinchao updated HBASE-5545:
--

Description: 
Scenario:

1. File is created 
2. But while writing data, all datanodes might have crashed. So writing data 
will fail.
3. Now even if close is called in finally block, close also will fail and throw 
the Exception because writing data failed.
4. After this if RS try to create the same file again, then 
AlreadyBeingCreatedException will come.

Suggestion to handle this scenario.
---
1. Check for the existence of the file, if exists delete the file and create 
new file. 

Here delete call for the file will not check whether the file is open or closed.

Overwrite Option:

1. Overwrite option will be applicable if you are trying to overwrite a closed 
file.
2. If the file is not closed, then even with overwrite option Same 
AlreadyBeingCreatedException will be thrown.
This is the expected behaviour to avoid the Multiple clients writing to same 
file.


Region server logs:

org.apache.hadoop.ipc.RemoteException: 
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create 
file /hbase/test1/12c01902324218d14b17a5880f24f64b/.tmp/.regioninfo for 
DFSClient_hb_rs_158-1-131-48,20020,1331107668635_1331107669061_-252463556_25 on 
client 158.1.132.19 because current leaseholder is trying to recreate file.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1570)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1440)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1382)
at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:658)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:547)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1137)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1133)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1131)

at org.apache.hadoop.ipc.Client.call(Client.java:961)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:245)
at $Proxy6.create(Unknown Source)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at $Proxy6.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:3643)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:778)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:630)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:611)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:518)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:424)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2672)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2658)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:330)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:116)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-03-07 20:51:45,858] [WARN ] 
[RS_OPEN_REGION-158-1-131-48,20020,1331107668635-23] 
[com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker 131] Retrying the 
method call: public abstract void 
org.apache.hadoop.hdfs.protocol.ClientProtocol.create(java.lang.String,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,boolean,boolean,short,long)
 throws java.io.IOException with arguments of length: 7. The exisiting 
ActiveServerConnection is:
ActiveServerConnectionInfo:
Metadata:158-1-131-48/158.1.132.19:9000
Version:145720623220907

[2012-03-07 20:51:45,872] [DEBUG] 
[RS_OPEN_REGION-158-1-131-48,20020,1331107668635-20] 
[org.apache.hadoop.hbase.zookeeper.ZKAssign 849] 
regionserver:20020-0x135ec32b39e0002-0x135ec32b39e0002 Successfully 
transitioned node 91bf3e6f8adb2e4b335f061036353126 from M_ZK_REGION_OFFLINE to 
RS_ZK_REGION_OPENING
[2012-03-07 20:51:45,873] [DEBUG] 
[RS_OPEN_REGION-158-1-131-48,20020,1331107668635

[jira] [Updated] (HBASE-5545) region can't be opened for a long time. Because the creating File failed.

2012-03-08 Thread gaojinchao (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

gaojinchao updated HBASE-5545:
--

Description: 
Scenario:

1. File is created 
2. But while writing data, all datanodes might have crashed. So writing data 
will fail.
3. Now even if close is called in finally block, close also will fail and throw 
the Exception because writing data failed.
4. After this if RS try to create the same file again, then 
AlreadyBeingCreatedException will come.

Suggestion to handle this scenario.
---
1. Check for the existence of the file, if exists delete the file and create 
new file. 

Here delete call for the file will not check whether the file is open or closed.

Overwrite Option:

1. Overwrite option will be applicable if you are trying to overwrite a closed 
file.
2. If the file is not closed, then even with overwrite option Same 
AlreadyBeingCreatedException will be thrown.
This is the expected behaviour to avoid the Multiple clients writing to same 
file.


Region server logs:

org.apache.hadoop.ipc.RemoteException: 
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create 
file /hbase/test1/12c01902324218d14b17a5880f24f64b/.tmp/.regioninfo for 
DFSClient_hb_rs_158-1-131-48,20020,1331107668635_1331107669061_-252463556_25 on 
client 158.1.132.19 because current leaseholder is trying to recreate file.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1570)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1440)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1382)
at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:658)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:547)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1137)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1133)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1131)

at org.apache.hadoop.ipc.Client.call(Client.java:961)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:245)
at $Proxy6.create(Unknown Source)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker.invokeMethod(RPCRetryAndSwitchInvoker.java:201)
at 
com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker.invokeMethod(RPCRetryAndSwitchInvoker.java:189)
at 
com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker.invoke(RPCRetryAndSwitchInvoker.java:77)
at $Proxy6.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:3643)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:778)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:630)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:611)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:518)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:424)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2672)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2658)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:330)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:116)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-03-07 20:51:45,858] [WARN ] 
[RS_OPEN_REGION-158-1-131-48,20020,1331107668635-23] 
[com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker 131] Retrying the 
method call: public abstract void 
org.apache.hadoop.hdfs.protocol.ClientProtocol.create(java.lang.String,org.apache.hadoop.fs.permission.FsPermission,java.lang.String,boolean,boolean,short,long)
 throws java.io.IOException with arguments of length: 7. The exisiting 
ActiveServerConnection is:
ActiveServerConnectionInfo:
Metadata:158-1-131-48/158.1.132.19:9000
Version:145720623220907

[2012-03-07 20:51:45,872] [DEBUG] 
[RS_OPEN_REGION-158-1-131-48,20020,13311076

[jira] [Created] (HBASE-5546) Master assigns region in the original region server when opening region failed

2012-03-08 Thread gaojinchao (Created) (JIRA)

Master assigns region in the original region server when opening region failed  


 Key: HBASE-5546
 URL: https://issues.apache.org/jira/browse/HBASE-5546
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.92.0
Reporter: gaojinchao
Priority: Minor
 Fix For: 0.92.1


Master assigns region in the original region server when 
RS_ZK_REGION_FAILED_OPEN envent was coming.
Maybe we should choose other region server.

[2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053
[2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
[org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
region=c70e98bdca98a0657a56436741523053



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5325) Expose basic information about the master-status through jmx beans

2012-03-08 Thread Jean-Daniel Cryans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225794#comment-13225794
 ] 

Jean-Daniel Cryans commented on HBASE-5325:
---

Just a note here, this patch added yet another service to our mbeans. Before we 
had "HBase" and "RegionServer", now this patches adds "org.apache.hbase".

> Expose basic information about the master-status through jmx beans 
> ---
>
> Key: HBASE-5325
> URL: https://issues.apache.org/jira/browse/HBASE-5325
> Project: HBase
>  Issue Type: Improvement
>Reporter: Hitesh Shah
>Assignee: Hitesh Shah
>Priority: Minor
> Fix For: 0.92.1, 0.94.0
>
> Attachments: HBASE-5325.1.patch, HBASE-5325.2.patch, 
> HBASE-5325.3.branch-0.92.patch, HBASE-5325.3.patch, HBASE-5325.wip.patch
>
>
> Similar to the Namenode and Jobtracker, it would be good if the hbase master 
> could expose some information through mbeans.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5545) region can't be opened for a long time. Because the creating File failed.

2012-03-08 Thread gaojinchao (Created) (JIRA)

region can't be opened for a long time. Because the creating File failed.
-

 Key: HBASE-5545
 URL: https://issues.apache.org/jira/browse/HBASE-5545
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.90.6
Reporter: gaojinchao
Assignee: gaojinchao
 Fix For: 0.90.7


Scenario:

1. File is created 
2. But while writing data, all datanodes might have crashed. So writing data 
will fail.
3. Now even if close is called in finally block, close also will fail and throw 
the Exception because writing data failed.
4. After this if RS try to create the same file again, then 
AlreadyBeingCreatedException will come.

Suggestion to handle this scenario.
---
1. Check for the existence of the file, if exists delete the file and create 
new file. 

Here delete call for the file will not check whether the file is open or closed.

Overwrite Option:

1. Overwrite option will be applicable if you are trying to overwrite a closed 
file.
2. If the file is not closed, then even with overwrite option Same 
AlreadyBeingCreatedException will be thrown.
This is the expected behaviour to avoid the Multiple clients writing to same 
file.


Region server logs:

[2012-03-07 20:51:45,858] [ERROR] 
[RS_OPEN_REGION-158-1-131-48,20020,1331107668635-23] 
[com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker 105] Exception occured 
while connecting to server : 
ActiveServerConnectionInfo:
Metadata:158-1-131-48/158.1.132.19:9000
Version:145720623220907

org.apache.hadoop.ipc.RemoteException: 
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create 
file /hbase/test1/12c01902324218d14b17a5880f24f64b/.tmp/.regioninfo for 
DFSClient_hb_rs_158-1-131-48,20020,1331107668635_1331107669061_-252463556_25 on 
client 158.1.132.19 because current leaseholder is trying to recreate file.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1570)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1440)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1382)
at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:658)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:547)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1137)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1133)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1131)

at org.apache.hadoop.ipc.Client.call(Client.java:961)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:245)
at $Proxy6.create(Unknown Source)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker.invokeMethod(RPCRetryAndSwitchInvoker.java:201)
at 
com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker.invokeMethod(RPCRetryAndSwitchInvoker.java:189)
at 
com.huawei.isap.ump.ha.client.RPCRetryAndSwitchInvoker.invoke(RPCRetryAndSwitchInvoker.java:77)
at $Proxy6.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:3643)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:778)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:630)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:611)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:518)
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkRegioninfoOnFilesystem(HRegion.java:424)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:340)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2672)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2658)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:330)
at 
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:116)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:158)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-03-07 20:51:45,858] [WARN ] 
[RS_OPEN_REGION-158-1-131-48,20020,133110766

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225786#comment-13225786
 ] 

Lars Hofhansl commented on HBASE-5541:
--

Re: Duplicate code. Yeah there is a bunch of that.
Between mutateRowsWithLocks, doMiniBatchPut, and processRow there is lot of 
duplicated code.
Scott started to make it better in HBASE-5542. Need to go through and see how 
we can share code from doMiniBatchPut too.

They are slightly different, though. doMiniBatchPut does a best effort write. 
If some puts fail it continues with the ones that succeed. For 
mutateRowsWithLocks and processRow all ops need to succeed.


> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225779#comment-13225779
 ] 

Lars Hofhansl commented on HBASE-5541:
--

@Stack: Sorry should've waited with the commit a bit.

The mvcc.completeMemstoreInsert(w); is still done in the finally block. It's 
attempted once in the try block and then if w is still not-null in the finally 
block, it is redone (that means the try block didn't get to that statement, 
because the WAL sync failed).

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225775#comment-13225775
 ] 

Hudson commented on HBASE-5541:
---

Integrated in HBase-0.94 #22 (See 
[https://builds.apache.org/job/HBase-0.94/22/])
HBASE-5541 Avoid holding the rowlock during HLog sync in 
HRegion.mutateRowWithLocks (Revision 1298680)

 Result = SUCCESS
larsh : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5505) if max file size set to 256M, Per table hbase.hregion.max.filesize setting(at htable creation) overwritten by value set in hbase-site.conf(which might not be 256MB)

2012-03-08 Thread steven zhuang (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225773#comment-13225773
 ] 

steven zhuang commented on HBASE-5505:
--

sure, I am glad if I can help. I will upload a fix patch later. 

> if max file size set to 256M, Per table hbase.hregion.max.filesize setting(at 
> htable creation) overwritten by value set in hbase-site.conf(which might not 
> be 256MB)
> 
>
> Key: HBASE-5505
> URL: https://issues.apache.org/jira/browse/HBASE-5505
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4, 0.92.0
>Reporter: steven zhuang
>
> seems to me that when set to DEFAULT_MAX_FILE_SIZE, the max region file size 
> set at Htable creation time may later be overwritten by value set in 
> hbase-site.conf, i.e. no "Per table" hbase.hregion.max.filesize at all.
> in this case, on Web UI in the table description, max file is still 256M, but 
> we can find region file much bigger on HDFS.
> In 0.92.0, o.a.h.hbase.regionserver.ConstantSizeRegionSplitPolicy, around 
> line 33:
> long maxFileSize = region.getTableDesc().getMaxFileSize();
> // By default we split region if a file > 
> HConstants.DEFAULT_MAX_FILE_SIZE.
> if (maxFileSize == HConstants.DEFAULT_MAX_FILE_SIZE) { //zx: here 
> setting is over written by configuaration from hbase-site.conf
>   maxFileSize = getConf().getLong("hbase.hregion.max.filesize",
> HConstants.DEFAULT_MAX_FILE_SIZE);
> }
> In 0.90.4(cdh3u3) or earlier version, this is found in 
> o.a.h.hbase.regionserver.Store, around line 188:
> long maxFileSize = info.getTableDesc().getMaxFileSize();
> if (maxFileSize == HConstants.DEFAULT_MAX_FILE_SIZE) {
>   maxFileSize = conf.getLong("hbase.hregion.max.filesize",
> HConstants.DEFAULT_MAX_FILE_SIZE);
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225772#comment-13225772
 ] 

Hadoop QA commented on HBASE-5541:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12517655/5541-v2.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -123 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 159 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestImportTsv

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1142//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1142//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1142//console

This message is automatically generated.

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5535) Make the functions in task monitor synchronized


[ 
https://issues.apache.org/jira/browse/HBASE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225769#comment-13225769
 ] 

Hadoop QA commented on HBASE-5535:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12517652/HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -123 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 159 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1141//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1141//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1141//console

This message is automatically generated.

> Make the functions in task monitor synchronized
> ---
>
> Key: HBASE-5535
> URL: https://issues.apache.org/jira/browse/HBASE-5535
> Project: HBase
>  Issue Type: Bug
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: 
> HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch
>
>
> There are some potential race condition in the task monitor. So update the 
> functions in task monitor to be synchronized.
> The example of the problem caused by the race condition:
> ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush 
> failed for region 
> java.lang.IndexOutOfBoundsException: Index: 1745, Size: 1744
> at java.util.ArrayList.add(ArrayList.java:367)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.AbstractList.add(AbstractList.java:91)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:74)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1139)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:260)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:234)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225765#comment-13225765
 ] 

Hudson commented on HBASE-5074:
---

Integrated in HBase-0.94 #21 (See 
[https://builds.apache.org/job/HBase-0.94/21/])
HBASE-5074 Support checksums in HBase block cache (Dhruba) (Revision 
1298666)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java


> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5074-0.9

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225766#comment-13225766
 ] 

Hudson commented on HBASE-5526:
---

Integrated in HBase-0.94 #21 (See 
[https://builds.apache.org/job/HBase-0.94/21/])
HBASE-5526 Configurable file and directory based umask (Jesse Yates) 
(Revision 1298658)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/branches/0.94/src/main/resources/hbase-default.xml
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java


> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225762#comment-13225762
 ] 

stack commented on HBASE-5541:
--

Why we have another variable for saying if locked?  Can't we ask the locked 
object if locked?

{code}
   this.updatesLock.readLock().lock();
+  locked = true;
{code}

this.updatesLock.readLock().lock(); returns a lock.  Use that?  Set it to null 
when unlocked?  If non-null in finally, unlock it?

Hmm... you are trying to make it same as doMiniBatchPut... ok.  Ignore above.

Could this patch not make it so doMiniBatchPut and the update to this method 
shared code?  They seem pretty similar?

This was done in a finally block:

{code}
-mvcc.completeMemstoreInsert(w);
{code}

Its ok that its no longer in a finally block?  Could we skip out during the 
sync w/o updating this?  I suppose thats ok... then these new edits won't be 
seen by a running scanner?  If exception up in cp calling postdelete, postput, 
we'll rollback edits from memstore but mvcc has been advanced and won't go 
backward?  I see you update mvcc if w != null... but maybe the update of mvcc 
should happen after call to cp both here and in the doMiniBatchPut?

Otherwise +1 on commit

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225759#comment-13225759
 ] 

Zhihong Yu commented on HBASE-5542:
---

Please refresh patch as Lars recently integrated HBASE-5541

> Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
> 
>
> Key: HBASE-5542
> URL: https://issues.apache.org/jira/browse/HBASE-5542
> Project: HBase
>  Issue Type: Improvement
>Reporter: Scott Chen
>Assignee: Scott Chen
> Attachments: HBASE-5542.D2217.1.patch
>
>
> mutateRowsWithLocks() does atomic mutations on multiple rows.
> processRow() does atomic read-modify-writes on a single row.
> It will be useful to generalize both and have a
> processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
> This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()


[ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225757#comment-13225757
 ] 

Phabricator commented on HBASE-5542:


tedyu has commented on the revision "HBASE-5542 [jira] Unify 
HRegion.mutateRowsWithLocks() and HRegion.processRow()".

  Thanks for the fast turn-around.

INLINE COMMENTS
  
src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java:35
 Add javadoc for this class, please.
  
src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java:85
 We can store m.getWriteToWAL() in a boolean before entering the loop and use 
the boolean variable, right ?

REVISION DETAIL
  https://reviews.facebook.net/D2217


> Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
> 
>
> Key: HBASE-5542
> URL: https://issues.apache.org/jira/browse/HBASE-5542
> Project: HBase
>  Issue Type: Improvement
>Reporter: Scott Chen
>Assignee: Scott Chen
> Attachments: HBASE-5542.D2217.1.patch
>
>
> mutateRowsWithLocks() does atomic mutations on multiple rows.
> processRow() does atomic read-modify-writes on a single row.
> It will be useful to generalize both and have a
> processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
> This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5213) "hbase master stop" does not bring down backup masters


[ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225748#comment-13225748
 ] 

Hadoop QA commented on HBASE-5213:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12517644/HBASE-5213-v2-trunk.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 5 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -123 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 159 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.coprocessor.TestMasterObserver
  org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1140//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1140//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1140//console

This message is automatically generated.

> "hbase master stop" does not bring down backup masters
> --
>
> Key: HBASE-5213
> URL: https://issues.apache.org/jira/browse/HBASE-5213
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, 
> HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch
>
>
> Typing "hbase master stop" produces the following message:
> "stop   Start cluster shutdown; Master signals RegionServer shutdown"
> It seems like backup masters should be considered part of the cluster, but 
> they are not brought down by "hbase master stop".
> "stop-hbase.sh" does correctly bring down the backup masters.
> The same behavior is observed when a client app makes use of the client API 
> HBaseAdmin.shutdown() 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
>  -- this isn't too surprising since I think "hbase master stop" just calls 
> this API.
> It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


 [ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5541:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed 0.94 and trunk

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5542) Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()

2012-03-08 Thread Phabricator (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5542:
---

Attachment: HBASE-5542.D2217.1.patch

sc requested code review of "HBASE-5542 [jira] Unify 
HRegion.mutateRowsWithLocks() and HRegion.processRow()".
Reviewers: tedyu, lhofhansl, JIRA

  Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()

  Task ID: #

  Blame Rev:

  mutateRowsWithLocks() does atomic mutations on multiple rows.
  processRow() does atomic read-modify-writes on a single row.

  It will be useful to generalize both and have a
  processRowsWithLocks() that does atomic read-modify-writes on multiple rows.

  This also helps reduce some redundancy in the codes.

TEST PLAN
  Revert Plan:

  Tags:

REVISION DETAIL
  https://reviews.facebook.net/D2217

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  
src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java
  src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowProcessor.java
  src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestProcessRowEndpoint.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/4851/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


> Unify HRegion.mutateRowsWithLocks() and HRegion.processRow()
> 
>
> Key: HBASE-5542
> URL: https://issues.apache.org/jira/browse/HBASE-5542
> Project: HBase
>  Issue Type: Improvement
>Reporter: Scott Chen
>Assignee: Scott Chen
> Attachments: HBASE-5542.D2217.1.patch
>
>
> mutateRowsWithLocks() does atomic mutations on multiple rows.
> processRow() does atomic read-modify-writes on a single row.
> It will be useful to generalize both and have a
> processRowsWithLocks() that does atomic read-modify-writes on multiple rows.
> This also helps reduce some redundancy in the codes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5514) Compile against hadoop 0.24-SNAPSHOT


[ 
https://issues.apache.org/jira/browse/HBASE-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225737#comment-13225737
 ] 

Hudson commented on HBASE-5514:
---

Integrated in HBase-TRUNK #2674 (See 
[https://builds.apache.org/job/HBase-TRUNK/2674/])
HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1298488)

 Result = FAILURE
tedyu : 
Files : 
* /hbase/trunk/pom.xml
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java


> Compile against hadoop 0.24-SNAPSHOT
> 
>
> Key: HBASE-5514
> URL: https://issues.apache.org/jira/browse/HBASE-5514
> Project: HBase
>  Issue Type: Bug
>  Components: build, test
>Affects Versions: 0.92.0, 0.94.0
>Reporter: Mingjie Lai
>Assignee: Mingjie Lai
>Priority: Minor
> Fix For: 0.94.0
>
> Attachments: HBASE-5514-2.patch, HBASE-5514-3.patch, 
> HBASE-5514-4.patch, HBASE-5514.patch
>
>
> Need to compile hbase against the latest hadoop trunk which just had NN HA 
> merged in. 
> 1) add a hadoop 0.24 profile
> 2) HBASE-5480
> 3) HADOOP-8124 removed deprecated Syncable.sync(). It brings compile errors 
> for hbase against hadoop trunk(0.24). TestHLogSplit and TestHLog still call 
> the deprecated sync(). Need to replace it with hflush() so the compilation 
> can pass. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225736#comment-13225736
 ] 

Hudson commented on HBASE-5526:
---

Integrated in HBase-TRUNK #2674 (See 
[https://builds.apache.org/job/HBase-TRUNK/2674/])
HBASE-5526 ^Cnfigurable file and directory based umask (Jesse Yates) 
(Revision 1298657)

 Result = FAILURE
larsh : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* /hbase/trunk/src/main/resources/hbase-default.xml
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java


> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set


[ 
https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225738#comment-13225738
 ] 

Hudson commented on HBASE-5529:
---

Integrated in HBase-TRUNK #2674 (See 
[https://builds.apache.org/job/HBase-TRUNK/2674/])
HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory 
Chanan) (Revision 1298574)

 Result = FAILURE
larsh : 
Files : 
* /hbase/trunk/pom.xml


> MR test failures becuase MALLOC_ARENA_MAX is not set
> 
>
> Key: HBASE-5529
> URL: https://issues.apache.org/jira/browse/HBASE-5529
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce, test
>Affects Versions: 0.92.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.92.1, 0.94.0, 0.96.0
>
> Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch
>
>
> When running unit tests on CentOS 6 I get a bunch of unit test failures in 
> mapreduce-related tests due to:
> 2012-03-03 00:14:18,776 WARN  [Container Monitor]
> monitor.ContainersMonitorImpl$MonitoringThread(436): Container
> [pid=21446,containerID=container_1330762435849_0002_01_01] is
> running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb
> physical memory used; 6.9gb of 4.2gb virtual memory used. Killing
> container.
> Note: this also came up in the mapreduce project. See: 
> https://issues.apache.org/jira/browse/MAPREDUCE-3933
> Patch coming shortly

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5480) Fixups to MultithreadedTableMapper for Hadoop 0.23.2+


[ 
https://issues.apache.org/jira/browse/HBASE-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225735#comment-13225735
 ] 

Hudson commented on HBASE-5480:
---

Integrated in HBase-TRUNK #2674 (See 
[https://builds.apache.org/job/HBase-TRUNK/2674/])
HBASE-5480. Fixups to MultithreadedTableMapper for Hadoop 0.23 (Revision 
1298259)

 Result = FAILURE
apurtell : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java


> Fixups to MultithreadedTableMapper for Hadoop 0.23.2+
> -
>
> Key: HBASE-5480
> URL: https://issues.apache.org/jira/browse/HBASE-5480
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.94.0
>
> Attachments: HBASE-5480.patch
>
>
> There are two issues:
> - StatusReporter has a new method getProgress()
> - Mapper and reducer context objects can no longer be directly instantiated.
> See attached patch. I'm not thrilled with the added reflection but it was the 
> minimally intrusive change.
> Raised the priority to critical because compilation fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4890) fix possible NPE in HConnectionManager


[ 
https://issues.apache.org/jira/browse/HBASE-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225734#comment-13225734
 ] 

Hudson commented on HBASE-4890:
---

Integrated in HBase-TRUNK #2674 (See 
[https://builds.apache.org/job/HBase-TRUNK/2674/])
HBASE-4890 fix possible NPE in HConnectionManager (Revision 1298272)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java


> fix possible NPE in HConnectionManager
> --
>
> Key: HBASE-4890
> URL: https://issues.apache.org/jira/browse/HBASE-4890
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Jonathan Hsieh
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.1, 0.94.0
>
> Attachments: 4890-v2.txt, 4890-v3.txt, 4890-v3.txt, 4890-v3.txt, 
> 4890.txt, splits.txt
>
>
> I was running YCSB against a 0.92 branch and encountered this error message:
> {code}
> 11/11/29 08:47:16 WARN client.HConnectionManager$HConnectionImplementation: 
> Failed all from 
> region=usertable,user3917479014967760871,1322555655231.f78d161e5724495a9723bcd972f97f41.,
>  hostname=c0316.hal.cloudera.com, port=57020
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> java.lang.NullPointerException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1501)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1353)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:898)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:775)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:750)
> at com.yahoo.ycsb.db.HBaseClient.update(Unknown Source)
> at com.yahoo.ycsb.DBWrapper.update(Unknown Source)
> at com.yahoo.ycsb.workloads.CoreWorkload.doTransactionUpdate(Unknown 
> Source)
> at com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(Unknown Source)
> at com.yahoo.ycsb.ClientThread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1315)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1327)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1325)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:158)
> at $Proxy4.multi(Unknown Source)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1330)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1328)
> at 
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithoutRetries(HConnectionManager.java:1309)
> ... 7 more
> {code}
> It looks like the NPE is caused by server being null in the MultiRespone 
> call() method.
> {code}
>  public MultiResponse call() throws IOException {
>  return getRegionServerWithoutRetries(
>  new ServerCallable(connection, tableName, null) {
>public MultiResponse call() throws IOException {
>  return server.multi(multi);
>}
>@Override
>public void connect(boolean reload) throws IOException {
>  server =
>connection.getHRegionConnection(loc.getHostname(), 
> loc.getPort());
>}
>  }
>  );
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache

[
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225733#comment-13225733
]

Hudson commented on HBASE-5074:
---

Integrated in HBase-TRUNK #2674 (See
[https://builds.apache.org/job/HBase-TRUNK/2674/])
[jira] [HBASE-5074] Support checksums in HBase block cache

Author: Dhruba

The salient points of this patch:

6. By default, hbase-level checksums are switched on and hdfs level checksums
are switched off for hfile-reads. No changes to Hlog code path here.

Reviewers: mbautin

Reviewed By: mbautin

CC: JIRA, tedyu, mbautin, dhruba, todd, stack

Differential Revision: https://reviews.facebook.net/D1521 (Revision 1298641)

[jira] [Commented] (HBASE-5213) "hbase master stop" does not bring down backup masters


[ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225731#comment-13225731
 ] 

Zhihong Yu commented on HBASE-5213:
---

Will integrated TRUNK patch to 0.94 and TRUNK if there is no objection.

> "hbase master stop" does not bring down backup masters
> --
>
> Key: HBASE-5213
> URL: https://issues.apache.org/jira/browse/HBASE-5213
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, 
> HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch
>
>
> Typing "hbase master stop" produces the following message:
> "stop   Start cluster shutdown; Master signals RegionServer shutdown"
> It seems like backup masters should be considered part of the cluster, but 
> they are not brought down by "hbase master stop".
> "stop-hbase.sh" does correctly bring down the backup masters.
> The same behavior is observed when a client app makes use of the client API 
> HBaseAdmin.shutdown() 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
>  -- this isn't too surprising since I think "hbase master stop" just calls 
> this API.
> It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5515) Add a processRow API that supports atomic multiple reads and writes on a row


[ 
https://issues.apache.org/jira/browse/HBASE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225732#comment-13225732
 ] 

Hudson commented on HBASE-5515:
---

Integrated in HBase-TRUNK #2674 (See 
[https://builds.apache.org/job/HBase-TRUNK/2674/])
HBASE-5515 Add a processRow API that supports atomic multiple reads and 
writes on a row (Scott Chen) (Revision 1298533)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/RowProcessor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestProcessRowEndpoint.java


> Add a processRow API that supports atomic multiple reads and writes on a row
> 
>
> Key: HBASE-5515
> URL: https://issues.apache.org/jira/browse/HBASE-5515
> Project: HBase
>  Issue Type: New Feature
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.96.0
>
> Attachments: HBASE-5515.D2067.1.patch, HBASE-5515.D2067.10.patch, 
> HBASE-5515.D2067.11.patch, HBASE-5515.D2067.12.patch, 
> HBASE-5515.D2067.13.patch, HBASE-5515.D2067.14.patch, 
> HBASE-5515.D2067.15.patch, HBASE-5515.D2067.16.patch, 
> HBASE-5515.D2067.17.patch, HBASE-5515.D2067.18.patch, 
> HBASE-5515.D2067.19.patch, HBASE-5515.D2067.2.patch, 
> HBASE-5515.D2067.20.patch, HBASE-5515.D2067.21.patch, 
> HBASE-5515.D2067.22.patch, HBASE-5515.D2067.3.patch, 
> HBASE-5515.D2067.4.patch, HBASE-5515.D2067.5.patch, HBASE-5515.D2067.6.patch, 
> HBASE-5515.D2067.7.patch, HBASE-5515.D2067.8.patch, HBASE-5515.D2067.9.patch
>
>
> We have modified HRegion.java internally to do some atomic row processing. It 
> will be nice to have a plugable API for this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


 [ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5541:
-

Attachment: 5541-v2.txt

Just comment changes

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225729#comment-13225729
 ] 

Zhihong Yu commented on HBASE-5541:
---

+1 from me.

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541-v2.txt, 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225725#comment-13225725
 ] 

Lars Hofhansl commented on HBASE-5541:
--

I'll wait for an official +1.

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5074) support checksums in HBase block cache


 [ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5074:
-

Fix Version/s: 0.96.0

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5074-0.94.txt, D1521.1.patch, D1521.1.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, 
> D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, 
> D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, 
> D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, 
> D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, 
> D1521.9.patch, D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5535) Make the functions in task monitor synchronized

2012-03-08 Thread Mikhail Bautin (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-5535:
--

Status: Patch Available  (was: Open)

> Make the functions in task monitor synchronized
> ---
>
> Key: HBASE-5535
> URL: https://issues.apache.org/jira/browse/HBASE-5535
> Project: HBase
>  Issue Type: Bug
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: 
> HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch
>
>
> There are some potential race condition in the task monitor. So update the 
> functions in task monitor to be synchronized.
> The example of the problem caused by the race condition:
> ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush 
> failed for region 
> java.lang.IndexOutOfBoundsException: Index: 1745, Size: 1744
> at java.util.ArrayList.add(ArrayList.java:367)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.AbstractList.add(AbstractList.java:91)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:74)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1139)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:260)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:234)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5535) Make the functions in task monitor synchronized

2012-03-08 Thread Mikhail Bautin (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Bautin updated HBASE-5535:
--

Attachment: 
HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch

Liyin's two-line patch from our internal 89-fb repository.

> Make the functions in task monitor synchronized
> ---
>
> Key: HBASE-5535
> URL: https://issues.apache.org/jira/browse/HBASE-5535
> Project: HBase
>  Issue Type: Bug
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: 
> HBASE-5535-Make-the-functions-in-task-monitor-synchr-2012-03-08_16_33_42.patch
>
>
> There are some potential race condition in the task monitor. So update the 
> functions in task monitor to be synchronized.
> The example of the problem caused by the race condition:
> ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush 
> failed for region 
> java.lang.IndexOutOfBoundsException: Index: 1745, Size: 1744
> at java.util.ArrayList.add(ArrayList.java:367)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.SubList.add(AbstractList.java:633)
> at java.util.AbstractList.add(AbstractList.java:91)
> at 
> org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:74)
> at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1139)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:260)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:234)
> at 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-5074) support checksums in HBase block cache

2012-03-08 Thread Lars Hofhansl (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl resolved HBASE-5074.
--

  Resolution: Fixed
Hadoop Flags: Reviewed

Comitted to 0.94 as well.

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 5074-0.94.txt, D1521.1.patch, D1521.1.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, 
> D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, 
> D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, 
> D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, 
> D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, 
> D1521.9.patch, D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5074) support checksums in HBase block cache


 [ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5074:
-

Status: Open  (was: Patch Available)

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 5074-0.94.txt, D1521.1.patch, D1521.1.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, 
> D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, 
> D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, 
> D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, 
> D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, 
> D1521.9.patch, D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5074) support checksums in HBase block cache


 [ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5074:
-

Attachment: 5074-0.94.txt

Here's the 0.94 version.
Applied mostly with some offsets, just had to fix up some imports.

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 5074-0.94.txt, D1521.1.patch, D1521.1.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.10.patch, D1521.11.patch, D1521.11.patch, D1521.12.patch, 
> D1521.12.patch, D1521.13.patch, D1521.13.patch, D1521.14.patch, 
> D1521.14.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, 
> D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, 
> D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, 
> D1521.9.patch, D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5213) "hbase master stop" does not bring down backup masters


[ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225712#comment-13225712
 ] 

Zhihong Yu commented on HBASE-5213:
---

I ran TestAdmin three times based on patch v2.
They passed.

> "hbase master stop" does not bring down backup masters
> --
>
> Key: HBASE-5213
> URL: https://issues.apache.org/jira/browse/HBASE-5213
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, 
> HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch
>
>
> Typing "hbase master stop" produces the following message:
> "stop   Start cluster shutdown; Master signals RegionServer shutdown"
> It seems like backup masters should be considered part of the cluster, but 
> they are not brought down by "hbase master stop".
> "stop-hbase.sh" does correctly bring down the backup masters.
> The same behavior is observed when a client app makes use of the client API 
> HBaseAdmin.shutdown() 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
>  -- this isn't too surprising since I think "hbase master stop" just calls 
> this API.
> It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache

2012-03-08 Thread Mikhail Bautin (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225703#comment-13225703
 ] 

Mikhail Bautin commented on HBASE-5074:
---

@Lars: yes. I have also re-run unit tests one more time.

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225704#comment-13225704
 ] 

Lars Hofhansl commented on HBASE-5074:
--

Thanks I'll apply and commit the patch to 0.94.

Thanks for the great work guys!!

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5213) "hbase master stop" does not bring down backup masters

2012-03-08 Thread Gregory Chanan (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated HBASE-5213:
--

Attachment: HBASE-5213-v2-trunk.patch
HBASE-5213-v2-92.patch
HBASE-5213-v2-90.patch

* Attached v2 for trunk,92, and 90 *

For trunk, only small differences from previous version (comments and spacing).

Tested that trunk patch works with 94 as well.

Created patches for 92 and 90.

> "hbase master stop" does not bring down backup masters
> --
>
> Key: HBASE-5213
> URL: https://issues.apache.org/jira/browse/HBASE-5213
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.5, 0.92.0, 0.94.0, 0.96.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HBASE-5213-v0-trunk.patch, HBASE-5213-v1-trunk.patch, 
> HBASE-5213-v2-90.patch, HBASE-5213-v2-92.patch, HBASE-5213-v2-trunk.patch
>
>
> Typing "hbase master stop" produces the following message:
> "stop   Start cluster shutdown; Master signals RegionServer shutdown"
> It seems like backup masters should be considered part of the cluster, but 
> they are not brought down by "hbase master stop".
> "stop-hbase.sh" does correctly bring down the backup masters.
> The same behavior is observed when a client app makes use of the client API 
> HBaseAdmin.shutdown() 
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#shutdown()
>  -- this isn't too surprising since I think "hbase master stop" just calls 
> this API.
> It seems like HBASE-1448 address this; perhaps there was a regression?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-5544) Add metrics to HRegion.processRow()

2012-03-08 Thread Scott Chen (Created) (JIRA)

Add metrics to HRegion.processRow()
---

 Key: HBASE-5544
 URL: https://issues.apache.org/jira/browse/HBASE-5544
 Project: HBase
  Issue Type: New Feature
Reporter: Scott Chen
Assignee: Scott Chen


Add metrics of
1. time for waiting for the lock
2. processing time (scan time)
3. time spent while holding the lock
4. total call time
5. number of failures / calls

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5526) Configurable file and directory based umask


 [ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5526:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to 0.94 and trunk

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225689#comment-13225689
 ] 

Lars Hofhansl commented on HBASE-5526:
--

Can file a child bug for documentation and why this might be useful.

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225686#comment-13225686
 ] 

Lars Hofhansl commented on HBASE-5074:
--

Is "D1521.14.patch" the that was applied to trunk?

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set


[ 
https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225679#comment-13225679
 ] 

Hudson commented on HBASE-5529:
---

Integrated in HBase-0.92 #322 (See 
[https://builds.apache.org/job/HBase-0.92/322/])
HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory 
Chanan) (Revision 1298582)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/pom.xml


> MR test failures becuase MALLOC_ARENA_MAX is not set
> 
>
> Key: HBASE-5529
> URL: https://issues.apache.org/jira/browse/HBASE-5529
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce, test
>Affects Versions: 0.92.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.92.1, 0.94.0, 0.96.0
>
> Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch
>
>
> When running unit tests on CentOS 6 I get a bunch of unit test failures in 
> mapreduce-related tests due to:
> 2012-03-03 00:14:18,776 WARN  [Container Monitor]
> monitor.ContainersMonitorImpl$MonitoringThread(436): Container
> [pid=21446,containerID=container_1330762435849_0002_01_01] is
> running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb
> physical memory used; 6.9gb of 4.2gb virtual memory used. Killing
> container.
> Note: this also came up in the mapreduce project. See: 
> https://issues.apache.org/jira/browse/MAPREDUCE-3933
> Patch coming shortly

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set


[ 
https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225678#comment-13225678
 ] 

Hudson commented on HBASE-5529:
---

Integrated in HBase-0.92-security #98 (See 
[https://builds.apache.org/job/HBase-0.92-security/98/])
HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory 
Chanan) (Revision 1298582)

 Result = FAILURE
larsh : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/pom.xml


> MR test failures becuase MALLOC_ARENA_MAX is not set
> 
>
> Key: HBASE-5529
> URL: https://issues.apache.org/jira/browse/HBASE-5529
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce, test
>Affects Versions: 0.92.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.92.1, 0.94.0, 0.96.0
>
> Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch
>
>
> When running unit tests on CentOS 6 I get a bunch of unit test failures in 
> mapreduce-related tests due to:
> 2012-03-03 00:14:18,776 WARN  [Container Monitor]
> monitor.ContainersMonitorImpl$MonitoringThread(436): Container
> [pid=21446,containerID=container_1330762435849_0002_01_01] is
> running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb
> physical memory used; 6.9gb of 4.2gb virtual memory used. Killing
> container.
> Note: this also came up in the mapreduce project. See: 
> https://issues.apache.org/jira/browse/MAPREDUCE-3933
> Patch coming shortly

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225667#comment-13225667
 ] 

Lars Hofhansl commented on HBASE-5541:
--

Yep :)

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask

2012-03-08 Thread Jesse Yates (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225666#comment-13225666
 ] 

Jesse Yates commented on HBASE-5526:


Btw, all failed tests passed locally.

Do we want to add any extra documentation to ref guide or do you think the 
standard stuff from hbase-defaults.xml is going to suffice?

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225661#comment-13225661
 ] 

Zhihong Yu commented on HBASE-5541:
---

Redundant step 11 comment would be removed, right ?

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225658#comment-13225658
 ] 

Lars Hofhansl commented on HBASE-5074:
--

and HBASE-5526, but that's it from my side.

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225660#comment-13225660
 ] 

Lars Hofhansl commented on HBASE-5541:
--

Looks good to me. Any objections to committing this?

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225659#comment-13225659
 ] 

Lars Hofhansl commented on HBASE-5526:
--

Ok... last chance to object.

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225655#comment-13225655
 ] 

Lars Hofhansl commented on HBASE-5074:
--

Yes sir. Still waiting for HBASE-4608.
And HBASE-5541 :)


> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225654#comment-13225654
 ] 

stack commented on HBASE-5074:
--

Wahoo!!

Lars, you want to pull it into 0.94? (Does this mean 0.94 is good to go?  
Should we put up an RC?)

> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225641#comment-13225641
 ] 

Phabricator commented on HBASE-5074:


mbautin has committed the revision "[jira] [HBASE-5074] Support checksums in 
HBase block cache".

REVISION DETAIL
  https://reviews.facebook.net/D1521

COMMIT
  https://reviews.facebook.net/rHBASE1298641


> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5074) support checksums in HBase block cache


[ 
https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225640#comment-13225640
 ] 

Phabricator commented on HBASE-5074:


mbautin has committed the revision "[jira] [HBASE-5074] Support checksums in 
HBase block cache".

REVISION DETAIL
  https://reviews.facebook.net/D1521

COMMIT
  https://reviews.facebook.net/rHBASE1298641


> support checksums in HBase block cache
> --
>
> Key: HBASE-5074
> URL: https://issues.apache.org/jira/browse/HBASE-5074
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, 
> D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, 
> D1521.11.patch, D1521.11.patch, D1521.12.patch, D1521.12.patch, 
> D1521.13.patch, D1521.13.patch, D1521.14.patch, D1521.14.patch, 
> D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, 
> D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, 
> D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, 
> D1521.9.patch
>
>
> The current implementation of HDFS stores the data in one block file and the 
> metadata(checksum) in another block file. This means that every read into the 
> HBase block cache actually consumes two disk iops, one to the datafile and 
> one to the checksum file. This is a major problem for scaling HBase, because 
> HBase is usually bottlenecked on the number of random disk iops that the 
> storage-hardware offers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5517) Region Server Coprocessor : Suggestion for change when next() call with nbRows>1

2012-03-08 Thread Andrew Purtell (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225624#comment-13225624
 ] 

Andrew Purtell commented on HBASE-5517:
---

You should be able to achieve what you want with a custom scanner that 
implements RegionScanner. 

Use the coprocessor preScannerOpen. In that hook, wrap the passed in 
RegionScanner with an instance of a custom wrapper class that extends 
RegionScanner, and return that instance of your scanner wrapper. 

Then in your wrapper, in the implementation of next(List result, int 
limit), use a for loop to call next(List results).

Then in your wrapper, in the implementation of next(List results), 
call reseek or whatever, and then call the next(List results) method 
of the scanner you are wrapping.

> Region Server Coprocessor : Suggestion for change when next() call with 
> nbRows>1
> 
>
> Key: HBASE-5517
> URL: https://issues.apache.org/jira/browse/HBASE-5517
> Project: HBase
>  Issue Type: Improvement
>  Components: coprocessors
>Affects Versions: 0.92.0
>Reporter: Anoop Sam John
>
> Originated from the discussion under HBASE-2038 [Coprocessor based IHBase]
> Currently preNext() and postNext() will be called once for a next() call into 
> HRegionServer.
> But if the next() is being called with nbRows>1, co processor should provide 
> a chance to do some operation before, after every next() calls into region as 
> part of call next(int scannerId, int nbRows).
> In case of usage of coprocessor with IHBase, before making any calls of 
> next() into a Region, we need to make a reseek() to a row based on the index 
> information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5538) A metric to measure the size of the response queue in the hbase rpc server


[ 
https://issues.apache.org/jira/browse/HBASE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225619#comment-13225619
 ] 

Hadoop QA commented on HBASE-5538:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12517613/D2199.1.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -129 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 155 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestImportTsv

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1139//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1139//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1139//console

This message is automatically generated.

> A metric to measure the size of the response queue in the hbase rpc server
> --
>
> Key: HBASE-5538
> URL: https://issues.apache.org/jira/browse/HBASE-5538
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: D2199.1.patch
>
>
> The HbaseServer queues responses to client (if the client is slow). Expose a 
> metric that records the size of the response queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225604#comment-13225604
 ] 

Hadoop QA commented on HBASE-5526:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12517616/java_HBASE-5526-v7.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 javadoc.  The javadoc tool appears to have generated -127 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 155 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
  org.apache.hadoop.hbase.TestZooKeeper

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1138//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1138//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1138//console

This message is automatically generated.

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5522) hbase 0.92 test artifacts are missing from Maven central

2012-03-08 Thread Enis Soztutar (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225596#comment-13225596
 ] 

Enis Soztutar commented on HBASE-5522:
--

@Roman, 
Are you able to fetch the test artifacts now? 

> hbase 0.92 test artifacts are missing from Maven central
> 
>
> Key: HBASE-5522
> URL: https://issues.apache.org/jira/browse/HBASE-5522
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.92.0
>Reporter: Roman Shaposhnik
>Assignee: stack
> Fix For: 0.92.1, 0.94.0
>
> Attachments: 5522.txt
>
>
> Could someone with enough karma, please, publish the test artifacts for 
> 0.92.0?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5509) MR based copier for copying HFiles (trunk version)


[ 
https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225580#comment-13225580
 ] 

Zhihong Yu commented on HBASE-5509:
---

I put a few comments on RB.

> MR based copier for copying HFiles (trunk version)
> --
>
> Key: HBASE-5509
> URL: https://issues.apache.org/jira/browse/HBASE-5509
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, regionserver
>Reporter: Karthik Ranganathan
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5509-v2.txt, 5509.txt
>
>
> This copier is a modification of the distcp tool in HDFS. It does the 
> following:
> 1. List out all the regions in the HBase cluster for the required table
> 2. Write the above out to a file
> 3. Each mapper 
>3.1 lists all the HFiles for a given region by querying the regionserver
>3.2 copies all the HFiles
>3.3 outputs success if the copy succeeded, failure otherwise. Failed 
> regions are retried in another loop
> 4. Mappers are placed on nodes which have maximum locality for a given region 
> to speed up copying

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5529) MR test failures becuase MALLOC_ARENA_MAX is not set


[ 
https://issues.apache.org/jira/browse/HBASE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225574#comment-13225574
 ] 

Hudson commented on HBASE-5529:
---

Integrated in HBase-0.94 #20 (See 
[https://builds.apache.org/job/HBase-0.94/20/])
HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory 
Chanan) (Revision 1298575)

 Result = SUCCESS
larsh : 
Files : 
* /hbase/branches/0.94/pom.xml


> MR test failures becuase MALLOC_ARENA_MAX is not set
> 
>
> Key: HBASE-5529
> URL: https://issues.apache.org/jira/browse/HBASE-5529
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce, test
>Affects Versions: 0.92.0
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.92.1, 0.94.0, 0.96.0
>
> Attachments: HBASE-5529-to92.patch, HBASE-5529-trunk.patch
>
>
> When running unit tests on CentOS 6 I get a bunch of unit test failures in 
> mapreduce-related tests due to:
> 2012-03-03 00:14:18,776 WARN  [Container Monitor]
> monitor.ContainersMonitorImpl$MonitoringThread(436): Container
> [pid=21446,containerID=container_1330762435849_0002_01_01] is
> running beyond virtual memory limits. Current usage: 223.1mb of 2.0gb
> physical memory used; 6.9gb of 4.2gb virtual memory used. Killing
> container.
> Note: this also came up in the mapreduce project. See: 
> https://issues.apache.org/jira/browse/MAPREDUCE-3933
> Patch coming shortly

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5543) Add a keepalive option for IPC connections

2012-03-08 Thread Andrew Purtell (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225575#comment-13225575
 ] 

Andrew Purtell commented on HBASE-5543:
---

bq. We have "ipc.client.ping", right? What we want is sort of the opposite 
direction of that?

That's what I had in mind.

> Add a keepalive option for IPC connections
> --
>
> Key: HBASE-5543
> URL: https://issues.apache.org/jira/browse/HBASE-5543
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors, ipc
>Reporter: Andrew Purtell
>
> On the user list someone wrote in with a connection failure due to a long 
> running coprocessor:
> {quote}
> On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
> 2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1, 
> client version=0, methodsFingerPrint=0), rpc version=1, client version=29, 
> methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
> 2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
> {quote}
> I suggested in response we might consider give our RPC a keepalive option for 
> calls that may run for a long time (like execCoprocessor).
> LarsH +1ed the idea:
> {quote}
> +1 on "keepalive". It's a shame (especially for long running server code) to 
> do all the work, just to find out at the end that the client has given up.
> Or maybe there should be a way to cancel an operation if the clients decides 
> it does not want to wait any longer (PostgreSQL does that for example). Here 
> that would mean the server would need to check periodically and coprocessors 
> would need to be written to support that - so maybe that's no-starter.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5543) Add a keepalive option for IPC connections

2012-03-08 Thread Todd Lipcon (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225573#comment-13225573
 ] 

Todd Lipcon commented on HBASE-5543:


We have "ipc.client.ping", right? What we want is sort of the opposite 
direction of that?

> Add a keepalive option for IPC connections
> --
>
> Key: HBASE-5543
> URL: https://issues.apache.org/jira/browse/HBASE-5543
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors, ipc
>Reporter: Andrew Purtell
>
> On the user list someone wrote in with a connection failure due to a long 
> running coprocessor:
> {quote}
> On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
> 2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1, 
> client version=0, methodsFingerPrint=0), rpc version=1, client version=29, 
> methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
> 2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
> {quote}
> I suggested in response we might consider give our RPC a keepalive option for 
> calls that may run for a long time (like execCoprocessor).
> LarsH +1ed the idea:
> {quote}
> +1 on "keepalive". It's a shame (especially for long running server code) to 
> do all the work, just to find out at the end that the client has given up.
> Or maybe there should be a way to cancel an operation if the clients decides 
> it does not want to wait any longer (PostgreSQL does that for example). Here 
> that would mean the server would need to check periodically and coprocessors 
> would need to be written to support that - so maybe that's no-starter.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5543) Add a keepalive option for IPC connections


[ 
https://issues.apache.org/jira/browse/HBASE-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225566#comment-13225566
 ] 

stack commented on HBASE-5543:
--

Yeah, it looks like its inevitable that we'll ask the server to do legitimate 
stuff that will take longer than the rpctimeout yet the server is making 
headway: e.g. the reproducing test case, though a little artificial, for 
"HBASE-4890  fix possible NPE in HConnectionManager" was asking the 
regionserver to open 3k regions.

If its a task like the above, there should be a facility for telling client 
we're alive still or we should just refuse the request because it will take too 
long (The latter we need to do t -- from Benoiit.  If server is going to 
take too long servicing a request, so long the client will be gone by the time 
its done its work, then refuse the request... don't do the increment or update 
that the updating client will not be around to see).

> Add a keepalive option for IPC connections
> --
>
> Key: HBASE-5543
> URL: https://issues.apache.org/jira/browse/HBASE-5543
> Project: HBase
>  Issue Type: Improvement
>  Components: client, coprocessors, ipc
>Reporter: Andrew Purtell
>
> On the user list someone wrote in with a connection failure due to a long 
> running coprocessor:
> {quote}
> On Wed, Mar 7, 2012 at 10:59 PM, raghavendhra rahul wrote:
> 2012-03-08 12:03:09,475 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> Responder, call execCoprocessor([B@50cb21, getProjection(), rpc version=1, 
> client version=0, methodsFingerPrint=0), rpc version=1, client version=29, 
> methodsFingerPrint=54742778 from 10.184.17.26:46472: output error
> 2012-03-08 12:03:09,476 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server 
> handler 7 on 60020 caught: java.nio.channels.ClosedChannelException
> {quote}
> I suggested in response we might consider give our RPC a keepalive option for 
> calls that may run for a long time (like execCoprocessor).
> LarsH +1ed the idea:
> {quote}
> +1 on "keepalive". It's a shame (especially for long running server code) to 
> do all the work, just to find out at the end that the client has given up.
> Or maybe there should be a way to cancel an operation if the clients decides 
> it does not want to wait any longer (PostgreSQL does that for example). Here 
> that would mean the server would need to check periodically and coprocessors 
> would need to be written to support that - so maybe that's no-starter.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5538) A metric to measure the size of the response queue in the hbase rpc server


[ 
https://issues.apache.org/jira/browse/HBASE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225562#comment-13225562
 ] 

Phabricator commented on HBASE-5538:


stack has accepted the revision "[jira] [HBASE-5538] A metric to measure the 
size of the response queue in the hbase rpc server".

  +1

REVISION DETAIL
  https://reviews.facebook.net/D2199

BRANCH
  svn


> A metric to measure the size of the response queue in the hbase rpc server
> --
>
> Key: HBASE-5538
> URL: https://issues.apache.org/jira/browse/HBASE-5538
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: D2199.1.patch
>
>
> The HbaseServer queues responses to client (if the client is slow). Expose a 
> metric that records the size of the response queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5538) A metric to measure the size of the response queue in the hbase rpc server

2012-03-08 Thread stack (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5538:
-

Status: Patch Available  (was: Open)

> A metric to measure the size of the response queue in the hbase rpc server
> --
>
> Key: HBASE-5538
> URL: https://issues.apache.org/jira/browse/HBASE-5538
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: D2199.1.patch
>
>
> The HbaseServer queues responses to client (if the client is slow). Expose a 
> metric that records the size of the response queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5509) MR based copier for copying HFiles (trunk version)

2012-03-08 Thread Jesse Yates (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225559#comment-13225559
 ] 

Jesse Yates commented on HBASE-5509:


Looking at it today - should have my comments in by COB.

> MR based copier for copying HFiles (trunk version)
> --
>
> Key: HBASE-5509
> URL: https://issues.apache.org/jira/browse/HBASE-5509
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, regionserver
>Reporter: Karthik Ranganathan
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5509-v2.txt, 5509.txt
>
>
> This copier is a modification of the distcp tool in HDFS. It does the 
> following:
> 1. List out all the regions in the HBase cluster for the required table
> 2. Write the above out to a file
> 3. Each mapper 
>3.1 lists all the HFiles for a given region by querying the regionserver
>3.2 copies all the HFiles
>3.3 outputs success if the copy succeeded, failure otherwise. Failed 
> regions are retried in another loop
> 4. Mappers are placed on nodes which have maximum locality for a given region 
> to speed up copying

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5526) Configurable file and directory based umask

2012-03-08 Thread Jesse Yates (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesse Yates updated HBASE-5526:
---

Attachment: java_HBASE-5526-v7.patch

Rebased against 0.94.

I believe this also applies against trunk, if we want to commit it there as 
well.

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526-v7.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5541) Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks


[ 
https://issues.apache.org/jira/browse/HBASE-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225549#comment-13225549
 ] 

Hadoop QA commented on HBASE-5541:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12517600/5541.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 javadoc.  The javadoc tool appears to have generated -129 warning 
messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 155 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.coprocessor.TestMasterObserver
  org.apache.hadoop.hbase.mapreduce.TestImportTsv
  org.apache.hadoop.hbase.mapred.TestTableMapReduce
  org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1136//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1136//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1136//console

This message is automatically generated.

> Avoid holding the rowlock during HLog sync in HRegion.mutateRowWithLocks
> 
>
> Key: HBASE-5541
> URL: https://issues.apache.org/jira/browse/HBASE-5541
> Project: HBase
>  Issue Type: Sub-task
>  Components: client, regionserver
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.0
>
> Attachments: 5541.txt
>
>
> Currently mutateRowsWithLocks holds the row lock while the HLog is sync'ed.
> Similar to what we do in doMiniBatchPut, we should create the log entry with 
> the lock held, but only sync the HLog after the lock is released, along with 
> rollback logic in case the sync'ing fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5538) A metric to measure the size of the response queue in the hbase rpc server

2012-03-08 Thread Phabricator (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5538:
---

Attachment: D2199.1.patch

dhruba requested code review of "[jira] [HBASE-5538] A metric to measure the 
size of the response queue in the hbase rpc server".
Reviewers: JIRA

  Calculate the size of the rpc response queue. Expose it via the metrics api.

TEST PLAN
  run all unit tests

REVISION DETAIL
  https://reviews.facebook.net/D2199

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/4827/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.


> A metric to measure the size of the response queue in the hbase rpc server
> --
>
> Key: HBASE-5538
> URL: https://issues.apache.org/jira/browse/HBASE-5538
> Project: HBase
>  Issue Type: Improvement
>  Components: ipc
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: D2199.1.patch
>
>
> The HbaseServer queues responses to client (if the client is slow). Expose a 
> metric that records the size of the response queue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225546#comment-13225546
 ] 

Lars Hofhansl commented on HBASE-5526:
--

{code}
patching file src/main/java/org/apache/hadoop/hbase/HConstants.java
Hunk #1 succeeded at 610 (offset 4 lines).
patching file 
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
patching file src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Hunk #1 succeeded at 67 (offset 2 lines).
Hunk #2 FAILED at 79.
Hunk #3 FAILED at 89.
Hunk #4 succeeded at 342 (offset 4 lines).
Hunk #5 succeeded at 770 (offset 8 lines).
2 out of 5 hunks FAILED -- saving rejects to file 
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java.rej
patching file src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
patching file src/main/resources/hbase-default.xml
patching file src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
PATCH APPLICATION FAILED
{code}

HRegion was just recently changed.


> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask

2012-03-08 Thread Jesse Yates (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225541#comment-13225541
 ] 

Jesse Yates commented on HBASE-5526:


Yup, works for blocking out the hfiles and .regioninfo.

Odd that it didn't apply. Going to try rebasing and see if that was the problem.

> Configurable file and directory based umask
> ---
>
> Key: HBASE-5526
> URL: https://issues.apache.org/jira/browse/HBASE-5526
> Project: HBase
>  Issue Type: New Feature
>  Components: regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.94.0
>
> Attachments: java_HBASE-5526-v2.patch, java_HBASE-5526-v3.patch, 
> java_HBASE-5526-v5.patch, java_HBASE-5526.patch
>
>
> Currently many all the files created by the HBase user are just written using 
> the default file permissions granted by hdfs. However, to ensure only the 
> correct user/group views the files and directories, we need to be able to 
> apply a configurable umask to either directories or files. 
> This ticket covers setting permissions for files written to dfs, as opposed 
> to things like pid and log files.
> The impetus for this was to allow the web-user to view the directory 
> structure of hbase, but not to actually see any of the actual data hbase is 
> storing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5509) MR based copier for copying HFiles (trunk version)


[ 
https://issues.apache.org/jira/browse/HBASE-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225539#comment-13225539
 ] 

Lars Hofhansl commented on HBASE-5509:
--

Any takers for a review?
I assume +1 from the FB guys (right Karthik?) :)


> MR based copier for copying HFiles (trunk version)
> --
>
> Key: HBASE-5509
> URL: https://issues.apache.org/jira/browse/HBASE-5509
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation, regionserver
>Reporter: Karthik Ranganathan
>Assignee: Lars Hofhansl
> Fix For: 0.94.0, 0.96.0
>
> Attachments: 5509-v2.txt, 5509.txt
>
>
> This copier is a modification of the distcp tool in HDFS. It does the 
> following:
> 1. List out all the regions in the HBase cluster for the required table
> 2. Write the above out to a file
> 3. Each mapper 
>3.1 lists all the HFiles for a given region by querying the regionserver
>3.2 copies all the HFiles
>3.3 outputs success if the copy succeeded, failure otherwise. Failed 
> regions are retried in another loop
> 4. Mappers are placed on nodes which have maximum locality for a given region 
> to speed up copying

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5526) Configurable file and directory based umask