[jira] [Commented] (HBASE-9829) make the compaction logging less confusing
[ https://issues.apache.org/jira/browse/HBASE-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848279#comment-13848279 ] Hudson commented on HBASE-9829: --- FAILURE: Integrated in HBase-TRUNK #4723 (See [https://builds.apache.org/job/HBase-TRUNK/4723/]) HBASE-9829 make the compaction logging less confusing (sershe: rev 1550866) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionRequest.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java make the compaction logging less confusing -- Key: HBASE-9829 URL: https://issues.apache.org/jira/browse/HBASE-9829 Project: HBase Issue Type: Improvement Components: Compaction Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Fix For: 0.99.0 Attachments: HBASE-9829.01.patch, HBASE-9829.patch 1) One of the most popular question from HBase users has got to be I have scheduled major compactions to run once per week, why are there so many. We need to somehow tell the user, wherever we log that there is a major compaction, whether it's a major compaction because that's what was in the request (from regular major compaction or user request), or was it just promoted because it took all files. Esp. the latter should be clear. 2) small vs large compaction threads and minor vs major compactions is confusing. Maybe the threads can be named short and long compactions. We -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10151) No-op HeapMemoryTuner
[ https://issues.apache.org/jira/browse/HBASE-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848280#comment-13848280 ] Hudson commented on HBASE-10151: FAILURE: Integrated in HBase-TRUNK #4723 (See [https://builds.apache.org/job/HBase-TRUNK/4723/]) HBASE-10151 No-op HeapMemoryTuner (anoopsamjohn: rev 1550871) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NoOpHeapMemoryTuner.java No-op HeapMemoryTuner - Key: HBASE-10151 URL: https://issues.apache.org/jira/browse/HBASE-10151 Project: HBase Issue Type: New Feature Affects Versions: 0.99.0 Reporter: Andrew Purtell Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10151.patch Provide a no-op HeapMemoryTuner that does not change any memory settings, just enforces the old style fixed proportions. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10155) HRegion isRecovering state is wrongly coming in postOpen hook
[ https://issues.apache.org/jira/browse/HBASE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848326#comment-13848326 ] Hudson commented on HBASE-10155: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #9 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/9/]) HBASE-10155 HRegion isRecovering state is wrongly coming in postOpen hook (anoopsamjohn: rev 1550881) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java HRegion isRecovering state is wrongly coming in postOpen hook - Key: HBASE-10155 URL: https://issues.apache.org/jira/browse/HBASE-10155 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10155.patch, HBASE-10155_V2.patch, HBASE-10155_V3.patch in the postOpen() hook also we should say correctly the isRecovering status of the region. Now it always comes as false. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10155) HRegion isRecovering state is wrongly coming in postOpen hook
[ https://issues.apache.org/jira/browse/HBASE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848328#comment-13848328 ] Hudson commented on HBASE-10155: SUCCESS: Integrated in HBase-0.98 #12 (See [https://builds.apache.org/job/HBase-0.98/12/]) HBASE-10155 HRegion isRecovering state is wrongly coming in postOpen hook (anoopsamjohn: rev 1550881) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java HRegion isRecovering state is wrongly coming in postOpen hook - Key: HBASE-10155 URL: https://issues.apache.org/jira/browse/HBASE-10155 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10155.patch, HBASE-10155_V2.patch, HBASE-10155_V3.patch in the postOpen() hook also we should say correctly the isRecovering status of the region. Now it always comes as false. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10155) HRegion isRecovering state is wrongly coming in postOpen hook
[ https://issues.apache.org/jira/browse/HBASE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848350#comment-13848350 ] Hudson commented on HBASE-10155: SUCCESS: Integrated in HBase-TRUNK #4724 (See [https://builds.apache.org/job/HBase-TRUNK/4724/]) HBASE-10155 HRegion isRecovering state is wrongly coming in postOpen hook (anoopsamjohn: rev 1550880) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java HRegion isRecovering state is wrongly coming in postOpen hook - Key: HBASE-10155 URL: https://issues.apache.org/jira/browse/HBASE-10155 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10155.patch, HBASE-10155_V2.patch, HBASE-10155_V3.patch in the postOpen() hook also we should say correctly the isRecovering status of the region. Now it always comes as false. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10157) Provide CP hook post log replay
[
https://issues.apache.org/jira/browse/HBASE-10157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848383#comment-13848383
]
Anoop Sam John commented on HBASE-10157:
{code}
this.isRecovering = newState;
+if (wasRecovering !isRecovering) {
+ // Call only when log replay is over.
+ coprocessorHost.postLogReplay();
+}
{code}
In the patch calling the hook after setting the boolean state. Can change to
call the hook before setting the boolean? That will make sure that region will
show its status as recovering before even while postLogReplay() hook works.
Provide CP hook post log replay
---
Key: HBASE-10157
URL: https://issues.apache.org/jira/browse/HBASE-10157
Project: HBase
Issue Type: Improvement
Components: Coprocessors
Affects Versions: 0.96.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0, 0.96.2, 0.99.0
Attachments: HBASE-10157.patch, HBASE-10157_V2.patch
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10157) Provide CP hook post log replay
[ https://issues.apache.org/jira/browse/HBASE-10157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10157: --- Attachment: HBASE-10157_V3.patch Provide CP hook post log replay --- Key: HBASE-10157 URL: https://issues.apache.org/jira/browse/HBASE-10157 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.96.2, 0.99.0 Attachments: HBASE-10157.patch, HBASE-10157_V2.patch, HBASE-10157_V3.patch -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Issue Comment Deleted] (HBASE-10157) Provide CP hook post log replay
[
https://issues.apache.org/jira/browse/HBASE-10157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-10157:
---
Comment: was deleted
(was: {code}
this.isRecovering = newState;
+if (wasRecovering !isRecovering) {
+ // Call only when log replay is over.
+ coprocessorHost.postLogReplay();
+}
{code}
In the patch calling the hook after setting the boolean state. Can change to
call the hook before setting the boolean? That will make sure that region will
show its status as recovering before even while postLogReplay() hook works.)
Provide CP hook post log replay
---
Key: HBASE-10157
URL: https://issues.apache.org/jira/browse/HBASE-10157
Project: HBase
Issue Type: Improvement
Components: Coprocessors
Affects Versions: 0.96.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0, 0.96.2, 0.99.0
Attachments: HBASE-10157.patch, HBASE-10157_V2.patch
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10157) Provide CP hook post log replay
[ https://issues.apache.org/jira/browse/HBASE-10157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10157: --- Attachment: (was: HBASE-10157_V3.patch) Provide CP hook post log replay --- Key: HBASE-10157 URL: https://issues.apache.org/jira/browse/HBASE-10157 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.96.2, 0.99.0 Attachments: HBASE-10157.patch, HBASE-10157_V2.patch -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10076) Backport MapReduce over snapshot files [0.94]
[ https://issues.apache.org/jira/browse/HBASE-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848404#comment-13848404 ] Bryan Keller commented on HBASE-10076: -- Yes, there is some minor cut-n-pasting involved. We could easily package this to be available outside of the distribution if that is deemed necessary. Backport MapReduce over snapshot files [0.94] - Key: HBASE-10076 URL: https://issues.apache.org/jira/browse/HBASE-10076 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Jesse Yates Fix For: 0.94.15 Attachments: hbase-10076-v0.patch MapReduce over Snapshots would be valuable on 0.94. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10076) Backport MapReduce over snapshot files [0.94]
[ https://issues.apache.org/jira/browse/HBASE-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848405#comment-13848405 ] Bryan Keller commented on HBASE-10076: -- BTW I have made a few minor enhancements since submitting that patch. I'll check this weekend to see if it is anything worthwhile. Backport MapReduce over snapshot files [0.94] - Key: HBASE-10076 URL: https://issues.apache.org/jira/browse/HBASE-10076 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Jesse Yates Fix For: 0.94.15 Attachments: hbase-10076-v0.patch MapReduce over Snapshots would be valuable on 0.94. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10157) Provide CP hook post log replay
[
https://issues.apache.org/jira/browse/HBASE-10157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848413#comment-13848413
]
Hadoop QA commented on HBASE-10157:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618642/HBASE-10157_V2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8171//console
This message is automatically generated.
Provide CP hook post log replay
---
Key: HBASE-10157
URL: https://issues.apache.org/jira/browse/HBASE-10157
Project: HBase
Issue Type: Improvement
Components: Coprocessors
Affects Versions: 0.96.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0, 0.96.2, 0.99.0
Attachments: HBASE-10157.patch, HBASE-10157_V2.patch
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10076) Backport MapReduce over snapshot files [0.94]
[ https://issues.apache.org/jira/browse/HBASE-10076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848455#comment-13848455 ] Lars Hofhansl commented on HBASE-10076: --- Thanks Bryan. Much appreciated! Backport MapReduce over snapshot files [0.94] - Key: HBASE-10076 URL: https://issues.apache.org/jira/browse/HBASE-10076 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Jesse Yates Fix For: 0.94.15 Attachments: hbase-10076-v0.patch MapReduce over Snapshots would be valuable on 0.94. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848463#comment-13848463 ] Bryan Keller commented on HBASE-8369: - One reason it is much smaller is I'm not using any of the Protobuf code, which is pretty sizable. MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v11.patch, hbase-8369_v5.patch, hbase-8369_v6.patch, hbase-8369_v7.patch, hbase-8369_v8.patch, hbase-8369_v9.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848467#comment-13848467 ] Bryan Keller commented on HBASE-8369: - Btw my patch is based off of Enis's original patch. The functionality should be the same AFAIK, i.e. a new table directory is assembled from the snapshot. MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v11.patch, hbase-8369_v5.patch, hbase-8369_v6.patch, hbase-8369_v7.patch, hbase-8369_v8.patch, hbase-8369_v9.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10157) Provide CP hook post log replay
[ https://issues.apache.org/jira/browse/HBASE-10157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848468#comment-13848468 ] Andrew Purtell commented on HBASE-10157: +1, please just add javadoc on commit that bypass() won't work here. Provide CP hook post log replay --- Key: HBASE-10157 URL: https://issues.apache.org/jira/browse/HBASE-10157 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.96.2, 0.99.0 Attachments: HBASE-10157.patch, HBASE-10157_V2.patch -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848478#comment-13848478 ] stack commented on HBASE-8369: -- bq. Thinking about this more, it is a good exercise to keep the M/R code out of HBase anyway... That'd be coolio. [~eclark] looked at doing the first step, making a new mr module but apparently a web of circular depdencies (client and server) MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v11.patch, hbase-8369_v5.patch, hbase-8369_v6.patch, hbase-8369_v7.patch, hbase-8369_v8.patch, hbase-8369_v9.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848481#comment-13848481 ] Enis Soztutar commented on HBASE-8369: -- bq. Maybe Enis Soztutar can mention the logic on why for some of these kinds of things? These are the list of high level things in the final version(v11) of the patch, which are different from Bryan's version (trunk-v3) - ClientScanner / AbstractClientScanner / TableRecordReaderImpl changes: the ClientSideRegion scanner keeps track of ScanMetrics, and exports those via MR job counters or Scan. - CellUtil changes : these are at a different place in Bryan's patch. - PB of MR data - HDFSBlocksDistribution: in v3, we are providing 3 servers with highest locality to the input split. In v11, we are using all the servers with 80% of the locality for the top locality server. This ensures better locality. - ClientSideRegionScanner / TableSnapshotScanner: not present in v3. ClientSideRegionScanner is an internal class to do the scanning. Both TableSnapshotScanner and TableSnapshotInputFormat uses it. TableSnapshotScanner is a client API, to scan snapshots without MR. - TableMapreduceUtil changes (other than the new method): needed in case security is enabled. We should not talk with the HBase cluster at all. - HRegion changes: v3 patch does send the parent dir for the region snapshot by assuming that table dir is the parent dir of the region dir. We do not want to make that assumption in trunk. - RestoreSnapshotHelper / ModifyRegionUtils : code organization - Other than these, general test, integration test, or performance evaluation tools. For 0.94, we can do a less intrusive patch which combines some of the changes above (like RestoreSnapshotHelper changes going into the new classes), and get rid of some of the changes like HRegion changes. MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v11.patch, hbase-8369_v5.patch, hbase-8369_v6.patch, hbase-8369_v7.patch, hbase-8369_v8.patch, hbase-8369_v9.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10164:
---
Attachment: 10164-v1.txt
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10164:
---
Assignee: Ted Yu
Status: Patch Available (was: Open)
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848496#comment-13848496
]
Ted Yu commented on HBASE-10164:
Tried the following commands with patch - they all succeeded:
{code}
1010 HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
1011 HBASE_HEAPSIZE=50m /usr/lib/hbase/bin/hbase shell
1013 HBASE_HEAPSIZE=50 /usr/lib/hbase/bin/hbase shell
{code}
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848499#comment-13848499
]
rajeshbabu commented on HBASE-10164:
lgtm. +1
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Comment Edited] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848499#comment-13848499
]
rajeshbabu edited comment on HBASE-10164 at 12/15/13 1:27 AM:
--
lgtm. +1
nit. format at fi.
{code}
+ fi
{code}
can be
{code}
+ fi
{code}
was (Author: rajesh23):
lgtm. +1
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10155) HRegion isRecovering state is wrongly coming in postOpen hook
[ https://issues.apache.org/jira/browse/HBASE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848503#comment-13848503 ] Hudson commented on HBASE-10155: SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-1.1 #6 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/6/]) HBASE-10155 HRegion isRecovering state is wrongly coming in postOpen hook (anoopsamjohn: rev 1550880) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java HRegion isRecovering state is wrongly coming in postOpen hook - Key: HBASE-10155 URL: https://issues.apache.org/jira/browse/HBASE-10155 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10155.patch, HBASE-10155_V2.patch, HBASE-10155_V3.patch in the postOpen() hook also we should say correctly the isRecovering status of the region. Now it always comes as false. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10151) No-op HeapMemoryTuner
[ https://issues.apache.org/jira/browse/HBASE-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848502#comment-13848502 ] Hudson commented on HBASE-10151: SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-1.1 #6 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/6/]) HBASE-10151 No-op HeapMemoryTuner (anoopsamjohn: rev 1550871) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NoOpHeapMemoryTuner.java No-op HeapMemoryTuner - Key: HBASE-10151 URL: https://issues.apache.org/jira/browse/HBASE-10151 Project: HBase Issue Type: New Feature Affects Versions: 0.99.0 Reporter: Andrew Purtell Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10151.patch Provide a no-op HeapMemoryTuner that does not change any memory settings, just enforces the old style fixed proportions. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9829) make the compaction logging less confusing
[ https://issues.apache.org/jira/browse/HBASE-9829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848501#comment-13848501 ] Hudson commented on HBASE-9829: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-1.1 #6 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/6/]) HBASE-9829 make the compaction logging less confusing (sershe: rev 1550866) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionRequest.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java make the compaction logging less confusing -- Key: HBASE-9829 URL: https://issues.apache.org/jira/browse/HBASE-9829 Project: HBase Issue Type: Improvement Components: Compaction Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Fix For: 0.99.0 Attachments: HBASE-9829.01.patch, HBASE-9829.patch 1) One of the most popular question from HBase users has got to be I have scheduled major compactions to run once per week, why are there so many. We need to somehow tell the user, wherever we log that there is a major compaction, whether it's a major compaction because that's what was in the request (from regular major compaction or user request), or was it just promoted because it took all files. Esp. the latter should be clear. 2) small vs large compaction threads and minor vs major compactions is confusing. Maybe the threads can be named short and long compactions. We -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8369) MapReduce over snapshot files
[ https://issues.apache.org/jira/browse/HBASE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848504#comment-13848504 ] Lars Hofhansl commented on HBASE-8369: -- These seems inmportant: bq. TableMapreduceUtil changes (other than the new method): needed in case security is enabled. We should not talk with the HBase cluster at all. bq. HDFSBlocksDistribution: in v3, we are providing 3 servers with highest locality to the input split. In v11, we are using all the servers with 80% of the locality for the top locality server. This ensures better locality. This seems nice to have: bq. ClientScanner / AbstractClientScanner / TableRecordReaderImpl changes: the ClientSideRegion scanner keeps track of ScanMetrics, and exports those via MR job counters or Scan. To track what/how the mappers are doing. The rest we can do without in 0.94. Let's also move the discussion to HBASE-10076. MapReduce over snapshot files - Key: HBASE-8369 URL: https://issues.apache.org/jira/browse/HBASE-8369 Project: HBase Issue Type: New Feature Components: mapreduce, snapshots Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0 Attachments: HBASE-8369-0.94.patch, HBASE-8369-0.94_v2.patch, HBASE-8369-0.94_v3.patch, HBASE-8369-0.94_v4.patch, HBASE-8369-0.94_v5.patch, HBASE-8369-trunk_v1.patch, HBASE-8369-trunk_v2.patch, HBASE-8369-trunk_v3.patch, hbase-8369_v0.patch, hbase-8369_v11.patch, hbase-8369_v5.patch, hbase-8369_v6.patch, hbase-8369_v7.patch, hbase-8369_v8.patch, hbase-8369_v9.patch The idea is to add an InputFormat, which can run the mapreduce job over snapshot files directly bypassing hbase server layer. The IF is similar in usage to TableInputFormat, taking a Scan object from the user, but instead of running from an online table, it runs from a table snapshot. We do one split per region in the snapshot, and open an HRegion inside the RecordReader. A RegionScanner is used internally for doing the scan without any HRegionServer bits. Users have been asking and searching for ways to run MR jobs by reading directly from hfiles, so this allows new use cases if reading from stale data is ok: - Take snapshots periodically, and run MR jobs only on snapshots. - Export snapshots to remote hdfs cluster, run the MR jobs at that cluster without HBase cluster. - (Future use case) Combine snapshot data with online hbase data: Scan from yesterday's snapshot, but read today's data from online hbase cluster. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-9261) Add cp hooks after {start|close}RegionOperation
[
https://issues.apache.org/jira/browse/HBASE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-9261:
--
Fix Version/s: 0.99.0
0.98.0
Add cp hooks after {start|close}RegionOperation
---
Key: HBASE-9261
URL: https://issues.apache.org/jira/browse/HBASE-9261
Project: HBase
Issue Type: Sub-task
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.99.0
Attachments: 9261-v7.patch, HBASE-9261.patch, HBASE-9261_v2.patch,
HBASE-9261_v3.patch, HBASE-9261_v4.patch, HBASE-9261_v5.patch,
HBASE-9261_v6.patch
These hooks helps for checking Resources(blocking memstore size) and
necessary locking on index region while performing batch of mutations.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-9261) Add cp hooks after {start|close}RegionOperation
[
https://issues.apache.org/jira/browse/HBASE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-9261:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
committed to trunk and 0.98.
Thanks all for reviews.
Add cp hooks after {start|close}RegionOperation
---
Key: HBASE-9261
URL: https://issues.apache.org/jira/browse/HBASE-9261
Project: HBase
Issue Type: Sub-task
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.99.0
Attachments: 9261-v7.patch, HBASE-9261.patch, HBASE-9261_v2.patch,
HBASE-9261_v3.patch, HBASE-9261_v4.patch, HBASE-9261_v5.patch,
HBASE-9261_v6.patch
These hooks helps for checking Resources(blocking memstore size) and
necessary locking on index region while performing batch of mutations.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10164:
---
Fix Version/s: 0.99.0
0.98.0
Hadoop Flags: Reviewed
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0, 0.99.0
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10164:
---
Release Note:
HBASE_HEAPSIZE accepts input with units (m, M, g, G).
If there is no unit, an 'm' would be appended to the value - keeping backward
compatibility.
Thanks for the review, I will make the correction upon commit.
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0, 0.99.0
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848514#comment-13848514
]
Hadoop QA commented on HBASE-10164:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618794/10164-v1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8172//console
This message is automatically generated.
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0, 0.99.0
Attachments: 10164-v1.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10164:
---
Attachment: 10164-v2.txt
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0, 0.99.0
Attachments: 10164-v1.txt, 10164-v2.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10164:
---
Status: Open (was: Patch Available)
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0, 0.99.0
Attachments: 10164-v1.txt, 10164-v2.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10048: --- Fix Version/s: 0.94.15 Integrated to 0.94 Thanks Shaohui Add hlog number metric in regionserver -- Key: HBASE-10048 URL: https://issues.apache.org/jira/browse/HBASE-10048 Project: HBase Issue Type: Improvement Components: metrics Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 Attachments: 10048.096.v4.txt, HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff, HBASE-10048-0.94-v3.diff, HBASE-10048-trunk-v1.diff, HBASE-10048-trunk-v2.diff, HBASE-10048-trunk-v3.diff, HBASE-10048-trunk-v4.diff Add hlog number metric in regionserver. We can use this metric to alert about memstore flush because of too many hlogs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9261) Add cp hooks after {start|close}RegionOperation
[
https://issues.apache.org/jira/browse/HBASE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848528#comment-13848528
]
Hudson commented on HBASE-9261:
---
SUCCESS: Integrated in HBase-TRUNK #4726 (See
[https://builds.apache.org/job/HBase-TRUNK/4726/])
HBASE-9261 Add cp hooks after {start|close}RegionOperation (rajeshbabu: rev
1550990)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientSideRegionScanner.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
Add cp hooks after {start|close}RegionOperation
---
Key: HBASE-9261
URL: https://issues.apache.org/jira/browse/HBASE-9261
Project: HBase
Issue Type: Sub-task
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.99.0
Attachments: 9261-v7.patch, HBASE-9261.patch, HBASE-9261_v2.patch,
HBASE-9261_v3.patch, HBASE-9261_v4.patch, HBASE-9261_v5.patch,
HBASE-9261_v6.patch
These hooks helps for checking Resources(blocking memstore size) and
necessary locking on index region while performing batch of mutations.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9261) Add cp hooks after {start|close}RegionOperation
[
https://issues.apache.org/jira/browse/HBASE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848532#comment-13848532
]
Hudson commented on HBASE-9261:
---
SUCCESS: Integrated in HBase-0.98 #13 (See
[https://builds.apache.org/job/HBase-0.98/13/])
HBASE-9261 Add cp hooks after {start|close}RegionOperation (rajeshbabu: rev
1550991)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientSideRegionScanner.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
Add cp hooks after {start|close}RegionOperation
---
Key: HBASE-9261
URL: https://issues.apache.org/jira/browse/HBASE-9261
Project: HBase
Issue Type: Sub-task
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.99.0
Attachments: 9261-v7.patch, HBASE-9261.patch, HBASE-9261_v2.patch,
HBASE-9261_v3.patch, HBASE-9261_v4.patch, HBASE-9261_v5.patch,
HBASE-9261_v6.patch
These hooks helps for checking Resources(blocking memstore size) and
necessary locking on index region while performing batch of mutations.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848537#comment-13848537 ] Hudson commented on HBASE-10048: SUCCESS: Integrated in HBase-0.94 #1227 (See [https://builds.apache.org/job/HBase-0.94/1227/]) HBASE-10048 Add hlog number metric in regionserver (tedyu: rev 1550997) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java Add hlog number metric in regionserver -- Key: HBASE-10048 URL: https://issues.apache.org/jira/browse/HBASE-10048 Project: HBase Issue Type: Improvement Components: metrics Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 Attachments: 10048.096.v4.txt, HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff, HBASE-10048-0.94-v3.diff, HBASE-10048-trunk-v1.diff, HBASE-10048-trunk-v2.diff, HBASE-10048-trunk-v3.diff, HBASE-10048-trunk-v4.diff Add hlog number metric in regionserver. We can use this metric to alert about memstore flush because of too many hlogs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Resolved] (HBASE-10166) Backport HBASE-8143 to 0.94 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[ https://issues.apache.org/jira/browse/HBASE-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-10166. --- Resolution: Won't Fix Backport HBASE-8143 to 0.94 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM --- Key: HBASE-10166 URL: https://issues.apache.org/jira/browse/HBASE-10166 Project: HBase Issue Type: Sub-task Components: hadoop2 Reporter: stack Assignee: stack Fix For: 0.94.15 Attachments: 8143v2.094.txt You want this in 0.94 [~lhofhansl]? -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10166) Backport HBASE-8143 to 0.94 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[ https://issues.apache.org/jira/browse/HBASE-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848539#comment-13848539 ] stack commented on HBASE-10166: --- bq. You have a good way to verify? Well, all is fine and then you OOME. I can doc for 0.94, np., or, rather, we already suggest down the buffer size when you set SSR in our doc so let me just close this as not relevant for 0.94. Backport HBASE-8143 to 0.94 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM --- Key: HBASE-10166 URL: https://issues.apache.org/jira/browse/HBASE-10166 Project: HBase Issue Type: Sub-task Components: hadoop2 Reporter: stack Assignee: stack Fix For: 0.94.15 Attachments: 8143v2.094.txt You want this in 0.94 [~lhofhansl]? -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848541#comment-13848541 ] Hudson commented on HBASE-10048: SUCCESS: Integrated in HBase-0.94-security #360 (See [https://builds.apache.org/job/HBase-0.94-security/360/]) HBASE-10048 Add hlog number metric in regionserver (tedyu: rev 1550997) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java Add hlog number metric in regionserver -- Key: HBASE-10048 URL: https://issues.apache.org/jira/browse/HBASE-10048 Project: HBase Issue Type: Improvement Components: metrics Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 Attachments: 10048.096.v4.txt, HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff, HBASE-10048-0.94-v3.diff, HBASE-10048-trunk-v1.diff, HBASE-10048-trunk-v2.diff, HBASE-10048-trunk-v3.diff, HBASE-10048-trunk-v4.diff Add hlog number metric in regionserver. We can use this metric to alert about memstore flush because of too many hlogs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10155) HRegion isRecovering state is wrongly coming in postOpen hook
[ https://issues.apache.org/jira/browse/HBASE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848542#comment-13848542 ] stack commented on HBASE-10155: --- +1 for 0.96. Thanks [~anoopsamjohn] HRegion isRecovering state is wrongly coming in postOpen hook - Key: HBASE-10155 URL: https://issues.apache.org/jira/browse/HBASE-10155 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10155.patch, HBASE-10155_V2.patch, HBASE-10155_V3.patch in the postOpen() hook also we should say correctly the isRecovering status of the region. Now it always comes as false. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10166) Backport HBASE-8143 to 0.94 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[ https://issues.apache.org/jira/browse/HBASE-10166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-10166: -- Fix Version/s: (was: 0.94.15) Backport HBASE-8143 to 0.94 HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM --- Key: HBASE-10166 URL: https://issues.apache.org/jira/browse/HBASE-10166 Project: HBase Issue Type: Sub-task Components: hadoop2 Reporter: stack Assignee: stack Attachments: 8143v2.094.txt You want this in 0.94 [~lhofhansl]? -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10048) Add hlog number metric in regionserver
[ https://issues.apache.org/jira/browse/HBASE-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848543#comment-13848543 ] Lars Hofhansl commented on HBASE-10048: --- Thanks Liu and Ted. Add hlog number metric in regionserver -- Key: HBASE-10048 URL: https://issues.apache.org/jira/browse/HBASE-10048 Project: HBase Issue Type: Improvement Components: metrics Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 Attachments: 10048.096.v4.txt, HBASE-10048-0.94-v1.diff, HBASE-10048-0.94-v2.diff, HBASE-10048-0.94-v3.diff, HBASE-10048-trunk-v1.diff, HBASE-10048-trunk-v2.diff, HBASE-10048-trunk-v3.diff, HBASE-10048-trunk-v4.diff Add hlog number metric in regionserver. We can use this metric to alert about memstore flush because of too many hlogs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9261) Add cp hooks after {start|close}RegionOperation
[
https://issues.apache.org/jira/browse/HBASE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848544#comment-13848544
]
Hudson commented on HBASE-9261:
---
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #10 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/10/])
HBASE-9261 Add cp hooks after {start|close}RegionOperation (rajeshbabu: rev
1550991)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientSideRegionScanner.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
Add cp hooks after {start|close}RegionOperation
---
Key: HBASE-9261
URL: https://issues.apache.org/jira/browse/HBASE-9261
Project: HBase
Issue Type: Sub-task
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.99.0
Attachments: 9261-v7.patch, HBASE-9261.patch, HBASE-9261_v2.patch,
HBASE-9261_v3.patch, HBASE-9261_v4.patch, HBASE-9261_v5.patch,
HBASE-9261_v6.patch
These hooks helps for checking Resources(blocking memstore size) and
necessary locking on index region while performing batch of mutations.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9927) ReplicationLogCleaner#stop() calls HConnectionManager#deleteConnection() unnecessarily
[
https://issues.apache.org/jira/browse/HBASE-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848547#comment-13848547
]
Lars Hofhansl commented on HBASE-9927:
--
+1
Seems fine to just remove it.
ReplicationLogCleaner#stop() calls HConnectionManager#deleteConnection()
unnecessarily
--
Key: HBASE-9927
URL: https://issues.apache.org/jira/browse/HBASE-9927
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Fix For: 0.94.15
Attachments: 9927.txt
When inspecting log, I found the following:
{code}
2013-11-08 18:23:48,472 ERROR [M:0;kiyo:42380.oldLogCleaner]
client.HConnectionManager(468): Connection not found in the list, can't
delete it (connection key=HConnectionKey{properties={hbase.rpc.timeout=6,
hbase.zookeeper.property.clientPort=59832, hbase.client.pause=100,
zookeeper.znode.parent=/hbase, hbase.client.retries.number=350,
hbase.zookeeper.quorum=localhost}, username='zy'}). May be the key was
modified?
java.lang.Exception
at
org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:468)
at
org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:404)
at
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.stop(ReplicationLogCleaner.java:141)
at
org.apache.hadoop.hbase.master.cleaner.CleanerChore.cleanup(CleanerChore.java:276)
{code}
The call to HConnectionManager#deleteConnection() is not needed.
Here is related code which has a comment for this effect:
{code}
// Not sure why we're deleting a connection that we never acquired or used
HConnectionManager.deleteConnection(this.getConf());
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9047) Tool to handle finishing replication when the cluster is offline
[ https://issues.apache.org/jira/browse/HBASE-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848548#comment-13848548 ] Lars Hofhansl commented on HBASE-9047: -- No automated way, currently. Need to run it yourself :( I'll run some selected tests. If there are no further objections, I'll commit this to all branches soon. Tool to handle finishing replication when the cluster is offline Key: HBASE-9047 URL: https://issues.apache.org/jira/browse/HBASE-9047 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0 Reporter: Jean-Daniel Cryans Assignee: Demai Ni Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 Attachments: HBASE-9047-0.94-v1.patch, HBASE-9047-0.94.9-v0.PATCH, HBASE-9047-trunk-v0.patch, HBASE-9047-trunk-v1.patch, HBASE-9047-trunk-v2.patch, HBASE-9047-trunk-v3.patch, HBASE-9047-trunk-v4.patch, HBASE-9047-trunk-v4.patch, HBASE-9047-trunk-v5.patch, HBASE-9047-trunk-v6.patch, HBASE-9047-trunk-v7.patch, HBASE-9047-trunk-v7.patch We're having a discussion on the mailing list about replicating the data on a cluster that was shut down in an offline fashion. The motivation could be that you don't want to bring HBase back up but still need that data on the slave. So I have this idea of a tool that would be running on the master cluster while it is down, although it could also run at any time. Basically it would be able to read the replication state of each master region server, finish replicating what's missing to all the slave, and then clear that state in zookeeper. The code that handles replication does most of that already, see ReplicationSourceManager and ReplicationSource. Basically when ReplicationSourceManager.init() is called, it will check all the queues in ZK and try to grab those that aren't attached to a region server. If the whole cluster is down, it will grab all of them. The beautiful thing here is that you could start that tool on all your machines and the load will be spread out, but that might not be a big concern if replication wasn't lagging since it would take a few seconds to finish replicating the missing data for each region server. I'm guessing when starting ReplicationSourceManager you'd give it a fake region server ID, and you'd tell it not to start its own source. FWIW the main difference in how replication is handled between Apache's HBase and Facebook's is that the latter is always done separately of HBase itself. This jira isn't about doing that. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9927) ReplicationLogCleaner#stop() calls HConnectionManager#deleteConnection() unnecessarily
[
https://issues.apache.org/jira/browse/HBASE-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848549#comment-13848549
]
Ted Yu commented on HBASE-9927:
---
Will integrate on Monday if there is no objection.
ReplicationLogCleaner#stop() calls HConnectionManager#deleteConnection()
unnecessarily
--
Key: HBASE-9927
URL: https://issues.apache.org/jira/browse/HBASE-9927
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Fix For: 0.94.15
Attachments: 9927.txt
When inspecting log, I found the following:
{code}
2013-11-08 18:23:48,472 ERROR [M:0;kiyo:42380.oldLogCleaner]
client.HConnectionManager(468): Connection not found in the list, can't
delete it (connection key=HConnectionKey{properties={hbase.rpc.timeout=6,
hbase.zookeeper.property.clientPort=59832, hbase.client.pause=100,
zookeeper.znode.parent=/hbase, hbase.client.retries.number=350,
hbase.zookeeper.quorum=localhost}, username='zy'}). May be the key was
modified?
java.lang.Exception
at
org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:468)
at
org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:404)
at
org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.stop(ReplicationLogCleaner.java:141)
at
org.apache.hadoop.hbase.master.cleaner.CleanerChore.cleanup(CleanerChore.java:276)
{code}
The call to HConnectionManager#deleteConnection() is not needed.
Here is related code which has a comment for this effect:
{code}
// Not sure why we're deleting a connection that we never acquired or used
HConnectionManager.deleteConnection(this.getConf());
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848552#comment-13848552
]
Hudson commented on HBASE-10164:
SUCCESS: Integrated in HBase-TRUNK #4727 (See
[https://builds.apache.org/job/HBase-TRUNK/4727/])
HBASE-10164 Allow heapsize of different units to be passed as HBASE_HEAPSIZE
(tedyu: rev 1550996)
* /hbase/trunk/bin/hbase
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0, 0.99.0
Attachments: 10164-v1.txt, 10164-v2.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10157) Provide CP hook post log replay
[ https://issues.apache.org/jira/browse/HBASE-10157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848553#comment-13848553 ] Anoop Sam John commented on HBASE-10157: Thanks for the review Andy. bq.please just add javadoc on commit that bypass() won't work here. Do we need? No post hook honor bypass and we normally not saying that explicitly. If you still feel we need to mention, I can add.. Provide CP hook post log replay --- Key: HBASE-10157 URL: https://issues.apache.org/jira/browse/HBASE-10157 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.96.2, 0.99.0 Attachments: HBASE-10157.patch, HBASE-10157_V2.patch -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10155) HRegion isRecovering state is wrongly coming in postOpen hook
[ https://issues.apache.org/jira/browse/HBASE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10155: --- Resolution: Fixed Fix Version/s: 0.99.0 0.96.2 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Stack. Committed to 96 branch also. HRegion isRecovering state is wrongly coming in postOpen hook - Key: HBASE-10155 URL: https://issues.apache.org/jira/browse/HBASE-10155 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.96.2, 0.99.0 Attachments: HBASE-10155.patch, HBASE-10155_V2.patch, HBASE-10155_V3.patch in the postOpen() hook also we should say correctly the isRecovering status of the region. Now it always comes as false. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10164) Allow heapsize of different units to be passed as HBASE_HEAPSIZE
[
https://issues.apache.org/jira/browse/HBASE-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848558#comment-13848558
]
Hudson commented on HBASE-10164:
FAILURE: Integrated in HBase-0.98 #14 (See
[https://builds.apache.org/job/HBase-0.98/14/])
HBASE-10164 Allow heapsize of different units to be passed as HBASE_HEAPSIZE
(tedyu: rev 1550994)
* /hbase/branches/0.98/bin/hbase
Allow heapsize of different units to be passed as HBASE_HEAPSIZE
Key: HBASE-10164
URL: https://issues.apache.org/jira/browse/HBASE-10164
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0, 0.99.0
Attachments: 10164-v1.txt, 10164-v2.txt
Currently HBASE_HEAPSIZE doesn't expect units.
{code}
$ HBASE_HEAPSIZE=5G /usr/lib/hbase/bin/hbase shell
Invalid maximum heap size: -Xmx5Gm
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
{code}
It would be more user friendly if different units of heapsize can be passed
through HBASE_HEAPSIZE
If unit ('m' or 'G') is specified, 'm' doesn't need to be appended.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
