date:20110907


[ 
https://issues.apache.org/jira/browse/HBASE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098790#comment-13098790
 ] 

Hudson commented on HBASE-4338:
---

Integrated in HBase-TRUNK #2180 (See 
[https://builds.apache.org/job/HBase-TRUNK/2180/])
HBASE-4338 Package build for rpm and deb are broken

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/pom.xml


> Package build for rpm and deb are broken
> 
>
> Key: HBASE-4338
> URL: https://issues.apache.org/jira/browse/HBASE-4338
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
> Environment: Java, Linux
>Reporter: Eric Yang
> Fix For: 0.92.0
>
> Attachments: HBASE-4338.patch
>
>
> Environment variable final.name was removed in HBASE-3629, and this prevents 
> rpm/deb packaging from building.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4327) Compile HBase against hadoop 0.22


[ 
https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098791#comment-13098791
 ] 

Hudson commented on HBASE-4327:
---

Integrated in HBase-TRUNK #2180 (See 
[https://builds.apache.org/job/HBase-TRUNK/2180/])
HBASE-4327 Compile HBase against hadoop 0.22

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/pom.xml


> Compile HBase against hadoop 0.22
> -
>
> Key: HBASE-4327
> URL: https://issues.apache.org/jira/browse/HBASE-4327
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.92.0
>Reporter: Joep Rottinghuis
> Fix For: 0.92.0
>
> Attachments: HBASE-4327-Michael.patch, HBASE-4327.patch, 
> HBASE-4327.patch, HBASE-4327.patch
>
>
> Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one 
> for hadoop-0.22.
> When overriding hadoop.version to 0.22, then the (compile-time) dependency on 
> hadoop-annotations cannot be met.
> That exists on 0.23 and 0.24/trunk, but not on 0.22.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4339) Improve eclipse documentation and project file generation


[ 
https://issues.apache.org/jira/browse/HBASE-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098789#comment-13098789
 ] 

Hudson commented on HBASE-4339:
---

Integrated in HBase-TRUNK #2180 (See 
[https://builds.apache.org/job/HBase-TRUNK/2180/])
HBASE-4339 Improve eclipse documentation and project file generation

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/pom.xml
* /hbase/trunk/src/docbkx/developer.xml


> Improve eclipse documentation and project file generation
> -
>
> Key: HBASE-4339
> URL: https://issues.apache.org/jira/browse/HBASE-4339
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.0
>Reporter: Eric Charles
>Assignee: Eric Charles
>Priority: Minor
> Attachments: HBASE-4339_book.patch, HBASE-4339_pom.patch
>
>
> Import via m2eclipse ask a few build path fixes. This should be documented in 
> the hbase book.
> mvn eclipse:eclipse is helped with the build-helper-maven-plugin plugin where 
> additional folder (target/...) are listed. The listed jamon folder is wrong.
> (Putting these 2 concerns on same jira as they are more or less related, 
> avoiding jira proliferation).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098828#comment-13098828
 ] 

jirapos...@reviews.apache.org commented on HBASE-4015:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1668/#review1783
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


Now i have moved it into AM itself.  



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


If you see the implementation of processOpeningState it is same as the 
handling of the previous OPENING state handling by timeout monitor.   Just that 
now we call it immediately instead of timeout monitor acting on it.

"Should we at least check the znode again before going ahead with assign? "

Now for the possibility of the znode moving to OPENED state before we try 
to assign to a new RS, our new logic of mulitiple checks that we do before a 
node can be forcefully moved to OFFLINE will handle the scenario that you have 
told.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


changed the method name



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


formatting done and the variable name has been changed from reassign to 
hijackAndPreempt.

Stack last time you asked to change the variable name to reassign :)).



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


invoke from timeout monitor has been removed everywhere.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


Before we try changing the znode state if some failure happens then it 
means RS succeeded.  So in this place we need not change the RS state. Correct 
me if am wrong 



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


Next time will not repeat.  When we refactor a piece of code to a new api 
and apply format on that the formatter does this.
Sometimes it gets missed out stack. Sorry



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


This is an existing code.  just the code has been moved into a method



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java


done



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java


This statement will increment the version number.  But the stat object is 
got before this step.
hence on successful creation of the node with offline state we create a new 
watcher and so the version+1 will give me the current exact version in the node.



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZK

[jira] [Commented] (HBASE-4195) Possible inconsistency in a memstore read after a reseek, possible performance improvement

2011-09-07 Thread nkeywal (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098944#comment-13098944
 ] 

nkeywal commented on HBASE-4195:


Synthesis:
 - the patch on the test case itself helps to understand where the error come 
from, but does not change anything else
 - the fix on the memstore fix the menstore part.
 - there is still another issue, not linked to the memstore but to the flush 
part (see message above from 23/Aug/11)
 - the test cas may fail for this reason. The probability of a failure is 
increased by increasing the test count and lowering the flush interval
 - Anyway, the flush issue is already handled in HBASE-2856.

So I think we can consider the patch as ok for the scope of this bug? @Ted : 
@Stack: do you agree? Do you need more info?


> Possible inconsistency in a memstore read after a reseek, possible 
> performance improvement
> --
>
> Key: HBASE-4195
> URL: https://issues.apache.org/jira/browse/HBASE-4195
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 20110824_4195_MemStore.patch, 
> 20110824_4195_TestHRegion.patch
>
>
> This follows the dicussion around HBASE-3855, and the random errors (20% 
> failure on trunk) on the unit test 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting
> I saw some points related to numIterReseek, used in the 
> MemStoreScanner#getNext (line 690):
> {noformat}679 protected KeyValue getNext(Iterator it) {
> 680 KeyValue ret = null;
> 681 long readPoint = ReadWriteConsistencyControl.getThreadReadPoint();
> 682 //DebugPrint.println( " MS@" + hashCode() + ": threadpoint = " + 
> readPoint);
> 683
> 684 while (ret == null && it.hasNext()) {
> 685   KeyValue v = it.next();
> 686   if (v.getMemstoreTS() <= readPoint) {
> 687 // keep it.
> 688 ret = v;
> 689   }
> 690   numIterReseek--;
> 691   if (numIterReseek == 0) {
> 692 break;
> 693}
> 694 }
> 695 return ret;
> 696   }{noformat}
> This function is called by seek, reseek, and next. The numIterReseek is only 
> usefull for reseek.
> There are some issues, I am not totally sure it's the root cause of the test 
> case error, but it could explain partly the randomness of the error, and one 
> point is for sure a bug.
> 1) In getNext, numIterReseek is decreased, then compared to zero. The seek 
> function sets numIterReseek to zero before calling getNext. It means that the 
> value will be actually negative, hence the test will always fail, and the 
> loop will continue. It is the expected behaviour, but it's quite smart.
> 2) In "reseek", numIterReseek is not set between the loops on the two 
> iterators. If the numIterReseek is equals to zero after the loop on the first 
> one, the loop on the second one will never call seek, as numIterReseek will 
> be negative.
> 3) Still in "reseek", the test to call "seek" is (kvsetNextRow == null && 
> numIterReseek == 0). In other words, if kvsetNextRow is not null when 
> numIterReseek equals zero, numIterReseek will start to be negative at the 
> next iteration and seek will never be called.
> 4) You can have side effects if reseek ends with a numIterReseek > 0: the 
> following calls to the "next" function will decrease numIterReseek to zero, 
> and getNext will break instead of continuing the loop. As a result, later 
> calls to next() may return null or not depending on how is configured the 
> default value for numIterReseek.
> To check if the issue comes from point 4, you can set the numIterReseek to 
> zero before returning in reseek:
> {noformat}  numIterReseek = 0;
>   return (kvsetNextRow != null || snapshotNextRow != null);
> }{noformat}
> On my env, on trunk, it seems to work, but as it's random I am not really 
> sure. I also had to modify the test (I added a loop) to make it fails more 
> often, the original test was working quite well here.
> It has to be confirmed that this totally fix (it could be partial or 
> unrelated) 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting 
> before implementing a complete solution.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4188) Make in-memory table scanning faster, part2 (continuation of hbase-1938)

2011-09-07 Thread nkeywal (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098948#comment-13098948
 ] 

nkeywal commented on HBASE-4188:


See HBASE-4195. It's the same test.

> Make in-memory table scanning faster, part2 (continuation of hbase-1938)
> 
>
> Key: HBASE-4188
> URL: https://issues.apache.org/jira/browse/HBASE-4188
> Project: HBase
>  Issue Type: Improvement
>  Components: performance
>Reporter: stack
>Assignee: nkeywal
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: 4188-mssp.txt
>
>
> This issue is a continuation of HBASE-1938 work (That issue is closed).  This 
> issue is about getting the last patch posted by nkeywal over in HBASE-1938 
> applied (assigned nkeywal since he's done the work).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4195) Possible inconsistency in a memstore read after a reseek, possible performance improvement


[ 
https://issues.apache.org/jira/browse/HBASE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098951#comment-13098951
 ] 

Ted Yu commented on HBASE-4195:
---

I already voted +1 on latest patch.

> Possible inconsistency in a memstore read after a reseek, possible 
> performance improvement
> --
>
> Key: HBASE-4195
> URL: https://issues.apache.org/jira/browse/HBASE-4195
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 20110824_4195_MemStore.patch, 
> 20110824_4195_TestHRegion.patch
>
>
> This follows the dicussion around HBASE-3855, and the random errors (20% 
> failure on trunk) on the unit test 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting
> I saw some points related to numIterReseek, used in the 
> MemStoreScanner#getNext (line 690):
> {noformat}679 protected KeyValue getNext(Iterator it) {
> 680 KeyValue ret = null;
> 681 long readPoint = ReadWriteConsistencyControl.getThreadReadPoint();
> 682 //DebugPrint.println( " MS@" + hashCode() + ": threadpoint = " + 
> readPoint);
> 683
> 684 while (ret == null && it.hasNext()) {
> 685   KeyValue v = it.next();
> 686   if (v.getMemstoreTS() <= readPoint) {
> 687 // keep it.
> 688 ret = v;
> 689   }
> 690   numIterReseek--;
> 691   if (numIterReseek == 0) {
> 692 break;
> 693}
> 694 }
> 695 return ret;
> 696   }{noformat}
> This function is called by seek, reseek, and next. The numIterReseek is only 
> usefull for reseek.
> There are some issues, I am not totally sure it's the root cause of the test 
> case error, but it could explain partly the randomness of the error, and one 
> point is for sure a bug.
> 1) In getNext, numIterReseek is decreased, then compared to zero. The seek 
> function sets numIterReseek to zero before calling getNext. It means that the 
> value will be actually negative, hence the test will always fail, and the 
> loop will continue. It is the expected behaviour, but it's quite smart.
> 2) In "reseek", numIterReseek is not set between the loops on the two 
> iterators. If the numIterReseek is equals to zero after the loop on the first 
> one, the loop on the second one will never call seek, as numIterReseek will 
> be negative.
> 3) Still in "reseek", the test to call "seek" is (kvsetNextRow == null && 
> numIterReseek == 0). In other words, if kvsetNextRow is not null when 
> numIterReseek equals zero, numIterReseek will start to be negative at the 
> next iteration and seek will never be called.
> 4) You can have side effects if reseek ends with a numIterReseek > 0: the 
> following calls to the "next" function will decrease numIterReseek to zero, 
> and getNext will break instead of continuing the loop. As a result, later 
> calls to next() may return null or not depending on how is configured the 
> default value for numIterReseek.
> To check if the issue comes from point 4, you can set the numIterReseek to 
> zero before returning in reseek:
> {noformat}  numIterReseek = 0;
>   return (kvsetNextRow != null || snapshotNextRow != null);
> }{noformat}
> On my env, on trunk, it seems to work, but as it's random I am not really 
> sure. I also had to modify the test (I added a loop) to make it fails more 
> often, the original test was working quite well here.
> It has to be confirmed that this totally fix (it could be partial or 
> unrelated) 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting 
> before implementing a complete solution.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098959#comment-13098959
 ] 

jirapos...@reviews.apache.org commented on HBASE-4015:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1668/#review1786
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java


I think preempt alone would suffice.
We don't need to place hijack and preempt together.


- Ted


On 2011-08-29 12:11:23, ramkrishna vasudevan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1668/
bq.  ---
bq.  
bq.  (Updated 2011-08-29 12:11:23)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Jean-Daniel Cryans, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  HBASE-4015 - updated patch.
bq.  
bq.  
bq.  invokeTimeOutManager(regionState.getRegion(),
bq.  TimeOutOperationType.ASSIGN);
bq.  
bq.  Instead of storing the state when timeout was deducted we process it
bq.  using future task.
bq.  
bq.  
bq.case RS_ZK_REGION_OPENING:
bq.  // TODO: Could check if it was on deadServers.  If it was, then we 
could
bq.  // do what happens in TimeoutMonitor when it sees this condition.
bq.  
bq.  // Just insert region into RIT
bq.  // If this never updates the timeout will trigger new assignment
bq.  if (regionInfo.isMetaRegion() || regionInfo.isRootRegion()) {
bq.regionsInTransition.put(encodedRegionName, new RegionState(
bq.regionInfo, RegionState.State.OPENING, data.getStamp(), data
bq..getOrigin()));
bq.processOpeningState(regionInfo);
bq.break;
bq.  }
bq.  regionsInTransition.put(encodedRegionName, new 
RegionState(regionInfo,
bq.  RegionState.State.OPENING, data.getStamp(), data.getOrigin()));
bq.  break;
bq.  
bq.  This change is for HBASE-4203. META and ROOT table need not wait till 
timeout
bq.  ==
bq.  
bq.  In forceRegionStateToOffline()
bq.  
bq.  } else {
bq.// If invoked from timeout monitor donot force it to OFFLINE. Based 
on the
bq.// state we will decide if to change in-memory state to OFFLINE or 
not.  It will
bq.// be done before setting the znode to OFFLINE state.
bq.if (!timeOutMonitorReAllocate) {
bq.  LOG.debug("Forcing OFFLINE; was=" + state);
bq.  state.update(RegionState.State.OFFLINE);
bq.}
bq.  If the timeout monitor tries to reallcoate the node then dont make
bq.  the inmemory state to OFFLINE.
bq.  But the noraml assign flow doesnot expect the inmemory state to OFFLINE.
bq.  Hence the above change.  This is continued with the check in 
bq.  assign()
bq.  int setOfflineInZooKeeper(final RegionState state,
bq.boolean timeOutMonitorReAllocate) {
bq.  // If invoked from timeoutmonitor the current state in memory need not 
be
bq.  // OFFLINE.  
bq.  if (!timeOutMonitorReAllocate && !state.isClosed() && 
!state.isOffline()) {
bq.this.master.abort("Unexpected state trying to OFFLINE; " + state,
bq.new IllegalStateException());
bq.return -1;
bq.  }
bq.  ==
bq.  
bq.  boolean allowCreation = false;
bq.  // If the isReAllocate is true and the current state is PENDING_OPEN
bq.  // or OPENING then update the inmemory state to PENDING_OPEN. This is
bq.  // important because
bq.  // if timeoutmonitor deducts that a region was in OPENING state for a 
long
bq.  // time but by the
bq.  // time timeout monitor tranits the node to OFFLINE the RS would have 
opened
bq.  // the node and the
bq.  // state in znode will be RS_ZK_REGION_OPENED. Inorder to invoke the
bq.  // OpenedRegionHandler
bq.  // we expect the inmemeory state to be PENDING_OPEN or OPENING.
bq.  // For all other cases we can change the inmemory state to OFFLINE.
bq.  if (timeOutMonitorReAllocate
bq.  && (state.getState().equals(RegionState.State.PENDING_OPEN) || 
state
bq.  .getState().equals(RegionState.State.OPENING))) {
bq.state.update(RegionState.State.PENDING_OPEN);
bq.allowCreation = false;
bq.  } else {
bq.state.update(RegionState.State.OFFLINE);
bq.allowCrea

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098991#comment-13098991
 ] 

jirapos...@reviews.apache.org commented on HBASE-4015:
--



bq.  On 2011-09-07 13:55:00, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java,
 line 1260
bq.  > 
bq.  >
bq.  > I think preempt alone would suffice.
bq.  > We don't need to place hijack and preempt together.

I can do this on commit .. no need to change Ram.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1668/#review1786
---


On 2011-08-29 12:11:23, ramkrishna vasudevan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1668/
bq.  ---
bq.  
bq.  (Updated 2011-08-29 12:11:23)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Jean-Daniel Cryans, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  HBASE-4015 - updated patch.
bq.  
bq.  
bq.  invokeTimeOutManager(regionState.getRegion(),
bq.  TimeOutOperationType.ASSIGN);
bq.  
bq.  Instead of storing the state when timeout was deducted we process it
bq.  using future task.
bq.  
bq.  
bq.case RS_ZK_REGION_OPENING:
bq.  // TODO: Could check if it was on deadServers.  If it was, then we 
could
bq.  // do what happens in TimeoutMonitor when it sees this condition.
bq.  
bq.  // Just insert region into RIT
bq.  // If this never updates the timeout will trigger new assignment
bq.  if (regionInfo.isMetaRegion() || regionInfo.isRootRegion()) {
bq.regionsInTransition.put(encodedRegionName, new RegionState(
bq.regionInfo, RegionState.State.OPENING, data.getStamp(), data
bq..getOrigin()));
bq.processOpeningState(regionInfo);
bq.break;
bq.  }
bq.  regionsInTransition.put(encodedRegionName, new 
RegionState(regionInfo,
bq.  RegionState.State.OPENING, data.getStamp(), data.getOrigin()));
bq.  break;
bq.  
bq.  This change is for HBASE-4203. META and ROOT table need not wait till 
timeout
bq.  ==
bq.  
bq.  In forceRegionStateToOffline()
bq.  
bq.  } else {
bq.// If invoked from timeout monitor donot force it to OFFLINE. Based 
on the
bq.// state we will decide if to change in-memory state to OFFLINE or 
not.  It will
bq.// be done before setting the znode to OFFLINE state.
bq.if (!timeOutMonitorReAllocate) {
bq.  LOG.debug("Forcing OFFLINE; was=" + state);
bq.  state.update(RegionState.State.OFFLINE);
bq.}
bq.  If the timeout monitor tries to reallcoate the node then dont make
bq.  the inmemory state to OFFLINE.
bq.  But the noraml assign flow doesnot expect the inmemory state to OFFLINE.
bq.  Hence the above change.  This is continued with the check in 
bq.  assign()
bq.  int setOfflineInZooKeeper(final RegionState state,
bq.boolean timeOutMonitorReAllocate) {
bq.  // If invoked from timeoutmonitor the current state in memory need not 
be
bq.  // OFFLINE.  
bq.  if (!timeOutMonitorReAllocate && !state.isClosed() && 
!state.isOffline()) {
bq.this.master.abort("Unexpected state trying to OFFLINE; " + state,
bq.new IllegalStateException());
bq.return -1;
bq.  }
bq.  ==
bq.  
bq.  boolean allowCreation = false;
bq.  // If the isReAllocate is true and the current state is PENDING_OPEN
bq.  // or OPENING then update the inmemory state to PENDING_OPEN. This is
bq.  // important because
bq.  // if timeoutmonitor deducts that a region was in OPENING state for a 
long
bq.  // time but by the
bq.  // time timeout monitor tranits the node to OFFLINE the RS would have 
opened
bq.  // the node and the
bq.  // state in znode will be RS_ZK_REGION_OPENED. Inorder to invoke the
bq.  // OpenedRegionHandler
bq.  // we expect the inmemeory state to be PENDING_OPEN or OPENING.
bq.  // For all other cases we can change the inmemory state to OFFLINE.
bq.  if (timeOutMonitorReAllocate
bq.  && (state.getState().equals(RegionState.State.PENDING_OPEN) || 
state
bq.  .getState().equals(RegionState.State.OPENING))) {
bq.s

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098995#comment-13098995
 ] 

jirapos...@reviews.apache.org commented on HBASE-4015:
--



bq.  On 2011-09-07 10:13:44, ramkrishna vasudevan wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java,
 line 415
bq.  > 
bq.  >
bq.  > If you see the implementation of processOpeningState it is same as 
the handling of the previous OPENING state handling by timeout monitor.   Just 
that now we call it immediately instead of timeout monitor acting on it.
bq.  > 
bq.  > "Should we at least check the znode again before going ahead with 
assign? "
bq.  > 
bq.  > Now for the possibility of the znode moving to OPENED state before 
we try to assign to a new RS, our new logic of mulitiple checks that we do 
before a node can be forcefully moved to OFFLINE will handle the scenario that 
you have told.

ok


bq.  On 2011-09-07 10:13:44, ramkrishna vasudevan wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java,
 line 1260
bq.  > 
bq.  >
bq.  > formatting done and the variable name has been changed from reassign 
to hijackAndPreempt.
bq.  > 
bq.  > Stack last time you asked to change the variable name to reassign 
:)).

Sorry if I did.  I thought I was commenting on use of word 'realloc' when it 
seemed like you meant reassign.  I think I spent more time reviewing this time 
than I did last time.


bq.  On 2011-09-07 10:13:44, ramkrishna vasudevan wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java,
 line 2335
bq.  > 
bq.  >
bq.  > Next time will not repeat.  When we refactor a piece of code to a 
new api and apply format on that the formatter does this.
bq.  > Sometimes it gets missed out stack. Sorry

No worries.  For next time.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1668/#review1783
---


On 2011-08-29 12:11:23, ramkrishna vasudevan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1668/
bq.  ---
bq.  
bq.  (Updated 2011-08-29 12:11:23)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Jean-Daniel Cryans, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  HBASE-4015 - updated patch.
bq.  
bq.  
bq.  invokeTimeOutManager(regionState.getRegion(),
bq.  TimeOutOperationType.ASSIGN);
bq.  
bq.  Instead of storing the state when timeout was deducted we process it
bq.  using future task.
bq.  
bq.  
bq.case RS_ZK_REGION_OPENING:
bq.  // TODO: Could check if it was on deadServers.  If it was, then we 
could
bq.  // do what happens in TimeoutMonitor when it sees this condition.
bq.  
bq.  // Just insert region into RIT
bq.  // If this never updates the timeout will trigger new assignment
bq.  if (regionInfo.isMetaRegion() || regionInfo.isRootRegion()) {
bq.regionsInTransition.put(encodedRegionName, new RegionState(
bq.regionInfo, RegionState.State.OPENING, data.getStamp(), data
bq..getOrigin()));
bq.processOpeningState(regionInfo);
bq.break;
bq.  }
bq.  regionsInTransition.put(encodedRegionName, new 
RegionState(regionInfo,
bq.  RegionState.State.OPENING, data.getStamp(), data.getOrigin()));
bq.  break;
bq.  
bq.  This change is for HBASE-4203. META and ROOT table need not wait till 
timeout
bq.  ==
bq.  
bq.  In forceRegionStateToOffline()
bq.  
bq.  } else {
bq.// If invoked from timeout monitor donot force it to OFFLINE. Based 
on the
bq.// state we will decide if to change in-memory state to OFFLINE or 
not.  It will
bq.// be done before setting the znode to OFFLINE state.
bq.if (!timeOutMonitorReAllocate) {
bq.  LOG.debug("Forcing OFFLINE; was=" + state);
bq.  state.update(RegionState.State.OFFLINE);
bq.}
bq.  If the timeout monitor tries to reallcoate the node then dont make
bq.  the inmemory state to OFFLINE.
bq.  But the

[jira] [Created] (HBASE-4342) Update Thrift to 0.7.0

2011-09-07 Thread Moaz Reyad (JIRA)

Update Thrift to 0.7.0
--

 Key: HBASE-4342
 URL: https://issues.apache.org/jira/browse/HBASE-4342
 Project: HBase
  Issue Type: Improvement
Reporter: Moaz Reyad
Priority: Minor


The new version of Thrift is 0.7.0 and it has features and bug fixes that could 
be useful to include in the next release of HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4342) Update Thrift to 0.7.0

2011-09-07 Thread Moaz Reyad (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Moaz Reyad updated HBASE-4342:
--

Attachment: HBASE-4342.patch.zip

> Update Thrift to 0.7.0
> --
>
> Key: HBASE-4342
> URL: https://issues.apache.org/jira/browse/HBASE-4342
> Project: HBase
>  Issue Type: Improvement
>Reporter: Moaz Reyad
>Priority: Minor
> Attachments: HBASE-4342.patch.zip
>
>
> The new version of Thrift is 0.7.0 and it has features and bug fixes that 
> could be useful to include in the next release of HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-2158) Change how high/low global limit works; start taking on writes as soon as we dip below high limit rather than block until low limit as we currently do.


 [ 
https://issues.apache.org/jira/browse/HBASE-2158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-2158.
--

Resolution: Not A Problem

Closed based on Anty and Gaojinchao's analysis above.  Thanks for digging in 
lads.

> Change how high/low global limit works; start taking on writes as soon as we 
> dip below high limit rather than block until low limit as we currently do.
> ---
>
> Key: HBASE-2158
> URL: https://issues.apache.org/jira/browse/HBASE-2158
> Project: HBase
>  Issue Type: Improvement
>Reporter: stack
>
> A Ryan Rawson suggestion.  See HBASE-2149 for more context.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1621) merge tool should work on online cluster, but disabled table

2011-09-07 Thread Daniel Einspanjer (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099015#comment-13099015
 ] 

Daniel Einspanjer commented on HBASE-1621:
--

@J-D

I noticed a potential small problem with the script while reading through it.
At the beginning, it calls HBaseAdmin.balanceSwitch(false), but after the 
initial scan, there are two Trollop:die statements that would prevent the 
balancer from being turned back on.

I think that you need to test the two break conditions in an if block, and then 
turn the balancer back on before you call die.

> merge tool should work on online cluster, but disabled table
> 
>
> Key: HBASE-1621
> URL: https://issues.apache.org/jira/browse/HBASE-1621
> Project: HBase
>  Issue Type: Bug
>Reporter: ryan rawson
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1621-trunk.txt, HBASE-1621-v2.patch, HBASE-1621.patch, 
> hbase-onlinemerge.patch, online_merge.rb
>
>
> taking down the entire cluster to merge 2 regions is a pain, i dont see why 
> the table or regions specifically couldnt be taken offline, then merged then 
> brought back up.
> this might need a new API to the regionservers so they can take direction 
> from not just the master.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099030#comment-13099030
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--



bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java,
 line 666
bq.  > 
bq.  >
bq.  > Hmm.. there is an fb patch that gives Get/Delete, etc. a common  
heritage.  Let me dig it up and commit so you can put this common code there.

I was thinking about this. Then I thought the code duplication is not 
significant enough. But now I realize Put and Delete already have a *lot* in 
common that could be factored out in addition to this.
A common superclass maybe "Mutation" would be nice.

I can make that change too, depending on how good the FB code is. Maybe in a 
separate jira?


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java,
 line 1400
bq.  > 
bq.  >
bq.  > What is happening here Lars?  Are we passing the clusterid from the 
HRegionServer down to the WAL by writing it into the Delete or Put?  Do we have 
to do that?  Can we not pass the clusterid to the WAL directly when 
HRegionServer creates one?
bq.  > 
bq.  > On above, nvm, I think I figure out why you do it this way later.

Yes :)
In a ring the Cluster Id needs to be passed on to the next link link.


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java,
 line 1001
bq.  > 
bq.  >
bq.  > Yeah, can't we pass the cluserid in on the HLog constructor rather 
than per append or do we need to allow for different clusterids coming in via 
append?

Ok.. I'll add a new constructor, then we won't need to call the setter.


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
 line 232
bq.  > 
bq.  >
bq.  > We could have read garbage that happened to be <0?  But yeah, this 
is going to go out with new major version and we'll just pronounce that it will 
not be able to read old WALs (this we've said in the past)
bq.  > 
bq.  > Why not VersionedWritable since it does this version stuff for you?

This was a good idea from Ted.
The first entry in HLogKey is the encodedRegionName which is written by 
Bytes.writeByteArray.
The first byte (or two bytes) encode the length of the array.

So we read that first. If the length is < 0 we know it couldn't have been a 
valid HLogKey, and hence it must be a new one and this is the now the version.


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
 line 235
bq.  > 
bq.  >
bq.  > Why double read of vint?  Is this how you do vints?

This one is now the length encodedRegionName


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java,
 line 374
bq.  > 
bq.  >
bq.  > This is good.
bq.  > 
bq.  > One thought is if these new constants -- the key name and the 
default cluster id -- belong in the general HConstants.  Are they used across 
packages?  If so, then this is the place for them.  Otherwise, I'd say put them 
in replication and then they'll have the replication class prefix when referred 
too which will give them some added context.  No biggie.  I'm just reacting 
against this fat HConstants.

I think DEFAULT_CLUSTER_ID belongs here.
CLUSTER_ID_ATTR should go somewhere else I agree. Best would be that common 
superclass to Get and Put you mention below.


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java,
 line 84
bq.  > 
bq.  >
bq.  > Is this needed?

Nope that was used outside of the constructor :)


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
 line 182
bq.  >

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099040#comment-13099040
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--



bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
 line 232
bq.  > 
bq.  >
bq.  > We could have read garbage that happened to be <0?  But yeah, this 
is going to go out with new major version and we'll just pronounce that it will 
not be able to read old WALs (this we've said in the past)
bq.  > 
bq.  > Why not VersionedWritable since it does this version stuff for you?
bq.  
bq.  Lars Hofhansl wrote:
bq.  This was a good idea from Ted.
bq.  The first entry in HLogKey is the encodedRegionName which is written 
by Bytes.writeByteArray.
bq.  The first byte (or two bytes) encode the length of the array.
bq.  
bq.  So we read that first. If the length is < 0 we know it couldn't have 
been a valid HLogKey, and hence it must be a new one and this is the now the 
version.

That sounds like a good idea to me too.  Does it need a comment so you and Ted 
don' t have to answer same question everytime some else looks at this bit of 
code? 

Good stuff Lars.


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java,
 line 666
bq.  > 
bq.  >
bq.  > Hmm.. there is an fb patch that gives Get/Delete, etc. a common  
heritage.  Let me dig it up and commit so you can put this common code there.
bq.  
bq.  Lars Hofhansl wrote:
bq.  I was thinking about this. Then I thought the code duplication is not 
significant enough. But now I realize Put and Delete already have a *lot* in 
common that could be factored out in addition to this.
bq.  A common superclass maybe "Mutation" would be nice.
bq.  
bq.  I can make that change too, depending on how good the FB code is. 
Maybe in a separate jira?

I notice that Put and Delete implement Operation (The Riley slow-query change 
has gone in already).  Could you put the dup code up in there?


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1780
---


On 2011-09-06 23:16:48, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-06 23:16:48)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1165864 
bq

[jira] [Commented] (HBASE-4342) Update Thrift to 0.7.0


[ 
https://issues.apache.org/jira/browse/HBASE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099046#comment-13099046
 ] 

stack commented on HBASE-4342:
--

Have you tried it out Moaz?  Do you know if it breaks compatibility with 0.6?  
Thanks.

> Update Thrift to 0.7.0
> --
>
> Key: HBASE-4342
> URL: https://issues.apache.org/jira/browse/HBASE-4342
> Project: HBase
>  Issue Type: Improvement
>Reporter: Moaz Reyad
>Priority: Minor
> Attachments: HBASE-4342.patch.zip
>
>
> The new version of Thrift is 0.7.0 and it has features and bug fixes that 
> could be useful to include in the next release of HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099048#comment-13099048
 ] 

jirapos...@reviews.apache.org commented on HBASE-4015:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1668/
---

(Updated 2011-09-07 15:52:26.864199)


Review request for Ted Yu, Michael Stack, Jean-Daniel Cryans, and Jonathan Gray.


Changes
---

Updated the patch as per the review comments.
Changed hijackAndPreempt to hijack.


Summary
---

HBASE-4015 - updated patch.


invokeTimeOutManager(regionState.getRegion(),
TimeOutOperationType.ASSIGN);

Instead of storing the state when timeout was deducted we process it
using future task.


  case RS_ZK_REGION_OPENING:
// TODO: Could check if it was on deadServers.  If it was, then we could
// do what happens in TimeoutMonitor when it sees this condition.

// Just insert region into RIT
// If this never updates the timeout will trigger new assignment
if (regionInfo.isMetaRegion() || regionInfo.isRootRegion()) {
  regionsInTransition.put(encodedRegionName, new RegionState(
  regionInfo, RegionState.State.OPENING, data.getStamp(), data
  .getOrigin()));
  processOpeningState(regionInfo);
  break;
}
regionsInTransition.put(encodedRegionName, new RegionState(regionInfo,
RegionState.State.OPENING, data.getStamp(), data.getOrigin()));
break;

This change is for HBASE-4203. META and ROOT table need not wait till timeout
==

In forceRegionStateToOffline()

} else {
  // If invoked from timeout monitor donot force it to OFFLINE. Based on the
  // state we will decide if to change in-memory state to OFFLINE or not.  
It will
  // be done before setting the znode to OFFLINE state.
  if (!timeOutMonitorReAllocate) {
LOG.debug("Forcing OFFLINE; was=" + state);
state.update(RegionState.State.OFFLINE);
  }
If the timeout monitor tries to reallcoate the node then dont make
the inmemory state to OFFLINE.
But the noraml assign flow doesnot expect the inmemory state to OFFLINE.
Hence the above change.  This is continued with the check in 
assign()
int setOfflineInZooKeeper(final RegionState state,
  boolean timeOutMonitorReAllocate) {
// If invoked from timeoutmonitor the current state in memory need not be
// OFFLINE.  
if (!timeOutMonitorReAllocate && !state.isClosed() && !state.isOffline()) {
  this.master.abort("Unexpected state trying to OFFLINE; " + state,
  new IllegalStateException());
  return -1;
}
==

boolean allowCreation = false;
// If the isReAllocate is true and the current state is PENDING_OPEN
// or OPENING then update the inmemory state to PENDING_OPEN. This is
// important because
// if timeoutmonitor deducts that a region was in OPENING state for a long
// time but by the
// time timeout monitor tranits the node to OFFLINE the RS would have opened
// the node and the
// state in znode will be RS_ZK_REGION_OPENED. Inorder to invoke the
// OpenedRegionHandler
// we expect the inmemeory state to be PENDING_OPEN or OPENING.
// For all other cases we can change the inmemory state to OFFLINE.
if (timeOutMonitorReAllocate
&& (state.getState().equals(RegionState.State.PENDING_OPEN) || state
.getState().equals(RegionState.State.OPENING))) {
  state.update(RegionState.State.PENDING_OPEN);
  allowCreation = false;
} else {
  state.update(RegionState.State.OFFLINE);
  allowCreation = true;
}
This change is quite tricky.  
In normal assign flow the unassigned node for the region will not be present
Hence we need to allow the creation of the node newly.
But in timeout monitor case we will have the node present in some state hence 
we decide whether to create node newly or not inside 
ZKAssign.createOrForceNodeOffline

The above code also updates the inmemory state of OFFLINE or PENDING_OPEN
which was not update in the previous forceRegionStateToOffline() call.
==
In ZKAssign.java()
if (version == -1) {
  // If timeoutmonitor deducts a node to be in OPENING state but before it
  // could
  // transit to OFFLINE state if RS had opened the region then the Master
  // deletes the
  // assigned region znode. In that case the znode will not exist. So we
  // should not

[jira] [Resolved] (HBASE-4309) slow query log metrics spewing warnings


 [ 
https://issues.apache.org/jira/browse/HBASE-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4309.
--

  Resolution: Fixed
Hadoop Flags: [Reviewed]

Applied the more adventurous patch.  Will revert to the alternative if gives us 
grief.  Thanks for patch Riley.

> slow query log metrics spewing warnings
> ---
>
> Key: HBASE-4309
> URL: https://issues.apache.org/jira/browse/HBASE-4309
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Riley Patterson
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-4309-alternative.patch, HBASE-4309.patch
>
>
> Lots of these in the logs in trunk:
> 2011-08-30 22:54:36,100 WARN org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Got 
> inc() request for method that doesnt exist: get.slowResponse.
> 2011-08-30 22:54:36,100 WARN org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Got 
> inc() request for method that doesnt exist: get.aboveOneSec.
> ...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1621) merge tool should work on online cluster, but disabled table

2011-09-07 Thread Jean-Daniel Cryans (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099089#comment-13099089
 ] 

Jean-Daniel Cryans commented on HBASE-1621:
---

bq. Will the java api be supported?

I'm not sure I understand the question, but the reason a script was built in 
the first place (instead of java code) is to be able to include it in a 0.90 
release without breaking compatibility, while offering development flexibility. 
It could eventually be included in hbck.

> merge tool should work on online cluster, but disabled table
> 
>
> Key: HBASE-1621
> URL: https://issues.apache.org/jira/browse/HBASE-1621
> Project: HBase
>  Issue Type: Bug
>Reporter: ryan rawson
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1621-trunk.txt, HBASE-1621-v2.patch, HBASE-1621.patch, 
> hbase-onlinemerge.patch, online_merge.rb
>
>
> taking down the entire cluster to merge 2 regions is a pain, i dont see why 
> the table or regions specifically couldnt be taken offline, then merged then 
> brought back up.
> this might need a new API to the regionservers so they can take direction 
> from not just the master.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-1621) merge tool should work on online cluster, but disabled table

2011-09-07 Thread Jean-Daniel Cryans (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099091#comment-13099091
 ] 

Jean-Daniel Cryans commented on HBASE-1621:
---

bq. I think that you need to test the two break conditions in an if block, and 
then turn the balancer back on before you call die.

True, also it needs to handle the case where the balancer was already switched 
off and shouldn't be turned back on at the end.

> merge tool should work on online cluster, but disabled table
> 
>
> Key: HBASE-1621
> URL: https://issues.apache.org/jira/browse/HBASE-1621
> Project: HBase
>  Issue Type: Bug
>Reporter: ryan rawson
>Assignee: stack
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 1621-trunk.txt, HBASE-1621-v2.patch, HBASE-1621.patch, 
> hbase-onlinemerge.patch, online_merge.rb
>
>
> taking down the entire cluster to merge 2 regions is a pain, i dont see why 
> the table or regions specifically couldnt be taken offline, then merged then 
> brought back up.
> this might need a new API to the regionservers so they can take direction 
> from not just the master.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099129#comment-13099129
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--

bq.  On 2011-09-07 03:59:28, Ted Yu wrote:
bq.  > Slight optimization outside the scope of this JIRA:
bq.  > I think the index for the loop in 
ReplicationSource.removeNonReplicableEdits()
bq.  > for (int i = 0; i < edit.size(); i++) {
bq.  > should start from end of kvs and decrement.

Because of the remove(i) from the ArrayList... Good point.
I'll fine another jira, unless you want to.

- Lars

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1777
---

On 2011-09-06 23:16:48, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-06 23:16:48)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no 
cycles > 2, yet), also passes.
bq.  Manually tested a cycle between 3 clusters.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099130#comment-13099130
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--



bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java,
 line 84
bq.  > 
bq.  >
bq.  > Is this needed?
bq.  
bq.  Lars Hofhansl wrote:
bq.  Nope that was used outside of the constructor :)

I meant: never used outside of the constructor


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java,
 line 666
bq.  > 
bq.  >
bq.  > Hmm.. there is an fb patch that gives Get/Delete, etc. a common  
heritage.  Let me dig it up and commit so you can put this common code there.
bq.  
bq.  Lars Hofhansl wrote:
bq.  I was thinking about this. Then I thought the code duplication is not 
significant enough. But now I realize Put and Delete already have a *lot* in 
common that could be factored out in addition to this.
bq.  A common superclass maybe "Mutation" would be nice.
bq.  
bq.  I can make that change too, depending on how good the FB code is. 
Maybe in a separate jira?
bq.  
bq.  Michael Stack wrote:
bq.  I notice that Put and Delete implement Operation (The Riley slow-query 
change has gone in already).  Could you put the dup code up in there?

He, I was thinking about that too. :)

Operation is also shared by Get, Scan, and MultiPut, though.
Get and Scan do have attributes and MultiPut does not.

The cluster id part really only makes sense for Put and Delete.

It almost seems we want two abstraction:
1. Attributes. These are common to Put, Delete, Get, and Scan. (Now that I look 
at it, I can't believe how often the same code is repeated for those :) ). 
Maybe have a "WithAttributes" class for the lack of a better name?
2. Cluster id, common to Put and Delete.

Put and Delete also have other stuff in common (FamilyMap, writeToWal, the row, 
the rowLock, toMap is very similar, etc, etc.)

Is there a jira for the FB change?

So for now... Get this checked in the way it is. Then have additional jira that 
refactor Put/Delete and maybe Get/Scan/Multiput?


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1780
---


On 2011-09-06 23:16:48, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-06 23:16:48)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1165864 
bq.
htt

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099135#comment-13099135
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--



bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
 line 232
bq.  > 
bq.  >
bq.  > We could have read garbage that happened to be <0?  But yeah, this 
is going to go out with new major version and we'll just pronounce that it will 
not be able to read old WALs (this we've said in the past)
bq.  > 
bq.  > Why not VersionedWritable since it does this version stuff for you?
bq.  
bq.  Lars Hofhansl wrote:
bq.  This was a good idea from Ted.
bq.  The first entry in HLogKey is the encodedRegionName which is written 
by Bytes.writeByteArray.
bq.  The first byte (or two bytes) encode the length of the array.
bq.  
bq.  So we read that first. If the length is < 0 we know it couldn't have 
been a valid HLogKey, and hence it must be a new one and this is the now the 
version.
bq.  
bq.  Michael Stack wrote:
bq.  That sounds like a good idea to me too.  Does it need a comment so you 
and Ted don' t have to answer same question everytime some else looks at this 
bit of code? 
bq.  
bq.  Good stuff Lars.

Yep... I'll add a comment.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1780
---


On 2011-09-06 23:16:48, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-06 23:16:48)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1165864 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no 
cycles > 2, yet), also passes.
bq.  Manually tested a cycle between 3 clusters.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.



> Support cyclic

[jira] [Created] (HBASE-4343) Get the TestAcidGuarantee unit test to fail consistently

Get the TestAcidGuarantee unit test to fail consistently


 Key: HBASE-4343
 URL: https://issues.apache.org/jira/browse/HBASE-4343
 Project: HBase
  Issue Type: Sub-task
Reporter: Amitanand Aiyer
Priority: Minor


We know that TestAcidGuarantee is broken in the current trunk. However, the 
unit-test passes more often than not.

In order to test out the solution we need to get it to fail consistently. This 
patch may not be committed/turned in. But,
required to test/accept the fix to 2856.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4344) Persist memstoreTS to disk

Persist memstoreTS to disk
--

 Key: HBASE-4344
 URL: https://issues.apache.org/jira/browse/HBASE-4344
 Project: HBase
  Issue Type: Sub-task
Reporter: Amitanand Aiyer


Atomicity can be achieved in two ways -- (i) by using  a multiversion 
concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
complete, until the "old" reads complete.

Currently, Memstore uses something along the lines of MVCC (called RWCC for 
read-write-consistency-control). But, this mechanism is not incorporated for 
the key-values written to the disk, as they do not include the memstore TS.

Let us make the two approaches be similar, by persisting the memstoreTS along 
with the key-value when it is written to the disk.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4345) Ensure that Scanners that read from the storefiles respect MVCC

Ensure that Scanners that read from the storefiles respect MVCC
---

 Key: HBASE-4345
 URL: https://issues.apache.org/jira/browse/HBASE-4345
 Project: HBase
  Issue Type: Sub-task
Reporter: Amitanand Aiyer


Currently, the key-values written to the disk do not include the MVCC (RWCC) 
version information. Once we add that
information, and make it persistent to disk; let us make the scanners respect 
the MVCC mechanism by ignoring 
"newer" writes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4309) slow query log metrics spewing warnings


[ 
https://issues.apache.org/jira/browse/HBASE-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099163#comment-13099163
 ] 

Hudson commented on HBASE-4309:
---

Integrated in HBase-TRUNK #2182 (See 
[https://builds.apache.org/job/HBase-TRUNK/2182/])
HBASE-4309 slow query log metrics spewing warnings

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java


> slow query log metrics spewing warnings
> ---
>
> Key: HBASE-4309
> URL: https://issues.apache.org/jira/browse/HBASE-4309
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Riley Patterson
>Priority: Critical
> Fix For: 0.92.0
>
> Attachments: HBASE-4309-alternative.patch, HBASE-4309.patch
>
>
> Lots of these in the logs in trunk:
> 2011-08-30 22:54:36,100 WARN org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Got 
> inc() request for method that doesnt exist: get.slowResponse.
> 2011-08-30 22:54:36,100 WARN org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Got 
> inc() request for method that doesnt exist: get.aboveOneSec.
> ...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4346) Optimise the storage that we use for storing MVCC information.

Optimise the storage that we use for storing MVCC information.
--

 Key: HBASE-4346
 URL: https://issues.apache.org/jira/browse/HBASE-4346
 Project: HBase
  Issue Type: Sub-task
 Environment: Writing the memstoreTS information is critical to 
ensuring Atomic semantics. However, it adds a storage overhead to the disk 
files. 

This information is useful only when there is a "concurrent" read operation 
that needs to ignore a particular key-value. 

Once it is known that a key-value is going to be included in all current and 
future read operations, MVCC memstoreTS information is
of no use anymore. 

We can optimize for storage on disk, by getting rid of these values during 
later compactions. 
Reporter: Amitanand Aiyer
Priority: Minor




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4343) Get the TestAcidGuarantee unit test to fail consistently


 [ 
https://issues.apache.org/jira/browse/HBASE-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4343:
---

Fix Version/s: (was: 0.94.0)
   0.89.20100924
 Assignee: Amitanand Aiyer
   Status: Patch Available  (was: Open)

> Get the TestAcidGuarantee unit test to fail consistently
> 
>
> Key: HBASE-4343
> URL: https://issues.apache.org/jira/browse/HBASE-4343
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
>Priority: Minor
> Fix For: 0.89.20100924
>
>
> We know that TestAcidGuarantee is broken in the current trunk. However, the 
> unit-test passes more often than not.
> In order to test out the solution we need to get it to fail consistently. 
> This patch may not be committed/turned in. But,
> required to test/accept the fix to 2856.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099177#comment-13099177
]

jirapos...@reviews.apache.org commented on HBASE-2195:
--

bq. On 2011-09-07 06:01:09, Michael Stack wrote:
bq. >
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
line 182
bq. >
bq. >
bq. > Yeah, if the clusterids are different, i'd think the edits are
different? That looks like oversight in original code.
bq.
bq. Lars Hofhansl wrote:
bq. I was not sure here, what does it means for two log keys to be
different. I just noticed that he cluster id was missing.
bq. We seems to use this specifically for sorting, maybe HLogKey even with
different cluster id need to order the same?

Should HLogKey should sort by sequenceid?

bq. On 2011-09-07 06:01:09, Michael Stack wrote:
bq. >
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java,
line 1001
bq. >
bq. >
bq. > Yeah, can't we pass the cluserid in on the HLog constructor rather
than per append or do we need to allow for different clusterids coming in via
append?
bq.
bq. Lars Hofhansl wrote:
bq. Ok.. I'll add a new constructor, then we won't need to call the setter.

Well, sounds like you don't need the constructor. I was only suggesting it if
clusterid was same for all edits that come into the WAL but later I saw that
the edits can come from another cluster. If you added clusterid to
constructor, what would you do? Add the passed clusterid to the WAL IFF the
passed Delete or Put were absent a clusterid?

bq. On 2011-09-07 06:01:09, Michael Stack wrote:
bq. >
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java,
line 666
bq. >
bq. >
bq. > Hmm.. there is an fb patch that gives Get/Delete, etc. a common
heritage. Let me dig it up and commit so you can put this common code there.
bq.
bq. Lars Hofhansl wrote:
bq. I was thinking about this. Then I thought the code duplication is not
significant enough. But now I realize Put and Delete already have a *lot* in
common that could be factored out in addition to this.
bq. A common superclass maybe "Mutation" would be nice.
bq.
bq. I can make that change too, depending on how good the FB code is.
Maybe in a separate jira?
bq.
bq. Michael Stack wrote:
bq. I notice that Put and Delete implement Operation (The Riley slow-query
change has gone in already). Could you put the dup code up in there?
bq.
bq. Lars Hofhansl wrote:
bq. He, I was thinking about that too. :)
bq.
bq. Operation is also shared by Get, Scan, and MultiPut, though.
bq. Get and Scan do have attributes and MultiPut does not.
bq.
bq. The cluster id part really only makes sense for Put and Delete.
bq.
bq. It almost seems we want two abstraction:
bq. 1. Attributes. These are common to Put, Delete, Get, and Scan. (Now
that I look at it, I can't believe how often the same code is repeated for
those :) ). Maybe have a "WithAttributes" class for the lack of a better name?
bq. 2. Cluster id, common to Put and Delete.
bq.
bq. Put and Delete also have other stuff in common (FamilyMap, writeToWal,
the row, the rowLock, toMap is very similar, etc, etc.)
bq.
bq. Is there a jira for the FB change?
bq.
bq. So for now... Get this checked in the way it is. Then have additional
jira that refactor Put/Delete and maybe Get/Scan/Multiput?

On 1., I'd say Attributes rather than WithAttributes as the added class name (I
think adding it would be good but can be in a different issue I'd say).

On 2., you talking about a ClusterId or should it be a Replication Interface?
If an Interface, you can mix it in easily. But then you don't have a super
class to put the commonality into. Back to square one (smile). Or some
utility that took the Interface and it did the set and get of clusterid.

But if Put and Delete have other commonality, that'd be cool figuring a
superclass they could share (Result too?)

But now we are into a different JIRA? Don't you think? Yes, lets do the
refactor elsewhere. The duplicated code don't look so bad after the above
exercise teasing stuff out.

The JIRA that added Operation is:

HBASE-4117 Slow Query Log and Client Operation Fingerprints
(Riley Patterson)

- Michael

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1780

[jira] [Updated] (HBASE-4343) Get the TestAcidGuarantee unit test to fail consistently


 [ 
https://issues.apache.org/jira/browse/HBASE-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4343:
---

Attachment: patch-1

This is a diff to get things to TestAcidGuarantee to fail consistently. We will 
not submit this diff as is -- esp, the changes to HRegion.java

> Get the TestAcidGuarantee unit test to fail consistently
> 
>
> Key: HBASE-4343
> URL: https://issues.apache.org/jira/browse/HBASE-4343
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
>Priority: Minor
> Fix For: 0.89.20100924
>
> Attachments: patch-1
>
>
> We know that TestAcidGuarantee is broken in the current trunk. However, the 
> unit-test passes more often than not.
> In order to test out the solution we need to get it to fail consistently. 
> This patch may not be committed/turned in. But,
> required to test/accept the fix to 2856.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk


 [ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4344:
---

Fix Version/s: (was: 0.94.0)
   0.89.20100924
 Assignee: Amitanand Aiyer
   Status: Patch Available  (was: Open)

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk


 [ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4344:
---

Attachment: patch-2

There are three parts to the changes made here:
 
The first one is in the write path, while writing the HFile (V2). We include 
the memstoreTS along with the other information already written during the 
KeyValue.

We maintain and append to the file info the maximum memstoreTS found among all 
the key-values written to the disk. The presence of this information in the 
file info represents the fact that the Key-values written to the HFile is in 
the new format that includes the memstoreTS.

Part 2: The reader is modified accordingly to see if the file info suggests the 
presence of the memstoreTS or not. If it does, then we also read the memstoreTS 
while preparing the KeyValues in the reader.

Part 3: On Region open/initialize, we need to ensure that the RWCC is 
initialized with a value bigger than the max of any value found in the 
storeFiles. this is necessary to ensure that the writes in the storefiles are 
seen by future reads.

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4345) Ensure that Scanners that read from the storefiles respect MVCC


 [ 
https://issues.apache.org/jira/browse/HBASE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4345:
---

Attachment: patch-3

To enforce the read-write consistency mechanism, we need to ignore all 
key-values that have a memstoreTS greater than the read point for the get/scan 
operation.

Filters provide an excellent mechanism to ignore values based on any desired 
condition. We implement the RWCC mechanism for the key-values read from the 
disk by including a filter for the scan object that ignores "newer" reads.

> Ensure that Scanners that read from the storefiles respect MVCC
> ---
>
> Key: HBASE-4345
> URL: https://issues.apache.org/jira/browse/HBASE-4345
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: patch-3
>
>
> Currently, the key-values written to the disk do not include the MVCC (RWCC) 
> version information. Once we add that
> information, and make it persistent to disk; let us make the scanners respect 
> the MVCC mechanism by ignoring 
> "newer" writes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4345) Ensure that Scanners that read from the storefiles respect MVCC


 [ 
https://issues.apache.org/jira/browse/HBASE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitanand Aiyer updated HBASE-4345:
---

Fix Version/s: (was: 0.94.0)
   0.89.20100924
 Assignee: Amitanand Aiyer
   Status: Patch Available  (was: Open)

> Ensure that Scanners that read from the storefiles respect MVCC
> ---
>
> Key: HBASE-4345
> URL: https://issues.apache.org/jira/browse/HBASE-4345
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: patch-3
>
>
> Currently, the key-values written to the disk do not include the MVCC (RWCC) 
> version information. Once we add that
> information, and make it persistent to disk; let us make the scanners respect 
> the MVCC mechanism by ignoring 
> "newer" writes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099199#comment-13099199
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/
---

(Updated 2011-09-07 18:49:31.631269)


Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.


Changes
---

Added comments.
Made originating-cluster-only operations (checkAndMutate, ICV) explicit.


Summary
---

For Master <-> Master replication (as well as cycles > 2) a replication sink 
needs to keep track who is was the originator of a change and
(1) do not replicate this data back to the originator and
(2) pass that information on to the next link in a cycle > 2

In order to do that, HlogKeys read from the WAL are tagged with the Cluster 
UUID at the ReplicationSource before the logs are shipped to the Sink. The sink 
writes the Cluster UUID into the WAL log (in HlogKey, so only once per edit). 
If the sink is also source for yet another sink the ClusterUUID is retained in 
the HLogKeys read from the local WAL and passed on to the sink.


This addresses bug HBASE-2195.
https://issues.apache.org/jira/browse/HBASE-2195


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 

Diff: https://reviews.apache.org/r/1730/diff


Testing
---

All tests pass.
New test: TestMasterReplication (which tests Master <-> Master, but no cycles > 
2, yet), also passes.
Manually tested a cycle between 3 clusters.


Thanks,

Lars



> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099202#comment-13099202
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1797
---



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java


fix sentence



http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java


maybe have a more helpful message



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


seems to me there's an opportunity for refactoring with TestReplication



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


fix those brackets in both methods



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


that's clever


- Jean-Daniel


On 2011-09-07 18:49:31, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 18:49:31)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSo

[jira] [Commented] (HBASE-4302) don't run SNAPPY compression test


[ 
https://issues.apache.org/jira/browse/HBASE-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099206#comment-13099206
 ] 

Todd Lipcon commented on HBASE-4302:


+1

> don't run SNAPPY compression test
> -
>
> Key: HBASE-4302
> URL: https://issues.apache.org/jira/browse/HBASE-4302
> Project: HBase
>  Issue Type: Test
>  Components: test
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HBASE-4302v1.patch, HBASE-4302v2.patch
>
>
> The presence of the Snappy CODEC it does not mean that the Snappy 
> JNI-bindings and native library are available.
> Because of this it is not possible to assert SNAPPY compression

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4014) Coprocessors: Flag the presence of coprocessors in logged exceptions

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099208#comment-13099208
 ] 

jirapos...@reviews.apache.org commented on HBASE-4014:
--

bq.  On 2011-08-02 05:38:49, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java, 
line 578
bq.  > 
bq.  >
bq.  > I saw in previous comments that this is needed because 
SortedCopyOnWriteSet doesn't implement toString().  So why not make 
SortedCopyOnWriteSet implement toString()?  Seems cleaner to me and more 
generic/reusable.
bq.  > 
bq.  > Wherever this is implemented, use a StringBuilder to create the 
string to return, not repeated string concatenation.

I ended up removing the toString() implementation from SortedCopyOnWriteSet 
since I am simply storing the coprocessor names (not the whole 
CoprocessorEnvironment). Thank you for the StringBuilder suggestion, though.  

bq.  On 2011-08-02 05:38:49, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java, 
line 594
bq.  > 
bq.  >
bq.  > Awfully long name, maybe just abortServer()?

Shortened to abortServer(); thanks for this.

bq.  On 2011-08-02 05:38:49, Gary Helmling wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java, 
line 80
bq.  > 
bq.  >
bq.  > This and the following method names are awfully long.  It's just 
personal preference, but I like to keep things shorter.

Shortened to abortServer() also; thanks.

bq.  On 2011-08-02 05:38:49, Gary Helmling wrote:
bq.  > 
src/test/java/org/apache/hadoop/hbase/coprocessor/BuggyRegionObserver.java, 
line 27
bq.  > 
bq.  >
bq.  > Minor nit, but why have this as a separate top-level class?  Seems 
like it could just be an inner class in TestRegionServerCoprocessorException 
same way that the BuggyMasterObserver is an inner class in 
TestMasterCoprocessorException.

Moved to interior of TestRegionServerCoprocessorException.java, thanks.

- Eugene

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/969/#review1261
---

On 2011-09-06 19:08:59, Eugene Koontz wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/969/
bq.  ---
bq.  
bq.  (Updated 2011-09-06 19:08:59)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Mingjie Lai.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  https://issues.apache.org/jira/browse/HBASE-4014 Coprocessors: Flag the 
presence of coprocessors in logged exceptions
bq.  
bq.  The general gist here is to wrap each of 
{Master,RegionServer}CoprocessorHost's coprocessor call inside a 
bq.  
bq.  "try { ... } catch (Throwable e) { handleCoprocessorThrowable(e) }"
bq.  
bq.  block. 
bq.  
bq.  handleCoprocessorThrowable() is responsible for either passing 'e' along 
to the client (if 'e' is an IOException) or, otherwise, aborting the service 
(Regionserver or Master).
bq.  
bq.  The abort message contains a list of the loaded coprocessors for crash 
analysis.
bq.  
bq.  
bq.  This addresses bug HBASE-4014.
bq.  https://issues.apache.org/jira/browse/HBASE-4014
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 
4e492e1 
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 3f60653 
bq.src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 
aa930f5 
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
8ff6e62 
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java 
5796413 
bq.src/main/resources/hbase-default.xml 2c8f44b 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorException.java
 PRE-CREATION 
bq.
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorException.java
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/969/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  patch includes two tests:
bq.  
bq.  TestMasterCoprocessorException.java
bq.  TestRegionServerCoprocessorException.java
bq.  
bq.  both tests pass in my build environment.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Eugene
bq.  
bq.

> Coprocessors: Flag the presence of coprocessors in logged exceptions
> -

[jira] [Updated] (HBASE-4302) Only run Snappy compression tests if Snappy is available


 [ 
https://issues.apache.org/jira/browse/HBASE-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4302:
---

Affects Version/s: 0.92.0
Fix Version/s: 0.92.0
 Hadoop Flags: [Reviewed]
  Summary: Only run Snappy compression tests if Snappy is available 
 (was: don't run SNAPPY compression test)

> Only run Snappy compression tests if Snappy is available
> 
>
> Key: HBASE-4302
> URL: https://issues.apache.org/jira/browse/HBASE-4302
> Project: HBase
>  Issue Type: Test
>  Components: test
>Affects Versions: 0.92.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Fix For: 0.92.0
>
> Attachments: HBASE-4302v1.patch, HBASE-4302v2.patch
>
>
> The presence of the Snappy CODEC it does not mean that the Snappy 
> JNI-bindings and native library are available.
> Because of this it is not possible to assert SNAPPY compression

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk


[ 
https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099214#comment-13099214
 ] 

Amitanand Aiyer commented on HBASE-2856:


I have created a few sub-tasks and submitted a diff for the same. Could y'll 
have a look at it, and let me know what you think? 

Thanks,
-Amit

> TestAcidGuarantee broken on trunk 
> --
>
> Key: HBASE-2856
> URL: https://issues.apache.org/jira/browse/HBASE-2856
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.89.20100621
>Reporter: ryan rawson
>Assignee: Amitanand Aiyer
>Priority: Blocker
> Fix For: 0.94.0
>
> Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 
> acid.txt
>
>
> TestAcidGuarantee has a test whereby it attempts to read a number of columns 
> from a row, and every so often the first column of N is different, when it 
> should be the same.  This is a bug deep inside the scanner whereby the first 
> peek() of a row is done at time T then the rest of the read is done at T+1 
> after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' 
> data becomes committed and flushed to disk.
> One possible solution is to introduce the memstoreTS (or similarly equivalent 
> value) to the HFile thus allowing us to preserve read consistency past 
> flushes.  Another solution involves fixing the scanners so that peek() is not 
> destructive (and thus might return different things at different times alas).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4302) Only run Snappy compression tests if Snappy is available


 [ 
https://issues.apache.org/jira/browse/HBASE-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4302:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk, thanks Alejandro

> Only run Snappy compression tests if Snappy is available
> 
>
> Key: HBASE-4302
> URL: https://issues.apache.org/jira/browse/HBASE-4302
> Project: HBase
>  Issue Type: Test
>  Components: test
>Affects Versions: 0.92.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Fix For: 0.92.0
>
> Attachments: HBASE-4302v1.patch, HBASE-4302v2.patch
>
>
> The presence of the Snappy CODEC it does not mean that the Snappy 
> JNI-bindings and native library are available.
> Because of this it is not possible to assert SNAPPY compression

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4346) Optimise the storage that we use for storing MVCC information.


[ 
https://issues.apache.org/jira/browse/HBASE-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099215#comment-13099215
 ] 

Amitanand Aiyer commented on HBASE-4346:


This task is a nice-to-have. Not really required for addressing the 
"correctness" issue with 2856.

Will work on this as the next step. Putting a place holder here.

> Optimise the storage that we use for storing MVCC information.
> --
>
> Key: HBASE-4346
> URL: https://issues.apache.org/jira/browse/HBASE-4346
> Project: HBase
>  Issue Type: Sub-task
> Environment: Writing the memstoreTS information is critical to 
> ensuring Atomic semantics. However, it adds a storage overhead to the disk 
> files. 
> This information is useful only when there is a "concurrent" read operation 
> that needs to ignore a particular key-value. 
> Once it is known that a key-value is going to be included in all current and 
> future read operations, MVCC memstoreTS information is
> of no use anymore. 
> We can optimize for storage on disk, by getting rid of these values during 
> later compactions. 
>Reporter: Amitanand Aiyer
>Priority: Minor
> Fix For: 0.94.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099224#comment-13099224
 ] 

jirapos...@reviews.apache.org commented on HBASE-4015:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1668/#review1798
---

Ship it!


I think this patch at a minimum narrows the races significantly, enough to 
warrant application.  Writing a test is a little tough.  If we could standup an 
AssignmentManager outside of its hosting Master context, that'd help.  Then we 
could set the AM and AM.TimeoutManager in competition over znode states.  This 
is something to do but I'd say in another issue.

I'd say we should let this issue hang out there another 24 hours in case anyone 
else has an opinion before we go ahead commit.

- Michael


On 2011-09-07 15:52:26, ramkrishna vasudevan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1668/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 15:52:26)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Jean-Daniel Cryans, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  HBASE-4015 - updated patch.
bq.  
bq.  
bq.  invokeTimeOutManager(regionState.getRegion(),
bq.  TimeOutOperationType.ASSIGN);
bq.  
bq.  Instead of storing the state when timeout was deducted we process it
bq.  using future task.
bq.  
bq.  
bq.case RS_ZK_REGION_OPENING:
bq.  // TODO: Could check if it was on deadServers.  If it was, then we 
could
bq.  // do what happens in TimeoutMonitor when it sees this condition.
bq.  
bq.  // Just insert region into RIT
bq.  // If this never updates the timeout will trigger new assignment
bq.  if (regionInfo.isMetaRegion() || regionInfo.isRootRegion()) {
bq.regionsInTransition.put(encodedRegionName, new RegionState(
bq.regionInfo, RegionState.State.OPENING, data.getStamp(), data
bq..getOrigin()));
bq.processOpeningState(regionInfo);
bq.break;
bq.  }
bq.  regionsInTransition.put(encodedRegionName, new 
RegionState(regionInfo,
bq.  RegionState.State.OPENING, data.getStamp(), data.getOrigin()));
bq.  break;
bq.  
bq.  This change is for HBASE-4203. META and ROOT table need not wait till 
timeout
bq.  ==
bq.  
bq.  In forceRegionStateToOffline()
bq.  
bq.  } else {
bq.// If invoked from timeout monitor donot force it to OFFLINE. Based 
on the
bq.// state we will decide if to change in-memory state to OFFLINE or 
not.  It will
bq.// be done before setting the znode to OFFLINE state.
bq.if (!timeOutMonitorReAllocate) {
bq.  LOG.debug("Forcing OFFLINE; was=" + state);
bq.  state.update(RegionState.State.OFFLINE);
bq.}
bq.  If the timeout monitor tries to reallcoate the node then dont make
bq.  the inmemory state to OFFLINE.
bq.  But the noraml assign flow doesnot expect the inmemory state to OFFLINE.
bq.  Hence the above change.  This is continued with the check in 
bq.  assign()
bq.  int setOfflineInZooKeeper(final RegionState state,
bq.boolean timeOutMonitorReAllocate) {
bq.  // If invoked from timeoutmonitor the current state in memory need not 
be
bq.  // OFFLINE.  
bq.  if (!timeOutMonitorReAllocate && !state.isClosed() && 
!state.isOffline()) {
bq.this.master.abort("Unexpected state trying to OFFLINE; " + state,
bq.new IllegalStateException());
bq.return -1;
bq.  }
bq.  ==
bq.  
bq.  boolean allowCreation = false;
bq.  // If the isReAllocate is true and the current state is PENDING_OPEN
bq.  // or OPENING then update the inmemory state to PENDING_OPEN. This is
bq.  // important because
bq.  // if timeoutmonitor deducts that a region was in OPENING state for a 
long
bq.  // time but by the
bq.  // time timeout monitor tranits the node to OFFLINE the RS would have 
opened
bq.  // the node and the
bq.  // state in znode will be RS_ZK_REGION_OPENED. Inorder to invoke the
bq.  // OpenedRegionHandler
bq.  // we expect the inmemeory state to be PENDING_OPEN or OPENING.
bq.  // For all other cases we can change the inmemory state to OFFLINE.
bq.  if (timeOutMonitorReAllocate
bq.  && (state.getState().equals(RegionState.State.PENDING_OPEN) || 
state

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099230#comment-13099230
 ] 

jirapos...@reviews.apache.org commented on HBASE-4015:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1668/#review1800
---


This patch is hard to review as it's very shotgun surgery-y, but that's 
probably because of the nature of the required change.

I'd like to know more about the testing that was done. How many regions on how 
many nodes?

- Jean-Daniel


On 2011-09-07 15:52:26, ramkrishna vasudevan wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1668/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 15:52:26)
bq.  
bq.  
bq.  Review request for Ted Yu, Michael Stack, Jean-Daniel Cryans, and Jonathan 
Gray.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  HBASE-4015 - updated patch.
bq.  
bq.  
bq.  invokeTimeOutManager(regionState.getRegion(),
bq.  TimeOutOperationType.ASSIGN);
bq.  
bq.  Instead of storing the state when timeout was deducted we process it
bq.  using future task.
bq.  
bq.  
bq.case RS_ZK_REGION_OPENING:
bq.  // TODO: Could check if it was on deadServers.  If it was, then we 
could
bq.  // do what happens in TimeoutMonitor when it sees this condition.
bq.  
bq.  // Just insert region into RIT
bq.  // If this never updates the timeout will trigger new assignment
bq.  if (regionInfo.isMetaRegion() || regionInfo.isRootRegion()) {
bq.regionsInTransition.put(encodedRegionName, new RegionState(
bq.regionInfo, RegionState.State.OPENING, data.getStamp(), data
bq..getOrigin()));
bq.processOpeningState(regionInfo);
bq.break;
bq.  }
bq.  regionsInTransition.put(encodedRegionName, new 
RegionState(regionInfo,
bq.  RegionState.State.OPENING, data.getStamp(), data.getOrigin()));
bq.  break;
bq.  
bq.  This change is for HBASE-4203. META and ROOT table need not wait till 
timeout
bq.  ==
bq.  
bq.  In forceRegionStateToOffline()
bq.  
bq.  } else {
bq.// If invoked from timeout monitor donot force it to OFFLINE. Based 
on the
bq.// state we will decide if to change in-memory state to OFFLINE or 
not.  It will
bq.// be done before setting the znode to OFFLINE state.
bq.if (!timeOutMonitorReAllocate) {
bq.  LOG.debug("Forcing OFFLINE; was=" + state);
bq.  state.update(RegionState.State.OFFLINE);
bq.}
bq.  If the timeout monitor tries to reallcoate the node then dont make
bq.  the inmemory state to OFFLINE.
bq.  But the noraml assign flow doesnot expect the inmemory state to OFFLINE.
bq.  Hence the above change.  This is continued with the check in 
bq.  assign()
bq.  int setOfflineInZooKeeper(final RegionState state,
bq.boolean timeOutMonitorReAllocate) {
bq.  // If invoked from timeoutmonitor the current state in memory need not 
be
bq.  // OFFLINE.  
bq.  if (!timeOutMonitorReAllocate && !state.isClosed() && 
!state.isOffline()) {
bq.this.master.abort("Unexpected state trying to OFFLINE; " + state,
bq.new IllegalStateException());
bq.return -1;
bq.  }
bq.  ==
bq.  
bq.  boolean allowCreation = false;
bq.  // If the isReAllocate is true and the current state is PENDING_OPEN
bq.  // or OPENING then update the inmemory state to PENDING_OPEN. This is
bq.  // important because
bq.  // if timeoutmonitor deducts that a region was in OPENING state for a 
long
bq.  // time but by the
bq.  // time timeout monitor tranits the node to OFFLINE the RS would have 
opened
bq.  // the node and the
bq.  // state in znode will be RS_ZK_REGION_OPENED. Inorder to invoke the
bq.  // OpenedRegionHandler
bq.  // we expect the inmemeory state to be PENDING_OPEN or OPENING.
bq.  // For all other cases we can change the inmemory state to OFFLINE.
bq.  if (timeOutMonitorReAllocate
bq.  && (state.getState().equals(RegionState.State.PENDING_OPEN) || 
state
bq.  .getState().equals(RegionState.State.OPENING))) {
bq.state.update(RegionState.State.PENDING_OPEN);
bq.allowCreation = false;
bq.  } else {
bq.state.update(RegionState.State.OFFLINE);
bq.allowCreation = true;
bq.  }
bq.  Thi

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099231#comment-13099231
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--



bq.  On 2011-09-07 18:51:35, Jean-Daniel Cryans wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java,
 line 96
bq.  > 
bq.  >
bq.  > seems to me there's an opportunity for refactoring with 
TestReplication

Yes, although I think the two will move in different directions over time.
For example I want to add testing with three clusters to setup a cycle soon, 
and then this would all change.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1797
---


On 2011-09-07 18:49:31, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 18:49:31)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no 
cycl

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099237#comment-13099237
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/
---

(Updated 2011-09-07 19:19:06.781064)


Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.


Changes
---

Addressed J-D issues


Summary
---

For Master <-> Master replication (as well as cycles > 2) a replication sink 
needs to keep track who is was the originator of a change and
(1) do not replicate this data back to the originator and
(2) pass that information on to the next link in a cycle > 2

In order to do that, HlogKeys read from the WAL are tagged with the Cluster 
UUID at the ReplicationSource before the logs are shipped to the Sink. The sink 
writes the Cluster UUID into the WAL log (in HlogKey, so only once per edit). 
If the sink is also source for yet another sink the ClusterUUID is retained in 
the HLogKeys read from the local WAL and passed on to the sink.


This addresses bug HBASE-2195.
https://issues.apache.org/jira/browse/HBASE-2195


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 

Diff: https://reviews.apache.org/r/1730/diff


Testing
---

All tests pass.
New test: TestMasterReplication (which tests Master <-> Master, but no cycles > 
2, yet), also passes.
Manually tested a cycle between 3 clusters.


Thanks,

Lars



> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4344) Persist memstoreTS to disk


[ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099244#comment-13099244
 ] 

stack commented on HBASE-4344:
--

Nice work Amit.

So, you'd just prepend the memstore to the front of the KV that is in hfile?  
Is the reason you put it here rather than integrate it better by inserting, 
say, after the current version/timestamp because you are trying to minimize 
changes?

Hmm... I see.  You have hfile decorate the KV bytes it picks from the file with 
the memstoreTS found in the metadata for hfile.

Its kinda ugly the fact that hfile knows about KV internals but its not your 
fault this is the case -- it has always been so.  Your addition here making the 
KV from bytes plus the metainfo that has the memstoreTs requires that hfile 
know about KV since its doing this merge of data from two sources (hfile data 
and hfile metadata).

Looks like you have to much logging going on.

Why '+  long maxMemstoreTS = -1;'?  Could do with a comment.

This is good '+ rwcc.initialize(maxMemstoreTS + 1);'

You have a note to yourself:

{code}
+/* Amit: Move this into the HFileReaderV2
+ * 
+ */
+b = metadataMap.get(HFileWriterV2.MAX_MEMSTORE_KEY);
+if (b != null) {
+  this.maxMemstoreTS = Bytes.toLong(b);
+}
{code}

Remove the commented out line:

{code}
+  //new KeyValue(kv.getBuffer(), kv.getOffset(), kv.getLength());
{code}

What you did w/ clone is improvement in line above.

So, this patch is not the prettiest I've seen but i think it could grow on me.  
You have done the difficult task of getting the memstoreTS into the filesystem 
BUT WITHOUT REDOING KV so change is minimally invasive.  I'm in favor of this 
patch.  A more radical redo of KV that includes the memstoreTS can happen later.

Good stuff Amit.

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4345) Ensure that Scanners that read from the storefiles respect MVCC


[ 
https://issues.apache.org/jira/browse/HBASE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099249#comment-13099249
 ] 

stack commented on HBASE-4345:
--

Interesting take, enforcing we only read at the read point or before by doing 
the functionality as a filter that is always used.  Intellectually I'd say that 
this should be core and not be done via a filter but if filter gets the job 
done, lets go for it.  Nice.

> Ensure that Scanners that read from the storefiles respect MVCC
> ---
>
> Key: HBASE-4345
> URL: https://issues.apache.org/jira/browse/HBASE-4345
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: patch-3
>
>
> Currently, the key-values written to the disk do not include the MVCC (RWCC) 
> version information. Once we add that
> information, and make it persistent to disk; let us make the scanners respect 
> the MVCC mechanism by ignoring 
> "newer" writes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4347) Remove duplicated code from Put, Delete, Get, Scan, MultiPut

2011-09-07 Thread Lars Hofhansl (JIRA)

Remove duplicated code from Put, Delete, Get, Scan, MultiPut


 Key: HBASE-4347
 URL: https://issues.apache.org/jira/browse/HBASE-4347
 Project: HBase
  Issue Type: Improvement
  Components: util
Affects Versions: 0.92.0
Reporter: Lars Hofhansl
Priority: Minor
 Fix For: 0.92.0


This came from discussion with Stack w.r.t. HBASE-2195.

There is currently a lot of duplicated code especially between Put and Delete, 
and also between all Operations.
For example all of Put/Delete/Get/Scan have attributes with exactly the same 
code in all classes.
Put and Delete also have the familyMap, Row, Rowlock, Timestamp, etc.

One way to do this is to introduce "OperationWithAttributes" which extends 
Operation, and have Put/Delete/Get/Scan extend that rather than Operation.
In addition Put and Delete could extends from Mutation (which itself would 
extend OperationWithAttributes).

If a static inheritance hierarchy is not desired here, we can use delegation.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099266#comment-13099266
 ] 

Ted Yu commented on HBASE-4015:
---

@J-D:
See Ramkrishna's comment @ 02/Sep/11 16:56

> Refactor the TimeoutMonitor to make it less racy
> 
>
> Key: HBASE-4015
> URL: https://issues.apache.org/jira/browse/HBASE-4015
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 0.90.3
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBASE-4015_1_trunk.patch, HBASE-4015_2_trunk.patch, 
> HBASE-4015_reprepared_trunk_2.patch, Timeoutmonitor with state diagrams.pdf
>
>
> The current implementation of the TimeoutMonitor acts like a race condition 
> generator, mostly making things worse rather than better. It does it's own 
> thing for a while without caring for what's happening in the rest of the 
> master.
> The first thing that needs to happen is that the regions should not be 
> processed in one big batch, because that sometimes can take minutes to 
> process (meanwhile a region that timed out opening might have opened, then 
> what happens is it will be reassigned by the TimeoutMonitor generating the 
> never ending PENDING_OPEN situation).
> Those operations should also be done more atomically, although I'm not sure 
> how to do it in a scalable way in this case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4342) Update Thrift to 0.7.0

2011-09-07 Thread Moaz Reyad (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099280#comment-13099280
 ] 

Moaz Reyad commented on HBASE-4342:
---

Yes, I tried it and it didn't break the compatibility. I didn't change any code 
at the client side other than updating the version of thrift in pom.xml.

> Update Thrift to 0.7.0
> --
>
> Key: HBASE-4342
> URL: https://issues.apache.org/jira/browse/HBASE-4342
> Project: HBase
>  Issue Type: Improvement
>Reporter: Moaz Reyad
>Priority: Minor
> Attachments: HBASE-4342.patch.zip
>
>
> The new version of Thrift is 0.7.0 and it has features and bug fixes that 
> could be useful to include in the next release of HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4345) Ensure that Scanners that read from the storefiles respect MVCC


[ 
https://issues.apache.org/jira/browse/HBASE-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099290#comment-13099290
 ] 

Ted Yu commented on HBASE-4345:
---

Minor comment:
{code}
+@Override
+public boolean hasFilterRow() {
+  if (existingFilter == null)
+return false;
+  else
+return existingFilter.hasFilterRow();
+}
{code}
else is not needed above.

What's the plan for all the sub-tasks of HBASE-2856 ?

> Ensure that Scanners that read from the storefiles respect MVCC
> ---
>
> Key: HBASE-4345
> URL: https://issues.apache.org/jira/browse/HBASE-4345
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: patch-3
>
>
> Currently, the key-values written to the disk do not include the MVCC (RWCC) 
> version information. Once we add that
> information, and make it persistent to disk; let us make the scanners respect 
> the MVCC mechanism by ignoring 
> "newer" writes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4344) Persist memstoreTS to disk


[ 
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099311#comment-13099311
 ] 

Ted Yu commented on HBASE-4344:
---

In HFileReaderV2.java, the following code isn't used:
{code}
+  private int keyValueLenSizeWithoutMemstoreTS() {
+  //KEY_VALUE_LEN_SIZE
+if (this.includeMemstoreTS)
+  return 2 * Bytes.SIZEOF_INT + Bytes.SIZEOF_LONG;
+else
+  return 2 * Bytes.SIZEOF_INT;
+  }
{code}

> Persist memstoreTS to disk
> --
>
> Key: HBASE-4344
> URL: https://issues.apache.org/jira/browse/HBASE-4344
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Amitanand Aiyer
>Assignee: Amitanand Aiyer
> Fix For: 0.89.20100924
>
> Attachments: patch-2
>
>
> Atomicity can be achieved in two ways -- (i) by using  a multiversion 
> concurrency system (MVCC), or (ii) by ensuring that "new" writes do not 
> complete, until the "old" reads complete.
> Currently, Memstore uses something along the lines of MVCC (called RWCC for 
> read-write-consistency-control). But, this mechanism is not incorporated for 
> the key-values written to the disk, as they do not include the memstore TS.
> Let us make the two approaches be similar, by persisting the memstoreTS along 
> with the key-value when it is written to the disk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4337) Update HBase directory structure layout to be aligned with Hadoop


[ 
https://issues.apache.org/jira/browse/HBASE-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099313#comment-13099313
 ] 

Eric Yang commented on HBASE-4337:
--

Can someone assign this to me?  Thanks

> Update HBase directory structure layout to be aligned with Hadoop
> -
>
> Key: HBASE-4337
> URL: https://issues.apache.org/jira/browse/HBASE-4337
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
>Reporter: Eric Yang
>
> In HADOOP-6255, a proposal was made for common directory layout for Hadoop 
> ecosystem.  This jira is to track the necessary work for making HBase 
> directory structure aligned with Hadoop for better integration.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4338) Package build for rpm and deb are broken


[ 
https://issues.apache.org/jira/browse/HBASE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099314#comment-13099314
 ] 

Eric Yang commented on HBASE-4338:
--

Stack, could you assign this issue to me?  Thanks

> Package build for rpm and deb are broken
> 
>
> Key: HBASE-4338
> URL: https://issues.apache.org/jira/browse/HBASE-4338
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
> Environment: Java, Linux
>Reporter: Eric Yang
> Fix For: 0.92.0
>
> Attachments: HBASE-4338.patch
>
>
> Environment variable final.name was removed in HBASE-3629, and this prevents 
> rpm/deb packaging from building.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4302) Only run Snappy compression tests if Snappy is available


[ 
https://issues.apache.org/jira/browse/HBASE-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099316#comment-13099316
 ] 

Hudson commented on HBASE-4302:
---

Integrated in HBase-TRUNK #2183 (See 
[https://builds.apache.org/job/HBase-TRUNK/2183/])
HBASE-4302  Only run Snappy compression tests if Snappy is available

todd : 
Files : 
* /hbase/trunk/CHANGES.txt
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestCompressionTest.java


> Only run Snappy compression tests if Snappy is available
> 
>
> Key: HBASE-4302
> URL: https://issues.apache.org/jira/browse/HBASE-4302
> Project: HBase
>  Issue Type: Test
>  Components: test
>Affects Versions: 0.92.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Fix For: 0.92.0
>
> Attachments: HBASE-4302v1.patch, HBASE-4302v2.patch
>
>
> The presence of the Snappy CODEC it does not mean that the Snappy 
> JNI-bindings and native library are available.
> Because of this it is not possible to assert SNAPPY compression

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3606) Create an package integration project


[ 
https://issues.apache.org/jira/browse/HBASE-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099318#comment-13099318
 ] 

Eric Yang commented on HBASE-3606:
--

Could someone assign this issue to me?  Thanks for helping my patch count. :)

> Create an package integration project
> -
>
> Key: HBASE-3606
> URL: https://issues.apache.org/jira/browse/HBASE-3606
> Project: HBase
>  Issue Type: New Feature
>  Components: build
> Environment: Java 6, Redhat 5.5/Ubuntu 10.10
>Reporter: Eric Yang
> Fix For: 0.92.0
>
> Attachments: HBASE-3606.patch
>
>
> For integrating Hadoop ecosystem more tightly and reduce the cost of hadoop 
> stack development, it would be nice to have a set of installable packages 
> which can setup hadoop stack with least amount effort.  The goal of this jira 
> is to create installable rpm and debian packages for HBase from the build 
> system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-2195) Support cyclic replication


 [ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-2195:
--

Attachment: 2195-v10.txt

The change to removeNonReplicableEdits() is included in version 10.

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4330) Fix races in slab cache

2011-09-07 Thread Li Pi (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099602#comment-13099602
 ] 

Li Pi commented on HBASE-4330:
--

Ran 3 instances of the tests in a loop for 24 hours+. No errors. 

mvn crashed many times, failing to run the tests at all, however. I never 
managed to get it to hang.

> Fix races in slab cache
> ---
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and 
> proposed fixes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4330) Fix races in slab cache


[ 
https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099610#comment-13099610
 ] 

Ted Yu commented on HBASE-4330:
---

How long did TestSingleSizeCache take, on average ?
Thanks

> Fix races in slab cache
> ---
>
> Key: HBASE-4330
> URL: https://issues.apache.org/jira/browse/HBASE-4330
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: 0.92.0
>
> Attachments: hbase-4330.txt
>
>
> A few races are still lingering in the slab cache. Here are some tests and 
> proposed fixes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4195) Possible inconsistency in a memstore read after a reseek, possible performance improvement


[ 
https://issues.apache.org/jira/browse/HBASE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099619#comment-13099619
 ] 

stack commented on HBASE-4195:
--

N:

We should remove:

{code}
this.reseekNumKeys = conf.getInt(RESEEKMAX_KEY, RESEEKMAX_DEFAULT);
{code}

and reseekNumKeys and the defines else someone will come along later and wonder 
what these were about?

Are seek and reseek the same now?  Or it seems like they have a bunch of common 
code... can we factor it out to common method if so?


To add to your synthesis:

+ We're removing the numIterReseek facility because we get new tailSet every 
time we reseek.
+ It looks like there is nice perf. benefit if we go w/ this patch
+ We're fixing a bug where we may miss a Put if a flush comes in in meantime 
because we won't have a running Iterator on new KVSet (but maybe this is not 
such a big deal -- perhaps -- because its unlikely the new Put will be within 
the purview of the current read point?

If you agree on the above changes and synthesis, no need of a new patch.  I"ll 
do the clean up on commit.  Thanks boss.



> Possible inconsistency in a memstore read after a reseek, possible 
> performance improvement
> --
>
> Key: HBASE-4195
> URL: https://issues.apache.org/jira/browse/HBASE-4195
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
> Environment: all
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Critical
> Fix For: 0.90.5
>
> Attachments: 20110824_4195_MemStore.patch, 
> 20110824_4195_TestHRegion.patch
>
>
> This follows the dicussion around HBASE-3855, and the random errors (20% 
> failure on trunk) on the unit test 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting
> I saw some points related to numIterReseek, used in the 
> MemStoreScanner#getNext (line 690):
> {noformat}679 protected KeyValue getNext(Iterator it) {
> 680 KeyValue ret = null;
> 681 long readPoint = ReadWriteConsistencyControl.getThreadReadPoint();
> 682 //DebugPrint.println( " MS@" + hashCode() + ": threadpoint = " + 
> readPoint);
> 683
> 684 while (ret == null && it.hasNext()) {
> 685   KeyValue v = it.next();
> 686   if (v.getMemstoreTS() <= readPoint) {
> 687 // keep it.
> 688 ret = v;
> 689   }
> 690   numIterReseek--;
> 691   if (numIterReseek == 0) {
> 692 break;
> 693}
> 694 }
> 695 return ret;
> 696   }{noformat}
> This function is called by seek, reseek, and next. The numIterReseek is only 
> usefull for reseek.
> There are some issues, I am not totally sure it's the root cause of the test 
> case error, but it could explain partly the randomness of the error, and one 
> point is for sure a bug.
> 1) In getNext, numIterReseek is decreased, then compared to zero. The seek 
> function sets numIterReseek to zero before calling getNext. It means that the 
> value will be actually negative, hence the test will always fail, and the 
> loop will continue. It is the expected behaviour, but it's quite smart.
> 2) In "reseek", numIterReseek is not set between the loops on the two 
> iterators. If the numIterReseek is equals to zero after the loop on the first 
> one, the loop on the second one will never call seek, as numIterReseek will 
> be negative.
> 3) Still in "reseek", the test to call "seek" is (kvsetNextRow == null && 
> numIterReseek == 0). In other words, if kvsetNextRow is not null when 
> numIterReseek equals zero, numIterReseek will start to be negative at the 
> next iteration and seek will never be called.
> 4) You can have side effects if reseek ends with a numIterReseek > 0: the 
> following calls to the "next" function will decrease numIterReseek to zero, 
> and getNext will break instead of continuing the loop. As a result, later 
> calls to next() may return null or not depending on how is configured the 
> default value for numIterReseek.
> To check if the issue comes from point 4, you can set the numIterReseek to 
> zero before returning in reseek:
> {noformat}  numIterReseek = 0;
>   return (kvsetNextRow != null || snapshotNextRow != null);
> }{noformat}
> On my env, on trunk, it seems to work, but as it's random I am not really 
> sure. I also had to modify the test (I added a loop) to make it fails more 
> often, the original test was working quite well here.
> It has to be confirmed that this totally fix (it could be partial or 
> unrelated) 
> org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting 
> before implementing a complete solution.

--
This message is automatically generated by JIRA.
For more information

[jira] [Assigned] (HBASE-3606) Create an package integration project


 [ 
https://issues.apache.org/jira/browse/HBASE-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reassigned HBASE-3606:


Assignee: Eric Yang

Assigned.  I added you as contributor Eric so you should be able to do it 
yourself going forward.  Thanks for the help.

> Create an package integration project
> -
>
> Key: HBASE-3606
> URL: https://issues.apache.org/jira/browse/HBASE-3606
> Project: HBase
>  Issue Type: New Feature
>  Components: build
> Environment: Java 6, Redhat 5.5/Ubuntu 10.10
>Reporter: Eric Yang
>Assignee: Eric Yang
> Fix For: 0.92.0
>
> Attachments: HBASE-3606.patch
>
>
> For integrating Hadoop ecosystem more tightly and reduce the cost of hadoop 
> stack development, it would be nice to have a set of installable packages 
> which can setup hadoop stack with least amount effort.  The goal of this jira 
> is to create installable rpm and debian packages for HBase from the build 
> system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4348) Add metrics for regions in transition

Add metrics for regions in transition
-

 Key: HBASE-4348
 URL: https://issues.apache.org/jira/browse/HBASE-4348
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Priority: Minor


The following metrics would be useful for monitoring the master:
- the number of regions in transition
- the number of regions in transition that have been in transition for more 
than a minute
- how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099623#comment-13099623
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1801
---

Ship it!

- Michael

On 2011-09-07 19:19:06, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 19:19:06)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no 
cycles > 2, yet), also passes.
bq.  Manually tested a cycle between 3 clusters.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v5.txt, 2195-v6.txt, 2195.

[jira] [Assigned] (HBASE-4337) Update HBase directory structure layout to be aligned with Hadoop


 [ 
https://issues.apache.org/jira/browse/HBASE-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reassigned HBASE-4337:


Assignee: Eric Yang

> Update HBase directory structure layout to be aligned with Hadoop
> -
>
> Key: HBASE-4337
> URL: https://issues.apache.org/jira/browse/HBASE-4337
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> In HADOOP-6255, a proposal was made for common directory layout for Hadoop 
> ecosystem.  This jira is to track the necessary work for making HBase 
> directory structure aligned with Hadoop for better integration.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4338) Package build for rpm and deb are broken


 [ 
https://issues.apache.org/jira/browse/HBASE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reassigned HBASE-4338:


Assignee: Eric Yang

> Package build for rpm and deb are broken
> 
>
> Key: HBASE-4338
> URL: https://issues.apache.org/jira/browse/HBASE-4338
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.90.3
> Environment: Java, Linux
>Reporter: Eric Yang
>Assignee: Eric Yang
> Fix For: 0.92.0
>
> Attachments: HBASE-4338.patch
>
>
> Environment variable final.name was removed in HBASE-3629, and this prevents 
> rpm/deb packaging from building.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-4349) Add metrics for executor services

Add metrics for executor services
-

 Key: HBASE-4349
 URL: https://issues.apache.org/jira/browse/HBASE-4349
 Project: HBase
  Issue Type: Improvement
  Components: metrics
Affects Versions: 0.92.0
Reporter: Todd Lipcon


Would be nice to add some metrics for each of the executor services in the 
master and region server:
- number of items in the queue
- age of the oldest yet-unprocessed item (ie in the queue or currently being 
handled)
- number of threads in the executor currently busy

These would allow us to monitor better for cases where things get "stuck"

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4342) Update Thrift to 0.7.0


 [ 
https://issues.apache.org/jira/browse/HBASE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-4342.
--

   Resolution: Fixed
Fix Version/s: 0.92.0
 Assignee: Moaz Reyad
 Hadoop Flags: [Reviewed]

Committed to TRUNK.  Thanks for the patch Moaz.

> Update Thrift to 0.7.0
> --
>
> Key: HBASE-4342
> URL: https://issues.apache.org/jira/browse/HBASE-4342
> Project: HBase
>  Issue Type: Improvement
>Reporter: Moaz Reyad
>Assignee: Moaz Reyad
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4342.patch.zip
>
>
> The new version of Thrift is 0.7.0 and it has features and bug fixes that 
> could be useful to include in the next release of HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099629#comment-13099629
 ] 

stack commented on HBASE-2195:
--

Lars you good on Ted's mod? Its this:

{code}
15:44 < tyu> a small change for removeNonReplicableEdits()
15:45 < tyu>protected void removeNonReplicableEdits(WALEdit edit) {
15:45 < tyu>  NavigableMap scopes = edit.getScopes();
15:45 < tyu>  List kvs = edit.getKeyValues();
15:45 < tyu> -for (int i = 0; i < edit.size(); i++) {
15:45 < tyu> +for (int i = edit.size()-1; i >= 0; i--) {
15:45 < tyu>  kvs.remove(i);
15:45 < tyu> -i--;
{code}

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099633#comment-13099633
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--

bq.  On 2011-09-07 22:37:48, Michael Stack wrote:
bq.  >

Cool... Let me add Ted's change, and a test does cyclic replication.

- Lars

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1801
---

On 2011-09-07 19:19:06, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 19:19:06)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no 
cycles > 2, yet), also passes.
bq.  Manually tested a cycle between 3 clusters.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Da

[jira] [Commented] (HBASE-4271) Clean up coprocessor's handlings of table operations

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099635#comment-13099635
 ] 

jirapos...@reviews.apache.org commented on HBASE-4271:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1691/#review1804
---

Ship it!

I took a quick look.  +1

- Michael

On 2011-09-03 01:47:00, Ming Ma wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1691/
bq.  ---
bq.  
bq.  (Updated 2011-09-03 01:47:00)
bq.  
bq.  
bq.  Review request for hbase, Gary Helmling and Ted Yu.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Along with coprocessor work, also fix HBaseAdmin.deleteTable so that it 
waits until the table is removed from HMaster's HTableDescriptor list.
bq.  
bq.  1. Fix preCreateTable, postCreateTable, preAssign, postAssign, preUnassign 
APIs.
bq.  2. Make coprocessor honor bypass request from preMove, preAddColumn, 
preModifyColumn, preDeleteColumn.
bq.  
bq.  
bq.  This addresses bug HBASE-4271.
bq.  https://issues.apache.org/jira/browse/HBASE-4271
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
 1164783 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
 1164783 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
 1164783 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
 1164783 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
 1164783 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
 1164783 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
 1164783 
bq.  
bq.  Diff: https://reviews.apache.org/r/1691/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  Unit tests
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Ming
bq.  
bq.

> Clean up coprocessor's handlings of table operations
> 
>
> Key: HBASE-4271
> URL: https://issues.apache.org/jira/browse/HBASE-4271
> Project: HBase
>  Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
>
> Couple fixes we can do w.r.t coprocessor's handlings of table operations.
> 1. Honor MasterObserver's requests to bypass default action.
> 2. Fix up the function signatures for preCreateTable to use HRegionInfo as 
> parameter instead.
> 3. Invoke postEnableTable, etc. methods after the operations are done.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4335) Splits can create temporary holes in .META. that confuse clients and regionservers


 [ 
https://issues.apache.org/jira/browse/HBASE-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-4335:
-

 Priority: Critical  (was: Major)
Fix Version/s: 0.92.0

> Splits can create temporary holes in .META. that confuse clients and 
> regionservers
> --
>
> Key: HBASE-4335
> URL: https://issues.apache.org/jira/browse/HBASE-4335
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.90.4
>Reporter: Joe Pallas
>Priority: Critical
> Fix For: 0.92.0
>
>
> When a SplitTransaction is performed, three updates are done to .META.:
> 1. The parent region is marked as splitting (and hence offline)
> 2. The first daughter region is added (same start key as parent)
> 3. The second daughter region is added (split key is start key)
> (later, the original parent region is deleted, but that's not important to 
> this discussion)
> Steps 2 and 3 are actually done concurrently by 
> SplitTransaction.DaughterOpener threads.  While the master is notified when a 
> split is complete, the only visibility that clients have is whether the 
> daughter regions have appeared in .META.
> If the second daughter is added to .META. first, then .META. will contain the 
> (offline) parent region followed by the second daughter region.  If the 
> client looks up a key that is greater than (or equal to) the split, the 
> client will find the second daughter region and use it.  If the key is less 
> than the split key, the client will find the parent region and see that it is 
> offline, triggering a retry.
> If the first daughter is added to .META. before the second daughter, there is 
> a window during which .META. has a hole: the first daughter effectively hides 
> the parent region (same start key), but there is no entry for the second 
> daughter.  A region lookup will find the first daughter for all keys in the 
> parent's range, but the first daughter does not include keys at or beyond the 
> split key.
> See HBASE-4333 and HBASE-4334 for details on how this causes problems and 
> suggestions for mitigating this in the client and regionserver.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy

2011-09-07 Thread Jean-Daniel Cryans (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099637#comment-13099637
 ] 

Jean-Daniel Cryans commented on HBASE-4015:
---

@Ted, thanks I didn't see it amid the rest.

@Ram, did you insert any data in those regions before killing the RSs? 
Replaying the edits usually a good chunk of time for the region to be reopened. 
You could also try doing a worst case cold startup by killing -9 all HBase 
components at the same time (more or less) and then restarting them all (also 
after data was added). Finally you could try setting a super low timeout 
setting, like 5 seconds, to trigger RIT timeouts by the hundreds.

> Refactor the TimeoutMonitor to make it less racy
> 
>
> Key: HBASE-4015
> URL: https://issues.apache.org/jira/browse/HBASE-4015
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 0.90.3
>Reporter: Jean-Daniel Cryans
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: HBASE-4015_1_trunk.patch, HBASE-4015_2_trunk.patch, 
> HBASE-4015_reprepared_trunk_2.patch, Timeoutmonitor with state diagrams.pdf
>
>
> The current implementation of the TimeoutMonitor acts like a race condition 
> generator, mostly making things worse rather than better. It does it's own 
> thing for a while without caring for what's happening in the rest of the 
> master.
> The first thing that needs to happen is that the regions should not be 
> processed in one big batch, because that sometimes can take minutes to 
> process (meanwhile a region that timed out opening might have opened, then 
> what happens is it will be reassigned by the TimeoutMonitor generating the 
> never ending PENDING_OPEN situation).
> Those operations should also be done more atomically, although I'm not sure 
> how to do it in a scalable way in this case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3606) Create an package integration project


[ 
https://issues.apache.org/jira/browse/HBASE-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099642#comment-13099642
 ] 

Eric Yang commented on HBASE-3606:
--

Thanks stack. :)

> Create an package integration project
> -
>
> Key: HBASE-3606
> URL: https://issues.apache.org/jira/browse/HBASE-3606
> Project: HBase
>  Issue Type: New Feature
>  Components: build
> Environment: Java 6, Redhat 5.5/Ubuntu 10.10
>Reporter: Eric Yang
>Assignee: Eric Yang
> Fix For: 0.92.0
>
> Attachments: HBASE-3606.patch
>
>
> For integrating Hadoop ecosystem more tightly and reduce the cost of hadoop 
> stack development, it would be nice to have a set of installable packages 
> which can setup hadoop stack with least amount effort.  The goal of this jira 
> is to create installable rpm and debian packages for HBase from the build 
> system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099653#comment-13099653
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/
---

(Updated 2011-09-07 23:19:53.225308)


Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.


Changes
---

Hopefully the last iteration.
Integrated Ted's optimization for ReplicationSource.removeNonReplicableEdits.
Better tests. Now tests Master - Master and a Cycle between 3 clusters.


Summary
---

For Master <-> Master replication (as well as cycles > 2) a replication sink 
needs to keep track who is was the originator of a change and
(1) do not replicate this data back to the originator and
(2) pass that information on to the next link in a cycle > 2

In order to do that, HlogKeys read from the WAL are tagged with the Cluster 
UUID at the ReplicationSource before the logs are shipped to the Sink. The sink 
writes the Cluster UUID into the WAL log (in HlogKey, so only once per edit). 
If the sink is also source for yet another sink the ClusterUUID is retained in 
the HLogKeys read from the local WAL and passed on to the sink.


This addresses bug HBASE-2195.
https://issues.apache.org/jira/browse/HBASE-2195


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
  
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 

Diff: https://reviews.apache.org/r/1730/diff


Testing
---

All tests pass.
New test: TestMasterReplication (which tests Master <-> Master, but no cycles > 
2, yet), also passes.
Manually tested a cycle between 3 clusters.


Thanks,

Lars



> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4331) Bypassing default actions in prePut fails sometimes with HTable client

2011-09-07 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099654#comment-13099654
 ] 

Lars Hofhansl commented on HBASE-4331:
--

What's the general feeling about this change?

> Bypassing default actions in prePut fails sometimes with HTable client
> --
>
> Key: HBASE-4331
> URL: https://issues.apache.org/jira/browse/HBASE-4331
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4331-v2.txt, 4331-v3.txt, 4331.txt
>
>
> While testing some other scenario I found calling 
> CoprocessorEnvironment.bypass() fails if all trailing puts in a batch are 
> bypassed that way. By extension a single bypassed put will also fail.
> The problem is that the puts are removed from the batch in a way that does 
> not align them with the result-status, and in addition the result is never 
> marked as success.
> A possible fix is to just mark bypassed puts as SUCCESS and filter them in 
> the following logic.
> (I also contemplated a new BYPASSED OperationStatusCode, but that turned out 
> to be not necessary).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4331) Bypassing default actions in prePut fails sometimes with HTable client

2011-09-07 Thread Gary Helmling (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099659#comment-13099659
 ] 

Gary Helmling commented on HBASE-4331:
--

Hi Lars,

I took a quick look and it seemed good.  I like the additional tests as well.

I just want to take some time to review the batch put handling in a little more 
detail.  Will dig in tonight.

> Bypassing default actions in prePut fails sometimes with HTable client
> --
>
> Key: HBASE-4331
> URL: https://issues.apache.org/jira/browse/HBASE-4331
> Project: HBase
>  Issue Type: Bug
>  Components: coprocessors
>Affects Versions: 0.92.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.92.0
>
> Attachments: 4331-v2.txt, 4331-v3.txt, 4331.txt
>
>
> While testing some other scenario I found calling 
> CoprocessorEnvironment.bypass() fails if all trailing puts in a batch are 
> bypassed that way. By extension a single bypassed put will also fail.
> The problem is that the puts are removed from the batch in a way that does 
> not align them with the result-status, and in addition the result is never 
> marked as success.
> A possible fix is to just mark bypassed puts as SUCCESS and filter them in 
> the following logic.
> (I also contemplated a new BYPASSED OperationStatusCode, but that turned out 
> to be not necessary).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4349) Add metrics for executor services


 [ 
https://issues.apache.org/jira/browse/HBASE-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4349:
---

Labels: noob  (was: )

> Add metrics for executor services
> -
>
> Key: HBASE-4349
> URL: https://issues.apache.org/jira/browse/HBASE-4349
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>  Labels: noob
>
> Would be nice to add some metrics for each of the executor services in the 
> master and region server:
> - number of items in the queue
> - age of the oldest yet-unprocessed item (ie in the queue or currently being 
> handled)
> - number of threads in the executor currently busy
> These would allow us to monitor better for cases where things get "stuck"

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4348) Add metrics for regions in transition


 [ 
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HBASE-4348:
---

Component/s: metrics
 Labels: noob  (was: )

> Add metrics for regions in transition
> -
>
> Key: HBASE-4348
> URL: https://issues.apache.org/jira/browse/HBASE-4348
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Priority: Minor
>  Labels: noob
>
> The following metrics would be useful for monitoring the master:
> - the number of regions in transition
> - the number of regions in transition that have been in transition for more 
> than a minute
> - how many seconds has the oldest region-in-transition been in transition

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4342) Update Thrift to 0.7.0


[ 
https://issues.apache.org/jira/browse/HBASE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099692#comment-13099692
 ] 

Hudson commented on HBASE-4342:
---

Integrated in HBase-TRUNK #2185 (See 
[https://builds.apache.org/job/HBase-TRUNK/2185/])
HBASE-4342 Update Thrift to 0.7.0

stack : 
Files : 
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/pom.xml
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/BatchMutation.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/ColumnDescriptor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/IOError.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/IllegalArgument.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/Mutation.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/TCell.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/TRegionInfo.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/TRowResult.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/generated/TScan.java


> Update Thrift to 0.7.0
> --
>
> Key: HBASE-4342
> URL: https://issues.apache.org/jira/browse/HBASE-4342
> Project: HBase
>  Issue Type: Improvement
>Reporter: Moaz Reyad
>Assignee: Moaz Reyad
>Priority: Minor
> Fix For: 0.92.0
>
> Attachments: HBASE-4342.patch.zip
>
>
> The new version of Thrift is 0.7.0 and it has features and bug fixes that 
> could be useful to include in the next release of HBase.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099719#comment-13099719
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1807
---


The new TestMasterReplication passes


http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java


I still don't see VersionedWritable.



http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


A short javadoc should be added, e.g.:

This class tests Master - Master replication and cyclic replication between 
3 clusters.

Can be done at time of commit.


- Ted


On 2011-09-07 23:19:53, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 23:19:53)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplicatio

[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails


[ 
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099720#comment-13099720
 ] 

Subbu M Iyer commented on HBASE-4301:
-

This patch addresses ROOT migration from older versions of HBASE to trunk, in 
addition to what is already done by Ted. I removed the table name derivation 
logic in HConnectionManager as we no longer need it.




> META migration from 0.90 to trunk fails
> ---
>
> Key: HBASE-4301
> URL: https://issues.apache.org/jira/browse/HBASE-4301
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 
> 4301-v3.txt, 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, 
> meta_migrate, meta_trunk, root_migrate, root_trunk
>
>
> I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table 
> is screwed. I can't scan it, etc, and other operations fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4301) META migration from 0.90 to trunk fails


 [ 
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subbu M Iyer updated HBASE-4301:


Attachment: 4301-Fixed_Root_migration_to_newer_HRI_format_.patch

> META migration from 0.90 to trunk fails
> ---
>
> Key: HBASE-4301
> URL: https://issues.apache.org/jira/browse/HBASE-4301
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 
> 4301-v3.txt, 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, 
> meta_migrate, meta_trunk, root_migrate, root_trunk
>
>
> I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table 
> is screwed. I can't scan it, etc, and other operations fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-4273) java.lang.NullPointerException when a table is being disabled and HMaster restarts

2011-09-07 Thread Ming Ma (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming Ma resolved HBASE-4273.


  Resolution: Fixed
Hadoop Flags: [Reviewed]

> java.lang.NullPointerException when a table is being disabled and HMaster 
> restarts
> --
>
> Key: HBASE-4273
> URL: https://issues.apache.org/jira/browse/HBASE-4273
> Project: HBase
>  Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 0.92.0
>
>
> This bug occurs in following scenario. 
> 1. For some reason, the regionLocation isn't set in .META. table for some 
> regions. Perhaps createTable didn't complete successfully.
> 1. The table of those regions is being disabled.
> 2. HMaster restarted.
> 3. At HMaster startup, it tries to transition from disabling to disabled 
> state. It got the following exception.
> java.lang.NullPointerException: Passed server is null
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.
> java:581)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager
> .java:1093)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager
> .java:1040)
> at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.r
> un(DisableTableHandler.java:132)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j
> ava:886)
> In AssignmentManager.rebuildUserRegions, it added such regions to its regions 
> list,
>   if (regionLocation == null) {
> // Region not being served, add to region map with no assignment
> // If this needs to be assigned out, it will also be in ZK as RIT
> // add if the table is not in disabled and enabling state
> if (false == checkIfRegionBelongsToDisabled(regionInfo)
> && false == checkIfRegionsBelongsToEnabling(regionInfo)) {
>   regions.put(regionInfo, regionLocation);
> }
> Perhaps, it should be
>   if (regionLocation == null) {
> // Region not being served, add to region map with no assignment
> // If this needs to be assigned out, it will also be in ZK as RIT
> // add if the table is not in disabled and enabling state
> if (true == checkIfRegionBelongsToEnabled(regionInfo) {
>   regions.put(regionInfo, regionLocation);
> }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-2195) Support cyclic replication


 [ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-2195:
--

Attachment: 2195-v12.txt

Version 12 makes HLogKey extend VersionedWritable

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v12.txt, 2195-v5.txt, 2195-v6.txt, 
> 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails


[ 
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099736#comment-13099736
 ] 

Subbu M Iyer commented on HBASE-4301:
-

Removed the log message.

> META migration from 0.90 to trunk fails
> ---
>
> Key: HBASE-4301
> URL: https://issues.apache.org/jira/browse/HBASE-4301
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 
> 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 
> 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, 
> meta_migrate, meta_trunk, root_migrate, root_trunk
>
>
> I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table 
> is screwed. I can't scan it, etc, and other operations fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4301) META migration from 0.90 to trunk fails


 [ 
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subbu M Iyer updated HBASE-4301:


Attachment: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch

> META migration from 0.90 to trunk fails
> ---
>
> Key: HBASE-4301
> URL: https://issues.apache.org/jira/browse/HBASE-4301
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 
> 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 
> 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, 
> meta_migrate, meta_trunk, root_migrate, root_trunk
>
>
> I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table 
> is screwed. I can't scan it, etc, and other operations fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099738#comment-13099738
 ] 

jirapos...@reviews.apache.org commented on HBASE-2195:
--



bq.  On 2011-09-08 02:45:44, Ted Yu wrote:
bq.  > The new TestMasterReplication passes

Of course it does :)


bq.  On 2011-09-08 02:45:44, Ted Yu wrote:
bq.  > 
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
 line 45
bq.  > 
bq.  >
bq.  > I still don't see VersionedWritable.

It does not really make sense to use VersionedWritable here. We can neither use 
readFields(...) not write(...) of VersionedWritable.


- Lars


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1807
---


On 2011-09-07 23:19:53, Lars Hofhansl wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  ---
bq.  
bq.  (Updated 2011-09-07 23:19:53)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication 
sink needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the 
Cluster UUID at the ReplicationSource before the logs are shipped to the Sink. 
The sink writes the Cluster UUID into the WAL log (in HlogKey, so only once per 
edit). If the sink is also source for yet another sink the ClusterUUID is 
retained in the HLogKeys read from the local WAL and passed on to the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.  https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
 PRE-CREATION 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java
 1166212 
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
 1166212 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no 
c

[jira] [Commented] (HBASE-2195) Support cyclic replication

2011-09-07 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099740#comment-13099740
 ] 

Lars Hofhansl commented on HBASE-2195:
--

Extending VersionedWritable makes no sense to me. We are not using any of its 
methods.
And VersionedWritable is not an interface so now you are enforcing a strict 
class hierarchy where that is not necessary.

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v12.txt, 2195-v5.txt, 2195-v6.txt, 
> 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails


[ 
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099744#comment-13099744
 ] 

stack commented on HBASE-4301:
--

This last patch looks good to me.

> META migration from 0.90 to trunk fails
> ---
>
> Key: HBASE-4301
> URL: https://issues.apache.org/jira/browse/HBASE-4301
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 
> 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 
> 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, 
> meta_migrate, meta_trunk, root_migrate, root_trunk
>
>
> I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table 
> is screwed. I can't scan it, etc, and other operations fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-4301) META migration from 0.90 to trunk fails


 [ 
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reassigned HBASE-4301:
-

Assignee: Subbu M Iyer  (was: Ted Yu)

Subbu provided majority of the patch.

> META migration from 0.90 to trunk fails
> ---
>
> Key: HBASE-4301
> URL: https://issues.apache.org/jira/browse/HBASE-4301
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Subbu M Iyer
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 
> 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 
> 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, 
> meta_migrate, meta_trunk, root_migrate, root_trunk
>
>
> I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table 
> is screwed. I can't scan it, etc, and other operations fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails


[ 
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099748#comment-13099748
 ] 

stack commented on HBASE-4301:
--

True, but you did bunch of debugging too.  You going to commit?

> META migration from 0.90 to trunk fails
> ---
>
> Key: HBASE-4301
> URL: https://issues.apache.org/jira/browse/HBASE-4301
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: Subbu M Iyer
>Priority: Blocker
> Fix For: 0.92.0
>
> Attachments: 4301-1-Fixed_Root_migration_to_newer_HRI_format_.patch, 
> 4301-Fixed_Root_migration_to_newer_HRI_format_.patch, 4301-v3.txt, 
> 4301-v4.txt, 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt, 
> meta_migrate, meta_trunk, root_migrate, root_trunk
>
>
> I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table 
> is screwed. I can't scan it, etc, and other operations fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099747#comment-13099747
 ] 

Ted Yu commented on HBASE-2195:
---

At this moment, we're not using getVersion() of VersionedWritable. But there 
might be future need for this method.
Here is an example from ThriftServer.getTableRegions() that uses 
HRI.getVersion():
{code}
  region.name = ByteBuffer.wrap(regionInfo.getRegionName());
  region.version = regionInfo.getVersion();
  regions.add(region);
{code}
In HRegionInfo, none of the methods from VersionedWritable is used.

This JIRA is targeting multi-cluster replication where we should expect 
different releases of HBase to be in use. Providing HLogKey.getVersion() has 
its benefits.

My two cents.

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v12.txt, 2195-v5.txt, 2195-v6.txt, 
> 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Issue Comment Edited] (HBASE-2195) Support cyclic replication


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099747#comment-13099747
 ] 

Ted Yu edited comment on HBASE-2195 at 9/8/11 4:09 AM:
---

At this moment, we're not using getVersion() of VersionedWritable. But there 
might be future need for this method.
Here is an example from ThriftServer.getTableRegions() that uses 
HRI.getVersion():
{code}
  region.name = ByteBuffer.wrap(regionInfo.getRegionName());
  region.version = regionInfo.getVersion();
  regions.add(region);
{code}

This JIRA is targeting multi-cluster replication where we should expect 
different releases of HBase to be in use. Providing HLogKey.getVersion() has 
its benefits.

My two cents.

  was (Author: yuzhih...@gmail.com):
At this moment, we're not using getVersion() of VersionedWritable. But 
there might be future need for this method.
Here is an example from ThriftServer.getTableRegions() that uses 
HRI.getVersion():
{code}
  region.name = ByteBuffer.wrap(regionInfo.getRegionName());
  region.version = regionInfo.getVersion();
  regions.add(region);
{code}
In HRegionInfo, none of the methods from VersionedWritable is used.

This JIRA is targeting multi-cluster replication where we should expect 
different releases of HBase to be in use. Providing HLogKey.getVersion() has 
its benefits.

My two cents.
  
> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v12.txt, 2195-v5.txt, 2195-v6.txt, 
> 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-2195) Support cyclic replication


[ 
https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099757#comment-13099757
 ] 

stack commented on HBASE-2195:
--

You think we should expose this version Ted?  I'd think it an internal 
implementation detail.   Hopefully, the case should be that the clients never 
have to worry what version their HLogKey is.  It always just works whatever 
data format is thrown at it because internally it'll do the right thing 
instantiating an HLogKey though given version 0 or version 1 data.  It seems 
like HLogKey does what VersionedWritable adds (but for the getVersion method)

> Support cyclic replication
> --
>
> Key: HBASE-2195
> URL: https://issues.apache.org/jira/browse/HBASE-2195
> Project: HBase
>  Issue Type: Sub-task
>  Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Lars Hofhansl
> Attachments: 2195-v10.txt, 2195-v12.txt, 2195-v5.txt, 2195-v6.txt, 
> 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey 
> and stop replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-4260) Expose a command to manually trigger an HLog roll