[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159787#comment-13159787 ] Hudson commented on HBASE-4308: --- Integrated in HBase-0.92 #163 (See [https://builds.apache.org/job/HBase-0.92/163/]) HBASE-4308 Race between RegionOpenedHandler and AssignmentManager (Ram) ramkrishna : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156630#comment-13156630 ] Hudson commented on HBASE-4308: --- Integrated in HBase-TRUNK-security #7 (See [https://builds.apache.org/job/HBase-TRUNK-security/7/]) HBASE-4308 Race between RegionOpenedHandler and AssignmentManager(Ram) ramkrishna : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156453#comment-13156453 ] Hudson commented on HBASE-4308: --- Integrated in HBase-0.92-security #10 (See [https://builds.apache.org/job/HBase-0.92-security/10/]) HBASE-4308 Race between RegionOpenedHandler and AssignmentManager (Ram) ramkrishna : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156258#comment-13156258 ] Hudson commented on HBASE-4308: --- Integrated in HBase-TRUNK #2475 (See [https://builds.apache.org/job/HBase-TRUNK/2475/]) HBASE-4308 Race between RegionOpenedHandler and AssignmentManager(Ram) ramkrishna : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156012#comment-13156012 ] ramkrishna.s.vasudevan commented on HBASE-4308: --- Thanks for the review Stack and Ted. Committed to Trunk and 0.92 branch. Do i need to commit in any other version also? > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155933#comment-13155933 ] stack commented on HBASE-4308: -- +1 on commit. I see now that the effect is the same. In ORH, we'd run the disabling code regardless whether we deleted znode or not and whether region in RIT or not. I see now that the disabling code will work for all three possible conditions still -- its just that one of the handlings has been moved up into AM; only two are done in ORH now. Good work Ram. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155795#comment-13155795 ] Hadoop QA commented on HBASE-4308: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504855/HBASE-4308_2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestInstantSchemaChange org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/345//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/345//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/345//console This message is automatically generated. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155792#comment-13155792
]
ramkrishna.s.vasudevan commented on HBASE-4308:
---
@Stack
Thanks for your review
{code}
+ private void makeRegionOnline(RegionState rs, HRegionInfo regionInfo) {
+regionOnline(regionInfo, rs.serverName);
+LOG.info("The master has opened the region "
++ regionInfo.getRegionNameAsString() + " that was online on "
++ rs.serverName);
+if (this.getZKTable().isDisablingOrDisabledTable(
+regionInfo.getTableNameAsString())) {
+ debugLog(regionInfo, "Opened region "
+ + regionInfo.getRegionNameAsString() + " but "
+ + "this table is disabled, triggering close of region");
+ unassign(regionInfo);
+}
+ }
{code}
I have not broken the logic of unassign if the table is disabled. In
OpenedRegionHandler also the same code is present even if deletion of the node
fails.
Same way if it the callback comes on successful deletion even there this code
is present. Is it ok Stack? I will commit after your confirmation :)
> Race between RegionOpenedHandler and AssignmentManager
> --
>
> Key: HBASE-4308
> URL: https://issues.apache.org/jira/browse/HBASE-4308
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0
>
> Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch
>
>
> When the master is processing a ZK event for REGION_OPENED, it calls delete()
> on the znode before it removes the node from RegionsInTransition. If the
> notification of that delete comes back into AssignmentManager before the
> region is removed from RIT, you see an error like:
> 2011-08-30 17:43:29,537 WARN [main-EventThread]
> master.AssignmentManager(861): Node deleted but still in RIT:
> .META.,,1.1028785192 state=OPEN, ts=1314751409532,
> server=todd-w510,55655,1314751396840
> Not certain if it causes issues, but it's a concerning log message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155762#comment-13155762
]
stack commented on HBASE-4308:
--
Is this check the wrong way round Ram?
{code}
+if (!openedNodeDeleted) {
+ if (this.assignmentManager.getZKTable().isDisablingOrDisabledTable(
+ regionInfo.getTableNameAsString())) {
+debugLog(regionInfo, "Opened region "
++ regionInfo.getRegionNameAsString() + " but "
++ "this table is disabled, triggering close of region");
+assignmentManager.unassign(regionInfo);
+ }
}
{code}
If we failed to delete the znode, only then you check if disabled? Won't
openedNodeDeleted be true if all goes well and this is when you want to check
if region is of a disabling table?
It looks like in old code that we checked table disabling whether we succeeded
znode delete or not?
Otherwise, I'm +1 on this patch (You can do fixup if I'm right and go ahead and
commit)
> Race between RegionOpenedHandler and AssignmentManager
> --
>
> Key: HBASE-4308
> URL: https://issues.apache.org/jira/browse/HBASE-4308
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0
>
> Attachments: HBASE-4308.patch, HBASE-4308_1.patch, HBASE-4308_2.patch
>
>
> When the master is processing a ZK event for REGION_OPENED, it calls delete()
> on the znode before it removes the node from RegionsInTransition. If the
> notification of that delete comes back into AssignmentManager before the
> region is removed from RIT, you see an error like:
> 2011-08-30 17:43:29,537 WARN [main-EventThread]
> master.AssignmentManager(861): Node deleted but still in RIT:
> .META.,,1.1028785192 state=OPEN, ts=1314751409532,
> server=todd-w510,55655,1314751396840
> Not certain if it causes issues, but it's a concerning log message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155408#comment-13155408 ] Hadoop QA commented on HBASE-4308: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504781/HBASE-4308_1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestInstantSchemaChange Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/338//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/338//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/338//console This message is automatically generated. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155383#comment-13155383
]
stack commented on HBASE-4308:
--
So, we are moving the call of regionOnline out of OpenRegionHandler and up as a
reaction to the delete of znode in AM? That looks like a good change.
What is odd though is that the log message -- "Node deleted but still in RIT: "
-- gives the impression that there is something wrong when this log message
comes out though this is now the legit way of onlining a region in master. I'd
suggest that we change the log message to 'Node deleted ...'.
Should this test which is in makeRegionOnline be up in the caller (You test
SPLIT and SPLITTING in caller... it would make code easier to read):
{code}
if (rs.getState().equals(RegionState.State.OPEN))
{code}
Why don't we do rs.isOpened() instead of the above check?
Call the method makeRegionOnline instead regionOnline?
This log message seems extraneous given the above logging of delete:
{code}
+debugLog(regionInfo, "The znode of region "
++ regionInfo.getRegionNameAsString() + " has been deleted.");
{code}
Otherwise patch looks good.
> Race between RegionOpenedHandler and AssignmentManager
> --
>
> Key: HBASE-4308
> URL: https://issues.apache.org/jira/browse/HBASE-4308
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0
>
> Attachments: HBASE-4308.patch, HBASE-4308_1.patch
>
>
> When the master is processing a ZK event for REGION_OPENED, it calls delete()
> on the znode before it removes the node from RegionsInTransition. If the
> notification of that delete comes back into AssignmentManager before the
> region is removed from RIT, you see an error like:
> 2011-08-30 17:43:29,537 WARN [main-EventThread]
> master.AssignmentManager(861): Node deleted but still in RIT:
> .META.,,1.1028785192 state=OPEN, ts=1314751409532,
> server=todd-w510,55655,1314751396840
> Not certain if it causes issues, but it's a concerning log message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155346#comment-13155346 ] Ted Yu commented on HBASE-4308: --- +1 on patch v2. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch, HBASE-4308_1.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155299#comment-13155299
]
Ted Yu commented on HBASE-4308:
---
Patch makes sense.
Minor comment:
{code}
+boolean deleteOpenedNode = false;
{code}
I think openedNodeDeleted would be a better name.
> Race between RegionOpenedHandler and AssignmentManager
> --
>
> Key: HBASE-4308
> URL: https://issues.apache.org/jira/browse/HBASE-4308
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.0
>Reporter: Todd Lipcon
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.92.0
>
> Attachments: HBASE-4308.patch
>
>
> When the master is processing a ZK event for REGION_OPENED, it calls delete()
> on the znode before it removes the node from RegionsInTransition. If the
> notification of that delete comes back into AssignmentManager before the
> region is removed from RIT, you see an error like:
> 2011-08-30 17:43:29,537 WARN [main-EventThread]
> master.AssignmentManager(861): Node deleted but still in RIT:
> .META.,,1.1028785192 state=OPEN, ts=1314751409532,
> server=todd-w510,55655,1314751396840
> Not certain if it causes issues, but it's a concerning log message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155198#comment-13155198 ] Hadoop QA commented on HBASE-4308: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504738/HBASE-4308.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -162 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 66 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.client.TestInstantSchemaChange Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/332//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/332//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/332//console This message is automatically generated. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > Attachments: HBASE-4308.patch > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154290#comment-13154290 ] ramkrishna.s.vasudevan commented on HBASE-4308: --- Will upload patch tomorrow. Test cases are running > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon >Assignee: ramkrishna.s.vasudevan > Fix For: 0.92.0 > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096191#comment-13096191 ] Jean-Daniel Cryans commented on HBASE-4308: --- I was also mentioning that we have to keep in mind that ZK notifications can be delayed, sometimes by tens of seconds when there's a huge backup. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon > Fix For: 0.92.0 > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096190#comment-13096190 ] Todd Lipcon commented on HBASE-4308: I was chatting with JD a bit about this class of bugs, yesterday. In my opinion we should never modify the regionsInTransition map except in response to hearing something from ZK. That is to say, handling REGION_OPENED, we should just call delete on the ZK node. Then, when ZK tells us that the node was deleted, we remove from RIT. RIT should be entirely "reactive" to ZK. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon > Fix For: 0.92.0 > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096185#comment-13096185 ] stack commented on HBASE-4308: -- Any harm swapping the order in which we do the delete and RIT remove? > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon > Fix For: 0.92.0 > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095793#comment-13095793 ] ramkrishna.s.vasudevan commented on HBASE-4308: --- I too have noticed this. but it doesnt cause any issues as such. > Race between RegionOpenedHandler and AssignmentManager > -- > > Key: HBASE-4308 > URL: https://issues.apache.org/jira/browse/HBASE-4308 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.0 >Reporter: Todd Lipcon > Fix For: 0.92.0 > > > When the master is processing a ZK event for REGION_OPENED, it calls delete() > on the znode before it removes the node from RegionsInTransition. If the > notification of that delete comes back into AssignmentManager before the > region is removed from RIT, you see an error like: > 2011-08-30 17:43:29,537 WARN [main-EventThread] > master.AssignmentManager(861): Node deleted but still in RIT: > .META.,,1.1028785192 state=OPEN, ts=1314751409532, > server=todd-w510,55655,1314751396840 > Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
