[jira] [Updated] (HBASE-5243) LogSyncerThread not getting shutdown waiting for the interrupted flag
[ https://issues.apache.org/jira/browse/HBASE-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5243: -- Attachment: HBASE-5243_0.90.patch Patch for 0.90. Pls review so that i can prepare a patch for trunk. LogSyncerThread not getting shutdown waiting for the interrupted flag - Key: HBASE-5243 URL: https://issues.apache.org/jira/browse/HBASE-5243 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.1, 0.90.6 Attachments: HBASE-5243_0.90.patch In the LogSyncer run() we keep looping till this.isInterrupted flag is set. But in some cases the DFSclient is consuming the Interrupted exception. So we are running into infinite loop in some shutdown cases. I would suggest that as we are the ones who tries to close down the LogSyncerThread we can introduce a variable like Close or shutdown and based on the state of this flag along with isInterrupted() we can make the thread stop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5243) LogSyncerThread not getting shutdown waiting for the interrupted flag
[ https://issues.apache.org/jira/browse/HBASE-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5243: -- Attachment: HBASE-5243_0.90_1.patch Corrected the patch. Replaced '||' with ''. LogSyncerThread not getting shutdown waiting for the interrupted flag - Key: HBASE-5243 URL: https://issues.apache.org/jira/browse/HBASE-5243 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.1, 0.90.6 Attachments: HBASE-5243_0.90.patch, HBASE-5243_0.90_1.patch In the LogSyncer run() we keep looping till this.isInterrupted flag is set. But in some cases the DFSclient is consuming the Interrupted exception. So we are running into infinite loop in some shutdown cases. I would suggest that as we are the ones who tries to close down the LogSyncerThread we can introduce a variable like Close or shutdown and based on the state of this flag along with isInterrupted() we can make the thread stop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5243) LogSyncerThread not getting shutdown waiting for the interrupted flag
[ https://issues.apache.org/jira/browse/HBASE-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5243: -- Attachment: HBASE-5243_trunk.patch Patch for trunk. LogSyncerThread not getting shutdown waiting for the interrupted flag - Key: HBASE-5243 URL: https://issues.apache.org/jira/browse/HBASE-5243 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.1, 0.90.6 Attachments: HBASE-5243_0.90.patch, HBASE-5243_0.90_1.patch, HBASE-5243_trunk.patch In the LogSyncer run() we keep looping till this.isInterrupted flag is set. But in some cases the DFSclient is consuming the Interrupted exception. So we are running into infinite loop in some shutdown cases. I would suggest that as we are the ones who tries to close down the LogSyncerThread we can introduce a variable like Close or shutdown and based on the state of this flag along with isInterrupted() we can make the thread stop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5243) LogSyncerThread not getting shutdown waiting for the interrupted flag
[ https://issues.apache.org/jira/browse/HBASE-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5243: -- Status: Patch Available (was: Open) LogSyncerThread not getting shutdown waiting for the interrupted flag - Key: HBASE-5243 URL: https://issues.apache.org/jira/browse/HBASE-5243 Project: HBase Issue Type: Bug Affects Versions: 0.90.5 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.1, 0.90.6 Attachments: HBASE-5243_0.90.patch, HBASE-5243_0.90_1.patch, HBASE-5243_trunk.patch In the LogSyncer run() we keep looping till this.isInterrupted flag is set. But in some cases the DFSclient is consuming the Interrupted exception. So we are running into infinite loop in some shutdown cases. I would suggest that as we are the ones who tries to close down the LogSyncerThread we can introduce a variable like Close or shutdown and based on the state of this flag along with isInterrupted() we can make the thread stop. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5235:
--
Attachment: HBASE-5235_0.90_1.patch
Addressing Ted's comments. Here the logWriter.values() will be iterated twice.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5235:
--
Attachment: HBASE-5235_trunk.patch
Patch for trunk.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5235:
--
Status: Patch Available (was: Open)
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5, 0.92.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5235:
--
Summary: HLogSplitter writer thread's streams not getting closed when any
of the writer threads has exceptions. (was: HLogSplitter writer threads not
getting closed when any of the writer threads has exceptions.)
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5237) Addendum for HBASE-5160 and HBASE-4397
[
https://issues.apache.org/jira/browse/HBASE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5237:
--
Attachment: HBASE-5237_0.90.patch
Addendum for HBASE-5160 and HBASE-4397
--
Key: HBASE-5237
URL: https://issues.apache.org/jira/browse/HBASE-5237
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.0, 0.90.6
Attachments: HBASE-5237_0.90.patch, HBASE-5237_trunk.patch
As part of HBASE-4397 there is one more scenario where the patch has to be
applied.
{code}
RegionPlan plan = getRegionPlan(state, forceNewPlan);
if (plan == null) {
debugLog(state.getRegion(),
Unable to determine a plan to assign + state);
return; // Should get reassigned later when RIT times out.
}
{code}
I think in this scenario also
{code}
this.timeoutMonitor.setAllRegionServersOffline(true);
{code}
this should be done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5237) Addendum for HBASE-5160 and HBASE-4397
[
https://issues.apache.org/jira/browse/HBASE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5237:
--
Attachment: HBASE-5237_trunk.patch
Addendum for HBASE-5160 and HBASE-4397
--
Key: HBASE-5237
URL: https://issues.apache.org/jira/browse/HBASE-5237
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.0, 0.90.6
Attachments: HBASE-5237_0.90.patch, HBASE-5237_trunk.patch
As part of HBASE-4397 there is one more scenario where the patch has to be
applied.
{code}
RegionPlan plan = getRegionPlan(state, forceNewPlan);
if (plan == null) {
debugLog(state.getRegion(),
Unable to determine a plan to assign + state);
return; // Should get reassigned later when RIT times out.
}
{code}
I think in this scenario also
{code}
this.timeoutMonitor.setAllRegionServersOffline(true);
{code}
this should be done.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5225) Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits
[ https://issues.apache.org/jira/browse/HBASE-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5225: -- Attachment: HBASE-3845_0.90_1.patch Update patch addressing Ted's comments. Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits - Key: HBASE-5225 URL: https://issues.apache.org/jira/browse/HBASE-5225 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-3845-90.patch, HBASE-3845_0.90_1.patch Critical defect. Patch from HBASE-3845 was not integrated to 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5225) Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits
[ https://issues.apache.org/jira/browse/HBASE-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5225: -- Affects Version/s: 0.90.4 Fix Version/s: 0.90.6 Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits - Key: HBASE-5225 URL: https://issues.apache.org/jira/browse/HBASE-5225 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.6 Critical defect. not merged to 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5200) AM.ProcessRegionInTransition() and AM.handleRegion() races thus leaving the region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-5200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5200:
--
Affects Version/s: 0.90.5
Fix Version/s: 0.90.7
0.92.1
AM.ProcessRegionInTransition() and AM.handleRegion() races thus leaving the
region assignment inconsistent.
---
Key: HBASE-5200
URL: https://issues.apache.org/jira/browse/HBASE-5200
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.7
This is the scenario
Consider a case where the balancer is going on thus trying to close regions
in a RS.
Before we could close a master switch happens.
On Master switch the set of nodes that are in RIT is collected and we first
get Data and start watching the node
After that the node data is added into RIT.
Now by this time (before adding to RIT) if the RS to which close was called
does a transition in AM.handleRegion() we miss the handling saying RIT state
was null.
{code}
2012-01-13 10:50:46,358 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
a66d281d231dfcaea97c270698b26b6f from server
HOST-192-168-47-205,20020,1326363111288 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,358 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
c12e53bfd48ddc5eec507d66821c4d23 from server
HOST-192-168-47-205,20020,1326363111288 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,358 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
59ae13de8c1eb325a0dd51f4902d2052 from server
HOST-192-168-47-205,20020,1326363111288 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,359 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
f45bc9614d7575f35244849af85aa078 from server
HOST-192-168-47-205,20020,1326363111288 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,359 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
cc3ecd7054fe6cd4a1159ed92fd62641 from server
HOST-192-168-47-204,20020,1326342744518 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,359 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
3af40478a17fee96b4a192b22c90d5a2 from server
HOST-192-168-47-205,20020,1326363111288 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,359 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
e6096a8466e730463e10d3d61f809b92 from server
HOST-192-168-47-204,20020,1326342744518 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,359 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
4806781a1a23066f7baed22b4d237e24 from server
HOST-192-168-47-204,20020,1326342744518 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
2012-01-13 10:50:46,359 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received CLOSED for region
d69e104131accaefe21dcc01fddc7629 from server
HOST-192-168-47-205,20020,1326363111288 but region was in the state null and
not in expected PENDING_CLOSE or CLOSING states
{code}
In branch the CLOSING node is created by RS thus leading to more
inconsistency.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5179: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510674/5179-90v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/773//console This message is automatically generated.) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5153) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers
[ https://issues.apache.org/jira/browse/HBASE-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5153: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510683/HBASE-5153-V6-90-minorchange.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/775//console This message is automatically generated.) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers --- Key: HBASE-5153 URL: https://issues.apache.org/jira/browse/HBASE-5153 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.4 Reporter: Jieshan Bean Assignee: Jieshan Bean Fix For: 0.90.6 Attachments: 5153-trunk.txt, HBASE-5153-V2.patch, HBASE-5153-V3.patch, HBASE-5153-V4-90.patch, HBASE-5153-V5-90.patch, HBASE-5153-V6-90-minorchange.patch, HBASE-5153-V6-90.txt, HBASE-5153-trunk-v2.patch, HBASE-5153-trunk.patch, HBASE-5153.patch HBASE-4893 is related to this issue. In that issue, we know, if multi-threads share a same connection, once this connection got abort in one thread, the other threads will got a HConnectionManager$HConnectionImplementation@18fb1f7 closed exception. It solve the problem of stale connection can't removed. But the orignal HTable instance cann't be continue to use. The connection in HTable should be recreated. Actually, there's two aproach to solve this: 1. In user code, once catch an IOE, close connection and re-create HTable instance. We can use this as a workaround. 2. In HBase Client side, catch this exception, and re-create connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: HBASE-5155_7.patch
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_1.patch, HBASE-5155_7.patch,
HBASE-5155_latest.patch, hbase-5155_6.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5179: -- Comment: was deleted (was: Meta Data issue is very difficult to deal. Some scenarios doesnot work. Please resolve this issue firstly. ) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v5.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: (was: HBASE-5155_7.patch)
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_1.patch, HBASE-5155_latest.patch,
hbase-5155_6.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: HBASE-5155_2.patch
Addressing comments and also avoided unnecessary zookeeper check to enable a
table under process fail over flow
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_1.patch, HBASE-5155_2.patch,
HBASE-5155_latest.patch, hbase-5155_6.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: HBASE-5155_3.patch
{code}
if (hri.isOffline() hri.isSplit()) {
- LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
-; checking daughter presence);
+ LOG.debug(Offlined and split region + hri.getRegionNameAsString()
+ + ; checking daughter presence);
+ if (MetaReader.getRegion(catalogTracker, hri.getRegionName()) == null) {
+return false;
+ }
{code}
Just added the above code as Stack commented.
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_1.patch, HBASE-5155_2.patch,
HBASE-5155_3.patch, HBASE-5155_latest.patch, hbase-5155_6.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: HBASE-5120_5.patch
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch, HBASE-5120_4.patch,
HBASE-5120_5.patch, HBASE-5120_5.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
[jira] [Updated] (HBASE-4236) Don't lock the stream while serializing the response
[ https://issues.apache.org/jira/browse/HBASE-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4236: -- Fix Version/s: 0.92.0 Don't lock the stream while serializing the response Key: HBASE-4236 URL: https://issues.apache.org/jira/browse/HBASE-4236 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor Fix For: 0.92.0 Attachments: 4236.txt It is not necessary to hold the lock on the stream while the response is being serialized. This unnecessarily prevents serializing responses in parallel. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: hbase-5155_new.patch
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_latest.patch, hbase-5155_new.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: (was: hbase-5155_new.patch)
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_latest.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5192) Backport HBASE-4236 Don't lock the stream while serializing the response
[ https://issues.apache.org/jira/browse/HBASE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5192: -- Attachment: HBASE-4236_0.90.patch Backport HBASE-4236 Don't lock the stream while serializing the response Key: HBASE-5192 URL: https://issues.apache.org/jira/browse/HBASE-5192 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-4236_0.90.patch Backporting to 0.90.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: hbase-5155_6.patch
The latest patch addresses the rolling restart scenarios also. One thing is as
HBASE-4083 is not checked into 0.90 the scenario pertaining to that defect will
not be supported.
Tested the following
- Master fail over with and with out patch
- RS fail over
- RS with parital disable state
One thing is this patch should get applied on the master for the patch to take
into effect because enabling, disabling of tables is started by the master.
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_latest.patch, hbase-5155_6.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: HBASE-5155_1.patch
Updated patch addressing Ted's comments.
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_1.patch, HBASE-5155_latest.patch,
hbase-5155_6.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5160) Backport HBASE-4397 - -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time
[ https://issues.apache.org/jira/browse/HBASE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5160: -- Attachment: HBASE-5160_2.patch My check in caused compilation error as i removed an api. Sorry about that. Backport HBASE-4397 - -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time --- Key: HBASE-5160 URL: https://issues.apache.org/jira/browse/HBASE-5160 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-5160-AssignmentManager.patch, HBASE-5160_2.patch Backporting to 0.90.6 considering the importance of the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5179: -- Comment: was deleted (was: sorry,the patch v3 does not work. when master is starting, shutdownhandler flow can't get the meta flag. I didn't find a good way to solve this problem. so I need analyze the flow deeply and need some time.) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v5.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5160) Backport HBASE-4397 - -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time
[ https://issues.apache.org/jira/browse/HBASE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5160: -- Attachment: HBASE-5160-AssignmentManager.patch Backport HBASE-4397 - -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time --- Key: HBASE-5160 URL: https://issues.apache.org/jira/browse/HBASE-5160 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-5160-AssignmentManager.patch Backporting to 0.90.6 considering the importance of the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5184) Backport HBASE-5152 - Region is on service before completing initialization when doing rollback of split, it will affect read correctness
[ https://issues.apache.org/jira/browse/HBASE-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5184: -- Attachment: HBASE-5152_0.90.patch Backport HBASE-5152 - Region is on service before completing initialization when doing rollback of split, it will affect read correctness -- Key: HBASE-5184 URL: https://issues.apache.org/jira/browse/HBASE-5184 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-5152_0.90.patch Important issue to be merged into 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Attachment: HBASE-5155_latest.patch
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Attachments: HBASE-5155_latest.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Fix Version/s: 0.90.6
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_latest.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: HBASE-5120_4.patch
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch, HBASE-5120_4.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328,
[jira] [Updated] (HBASE-5168) Backport HBASE-5100 - Rollback of split could cause closed region to be opened again
[ https://issues.apache.org/jira/browse/HBASE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5168: -- Attachment: HBASE-5100_0.90.patch Backport HBASE-5100 - Rollback of split could cause closed region to be opened again Key: HBASE-5168 URL: https://issues.apache.org/jira/browse/HBASE-5168 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Attachments: HBASE-5100_0.90.patch Considering the importance of the defect merging it to 0.90.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Status: Patch Available (was: Open)
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch, HBASE-5120_4.patch, HBASE-5120_5.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: HBASE-5120_5.patch
Changed debug to error.
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch, HBASE-5120_4.patch, HBASE-5120_5.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Status: Open (was: Patch Available)
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch, HBASE-5120_4.patch, HBASE-5120_5.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
[jira] [Updated] (HBASE-5156) Backport HBASE-4899 - Region would be assigned twice easily with continually killing server and moving region in testing environment
[ https://issues.apache.org/jira/browse/HBASE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5156: -- Fix Version/s: 0.90.6 Backport HBASE-4899 - Region would be assigned twice easily with continually killing server and moving region in testing environment - Key: HBASE-5156 URL: https://issues.apache.org/jira/browse/HBASE-5156 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-4899_Branch90_1.patch Need to backport to 0.90.6 considering the criticality of the issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5157) Backport HBASE-4880- Region is on service before openRegionHandler completes, may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5157: -- Fix Version/s: 0.90.6 Backport HBASE-4880- Region is on service before openRegionHandler completes, may cause data loss - Key: HBASE-5157 URL: https://issues.apache.org/jira/browse/HBASE-5157 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-4880_branch90_1.patch Backporting to 0.90.6 considering the importance of the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5158) Backport HBASE-4878 - Master crash when splitting hlog may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5158: -- Fix Version/s: 0.90.6 Backport HBASE-4878 - Master crash when splitting hlog may cause data loss -- Key: HBASE-5158 URL: https://issues.apache.org/jira/browse/HBASE-5158 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-4878_branch90_1.patch Backporting to 0.90.6 considering the importance of the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5168) Backport HBASE-5100 - Rollback of split could cause closed region to be opened again
[ https://issues.apache.org/jira/browse/HBASE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5168: -- Fix Version/s: 0.90.6 Backport HBASE-5100 - Rollback of split could cause closed region to be opened again Key: HBASE-5168 URL: https://issues.apache.org/jira/browse/HBASE-5168 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-5100_0.90.patch Considering the importance of the defect merging it to 0.90.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5160) Backport HBASE-4397 - -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time
[ https://issues.apache.org/jira/browse/HBASE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5160: -- Fix Version/s: 0.90.6 Backport HBASE-4397 - -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time --- Key: HBASE-5160 URL: https://issues.apache.org/jira/browse/HBASE-5160 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Backporting to 0.90.6 considering the importance of the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5184) Backport HBASE-5152 - Region is on service before completing initialization when doing rollback of split, it will affect read correctness
[ https://issues.apache.org/jira/browse/HBASE-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5184: -- Fix Version/s: 0.90.6 Backport HBASE-5152 - Region is on service before completing initialization when doing rollback of split, it will affect read correctness -- Key: HBASE-5184 URL: https://issues.apache.org/jira/browse/HBASE-5184 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Important issue to be merged into 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5178) Backport HBASE-4101 - Regionserver Deadlock
[ https://issues.apache.org/jira/browse/HBASE-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5178: -- Fix Version/s: 0.90.6 Backport HBASE-4101 - Regionserver Deadlock --- Key: HBASE-5178 URL: https://issues.apache.org/jira/browse/HBASE-5178 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.90.6 Attachments: HBASE-4101_0.90_1.patch Critical issue not merged to 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: HBASE-5120_2.patch
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch, HBASE-5120_2.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
2012-01-04 00:20:39,864 INFO
[jira] [Updated] (HBASE-5157) Backport HBASE-4880- Region is on service before openRegionHandler completes, may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5157: -- Attachment: HBASE-4880_branch90_1.patch Backport HBASE-4880- Region is on service before openRegionHandler completes, may cause data loss - Key: HBASE-5157 URL: https://issues.apache.org/jira/browse/HBASE-5157 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Attachments: HBASE-4880_branch90_1.patch Backporting to 0.90.6 considering the importance of the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5158) Backport HBASE-4878 - Master crash when splitting hlog may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5158: -- Attachment: HBASE-4878_branch90_1.patch Backport HBASE-4878 - Master crash when splitting hlog may cause data loss -- Key: HBASE-5158 URL: https://issues.apache.org/jira/browse/HBASE-5158 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Attachments: HBASE-4878_branch90_1.patch Backporting to 0.90.6 considering the importance of the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5159) Backport HBASE-4079 - HTableUtil - helper class for loading data
[ https://issues.apache.org/jira/browse/HBASE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5159: -- Attachment: HBASE-4079.patch Backport HBASE-4079 - HTableUtil - helper class for loading data - Key: HBASE-5159 URL: https://issues.apache.org/jira/browse/HBASE-5159 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Attachments: HBASE-4079.patch Backporting to 0.90.6 considering the usefulness of the feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5156) Backport HBASE-4899 - Region would be assigned twice easily with continually killing server and moving region in testing environment
[ https://issues.apache.org/jira/browse/HBASE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5156: -- Attachment: HBASE-4899_Branch90_1.patch Backport HBASE-4899 - Region would be assigned twice easily with continually killing server and moving region in testing environment - Key: HBASE-5156 URL: https://issues.apache.org/jira/browse/HBASE-5156 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Attachments: HBASE-4899_Branch90_1.patch Need to backport to 0.90.6 considering the criticality of the issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: HBASE-5120_3.patch
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
2012-01-04
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Status: Open (was: Patch Available)
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Status: Patch Available (was: Open)
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
[jira] [Updated] (HBASE-5137) MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws IOException
[
https://issues.apache.org/jira/browse/HBASE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5137:
--
Fix Version/s: 0.90.6
0.92.1
Committed to 0.90 and trunk. Do we need to commit in 0.92 also?
MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws
IOException
Key: HBASE-5137
URL: https://issues.apache.org/jira/browse/HBASE-5137
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: 5137-trunk.txt, HBASE-5137.patch, HBASE-5137.patch
I am not sure if this bug was already raised in JIRA.
In our test cluster we had a scenario where the RS had gone down and
ServerShutDownHandler started with splitLog.
But as the HDFS was down the check waitOnSafeMode throws IOException.
{code}
try {
// If FS is in safe mode, just wait till out of it.
FSUtils.waitOnSafeMode(conf,
conf.getInt(HConstants.THREAD_WAKE_FREQUENCY, 1000));
splitter.splitLog();
} catch (OrphanHLogAfterSplitException e) {
{code}
We catch the exception
{code}
} catch (IOException e) {
checkFileSystem();
LOG.error(Failed splitting + logDir.toString(), e);
}
{code}
So the HLog split itself did not happen. We encontered like 4 regions that
was recently splitted in the crashed RS was lost.
Can we abort the Master in such scenarios? Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4101) Regionserver Deadlock
[ https://issues.apache.org/jira/browse/HBASE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4101: -- Fix Version/s: (was: 0.90.4) 0.92.0 Regionserver Deadlock - Key: HBASE-4101 URL: https://issues.apache.org/jira/browse/HBASE-4101 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.90.3 Environment: CentOS 5.5, CDH3 u0 Hadoop, HBase 0.90.3 Reporter: Matt Davies Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: HBASE-4101_0.90.patch, HBASE-4101_0.90_1.patch, HBASE-4101_trunk.patch, HBASE-4101_trunk_1.patch, jstack.txt We periodically see a situation where the regionserver process exists in the process list, zookeeper thread sends the keepalive so the master won't remove it from the active list, yet the regionserver will not serve data. Hadoop(cdh3u0), HBase 0.90.3 (Apache version), under load from an internal testing tool. Attached is the full JStack -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5155:
--
Hadoop Flags: (was: Reviewed)
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[
https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5094:
--
Fix Version/s: 0.92.0
Updated the fix versions.
The META can hold an entry for a region with a different server name from the
one actually in the AssignmentManager thus making the region inaccessible.
Key: HBASE-5094
URL: https://issues.apache.org/jira/browse/HBASE-5094
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.92.0
Attachments: HBASE-5094_1.patch
{code}
RegionState rit =
this.services.getAssignmentManager().isRegionInTransition(e.getKey());
ServerName addressFromAM = this.services.getAssignmentManager()
.getRegionServerOfRegion(e.getKey());
if (rit != null !rit.isClosing() !rit.isPendingClose()) {
// Skip regions that were in transition unless CLOSING or
// PENDING_CLOSE
LOG.info(Skip assigning region + rit.toString());
} else if (addressFromAM != null
!addressFromAM.equals(this.serverName)) {
LOG.debug(Skip assigning region
+ e.getKey().getRegionNameAsString()
+ because it has been opened in
+ addressFromAM.getServerName());
}
{code}
In ServerShutDownHandler we try to get the address in the AM. This address
is initially null because it is not yet updated after the region was opened
.i.e. the CAll back after node deletion is not yet done in the master side.
But removal from RIT is completed on the master side. So this will trigger a
new assignment.
So there is a small window between the online region is actually added in to
the online list and the ServerShutdownHandler where we check the existing
address in AM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Status: Patch Available (was: Open)
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
2012-01-04 00:20:39,864 INFO
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: HBASE-5120_1.patch
Patch addresses the issue. But does not make the code cleaner as Stack wanted
it to be.
@Stack
We may need to make the TM and unassign flow lighter. This patch addresses the
problem in this JIRA.
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Fix For: 0.94.0, 0.92.1
Attachments: HBASE-5120.patch, HBASE-5120_1.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
[jira] [Updated] (HBASE-5152) Region is on service before completing initialization when doing rollback of split, it will affect read correctness
[ https://issues.apache.org/jira/browse/HBASE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5152: -- Resolution: Fixed Status: Resolved (was: Patch Available) Region is on service before completing initialization when doing rollback of split, it will affect read correctness Key: HBASE-5152 URL: https://issues.apache.org/jira/browse/HBASE-5152 Project: HBase Issue Type: Bug Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.92.0, 0.94.0 Attachments: 5152-v2.txt, hbase-5152.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5137) MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws IOException
[
https://issues.apache.org/jira/browse/HBASE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5137:
--
Attachment: HBASE-5137.patch
MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws
IOException
Key: HBASE-5137
URL: https://issues.apache.org/jira/browse/HBASE-5137
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5137.patch
I am not sure if this bug was already raised in JIRA.
In our test cluster we had a scenario where the RS had gone down and
ServerShutDownHandler started with splitLog.
But as the HDFS was down the check waitOnSafeMode throws IOException.
{code}
try {
// If FS is in safe mode, just wait till out of it.
FSUtils.waitOnSafeMode(conf,
conf.getInt(HConstants.THREAD_WAKE_FREQUENCY, 1000));
splitter.splitLog();
} catch (OrphanHLogAfterSplitException e) {
{code}
We catch the exception
{code}
} catch (IOException e) {
checkFileSystem();
LOG.error(Failed splitting + logDir.toString(), e);
}
{code}
So the HLog split itself did not happen. We encontered like 4 regions that
was recently splitted in the crashed RS was lost.
Can we abort the Master in such scenarios? Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5137) MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws IOException
[
https://issues.apache.org/jira/browse/HBASE-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5137:
--
Attachment: HBASE-5137.patch
Patch for 0.90 addressing Ted's comment of adding braces. But did not handle
interrupted exception.
@Ted
Pls check if it is ok.
MasterFileSystem.splitLog() should abort even if waitOnSafeMode() throws
IOException
Key: HBASE-5137
URL: https://issues.apache.org/jira/browse/HBASE-5137
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: 5137-trunk.txt, HBASE-5137.patch, HBASE-5137.patch
I am not sure if this bug was already raised in JIRA.
In our test cluster we had a scenario where the RS had gone down and
ServerShutDownHandler started with splitLog.
But as the HDFS was down the check waitOnSafeMode throws IOException.
{code}
try {
// If FS is in safe mode, just wait till out of it.
FSUtils.waitOnSafeMode(conf,
conf.getInt(HConstants.THREAD_WAKE_FREQUENCY, 1000));
splitter.splitLog();
} catch (OrphanHLogAfterSplitException e) {
{code}
We catch the exception
{code}
} catch (IOException e) {
checkFileSystem();
LOG.error(Failed splitting + logDir.toString(), e);
}
{code}
So the HLog split itself did not happen. We encontered like 4 regions that
was recently splitted in the crashed RS was lost.
Can we abort the Master in such scenarios? Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4397) -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time
[ https://issues.apache.org/jira/browse/HBASE-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4397: -- Resolution: Fixed Status: Resolved (was: Patch Available) Resolving as committed to trunk and 0.92 long back. -ROOT-, .META. tables stay offline for too long in recovery phase after all RSs are shutdown at the same time - Key: HBASE-4397 URL: https://issues.apache.org/jira/browse/HBASE-4397 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Fix For: 0.92.0, 0.94.0 Attachments: HBASE-4397-0.92.patch 1. Shutdown all RSs. 2. Bring all RS back online. The -ROOT-, .META. stay in offline state until timeout monitor force assignment 30 minutes later. That is because HMaster can't find a RS to assign the tables to in assign operation. 011-09-13 13:25:52,743 WARN org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of -ROOT-,,0.70236052 to sea-lab-4,60020,1315870341387, trying to assign elsewhere instead; retry=0 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:373) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:345) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1002) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:854) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:148) at $Proxy9.openRegion(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:407) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1408) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1153) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1128) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1123) at org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1788) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.verifyAndAssignRoot(ServerShutdownHandler.java:100) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.verifyAndAssignRootWithRetries(ServerShutdownHandler.java:118) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:181) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:167) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2011-09-13 13:25:52,743 WARN org.apache.hadoop.hbase.master.AssignmentManager: Unable to find a viable location to assign region -ROOT-,,0.70236052 Possible fixes: 1. Have serverManager handle server online event similar to how RegionServerTracker.java calls servermanager.expireServer in the case server goes down. 2. Make timeoutMonitor handle the situation better. This is a special situation in the cluster. 30 minutes timeout can be skipped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5088) A concurrency issue on SoftValueSortedMap
[ https://issues.apache.org/jira/browse/HBASE-5088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5088: -- Fix Version/s: 0.90.6 A concurrency issue on SoftValueSortedMap - Key: HBASE-5088 URL: https://issues.apache.org/jira/browse/HBASE-5088 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.4, 0.94.0 Reporter: Jieshan Bean Assignee: Jieshan Bean Priority: Critical Fix For: 0.92.0, 0.90.6 Attachments: 5088-syncObj.txt, 5088-useMapInterfaces.txt, 5088.generics.txt, HBase-5088-90.patch, HBase-5088-trunk.patch, HBase5088-90-replaceSoftValueSortedMap.patch, HBase5088-90-replaceTreeMap.patch, HBase5088-trunk-replaceTreeMap.patch, HBase5088Reproduce.java, PerformanceTestResults.png SoftValueSortedMap is backed by a TreeMap. All the methods in this class are synchronized. If we use this method to add/delete elements, it's ok. But in HConnectionManager#getCachedLocation, it use headMap to get a view from SoftValueSortedMap#internalMap. Once we operate on this view map(like add/delete) in other threads, a concurrency issue may occur. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4899) Region would be assigned twice easily with continually killing server and moving region in testing environment
[
https://issues.apache.org/jira/browse/HBASE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-4899:
--
Affects Version/s: (was: 0.92.1)
0.92.0
Fix Version/s: (was: 0.90.0)
0.92.0
Region would be assigned twice easily with continually killing server and
moving region in testing environment
---
Key: HBASE-4899
URL: https://issues.apache.org/jira/browse/HBASE-4899
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.92.0
Attachments: hbase-4899.patch, hbase-4899v2.patch, hbase-4899v3.patch
Before assigning region in ServerShutdownHandler#process, it will check
whether region is in RIT,
however, this checking doesn't work as the excepted in the following case:
1.move region A from server B to server C
2.kill server B
3.start server B immediately
Let's see what happen in the code for the above case
{code}
for step1:
1.1 server B close the region A,
1.2 master setOffline for region
A,(AssignmentManager#setOffline:this.regions.remove(regionInfo))
1.3 server C start to open region A.(Not completed)
for step3:
master ServerShutdownHandler#process() for server B
{
..
splitlog()
...
ListRegionState regionsInTransition =
this.services.getAssignmentManager()
.processServerShutdown(this.serverName);
...
Skip regions that were in transition unless CLOSING or PENDING_CLOSE
...
assign region
}
{code}
In fact, when running
ServerShutdownHandler#process()#this.services.getAssignmentManager().processServerShutdown(this.serverName),
region A is in RIT (step1.3 not completed), but the return ListRegionState
regionsInTransition doesn't contain it, because region A has removed from
AssignmentManager.regions by AssignmentManager#setOffline in step 1.2
Therefore, region A will be assigned twice.
Actually, one server killed and started twice will also easily cause region
assigned twice.
Exclude the above reason, another probability :
when execute ServerShutdownHandler#process()#MetaReader.getServerUserRegions
,region is included which is in RIT now.
But after completing MetaReader.getServerUserRegions, the region has been
opened in other server and is not in RIT now.
In our testing environment where balancing,moving and killing are executed
periodly, assigning region twice often happens, and it is hateful because it
will affect other test cases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Comment: was deleted
(was: Patch is attached so that i can access it at home. Not the final one and
not fully tested in cluster.)
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Attachments: HBASE-5120.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: HBASE-5120.patch
Attaching the patch so that i can access it at home. First cut versiion and
not fully tested.
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Attachments: HBASE-5120.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5120:
--
Attachment: (was: HBASE-5120.patch)
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Priority: Blocker
Attachments: HBASE-5120.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
2012-01-04 00:20:39,864 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
[jira] [Updated] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Status: Open (was: Patch Available)
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch, HBASE-5097_1.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Attachment: HBASE-5097_2.patch
Indention problem occured by mistake. Corrected patch uploaded.
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch, HBASE-5097_1.patch, HBASE-5097_2.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4773) HBaseAdmin may leak ZooKeeper connections
[
https://issues.apache.org/jira/browse/HBASE-4773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-4773:
--
Resolution: Fixed
Fix Version/s: 0.92.0
Status: Resolved (was: Patch Available)
Committed sometime back.
HBaseAdmin may leak ZooKeeper connections
-
Key: HBASE-4773
URL: https://issues.apache.org/jira/browse/HBASE-4773
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.90.4
Reporter: gaojinchao
Assignee: xufeng
Priority: Critical
Fix For: 0.92.0, 0.90.6
Attachments: 4773.patch, branches_4773.patch, trunk_4773_patch.patch
When master crashs, HBaseAdmin will leaks ZooKeeper connections
I think we should close the zk connetion when throw MasterNotRunningException
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException {
this.conf = HBaseConfiguration.create(c);
this.connection = HConnectionManager.getConnection(this.conf);
this.pause = this.conf.getLong(hbase.client.pause, 1000);
this.numRetries = this.conf.getInt(hbase.client.retries.number, 10);
this.retryLongerMultiplier =
this.conf.getInt(hbase.client.retries.longer.multiplier, 10);
//we should add this code and close the zk connection
try{
this.connection.getMaster();
}catch(MasterNotRunningException e){
HConnectionManager.deleteConnection(conf, false);
throw e;
}
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Status: Patch Available (was: Open)
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch, HBASE-5097_1.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Attachment: HBASE-5097_1.patch
Addressed comments from Lars and Ted.
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch, HBASE-5097_1.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Status: Open (was: Patch Available)
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch, HBASE-5097_1.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[
https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5094:
--
Resolution: Fixed
Assignee: ramkrishna.s.vasudevan
Status: Resolved (was: Patch Available)
Committed to 0.92 and trunk.
The META can hold an entry for a region with a different server name from the
one actually in the AssignmentManager thus making the region inaccessible.
Key: HBASE-5094
URL: https://issues.apache.org/jira/browse/HBASE-5094
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Attachments: HBASE-5094_1.patch
{code}
RegionState rit =
this.services.getAssignmentManager().isRegionInTransition(e.getKey());
ServerName addressFromAM = this.services.getAssignmentManager()
.getRegionServerOfRegion(e.getKey());
if (rit != null !rit.isClosing() !rit.isPendingClose()) {
// Skip regions that were in transition unless CLOSING or
// PENDING_CLOSE
LOG.info(Skip assigning region + rit.toString());
} else if (addressFromAM != null
!addressFromAM.equals(this.serverName)) {
LOG.debug(Skip assigning region
+ e.getKey().getRegionNameAsString()
+ because it has been opened in
+ addressFromAM.getServerName());
}
{code}
In ServerShutDownHandler we try to get the address in the AM. This address
is initially null because it is not yet updated after the region was opened
.i.e. the CAll back after node deletion is not yet done in the master side.
But removal from RIT is completed on the master side. So this will trigger a
new assignment.
So there is a small window between the online region is actually added in to
the online list and the ServerShutdownHandler where we check the existing
address in AM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5100) Rollback of split could cause closed region to be opened again
[
https://issues.apache.org/jira/browse/HBASE-5100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5100:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Resolved as committed.
Rollback of split could cause closed region to be opened again
--
Key: HBASE-5100
URL: https://issues.apache.org/jira/browse/HBASE-5100
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.92.0, 0.94.0
Attachments: 5100-double-exeception.txt, 5100-v2.txt, hbase-5100.patch
If master sending close region to rs and region's split transaction
concurrently happen,
it may cause closed region to opened.
See the detailed code in SplitTransaction#createDaughters
{code}
ListStoreFile hstoreFilesToSplit = null;
try{
hstoreFilesToSplit = this.parent.close(false);
if (hstoreFilesToSplit == null) {
// The region was closed by a concurrent thread. We can't continue
// with the split, instead we must just abandon the split. If we
// reopen or split this could cause problems because the region has
// probably already been moved to a different server, or is in the
// process of moving to a different server.
throw new IOException(Failed to close region: already closed by +
another thread);
}
} finally {
this.journal.add(JournalEntry.CLOSED_PARENT_REGION);
}
{code}
when rolling back, the JournalEntry.CLOSED_PARENT_REGION causes
this.parent.initialize();
Although this region is not onlined in the regionserver, it may bring some
potential problem.
For example, in our environment, the closed parent region is rolled back
sucessfully , and then starting compaction and split again.
The parent region is f892dd6107b6b4130199582abc78e9c1
master log
{code}
2011-12-26 00:24:42,693 INFO org.apache.hadoop.hbase.master.HMaster: balance
hri=writetest,8ZW417DZP93OU6SZ0QQMKTALTDP4883KW5AXSAFMQ952Y6J6VPPXEXRRPCWBR2PK7DQV3RKK28222JMOJSW3JJ8AB05MIREM1CL6,1324829936318.f892dd6107b6b4130199582abc78e9c1.,
src=dw87.kgb.sqa.cm4,60020,1324827866085,
dest=dw80.kgb.sqa.cm4,60020,1324827865780
2011-12-26 00:24:42,693 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region
writetest,8ZW417DZP93OU6SZ0QQMKTALTDP4883KW5AXSAFMQ952Y6J6VPPXEXRRPCWBR2PK7DQV3RKK28222JMOJSW3JJ8AB05MIREM1CL6,1324829936318.f892dd6107b6b4130199582abc78e9c1.
(offlining)
2011-12-26 00:24:42,694 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
serverName=dw87.kgb.sqa.cm4,60020,1324827866085, load=(requests=0, regions=0,
usedHeap=0, maxHeap=0) for region
writetest,8ZW417DZP93OU6SZ0QQMKTALTDP4883KW5AXSAFMQ952Y6J6VPPXEXRRPCWBR2PK7DQV3RKK28222JMOJSW3JJ8AB05MIREM1CL6,1324829936318.f892dd6107b6b4130199582abc78e9c1.
2011-12-26 00:24:42,699 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling new unassigned
node: /hbase-tbfs/unassigned/f892dd6107b6b4130199582abc78e9c1
(region=writetest,8ZW417DZP93OU6SZ0QQMKTALTDP4883KW5AXSAFMQ952Y6J6VPPXEXRRPCWBR2PK7DQV3RKK28222JMOJSW3JJ8AB05MIREM1CL6,1324829936318.f892dd6107b6b4130199582abc78e9c1.,
server=dw87.kgb.sqa.cm4,60020,1324827866085, state=RS_ZK_REGION_CLOSING)
2011-12-26 00:24:42,699 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSING, server=dw87.kgb.sqa.cm4,60020,1324827866085,
region=f892dd6107b6b4130199582abc78e9c1
2011-12-26 00:24:45,348 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=dw87.kgb.sqa.cm4,60020,1324827866085,
region=f892dd6107b6b4130199582abc78e9c1
2011-12-26 00:24:45,349 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for f892dd6107b6b4130199582abc78e9c1
2011-12-26 00:24:45,349 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=writetest,8ZW417DZP93OU6SZ0QQMKTALTDP4883KW5AXSAFMQ952Y6J6VPPXEXRRPCWBR2PK7DQV3RKK28222JMOJSW3JJ8AB05MIREM1CL6,1324829936318.f892dd6107b6b4130199582abc78e9c1.
state=CLOSED, ts=1324830285347
2011-12-26 00:24:45,349 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x13447f283f40e73 Creating (or updating) unassigned node for
f892dd6107b6b4130199582abc78e9c1 with OFFLINE state
2011-12-26 00:24:45,354 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=dw75.kgb.sqa.cm4:6,
region=f892dd6107b6b4130199582abc78e9c1
2011-12-26 00:24:45,354 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for
[jira] [Updated] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Attachment: HBASE-5097.patch
Trunk patch
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Status: Patch Available (was: Open)
RegionObserver implementation whose preScannerOpen and postScannerOpen Impl
return null can stall the system initialization through NPE
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5097.patch
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5102) Change the default value of the property hbase.connection.per.config to false in hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5102: -- Status: Patch Available (was: Open) Change the default value of the property hbase.connection.per.config to false in hbase-default.xml - Key: HBASE-5102 URL: https://issues.apache.org/jira/browse/HBASE-5102 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Priority: Minor Attachments: HBASE-5102.patch The property hbase.connection.per.config has a default value of true in hbase-default.xml. In HConnectionManager we try to assign false as the default value if no value is specified. Better to make it uniform. As per Ted's suggestion making it false in the hbase-default.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[
https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5094:
--
Attachment: HBASE-5094_1.patch
The META can hold an entry for a region with a different server name from the
one actually in the AssignmentManager thus making the region inaccessible.
Key: HBASE-5094
URL: https://issues.apache.org/jira/browse/HBASE-5094
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: ramkrishna.s.vasudevan
Priority: Critical
Attachments: HBASE-5094_1.patch
{code}
RegionState rit =
this.services.getAssignmentManager().isRegionInTransition(e.getKey());
ServerName addressFromAM = this.services.getAssignmentManager()
.getRegionServerOfRegion(e.getKey());
if (rit != null !rit.isClosing() !rit.isPendingClose()) {
// Skip regions that were in transition unless CLOSING or
// PENDING_CLOSE
LOG.info(Skip assigning region + rit.toString());
} else if (addressFromAM != null
!addressFromAM.equals(this.serverName)) {
LOG.debug(Skip assigning region
+ e.getKey().getRegionNameAsString()
+ because it has been opened in
+ addressFromAM.getServerName());
}
{code}
In ServerShutDownHandler we try to get the address in the AM. This address
is initially null because it is not yet updated after the region was opened
.i.e. the CAll back after node deletion is not yet done in the master side.
But removal from RIT is completed on the master side. So this will trigger a
new assignment.
So there is a small window between the online region is actually added in to
the online list and the ServerShutdownHandler where we check the existing
address in AM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5102) Change the default value of the property hbase.connection.per.config to false in hbase-default.xml
[ https://issues.apache.org/jira/browse/HBASE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5102: -- Attachment: HBASE-5102.patch Change the default value of the property hbase.connection.per.config to false in hbase-default.xml - Key: HBASE-5102 URL: https://issues.apache.org/jira/browse/HBASE-5102 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Priority: Minor Attachments: HBASE-5102.patch The property hbase.connection.per.config has a default value of true in hbase-default.xml. In HConnectionManager we try to assign false as the default value if no value is specified. Better to make it uniform. As per Ted's suggestion making it false in the hbase-default.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[
https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5094:
--
Attachment: 5094.patch
Patch for trunk
The META can hold an entry for a region with a different server name from the
one actually in the AssignmentManager thus making the region inaccessible.
Key: HBASE-5094
URL: https://issues.apache.org/jira/browse/HBASE-5094
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: ramkrishna.s.vasudevan
Priority: Critical
Attachments: 5094.patch
{code}
RegionState rit =
this.services.getAssignmentManager().isRegionInTransition(e.getKey());
ServerName addressFromAM = this.services.getAssignmentManager()
.getRegionServerOfRegion(e.getKey());
if (rit != null !rit.isClosing() !rit.isPendingClose()) {
// Skip regions that were in transition unless CLOSING or
// PENDING_CLOSE
LOG.info(Skip assigning region + rit.toString());
} else if (addressFromAM != null
!addressFromAM.equals(this.serverName)) {
LOG.debug(Skip assigning region
+ e.getKey().getRegionNameAsString()
+ because it has been opened in
+ addressFromAM.getServerName());
}
{code}
In ServerShutDownHandler we try to get the address in the AM. This address
is initially null because it is not yet updated after the region was opened
.i.e. the CAll back after node deletion is not yet done in the master side.
But removal from RIT is completed on the master side. So this will trigger a
new assignment.
So there is a small window between the online region is actually added in to
the online list and the ServerShutdownHandler where we check the existing
address in AM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5009) Failure of creating split dir if it already exists prevents splits from happening further
[
https://issues.apache.org/jira/browse/HBASE-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5009:
--
Status: Open (was: Patch Available)
Cancelling and resubmitting as the tests hanged
Failure of creating split dir if it already exists prevents splits from
happening further
-
Key: HBASE-5009
URL: https://issues.apache.org/jira/browse/HBASE-5009
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5009.patch, HBASE-5009_Branch90.patch
The scenario is
- The split of a region takes a long time
- The deletion of the splitDir fails due to HDFS problems.
- Subsequent splits also fail after that.
{code}
private static void createSplitDir(final FileSystem fs, final Path splitdir)
throws IOException {
if (fs.exists(splitdir)) throw new IOException(Splitdir already exits?
+ splitdir);
if (!fs.mkdirs(splitdir)) throw new IOException(Failed create of +
splitdir);
}
{code}
Correct me if am wrong? If it is an issue can we change the behaviour of
throwing exception?
Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5009) Failure of creating split dir if it already exists prevents splits from happening further
[
https://issues.apache.org/jira/browse/HBASE-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5009:
--
Fix Version/s: 0.90.6
0.92.1
Affects Version/s: (was: 0.90.6)
Status: Patch Available (was: Open)
Failure of creating split dir if it already exists prevents splits from
happening further
-
Key: HBASE-5009
URL: https://issues.apache.org/jira/browse/HBASE-5009
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5009.patch, HBASE-5009_Branch90.patch
The scenario is
- The split of a region takes a long time
- The deletion of the splitDir fails due to HDFS problems.
- Subsequent splits also fail after that.
{code}
private static void createSplitDir(final FileSystem fs, final Path splitdir)
throws IOException {
if (fs.exists(splitdir)) throw new IOException(Splitdir already exits?
+ splitdir);
if (!fs.mkdirs(splitdir)) throw new IOException(Failed create of +
splitdir);
}
{code}
Correct me if am wrong? If it is an issue can we change the behaviour of
throwing exception?
Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[ https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5094: -- Description: R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2 even though .META. indicates it is on RS3. 1) Region R1 - Assigned from RS1 to RS2. 2) RS1 goes down and ServerShutDownHandler. ServerShutDwonHandler finds RS1 with R1 from META as still META is not yet updated to RS2. 3) As RS1 goes down R1 is assigned from RS1 to RS3. 4) RS3 goes down. ServerShutdownHandler processes R1 and tries to assign it to RS2. 5) RS2 says ALREADY_OPENED but META shows RS3. I was able to reproduce the scenario. was: R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2 even though .META. indicates it is on RS3. T1: Load balancer tried to move R1 from RS1 to RS2 . 2011-11-21 14:03:20,812 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df., src=skynet-1,60020,1321912978281, dest=skynet-4,60020,1321912999305 T2: RS1 shutdown. 2011-11-21 14:03:24,759 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=skynet-1,60020,1321912978281 to dead servers, submitted shutdown handler to be executed, root=false, meta=true T3:R1 is opened on RS2. 2011-11-21 14:03:26,131 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The master has opened the region tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df. that was online on skynet-4,60020,1321912999305 T4: As part of RS1 shutdown handling, region reassignment starts. It uses the region location captured at T2. 2011-11-21 14:03:26,152 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 32 region(s) that skynet-1,60020,1321912978281 was carrying (skipping 0 regions(s) that are already in transition) T5: R1 is assigned to RS3. 2011-11-21 14:03:27,404 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:6-0x133b84f9f49 Retrieved 115 byte(s) of data from znode /hbase/unassigned/ee2e205a60f1bb06cc73bc9df06289df; data=region=tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df., origin=skynet-3,60020,1321912991430, state=RS_ZK_REGION_OPENED T6: RS3 shutdown. R1 is reassigned to RS2. 2011-11-21 14:03:37,899 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: ALREADY_OPENED region tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df. to skynet-4,60020,1321912999305 From AssignmentManager point of view, the R1 is assigned to RS2. The .META. table indicates the location is RS3. The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible. Key: HBASE-5094 URL: https://issues.apache.org/jira/browse/HBASE-5094 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2 even though .META. indicates it is on RS3. 1) Region R1 - Assigned from RS1 to RS2. 2) RS1 goes down and ServerShutDownHandler. ServerShutDwonHandler finds RS1 with R1 from META as still META is not yet updated to RS2. 3) As RS1 goes down R1 is assigned from RS1 to RS3. 4) RS3 goes down. ServerShutdownHandler processes R1 and tries to assign it to RS2. 5) RS2 says ALREADY_OPENED but META shows RS3. I was able to reproduce the scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[ https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5094: -- Description: R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2 even though .META. indicates it is on RS3. 1) Region R1 - Assigned from RS1 to RS2. 2) RS1 goes down and ServerShutDownHandler. ServerShutDwonHandler finds R1 with RS1 from META as still META is not yet updated to RS2. 3) As RS1 goes down R1 is assigned from RS1 to RS3. 4) RS3 goes down. ServerShutdownHandler processes R1 and tries to assign it to RS2. 5) RS2 says ALREADY_OPENED but META shows RS3. I was able to reproduce the scenario in 0.92 was: R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2 even though .META. indicates it is on RS3. 1) Region R1 - Assigned from RS1 to RS2. 2) RS1 goes down and ServerShutDownHandler. ServerShutDwonHandler finds RS1 with R1 from META as still META is not yet updated to RS2. 3) As RS1 goes down R1 is assigned from RS1 to RS3. 4) RS3 goes down. ServerShutdownHandler processes R1 and tries to assign it to RS2. 5) RS2 says ALREADY_OPENED but META shows RS3. I was able to reproduce the scenario. The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible. Key: HBASE-5094 URL: https://issues.apache.org/jira/browse/HBASE-5094 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2 even though .META. indicates it is on RS3. 1) Region R1 - Assigned from RS1 to RS2. 2) RS1 goes down and ServerShutDownHandler. ServerShutDwonHandler finds R1 with RS1 from META as still META is not yet updated to RS2. 3) As RS1 goes down R1 is assigned from RS1 to RS3. 4) RS3 goes down. ServerShutdownHandler processes R1 and tries to assign it to RS2. 5) RS2 says ALREADY_OPENED but META shows RS3. I was able to reproduce the scenario in 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[
https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5094:
--
Comment: was deleted
(was: Can we change the position removing RIT and adding to Region set?)
The META can hold an entry for a region with a different server name from the
one actually in the AssignmentManager thus making the region inaccessible.
Key: HBASE-5094
URL: https://issues.apache.org/jira/browse/HBASE-5094
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: ramkrishna.s.vasudevan
{code}
RegionState rit =
this.services.getAssignmentManager().isRegionInTransition(e.getKey());
ServerName addressFromAM = this.services.getAssignmentManager()
.getRegionServerOfRegion(e.getKey());
if (rit != null !rit.isClosing() !rit.isPendingClose()) {
// Skip regions that were in transition unless CLOSING or
// PENDING_CLOSE
LOG.info(Skip assigning region + rit.toString());
} else if (addressFromAM != null
!addressFromAM.equals(this.serverName)) {
LOG.debug(Skip assigning region
+ e.getKey().getRegionNameAsString()
+ because it has been opened in
+ addressFromAM.getServerName());
}
{code}
In ServerShutDownHandler we try to get the address in the AM. This address
is initially null because it is not yet updated after the region was opened
.i.e. the CAll back after node deletion is not yet done in the master side.
But removal from RIT is completed on the master side. So this will trigger a
new assignment.
So there is a small window between the online region is actually added in to
the online list and the ServerShutdownHandler where we check the existing
address in AM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5097) Coprocessor RegionObserver implementation without preScannerOpen and postScannerOpen Impl is throwing NPE and so failing the system initialization.
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Component/s: coprocessors
Coprocessor RegionObserver implementation without preScannerOpen and
postScannerOpen Impl is throwing NPE and so failing the system initialization.
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
Priority: Blocker
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5097) Coprocessor RegionObserver implementation without preScannerOpen and postScannerOpen Impl is throwing NPE and so failing the system initialization.
[
https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5097:
--
Priority: Major (was: Blocker)
Changing the priority.
Coprocessor RegionObserver implementation without preScannerOpen and
postScannerOpen Impl is throwing NPE and so failing the system initialization.
---
Key: HBASE-5097
URL: https://issues.apache.org/jira/browse/HBASE-5097
Project: HBase
Issue Type: Bug
Components: coprocessors
Reporter: ramkrishna.s.vasudevan
In HRegionServer.java openScanner()
{code}
r.prepareScanner(scan);
RegionScanner s = null;
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().preScannerOpen(scan);
}
if (s == null) {
s = r.getScanner(scan);
}
if (r.getCoprocessorHost() != null) {
s = r.getCoprocessorHost().postScannerOpen(scan, s);
}
{code}
If we dont have implemention for postScannerOpen the RegionScanner is null
and so throwing nullpointer
{code}
java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
{code}
Making this defect as blocker.. Pls feel free to change the priority if am
wrong. Also correct me if my way of trying out coprocessors without
implementing postScannerOpen is wrong. Am just a learner.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
[ https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5094: -- Summary: The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible. (was: The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region unaccessible.) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible. Key: HBASE-5094 URL: https://issues.apache.org/jira/browse/HBASE-5094 Project: HBase Issue Type: Bug Reporter: Ted Yu R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2 by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus the region is considered assigned to RS2 even though .META. indicates it is on RS3. T1: Load balancer tried to move R1 from RS1 to RS2 . 2011-11-21 14:03:20,812 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df., src=skynet-1,60020,1321912978281, dest=skynet-4,60020,1321912999305 T2: RS1 shutdown. 2011-11-21 14:03:24,759 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=skynet-1,60020,1321912978281 to dead servers, submitted shutdown handler to be executed, root=false, meta=true T3:R1 is opened on RS2. 2011-11-21 14:03:26,131 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The master has opened the region tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df. that was online on skynet-4,60020,1321912999305 T4: As part of RS1 shutdown handling, region reassignment starts. It uses the region location captured at T2. 2011-11-21 14:03:26,152 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 32 region(s) that skynet-1,60020,1321912978281 was carrying (skipping 0 regions(s) that are already in transition) T5: R1 is assigned to RS3. 2011-11-21 14:03:27,404 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:6-0x133b84f9f49 Retrieved 115 byte(s) of data from znode /hbase/unassigned/ee2e205a60f1bb06cc73bc9df06289df; data=region=tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df., origin=skynet-3,60020,1321912991430, state=RS_ZK_REGION_OPENED T6: RS3 shutdown. R1 is reassigned to RS2. 2011-11-21 14:03:37,899 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: ALREADY_OPENED region tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df. to skynet-4,60020,1321912999305 From AssignmentManager point of view, the R1 is assigned to RS2. The .META. table indicates the location is RS3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5092) Two adjacent assignments lead region is in PENDING_OPEN state and block table disable and enable actions.
[
https://issues.apache.org/jira/browse/HBASE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5092:
--
Comment: was deleted
(was: @Liu
Can we handle RIT exception also to retry the assignment? What do you think?)
Two adjacent assignments lead region is in PENDING_OPEN state and block table
disable and enable actions.
-
Key: HBASE-5092
URL: https://issues.apache.org/jira/browse/HBASE-5092
Project: HBase
Issue Type: Bug
Components: master, regionserver
Affects Versions: 0.92.0
Reporter: Liu Jia
Assignee: Liu Jia
Attachments: unhandled_PENDING_OPEN_lead_by_two_assignment.patch
Region is in PENDING_OPEN state and disable and enable are blocked.
We occasionally find if two assignments which have a short interval time will
lead to a PENDING_OPEN state staying in the regionInTransition map and
blocking the disable and enable table actions.
We found that the second assignment will set the zknode of this region to
M_ZK_REGION_OFFLINE then set the state in assignmentMananger's
regionInTransition map to PENDING_OPEN and abort its further operation
because of finding the the region is already in the regionserver by a
RegionAlreadyInTransitionException.
At the same time the first assignment is tickleOpening and find the version
of the zknode is messed up by the second assignment, so the
OpenRegionHandler print out the following two lines:
{noformat}
2011-12-23 22:12:15,197 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0]
zookeeper.ZKAssign(788): regionserver:59892-0x1346b43b91e0002 Attempt to
transition the unassigned node for 15237599c632752b8cfd3d5a86349768 from
RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING failed, the node existed but was
version 2 not the expected version 1
2011-12-23 22:12:15,197 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0]
handler.OpenRegionHandler(403): Failed refreshing OPENING;
region=15237599c632752b8cfd3d5a86349768, context=post_region_open
{noformat}
After that it tries to turn the state to FAILED_OPEN, but also failed due to
wrong version,
this is the output:
{noformat}
2011-12-23 22:12:15,199 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0]
zookeeper.ZKAssign(812): regionserver:59892-0x1346b43b91e0002 Attempt to
transition the unassigned node for 15237599c632752b8cfd3d5a86349768 from
RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN failed, the node existed but
was in the state M_ZK_REGION_OFFLINE set by the server
data16,59892,1324649528415
2011-12-23 22:12:15,199 WARN [RS_OPEN_REGION-data16,59892,1324649528415-0]
handler.OpenRegionHandler(307): Unable to mark region {NAME =
'table1,,1324649533045.15237599c632752b8cfd3d5a86349768.', STARTKEY = '',
ENDKEY = '', ENCODED = 15237599c632752b8cfd3d5a86349768,} as FAILED_OPEN.
It's likely that the master already timed out this open attempt, and thus
another RS already has the region.
{noformat}
So after all that, the PENDING_OPEN state is left in the assignmentMananger's
regionInTransition map and none will deal with it further,
This kind of situation will wait until the master find the state out of time.
The following is the test code:
{code:title=test.java|borderStyle=solid}
@Test
public void testDisableTables() throws IOException {
for (int i = 0; i 20; i++) {
HTableDescriptor des = admin.getTableDescriptor(Bytes.toBytes(table1));
ListHRegionInfo hris = TEST_UTIL.getHBaseCluster().getMaster()
.getAssignmentManager().getRegionsOfTable(Bytes.toBytes(table1));
TEST_UTIL.getHBaseCluster().getMaster()
.assign(hris.get(0).getRegionName());
TEST_UTIL.getHBaseCluster().getMaster()
.assign(hris.get(0).getRegionName());
admin.disableTable(Bytes.toBytes(table1));
admin.modifyTable(Bytes.toBytes(table1), des);
admin.enableTable(Bytes.toBytes(table1));
}
}
{code}
To fix this,we add a line to
public static int ZKAssign.transitionNode() to make
endState.RS_ZK_REGION_FAILED_OPEN transition pass.
{code:title=ZKAssign.java|borderStyle=solid}
if((!existingData.getEventType().equals(beginState))
//add the following line to make endState.RS_ZK_REGION_FAILED_OPEN
transition pass.
(!endState.equals(endState.RS_ZK_REGION_FAILED_OPEN))) {
LOG.warn(zkw.prefix(Attempt to transition the +
unassigned node for + encoded +
from + beginState + to + endState + failed, +
the node existed but was in the state +
existingData.getEventType() +
set by the server + serverName));
return -1;
}
{code}
Run the test case again we found that before the first
[jira] [Updated] (HBASE-4862) Splitting hlog and opening region concurrently may cause data loss
[ https://issues.apache.org/jira/browse/HBASE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4862: -- Fix Version/s: 0.92.0 Splitting hlog and opening region concurrently may cause data loss -- Key: HBASE-4862 URL: https://issues.apache.org/jira/browse/HBASE-4862 Project: HBase Issue Type: Bug Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.90.5 Attachments: 4862-0.92.txt, 4862-v6-90.txt, 4862-v6-trunk.patch, 4862.patch, 4862.txt, hbase-4862v1 for 0.90.diff, hbase-4862v1 for 0.90.diff, hbase-4862v1 for trunk.diff, hbase-4862v1 for trunk.diff, hbase-4862v2for0.90.diff, hbase-4862v2fortrunk.diff, hbase-4862v3for0.90.diff, hbase-4862v3fortrunk.diff, hbase-4862v5for0.90.diff, hbase-4862v5fortrunk.diff, hbase-4862v7for0.90.patch, hbase-4862v7fortrunk.patch Case Description: 1.Split hlog thread creat writer for the file region A/recoverd.edits/123456 and is appending log entry 2.Regionserver is opening region A now, and in the process replayRecoveredEditsIfAny() ,it will delete the file region A/recoverd.edits/123456 3.Split hlog thread catches the io exception, and stop parse this log file and if skipError = true , add it to the corrupt logsHowever, data in other regions in this log file will loss 4.Or if skipError = false, it will check filesystem.Of course, the file system is ok , and it only prints a error log, continue assigning regions. Therefore, data in other log files will also loss!! The case may happen in the following: 1.Move region from server A to server B 2.kill server A and Server B 3.restart server A and Server B We could prevent this exception throuth forbiding deleting recover.edits file which is appending by split hlog thread -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4951) master process can not be stopped when it is initializing
[ https://issues.apache.org/jira/browse/HBASE-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4951: -- Attachment: HBASE-4951_branch.patch Patch for branch0.90 master process can not be stopped when it is initializing - Key: HBASE-4951 URL: https://issues.apache.org/jira/browse/HBASE-4951 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.3 Reporter: xufeng Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.6 Attachments: HBASE-4951.patch, HBASE-4951_branch.patch It is easy to reproduce by following step: step1:start master process.(do not start regionserver process in the cluster). the master will wait the regionserver to check in: org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin step2:stop the master by sh command bin/hbase master stop result:the master process will never die because catalogTracker.waitForRoot() method will block unitl the root region assigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5073) Registered listeners not getting removed leading to memory leak in HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5073: -- Attachment: HBASE-5073.patch For branch patch Registered listeners not getting removed leading to memory leak in HBaseAdmin - Key: HBASE-5073 URL: https://issues.apache.org/jira/browse/HBASE-5073 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.5 Attachments: HBASE-5073.patch HBaseAdmin apis like tableExists(), flush, split, closeRegion uses catalog tracker. Every time Root node tracker and meta node tracker are started and a listener is registered. But after the operations are performed the listeners are not getting removed. Hence if the admin apis are consistently used then it may lead to memory leak. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5009) Failure of creating split dir if it already exists prevents splits from happening further
[
https://issues.apache.org/jira/browse/HBASE-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5009:
--
Attachment: HBASE-5009_Branch90.patch
Patch for 0.90
Failure of creating split dir if it already exists prevents splits from
happening further
-
Key: HBASE-5009
URL: https://issues.apache.org/jira/browse/HBASE-5009
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5009_Branch90.patch
The scenario is
- The split of a region takes a long time
- The deletion of the splitDir fails due to HDFS problems.
- Subsequent splits also fail after that.
{code}
private static void createSplitDir(final FileSystem fs, final Path splitdir)
throws IOException {
if (fs.exists(splitdir)) throw new IOException(Splitdir already exits?
+ splitdir);
if (!fs.mkdirs(splitdir)) throw new IOException(Failed create of +
splitdir);
}
{code}
Correct me if am wrong? If it is an issue can we change the behaviour of
throwing exception?
Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5009) Failure of creating split dir if it already exists prevents splits from happening further
[
https://issues.apache.org/jira/browse/HBASE-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5009:
--
Attachment: HBASE-5009.patch
Patch for trunk
Failure of creating split dir if it already exists prevents splits from
happening further
-
Key: HBASE-5009
URL: https://issues.apache.org/jira/browse/HBASE-5009
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-5009.patch, HBASE-5009_Branch90.patch
The scenario is
- The split of a region takes a long time
- The deletion of the splitDir fails due to HDFS problems.
- Subsequent splits also fail after that.
{code}
private static void createSplitDir(final FileSystem fs, final Path splitdir)
throws IOException {
if (fs.exists(splitdir)) throw new IOException(Splitdir already exits?
+ splitdir);
if (!fs.mkdirs(splitdir)) throw new IOException(Failed create of +
splitdir);
}
{code}
Correct me if am wrong? If it is an issue can we change the behaviour of
throwing exception?
Pls suggest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4951) master process can not be stopped when it is initializing
[ https://issues.apache.org/jira/browse/HBASE-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4951: -- Attachment: HBASE-4951.patch This is for trunk. master process can not be stopped when it is initializing - Key: HBASE-4951 URL: https://issues.apache.org/jira/browse/HBASE-4951 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.3 Reporter: xufeng Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.6 Attachments: HBASE-4951.patch It is easy to reproduce by following step: step1:start master process.(do not start regionserver process in the cluster). the master will wait the regionserver to check in: org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin step2:stop the master by sh command bin/hbase master stop result:the master process will never die because catalogTracker.waitForRoot() method will block unitl the root region assigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4878) Master crash when spliting hlog may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-4878:
--
Assignee: chunhui shen
Affects Version/s: 0.92.0
Status: Patch Available (was: Open)
Master crash when spliting hlog may cause data loss
---
Key: HBASE-4878
URL: https://issues.apache.org/jira/browse/HBASE-4878
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: chunhui shen
Assignee: chunhui shen
Attachments: hbase-4878.diff, hbase-4878v2.patch
Let's see the code of HlogSplitter#splitLog(final FileStatus[] logfiles)
{code}
private ListPath splitLog(final FileStatus[] logfiles) throws IOException {
try {
for (FileStatus log : logfiles) {
parseHLog(in, logPath, entryBuffers, fs, conf, skipErrors);
}
archiveLogs(srcDir, corruptedLogs, processedLogs, oldLogDir, fs, conf);
} finally {
status.setStatus(Finishing writing output logs and closing down.);
splits = outputSink.finishWritingAndClose();
}
}
{code}
If master is killed, after finishing archiveLogs(srcDir, corruptedLogs,
processedLogs, oldLogDir, fs, conf),
but before finishing splits = outputSink.finishWritingAndClose();
Log date would loss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4855) SplitLogManager hangs on cluster restart.
[ https://issues.apache.org/jira/browse/HBASE-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4855: -- Affects Version/s: 0.92.0 Fix Version/s: 0.92.0 SplitLogManager hangs on cluster restart. -- Key: HBASE-4855 URL: https://issues.apache.org/jira/browse/HBASE-4855 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.0 Start a master and RS RS goes down (kill -9) Wait for ServerShutDownHandler to create the splitlog nodes. As no RS is there it cannot be processed. Restart both master and bring up an RS. The master hangs in SplitLogManager.waitforTasks(). I feel that batch.done is not getting incremented properly. Not yet digged in fully. This may be the reason for occasional failure of TestDistributedLogSplitting.testWorkerAbort(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4855) SplitLogManager hangs on cluster restart.
[ https://issues.apache.org/jira/browse/HBASE-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4855: -- Fix Version/s: (was: 0.92.0) SplitLogManager hangs on cluster restart. -- Key: HBASE-4855 URL: https://issues.apache.org/jira/browse/HBASE-4855 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Start a master and RS RS goes down (kill -9) Wait for ServerShutDownHandler to create the splitlog nodes. As no RS is there it cannot be processed. Restart both master and bring up an RS. The master hangs in SplitLogManager.waitforTasks(). I feel that batch.done is not getting incremented properly. Not yet digged in fully. This may be the reason for occasional failure of TestDistributedLogSplitting.testWorkerAbort(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
