[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191359#comment-13191359
]
Hudson commented on HBASE-5235:
---
Integrated in HBase-0.92 #257 (See
[https://builds.apache.org/job/HBase-0.92/257/])
HBASE-5235 HLogSplitter writer thread's streams not getting closed when any
of the writer threads has exceptions. (Ram)
ramkrishna :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5, 0.92.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.90.6, 0.92.1
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_0.90_2.patch, HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191986#comment-13191986
]
Hudson commented on HBASE-5235:
---
Integrated in HBase-0.92-security #88 (See
[https://builds.apache.org/job/HBase-0.92-security/88/])
HBASE-5235 HLogSplitter writer thread's streams not getting closed when any
of the writer threads has exceptions. (Ram)
ramkrishna :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.5, 0.92.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.90.6, 0.92.1
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_0.90_2.patch, HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190666#comment-13190666
]
Hadoop QA commented on HBASE-5235:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511427/HBASE-5235_0.90_2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/833//console
This message is automatically generated.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_0.90_2.patch, HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190714#comment-13190714
]
Hudson commented on HBASE-5235:
---
Integrated in HBase-TRUNK #2644 (See
[https://builds.apache.org/job/HBase-TRUNK/2644/])
HBASE-5235 HLogSplitter writer thread's streams not getting closed when any
of the writer threads has exceptions.(Ram)
ramkrishna :
Files :
* /hbase/trunk/CHANGES.txt
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_0.90_2.patch, HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190883#comment-13190883
]
Hudson commented on HBASE-5235:
---
Integrated in HBase-TRUNK-security #85 (See
[https://builds.apache.org/job/HBase-TRUNK-security/85/])
HBASE-5235 HLogSplitter writer thread's streams not getting closed when any
of the writer threads has exceptions.(Ram)
ramkrishna :
Files :
* /hbase/trunk/CHANGES.txt
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_0.90_2.patch, HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190438#comment-13190438
]
Zhihong Yu commented on HBASE-5235:
---
{code}
+ // close the log writer streams only if they are not closed
+ // in closeStreams().
+ if (!closeCompleted !logWritersClosed) {
{code}
Do we need to check closeCompleted here ? It is set after logWritersClosed is
set to true.
I think closeAndCleanupCompleted would be a better name for hasClosed.
The following line should be in closeLogWriters():
{code}
+ logWritersClosed = true;
{code}
If I were you, I would put the loop from line 789 to 798 into closeLogWriters()
and let closeStreams() call closeLogWriters()
Then closeLogWriters() would start for loop to iterate over logWriters.values()
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190439#comment-13190439
]
Zhihong Yu commented on HBASE-5235:
---
The rationale for my above comment is that in patch v1, the following
assignment may be skipped if there is runtime exception thrown in the try block
of line 801:
{code}
+ logWritersClosed = true;
{code}
It would make the code cleaner if logWritersClosed only reflects the status of
log writer closing.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190481#comment-13190481
]
Zhihong Yu commented on HBASE-5235:
---
+1 on patch v2.
Minor comments which can be addressed at time of commit:
{code}
+LOG.info(Closed path + wap.p + (wrote + wap.editsWritten
++ edits in + (wap.nanosSpent / 1000 / 1000) + ms));
{code}
I think the above should be inside closeLogWriters().
{code}
+ // close the log writer streams only if they are not closed
+ // in closeStreams().
{code}
Since closeLogWriters() is called inside closeStreams(), the above comment can
be removed.
Please run through test suite.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190502#comment-13190502
]
Hadoop QA commented on HBASE-5235:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511401/HBASE-5235_trunk.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 javadoc. The javadoc tool appears to have generated -145 warning
messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 82 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/827//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/827//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/827//console
This message is automatically generated.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Attachments: HBASE-5235_0.90.patch, HBASE-5235_0.90_1.patch,
HBASE-5235_trunk.patch
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189853#comment-13189853
]
ramkrishna.s.vasudevan commented on HBASE-5235:
---
I think if any errors we should only close the Streams and not call
closeStreams().
Because here we do lot of other steps that completes the log split process.
Also even if one WriterThread has an exception can we completely abort the
master as we did for any failures of splitLog()? Pls suggest.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189961#comment-13189961
]
Zhihong Yu commented on HBASE-5235:
---
closeStreams() records all the IOException's and throw a MultipleIOException
before exiting.
I think the simplest solution is to wrap closeStreams() in a finally block in
finishWritingAndClose()
Thanks for reporting this issue, Ram.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189969#comment-13189969
]
ramkrishna.s.vasudevan commented on HBASE-5235:
---
But Ted, in closeStreams() we also do the steps related to renaming the .temp
files in recovered.edits. So we should only do the
{code}
for (WriterAndPath wap : logWriters.values()) {
try {
wap.w.close();
} catch (IOException ioe) {
LOG.error(Couldn't close log at + wap.p, ioe);
thrown.add(ioe);
continue;
}
{code}
and make the master abort so that the subsequent split can parse the HLog.
Correct me if am wrong.
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13189987#comment-13189987
]
Zhihong Yu commented on HBASE-5235:
---
How about introducing a boolean, logWritersClosed (similar to hasClosed) ?
We separate the above for loop into a new private method called
closeLogWriters() where logWritersClosed is checked upon entry. After
successful execution logWritersClosed would be set to true before exit.
closeStreams() calls closeLogWriters().
We also place closeLogWriters() in the finally block of finishWritingAndClose().
Is the above close to what you were thinking, Ram ?
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5235) HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.
[
https://issues.apache.org/jira/browse/HBASE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190326#comment-13190326
]
ramkrishna.s.vasudevan commented on HBASE-5235:
---
Yes. This what i was thinking too. Will upload a patch
HLogSplitter writer thread's streams not getting closed when any of the
writer threads has exceptions.
--
Key: HBASE-5235
URL: https://issues.apache.org/jira/browse/HBASE-5235
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.92.1, 0.90.6
Pls find the analysis. Correct me if am wrong
{code}
2012-01-15 05:14:02,374 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: WriterThread-9 Got
while writing log entry to log
java.io.IOException: All datanodes 10.18.40.200:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3373)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2811)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3026)
{code}
Here we have an exception in one of the writer threads. If any exception we
try to hold it in an Atomic variable
{code}
private void writerThreadError(Throwable t) {
thrown.compareAndSet(null, t);
}
{code}
In the finally block of splitLog we try to close the streams.
{code}
for (WriterThread t: writerThreads) {
try {
t.join();
} catch (InterruptedException ie) {
throw new IOException(ie);
}
checkForErrors();
}
LOG.info(Split writers finished);
return closeStreams();
{code}
Inside checkForErrors
{code}
private void checkForErrors() throws IOException {
Throwable thrown = this.thrown.get();
if (thrown == null) return;
if (thrown instanceof IOException) {
throw (IOException)thrown;
} else {
throw new RuntimeException(thrown);
}
}
So once we throw the exception the DFSStreamer threads are not getting closed.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
