[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17956023#comment-17956023
]
Istvan Toth commented on HBASE-28665:
-
Thank you.
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17956021#comment-17956021
]
Kiran Kumar Maturi commented on HBASE-28665:
[~stoty] [~vjasani] the master branch and branch-3
[https://github.com/apache/hbase/blob/branch-3/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java]
have different WAL implementation compared to branch-2 so we have not
backported it to branch-3 /master
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17955808#comment-17955808
]
Viraj Jasani commented on HBASE-28665:
--
Yes, master/branch-3 are not affected IIRC. Correct [~kiran.maturi]?
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17953320#comment-17953320
]
Istvan Toth commented on HBASE-28665:
-
[~kiran.maturi] [~vjasani]
I can see that this only on branch-2.
Is master/branch-3 not affected by this issues ?
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865305#comment-17865305
]
Hudson commented on HBASE-28665:
Results for branch branch-2.6
[build #163 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/163/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/163/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/163/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/163/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/163/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/163/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865219#comment-17865219
]
Hudson commented on HBASE-28665:
Results for branch branch-2.5
[build #565 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/565/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/565/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/565/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/565/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/565/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/565/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865017#comment-17865017
]
Kiran Kumar Maturi commented on HBASE-28665:
Thanks [~vjasani] for the the review
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863441#comment-17863441
]
Hudson commented on HBASE-28665:
Results for branch branch-2
[build #1097 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1097/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1097/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1097/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1097/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1097/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1097/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862531#comment-17862531
]
Kiran Kumar Maturi commented on HBASE-28665:
[~vjasani] created a pull request for branch-2 as well. Please review
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
> Labels: pull-request-available
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-28665) WALs not marked closed when there are errors in closing WALs
[
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855013#comment-17855013
]
Kiran Kumar Maturi commented on HBASE-28665:
We can add a check here if there are no unflushed entries then to allow the wal
to be cleaned up
{code:java}
if (isUnflushedEntries() || closeErrorCount.get() >= this.closeErrorsTolerated)
{
try {
closeWriter(this.writer, oldPath, true);
} finally {
if(!isUnflushedEntries()){
markClosedAndClean(oldPath)
}
inflightWALClosures.remove(oldPath.getName());
}
}
{code}
> WALs not marked closed when there are errors in closing WALs
>
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
>
> In our production clusters we have observed that when WAL close fails It
> causes the the oldWAL files not marked as close and not letting them cleaned.
> When a WAL close fails in closeWriter it increments the error count.
> {code:java}
> Span span = Span.current();
> try {
> span.addEvent("closing writer");
> writer.close();
> span.addEvent("writer closed");
> } catch (IOException ioe) {
> int errors = closeErrorCount.incrementAndGet();
> boolean hasUnflushedEntries = isUnflushedEntries();
> if (syncCloseCall && (hasUnflushedEntries || (errors >
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" +
> ioe.getMessage() + "\", errors="
> + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
> }
> LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK",
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> }
> }
> {code}
> as we don't mark them closed here like we do it here
>
> {code:java}
> Writer localWriter = this.writer;
> closeExecutor.execute(() -> {
> try {
> closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
> LOG.warn("close old writer failed", e);
> } finally {
> // call this even if the above close fails, as there is no
> other chance we can set
> // closed to true, it will not cause big problems.
> {color:red} markClosedAndClean(oldPath);{color}
> inflightWALClosures.remove(oldPath.getName());
> }
> });
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
