[
https://issues.apache.org/jira/browse/HBASE-28801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875354#comment-17875354
]
Kiran Kumar Maturi edited comment on HBASE-28801 at 8/21/24 6:00 AM:
---------------------------------------------------------------------
One thought I had is if we just mark the wal closed irrespective of the entries
and closure errors
{code:java}
if (isUnflushedEntries() || closeErrorCount.get() >= this.closeErrorsTolerated)
{
try {
closeWriter(this.writer, oldPath, true);
} finally {
inflightWALClosures.remove(oldPath.getName());
markClosedAndClean(oldPath);
}
{code}
My understanding of unflushed entries here is that the unsycned transactions
(which are there on the ring buffer) for which we have not sent a response yet
to the client will be lost if the server crashes.
With unflushed entries if we get an error during wal closure should that be
considered as data loss
If we are good with considering this is not a data loss scenario. We can
proceed with the above solution
was (Author: kiran.maturi):
One thought I had is if we just mark the wal closed irrespective of the entries
and closure errors
if (isUnflushedEntries() || closeErrorCount.get() >= this.closeErrorsTolerated)
{
try {
closeWriter(this.writer, oldPath, true);
} finally {
inflightWALClosures.remove(oldPath.getName());
markClosedAndClean(oldPath);
}
> WALs remain unclosed for long
> -----------------------------
>
> Key: HBASE-28801
> URL: https://issues.apache.org/jira/browse/HBASE-28801
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 2.5.6
> Reporter: Kiran Kumar Maturi
> Priority: Minor
>
> In our production fleet we have observed that WAL files are not cleaned up
> even when all the entries have been flushed. I have fixed the WAL close
> issue when there is issue with the wal closures as part of
> [HBASE-28665|https://issues.apache.org/jira/projects/HBASE/issues/HBASE-28665].
> There is a case in case of unflushed entried that can lead for the wal not
> be cleaned even after all the entries have been flushed
> [FSHLog.java
> |https://github.com/apache/hbase/blob/branch-2.6/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java#L388]
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >=
> this.closeErrorsTolerated) {
> try {
> closeWriter(this.writer, oldPath, true);
> } finally {
> inflightWALClosures.remove(oldPath.getName());
> if (!isUnflushedEntries()) {
> markClosedAndClean(oldPath);
> }
> }
> {code}
> If there are unflushed entries then wal will never be marked close and won't
> be cleaned further
> {code:java}
> private synchronized void cleanOldLogs() {
> List<Pair<Path, Long>> logsToArchive = null;
> // For each log file, look at its Map of regions to the highest sequence
> id; if all sequence ids
> // are older than what is currently in memory, the WAL can be GC'd.
> for (Map.Entry<Path, WALProps> e : this.walFile2Props.entrySet()) {
> if (!e.getValue().closed) {
> LOG.debug("{} is not closed yet, will try archiving it next time",
> e.getKey());
> continue;
> }
> Path log = e.getKey();
> Map<byte[], Long> sequenceNums =
> e.getValue().encodedName2HighestSequenceId;
> if (this.sequenceIdAccounting.areAllLower(sequenceNums)) {
> if (logsToArchive == null) {
> logsToArchive = new ArrayList<>();
> }
> logsToArchive.add(Pair.newPair(log, e.getValue().logSize));
> if (LOG.isTraceEnabled()) {
> LOG.trace("WAL file ready for archiving " + log);
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
