Jingxuan Fu created HBASE-27341:
-----------------------------------
Summary: when “hbase.regionserver.logroll.period” is set to a
negative number, Log cannot run normally
Key: HBASE-27341
URL: https://issues.apache.org/jira/browse/HBASE-27341
Project: HBase
Issue Type: Bug
Reporter: Jingxuan Fu
Assignee: Jingxuan Fu
In hbase- default.xml
<property>
<name>hbase.regionserver.logroll.period</name>
<value>3600000</value>
<description>Period at which we will roll the commit log regardless
of how many edits it has.</description>
</property>
Taking hbase.regionserver.logroll.period set to -1 in hbase-site.xml as an
example, it is normal to start hbase.
However, {*}there are a large number of repeated wal rolling operation records
in the log file{*}.
{code:java}
2022-08-06 15:39:37,172 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Rolled WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577156
with entries=0, filesize=83 B; new WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167
2022-08-06 15:39:37,172 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Archiving
file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577156
to
file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.1659771577156
2022-08-06 15:39:37,177 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Rolled WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577160.meta
with entries=0, filesize=83 B; new WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta
2022-08-06 15:39:37,177 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Archiving
file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577160.meta
to
file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577160.meta
2022-08-06 15:39:37,182 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Rolled WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167
with entries=0, filesize=83 B; new WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177
2022-08-06 15:39:37,182 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Archiving
file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167
to
file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.1659771577167
2022-08-06 15:39:37,188 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Rolled WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta
with entries=0, filesize=83 B; new WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta
2022-08-06 15:39:37,188 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Archiving
file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta
to
file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577172.meta
2022-08-06 15:39:37,196 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Rolled WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177
with entries=0, filesize=83 B; new WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577188
2022-08-06 15:39:37,197 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Archiving
file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177
to
file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.1659771577177
2022-08-06 15:39:37,201 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Rolled WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta
with entries=0, filesize=83 B; new WAL
/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577197.meta
2022-08-06 15:39:37,201 INFO [regionserver/xmw1-kb310-65:16020.logRoller]
wal.AbstractFSWAL: Archiving
file:/home/xmw/hbase-2.2.2/hbase-tmp/WALs/xmw1-kb310-65,16020,1659771570212/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta
to
file:/home/xmw/hbase-2.2.2/hbase-tmp/oldWALs/xmw1-kb310-65%2C16020%2C1659771570212.meta.1659771577182.meta{code}
When using the service, it works fine and close normally.
According to the above phenomenon, we believe that the phenomenon is related to
the WAL rolling of regionserver. After code review, we found that in the
constructor of
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java.
After reading the configuration item hbase.regionserver.logroll.period, {*}it
is not checked whether the value is normal{*}. In the subsequent run()
function, the current time is directly compared to the last rollback time with
hbase.regionserver.logroll.period to determine whether to perform the WAL
rollback operation, so that no matter what the current time is, The rollback
operation will be performed, {*}resulting in a large number of records in the
LOG{*}, *and it is very likely to occupy too many resources. And no prompt
message appears to remind us of the problem.*
Therefore, we propose to {*}add the judgment of
hbase.regionserver.logroll.period on the constructor of LogRoller{*}. If the
parameter LogRoller.rollPeriod affected by this configuration item is less than
zero, an error message will be output and he relevant exception can be thrown.
This way, if the hbase.regiononserver.logroll.period is less than 0, an
exception can be thrown and an error message logged, stopping the log from
having a large number of duplicate wal roll operation records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
