[ https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mallikarjun updated HBASE-25891: -------------------------------- Description: Currently WAL logs are stored in `backup:system` meta table {code:java} // code placeholder wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId, timestamp=1622003479895, value=backup_1622003358258 wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file, timestamp=1622003479895, value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175 wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root, timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId, timestamp=1622003479895, value=backup_1622003358258 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file, timestamp=1622003479895, value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root, timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 {code} This has several problems # Ever growing rows of wal's sourced for incremental backup is maintained and never cleaned up. # Unnecessary to have wal log listed for performing incremental backup or for performing logcleaner. # No support for rsgroup. Hence all rsgroups which doesn't have backup enabled tables, WAL's are retained forever. was: Currently WAL logs are stored in `backup:system` meta table {code:java} // code placeholder wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId, timestamp=1622003479895, value=backup_1622003358258 wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file, timestamp=1622003479895, value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175 wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root, timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId, timestamp=1622003479895, value=backup_1622003358258 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file, timestamp=1622003479895, value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280 wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root, timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 {code} This has several problems # Ever growing rows of wal's sourced for incremental backup is maintained and never cleaned up. # Unnecessary to have wal log listed for performing incremental backup or for performing logcleaner. # No support for rsgroup. Hence all rsgroups which doesn't have backup enabled, WAL's are retained forever. > Remove dependence storing WAL filenames for backup > -------------------------------------------------- > > Key: HBASE-25891 > URL: https://issues.apache.org/jira/browse/HBASE-25891 > Project: HBase > Issue Type: Improvement > Components: backup&restore > Reporter: Mallikarjun > Assignee: Mallikarjun > Priority: Major > > Currently WAL logs are stored in `backup:system` meta table > {code:java} > // code placeholder > wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId, > timestamp=1622003479895, value=backup_1622003358258 > wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file, > timestamp=1622003479895, > value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175 > wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root, > timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 > wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId, > timestamp=1622003479895, value=backup_1622003358258 > wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file, > timestamp=1622003479895, > value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280 > wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root, > timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1 > {code} > > This has several problems > # Ever growing rows of wal's sourced for incremental backup is maintained > and never cleaned up. > # Unnecessary to have wal log listed for performing incremental backup or > for performing logcleaner. > # No support for rsgroup. Hence all rsgroups which doesn't have backup > enabled tables, WAL's are retained forever. > -- This message was sent by Atlassian Jira (v8.3.4#803005)