[ https://issues.apache.org/jira/browse/HBASE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wellington Chevreuil resolved HBASE-28004. ------------------------------------------ Resolution: Fixed > Persistent cache map can get corrupt if crash happens midway through the write > ------------------------------------------------------------------------------ > > Key: HBASE-28004 > URL: https://issues.apache.org/jira/browse/HBASE-28004 > Project: HBase > Issue Type: Sub-task > Affects Versions: 2.6.0, 3.0.0-alpha-4, 4.0.0-alpha-1 > Reporter: Wellington Chevreuil > Assignee: Wellington Chevreuil > Priority: Major > Fix For: 2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1 > > > HBASE-27686 added a background thread for periodically saving the cache index > map, together with a list of completed cached files so that we can recover > the cache state in case of crash or restart. Problem is that the cache index > can become few GB large (a sample case with 1.6TB of used bucket cache would > map to between 8GB to 10GB indexes), and these writes take few seconds to > complete, causing any RS crash very likely to leave a corrupt index file that > can't be recovered when the RS starts again. Worse, since we store the list > of cached files on a separate file, this also leads to cache inconsistencies, > with files in the list of cached files never cached once the RS is restarted, > even though we have no cache index for those and every read ends up going to > the FS. > This task aims to refactor the cache persistent as follows: > 1) Write both the list of completely cached files and the cache indexes in a > single file, so that we can have this synced atomically; > 2) When writing the persistent cache file, use a temp name first, then once > the write is successfully finished, rename it to the actual name. This way, > if crash happens whilst the persistent cache is still being written, the temp > file would be corrupt, but we could still recover from the last successful > sync, and we would only lose the caching ops since the last sync. -- This message was sent by Atlassian Jira (v8.20.10#820010)