[ https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590904#comment-16590904 ]
Ted Yu commented on HBASE-21098: -------------------------------- {code} public static final String SNAPSHOT_WORKING_DIR = "hbase.snapshot.working.dir"; {code} Is there recommendation on where the working directory should reside ? > Improve Snapshot Performance with Temporary Snapshot Directory when rootDir > on S3 > --------------------------------------------------------------------------------- > > Key: HBASE-21098 > URL: https://issues.apache.org/jira/browse/HBASE-21098 > Project: HBase > Issue Type: Improvement > Affects Versions: 3.0.0, 2.1.1 > Reporter: Tyler Mi > Priority: Major > Attachments: HBASE-21098.master.001.patch, > HBASE-21098.master.002.patch, HBASE-21098.master.003.patch > > > When using Apache HBase, the snapshot feature can be used to make a point in > time recovery. To do this, HBase creates a manifest of all the files in all > of the Regions so that those files can be referenced again when a user > restores a snapshot. With HBase storage mode S3, developers can store their > data off-cluster in Amazon S3. However, utilizing S3 as a FileSystem is > inefficient in some operations, namely renames. Most Hadoop ecosystem > applications use an atomic rename as a method of committing data. However, > with S3, a rename is a separate copy and then a delete of every file which is > no longer atomic and, in fact, quite costly. In addition, puts and deletes on > S3 have latency issues that traditional filesystems do not encounter when > manipulating the region snapshots. When HBase on S3 customers have a > significant amount of regions, puts, deletes, and renames (the final commit > stage of the snapshot) become the bottleneck causing snapshots to take many > minutes or even hours to complete. > The purpose of this patch is to increase the overall performance of snapshots > while utilizing HBase on S3 through the use of a temporary directory for the > snapshots that exists on a traditional filesystem to circumvent the > bottlenecks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)