[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341446#comment-16341446 ] Ted Yu commented on HBASE-8963: --- There were different opinions above. If per-table hfile archive TTL is not favored, can I assume patch v9 is closer (sans table property) ? > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338985#comment-16338985 ] Jean-Marc Spaggiari commented on HBASE-8963: Folks, any idea if this will one day be done? Just got to drop a 400 000 HFiles table, it takes a while ;) Just want to skip this move... > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968218#comment-14968218 ] Enis Soztutar commented on HBASE-8963: -- Agreed with the comments above that per-table hfile archive TTL is hard to do and the benefits are not immediately clear to me. The current v10 patch does not respect the TTL time unless it is 0 which is very confusing. SKIP_ARCHIVE as a table property is debatable. Do we want to skip archiving hfiles for compacted files as well? If so, this is the only way. But if we want to skip archiving files on table deletion, then I prefer that we add a flag to the drop table command. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961395#comment-14961395 ] Ted Yu commented on HBASE-8963: --- Lars, Enis: Mind revisiting this issue ? Thanks > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950201#comment-14950201
]
Sebastian Cotarla commented on HBASE-8963:
--
I would like to help but I can't figure out which solution are you
implementing.
[~lhofhansl] initially proposed to "make the retention period configurable per
table, and have 0 indicate immediate deletion". I like the idea because from a
user point of view it is intuitive and consistent with
"hbase.master.hfilecleaner.ttl". If we go this route then I would expect both
config property and table attribute to be taken into account when deciding if a
file needs to be archived and when it needs to be deleted from the archive. I
would also expect table attribute to have priority over global configuration.
The new attribute HFILECLEANER_TTL follows the above reasoning so I am happy
with it (much better than SKIP_ARCHIVING_ON_DELETION), however looking at patch
v10 I don't see it used in two places I would expect it to be: HFileArchiver &
TimeToLiveHFileCleaner.
Later on Lars also added:
{quote}
It's not actually as simple as I thought. At the time we're archiving/deleting
HFiles we do not necessarily have access to the table's metadata, it's
expensive to read the table description (in the region or family case). When
the table is dropped, we'd need to keep the TTL information somewhere else.
{quote}
which worries me. The first part, related to access to table's metadata, was
addressed to some extent (ony in HFileArchiver) by v9 patch. The "table
dropped" case is probably harder to tackle - I would propose to fall back to
the global config when table does not exist anymore and we need to decide what
to do with remaining hfiles.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch,
> HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch,
> HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch,
> HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch,
> HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch
>
>
> Currently HFileArchiver is always called when a table is dropped or compacted.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted or compacted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949704#comment-14949704 ] Ted Yu commented on HBASE-8963: --- w.r.t. new table attribute, how about SKIP_ARCHIVING_ON_DELETION ? > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949464#comment-14949464 ] Ted Yu commented on HBASE-8963: --- I think so. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949461#comment-14949461 ] Sebastian Cotarla commented on HBASE-8963: -- [~yuzhih...@gmail.com] does v10 solves also the control of archiving/removing compacted files? Excuse my ignorance but I fail to see how this will work (in my defence I'm just a HBase user who would like to turn of archiving of compacted files). > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949015#comment-14949015 ] Ted Yu commented on HBASE-8963: --- I am open to better naming of the new table attribute. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949006#comment-14949006 ] Matteo Bertozzi commented on HBASE-8963: v10 looks ok to me > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14947207#comment-14947207 ] Ted Yu commented on HBASE-8963: --- Tentative patch - needs to retrieve table description based on given path to HFile. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 8963-v10.txt, HBASE-8963.trunk.v1.patch, > HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, > HBASE-8963.trunk.v4.patch, HBASE-8963.trunk.v5.patch, > HBASE-8963.trunk.v6.patch, HBASE-8963.trunk.v7.patch, > HBASE-8963.trunk.v8.patch, HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped or compacted. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted or compacted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680347#comment-14680347 ] Ted Yu commented on HBASE-8963: --- If [~bharathv] is busy, I have some cycles to carry this forward. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 2.0.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch, > HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625113#comment-14625113 ] Andrew Purtell commented on HBASE-8963: --- This has been stalled for a while waiting for a response from [~bharathv] addressing this: https://issues.apache.org/jira/browse/HBASE-8963?focusedCommentId=13962567&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13962567 Is there more work planned on this? Or should we resolve it as Later or Incomplete? > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 2.0.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch, > HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551307#comment-14551307 ] Jean-Marc Spaggiari commented on HBASE-8963: Stalled? > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 2.0.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch, > HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14229167#comment-14229167 ] Ted Yu commented on HBASE-8963: --- [~lhofhansl]: Do you think this JIRA can get some attention ? > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 2.0.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch, > HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962667#comment-13962667 ] bharath v commented on HBASE-8963: -- +1 [~lhofhansl]. Also we cannot afford to cache it in master's memory because we won't have it after master restarts/crashes and archiver doesn't know when to delete the files. So it needs to be persisted somewhere on to HDFS. Fwiw, i'll fix my patch to work for the 2 failed tests and post a new version soon. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch, > HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962567#comment-13962567 ] Lars Hofhansl commented on HBASE-8963: -- bq. I think Lars is right that making "hbase.master.hfilecleaner.ttl" configurable per table It's not actually as simple as I thought. At the time we're archiving/deleting HFiles we do not necessarily have access to the table's metadata, it's expensive to read the table description (in the region or family case). When the table is dropped, we'd need to keep the TTL information somewhere else. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch, > HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961811#comment-13961811
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12638959/HBASE-8963.trunk.v9.patch
against trunk revision .
ATTACHMENT ID: 12638959
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testHBase3583(TestRegionObserverInterface.java:244)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9212//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch,
> HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch,
> HBASE-8963.trunk.v9.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961774#comment-13961774 ] Enis Soztutar commented on HBASE-8963: -- I think Lars is right that making "hbase.master.hfilecleaner.ttl" configurable per table, with the global configuration used if not set is better than adding yet another config parameter. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch, > HBASE-8963.trunk.v9.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961337#comment-13961337
]
Ted Yu commented on HBASE-8963:
---
Please keep line length at 100 or shorter.
{code}
+setValue(SKIP_ARCHIVE_KEY,skip?TRUE:FALSE);
{code}
nit: insert space following comma, around question mark and colon.
{code}
+FileStatus currentStatus = fs.getFileStatus(currentFile.getPath());
+
+if(conf.getBoolean(HFILE_SKIP_ARCHIVE_CONF_KEY, true) || ((htd != null) &&
htd.isSkipArchiveEnabled())){
{code}
currentStatus is only used within the if block - you can move it inside the if
block.
{code}
+if(fs.delete(currentFile.getPath())){
{code}
Please insert space between if and '(' - applies to other if statements in
patch.
{code}
+final HTable t = new HTable(UTIL.getConfiguration(), TEST_TABLE);
{code}
Where is t closed ?
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch,
> HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13961326#comment-13961326
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12638896/HBASE-8963.trunk.v8.patch
against trunk revision .
ATTACHMENT ID: 12638896
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 1.3.9) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ public static void archiveRegion(Configuration conf, FileSystem fs,
HRegionInfo info, HTableDescriptor htd)
+ Path tableDir, byte[] family, Collection compactedFiles,
HTableDescriptor htd) throws IOException {
+ private static List resolveAndArchive(Configuration conf, FileSystem
fs, Path baseArchiveDir,
+if(conf.getBoolean(HFILE_SKIP_ARCHIVE_CONF_KEY, true) || ((htd != null) &&
htd.isSkipArchiveEnabled())){
+ HFileArchiver.archiveRegion(this.services.getConfiguration(), fs,
regionA, getTableDescriptor(regionA.getTable()));
+ HFileArchiver.archiveRegion(this.services.getConfiguration(), fs,
regionB, getTableDescriptor(regionB.getTable()));
+ HFileArchiver.archiveRegion(this.services.getConfiguration(), fs,
parent, getTableDescriptor(parent.getTable()));
+HFileArchiver.archiveRegion(conf, fs, region,
this.services.getTableDescriptors().get(region.getTable()));
+ public void removeStoreFile(final HTableDescriptor htd, final String
familyName, final Path filePath)
+ public void removeStoreFiles(final HTableDescriptor htd, final String
familyName, final Collection storeFiles)
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9207//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9207//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch,
> HBASE-8963.trunk.v7.patch, HBASE-8963.trunk.v8.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951534#comment-13951534 ] bharath v commented on HBASE-8963: -- [~lhofhansl] Sorry for the delay here. I was traveling this whole week and didn't get a chance to work on this jira. That sounds nice but seems to require a bit of code refactoring. With the current code base, it can be achieved by configuring TimeToLiveHFileCleaner and setting "hbase.master.hfilecleaner.ttl" accordingly, its a global setting though. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942019#comment-13942019 ] Lars Hofhansl commented on HBASE-8963: -- I see. So let's make the retention period configurable per table, and have 0 indicate immediate deletion. (although not sure how hard that is, as we do not have the information when archive cleaner runs - maybe it's too much of a refactoring) > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941572#comment-13941572 ] bharath v commented on HBASE-8963: -- [~larsh] The main motivation for this option are the following 1) Sometimes customers want to retain hfiles in the archive directory for a day or so after deletes, as a backup. So they configure the cleaner to run every 24hours to cleanup the files. This obviously takes up space. However not all tables require this feature and some can be deleted from disk as soon as the table is deleted / compactions are done. So this patch helps such situations where they can mark skip_archive for such test tables and not let it pass through archive so that they can reclaim the space immediately after deletion. 2) Avoid unnecessary NN operations for not-so-useful tables. If we move to the archive, instead of N file delete calls, we make N file moves and then do N file delete calls. Depending on the table size N can be very huge and we can avoid putting pressure on NN. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941373#comment-13941373 ] Lars Hofhansl commented on HBASE-8963: -- I still do not get the use case. The archiver does not keep the files forever. Is it a problem to keep the data around for 5 minutes (default) and then have it removed? Could always reduce the time from 5 minutes to 1 minute or so. (and maybe we can use 0 as an indicator to delete immediately - that would come naturally). But I really first want to hear why keeping the files for a short amount of time and then deleting them is a problem. Maybe it's obvious and I am missing something. In that case please add some more details to the description. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941345#comment-13941345
]
bharath v commented on HBASE-8963:
--
Thanks for your comments everyone. It makes sense to add a per table config.
Will write a patch for that.
[~enis] We run these files through all the configured cleaners, which checks
all the back links/snapshots associated hfiles and deletes only if its valid.
So that case is covered.
The motivation for a global flag is probably if someone wants to apply it for
all the tables and not go through the pain of setting it per table. They can
override specific tables with {SKIP_ARCHIVE = false} in which case, the global
parameter isn't taken into consideration.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch,
> HBASE-8963.trunk.v7.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941317#comment-13941317 ] Enis Soztutar commented on HBASE-8963: -- bq. What is the motivation? If you don't care about snapshots or other form of backup you can turn of the archiver I think the archive for hfiles as similar to the fs thrash. Normally, you do not want to run with that. Adding a global config is unnecessary. bq. so if you have skipArchive true but the file is referenced by a snapshot it will be archived anyway. ok good. bq. I was thinking at the table flag also to avoid stuff like "echo 'drop_table', 'skip_archive' | hbase shell" executed by mistake. If you explicitly set skip_archive, it is assumed you know what you are doing. Requiring explicit alter is fine, but the feature won't be as usable i feel. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941232#comment-13941232
]
Matteo Bertozzi commented on HBASE-8963:
{quote}I don't imagine anyone wanting to run with skip archive as a global
config on production. I think we should not do the global config at all{quote}
What is the motivation? If you don't care about snapshots or other form of
backup you can turn of the archiver
{quote}Doesn't snapshots refer to files in the archive? If we do SKIP_ARCHIVE
as a table property, the previous snapshots will be broken with compactions I
guess.{quote}
No, if you have skipArchive on.. we must check anyway if the file is referenced
by someone. And is exactly what this patch is doing. so if you have skipArchive
true but the file is referenced by a snapshot it will be archived anyway.
{quote}I think we should do a "rm -rf" kind of think in drop table. If the
files are not referred, they are not moved to archive, but deleted
instead.{quote}
I was thinking at the table flag also to avoid stuff like "echo 'drop_table',
'skip_archive' | hbase shell" executed by mistake.
since in that case if you want to switch from a table with skip archive false
to true you have to do an alter, which it will have 0 cost since the table is
already disabled and it will be only a master side operation.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch,
> HBASE-8963.trunk.v7.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941231#comment-13941231 ] Jean-Marc Spaggiari commented on HBASE-8963: Worse case, can we do the 2 options? ;) > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941142#comment-13941142 ] Enis Soztutar commented on HBASE-8963: -- I don't imagine anyone wanting to run with skip archive as a global config on production. I think we should not do the global config at all, but allow drop table to have an option to skip. Doesn't snapshots refer to files in the archive? If we do SKIP_ARCHIVE as a table property, the previous snapshots will be broken with compactions I guess. I think we should do a "rm -rf" kind of think in drop table. If the files are not referred, they are not moved to archive, but deleted instead. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940449#comment-13940449
]
Matteo Bertozzi commented on HBASE-8963:
I don't think is a drop table argument, since the compaction will archive stuff
anyway.
I like the global level property to have a general settings, but we should also
have a table level option
something like create 'testtb' {SKIP_ARCHIVE = true} that will work on both
compaction + delete
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch,
> HBASE-8963.trunk.v7.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940431#comment-13940431 ] Jean-Marc Spaggiari commented on HBASE-8963: I agree with Lars. Makes more sense to me too to have a parameter to drop table to allow skipping archiving instead of setting that at any required level. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940175#comment-13940175 ] Lars Hofhansl commented on HBASE-8963: -- Would make more sense to me if this was a parameter to an individual table-drop operation rather than a global and table level configuration. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13937986#comment-13937986 ] Matteo Bertozzi commented on HBASE-8963: instead of admin.disableTable() + admin.deleteTable() you can use TEST_UTIL.deleteTable() that does both for you. getTableRowCount() exists already in TEST_UTIL.countRows(), there is also SnapshotTestingUtils.verifyRowCount(TEST_UTIL, tableName, expectedRows) that does the count + assert for you. nit on formatting: there are a couple of places like "a,b, c" and a couple of places with 4 spaces instead of 2 I think.. test failures seems not related but there is a new report from findbug, you can find the details in the link above. other than that looks good to me, did you do some manual verification? can you post the steps and the conf settings. Thanks! > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936336#comment-13936336 ] Ted Yu commented on HBASE-8963: --- If you compare https://builds.apache.org/job/PreCommit-HBASE-Build/9015/artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html (for this JIRA) with https://builds.apache.org/job/PreCommit-HBASE-Build/9015/artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html , you should be able to find the new findbugs warning. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch, > HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch, > HBASE-8963.trunk.v7.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13936285#comment-13936285
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634925/HBASE-8963.trunk.v7.patch
against trunk revision .
ATTACHMENT ID: 12634925
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9015//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch,
> HBASE-8963.trunk.v7.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935671#comment-13935671
]
Ted Yu commented on HBASE-8963:
---
patch v6 looks good.
{code}
+ return true;
+}
+else{
{code}
nit: if you keep else keyword, it should be on the same line as }.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935676#comment-13935676
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634807/HBASE-8963.trunk.v6.patch
against trunk revision .
ATTACHMENT ID: 12634807
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9001//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch, HBASE-8963.trunk.v6.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935462#comment-13935462
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634791/HBASE-8963.trunk.v5.patch
against trunk revision .
ATTACHMENT ID: 12634791
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8999//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch,
> HBASE-8963.trunk.v5.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934815#comment-13934815
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634669/HBASE-8963.trunk.v4.patch
against trunk revision .
ATTACHMENT ID: 12634669
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934784#comment-13934784
]
Matteo Bertozzi commented on HBASE-8963:
{quote}All the archive cleaner plugins (except snapshot and hfilelink cleaner)
are written assuming the hfile is already in archive .{quote}
What about the custom ones that the user has written, or the ones that we will
write in the future?
I'll be the first to forget to update the archiver if I add a new cleaner like
the snapshot one.
{quote}Given this runs only when skipArchive is enabled, users are ok with
these hfiles getting deleted and won't expect these hfiles to be checked via
their custom cleaners.{quote}
I think users are ok to get unreferenced files deleted.
e.g. If I take a snapshot on the table and the delete the table, I want the
snapshot to be sane.
Also this settings is global and not per table which means.. that you may just
turn this on at some point in the future because your new table takes long time
archiving... but you don't want to lose snapshots (or other files that your
custom cleaner should keep around) for the other tables.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934726#comment-13934726 ] bharath v commented on HBASE-8963: -- [~yuzhih...@gmail.com] Added v4 as per your comments [~mbertozzi] I did it for 2 reasons 1) All the archive cleaner plugins (except snapshot and hfilelink cleaner) are written assuming the hfile is already in archive . For example TimToLiveHFileCleaner which check if the ttl of the hfile "inside the archive" directory exceeds the configured timeout. That doesn't need to be run on the hfiles which are not yet archived and will always return true. Also any further custom implementation of users will expect the store files to be already in archive directory and would probably fail if we run the wholeCleanerChain 2) Given this runs only when skipArchive is enabled, users are ok with these hfiles getting deleted and won't expect these hfiles to be checked via their custom cleaners. Are these assumptions valid? > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Fix For: 0.99.0 > > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934305#comment-13934305
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634534/HBASE-8963.trunk.v3.patch
against trunk revision .
ATTACHMENT ID: 12634534
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the trunk's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ public static boolean archiveRegion(Configuration conf,FileSystem fs,
Path rootdir, Path tableDir, Path regionDir)
+ private static List resolveAndArchive(Configuration conf, FileSystem
fs, Path baseArchiveDir,
+ private static boolean resolveAndArchiveFile(Configuration conf, Path
archiveDir, File currentFile,
+ if(snapshotCleaner.isFileDeletable(currentStatus) &&
fileLinkCleaner.isFileDeletable(currentStatus)){
+ LOG.debug("Attempting to delete file without archiving
:"+currentFile.getPath().getName());
+ LOG.debug("Attempt to delete file
"+currentFile.getPath().getName()+" failed. Moving it to the archive.");
+// Delete the table, since skiparchive is set unnecessary hfiles
created due to deletes are deleted
+// Delete a few rows from the original table, for this test we delete
all the rows starting with 'a'.
+cloneCount = getTableRowCount(new
HTable(TEST_UTIL.getConfiguration(),Bytes.toBytes(TABLENAME3.toString()+".clone")));
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8976//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934084#comment-13934084
]
Matteo Bertozzi commented on HBASE-8963:
Is there a reason to hardcode the SnapshotHFileCleaner and the FileLinkCleaner
and discard the others?
can't you reuse in some way
CleanerChore.initCleanerChain("hbase.master.hfilecleaner.plugins")
just to make sure that people with its own cleaner doesn't lose their old files
with the skip opt on
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Fix For: 0.99.0
>
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934004#comment-13934004
]
Ted Yu commented on HBASE-8963:
---
{code}
+ private static final String HFILE_SKIP_ARCHIVE_CONF =
"hbase.hfile.skip.archive";
{code}
Should the constant be named HFILE_SKIP_ARCHIVE_CONF_KEY ? It is key to the new
config, right ?
For initCleaners(), null check on snapshotCleaner alone would be enough because
both cleaners would be instantiated, right ?
{code}
+ if(fs.delete(currentFile.getPath())){
+ return true;
+ }
+ else{
+ LOG.debug("Attempt to delete file
"+currentFile.getPath().getName()+" failed. Moving it to the archive.");
{code}
Suggest changing the log level to error.
License header is needed for TestSkipArchiveTableDeleteHandler.java
In that file, indentation is not right - 2 spaces should be used for each
indent.
{code}
+rs.close();
+}finally {
+rs.close();
{code}
rs is closed twice, right ?
Thanks for taking this JIRA.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933987#comment-13933987
]
Ted Yu commented on HBASE-8963:
---
I started reviewing patch v2 and saw patch v3 coming.
Here're my comments based on v2
{code}
+ public static boolean archiveRegion(Configuration conf,FileSystem fs, Path
rootdir, Path tableDir, Path regionDir)
{code}
nit: leave a space following the comma between conf and FileSystem.
For resolveAndArchiveFile(), please add javadoc for parameter conf.
{code}
+ FileStatus currentStatus = fs.getFileStatus(currentFile.getPath());
+
+ if(conf.getBoolean(HFILE_SKIP_ARCHIVE_CONF, false)){
{code}
currentStatus is only used when the condition is true, move its assignment
inside the if block.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
> HBASE-8963.trunk.v3.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933983#comment-13933983 ] bharath v commented on HBASE-8963: -- Please ignore v2. It includes some unintended changes. Added v3 of patch without them. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, > HBASE-8963.trunk.v3.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933976#comment-13933976 ] bharath v commented on HBASE-8963: -- Attached v2 of the patch. This moves all the checks to the central code archiver. Had to make some changes in the archive() calls by adding the Configuration object and as a result, I changed the caller classes too, especially from compaction/ deletion. Add unitTests for table deletion with snapshots/file links. Tested compaction locally on my machine and confirmed that the code skips archives and deletes the storefiles directly (via debug logs). > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933282#comment-13933282 ] bharath v commented on HBASE-8963: -- I wrote this patch just keeping the DeleteHandler in mind as I got confused by the jira description. It makes sense to move the skipArchive check to archiver makes more sense so that other components like compactors can use it too. Will upload a new patch with changes shortly. Thanks for your quick review [~mbertozzi]. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Attachments: HBASE-8963.trunk.v1.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933274#comment-13933274
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634422/HBASE-8963.trunk.v1.patch
against trunk revision .
ATTACHMENT ID: 12634422
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8968//console
This message is automatically generated.
> Add configuration option to skip HFile archiving
>
>
> Key: HBASE-8963
> URL: https://issues.apache.org/jira/browse/HBASE-8963
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: bharath v
> Attachments: HBASE-8963.trunk.v1.patch
>
>
> Currently HFileArchiver is always called when a table is dropped.
> A configuration option (either global or per table) should be provided so
> that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933222#comment-13933222 ] Matteo Bertozzi commented on HBASE-8963: so, you are currently skipping the archive just for Delete Table? what about compactions and others? I think that you should move the if (skipArchive) inside the archiver, since everyone is calling the archiver... and the inside the archiver you check the "skipArchive" property. In this way you don't have to put if (skipArchive) all around. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: bharath v > Attachments: HBASE-8963.trunk.v1.patch > > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13710251#comment-13710251 ] Jesse Yates commented on HBASE-8963: This seems reasonable... would certainly save a bit of CPU time and disk space. Would need to be careful though to ensure it doesn't conflict with snapshotting. > Add configuration option to skip HFile archiving > > > Key: HBASE-8963 > URL: https://issues.apache.org/jira/browse/HBASE-8963 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu > > Currently HFileArchiver is always called when a table is dropped. > A configuration option (either global or per table) should be provided so > that archiving can be skipped when table is deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
