[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776583#comment-13776583
]
Jean-Marc Spaggiari commented on HBASE-9648:
Is the region really empty? From the logs provided by the user we can see this:
{code}
size=491.0; total size for store is 5.7m
{code}
491 bytes is very small, I agree, but might not be totally empty.
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776868#comment-13776868
]
Jean-Marc Spaggiari commented on HBASE-9648:
Here are the details for the file which was causing the issue.
{code}
jmspaggiari@t430s:~/workspace/hbase-0.94.10$ bin/hbase
org.apache.hadoop.hbase.io.hfile.HFile -m -s -v -f
fca0882dc7624342a8f4fce4b89420ff
13/09/24 14:54:09 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32
Scanning -> fca0882dc7624342a8f4fce4b89420ff
13/09/24 14:54:09 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum
size 247.9m
13/09/24 14:54:09 ERROR metrics.SchemaMetrics: Inconsistent configuration.
Previous configuration for using table name in metrics: true, new
configuration: false
13/09/24 14:54:09 WARN metrics.SchemaConfigured: Could not determine table and
column family of the HFile path fca0882dc7624342a8f4fce4b89420ff. Expecting at
least 5 path components.
13/09/24 14:54:09 WARN snappy.LoadSnappy: Snappy native library is available
13/09/24 14:54:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/09/24 14:54:09 INFO snappy.LoadSnappy: Snappy native library loaded
13/09/24 14:54:09 INFO compress.CodecPool: Got brand-new decompressor
Block index size as per heapsize: 336
reader=fca0882dc7624342a8f4fce4b89420ff,
compression=snappy,
cacheConf=CacheConfig:enabled [cacheDataOnRead=true]
[cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
[cacheEvictOnClose=false] [cacheCompressed=false],
firstKey=null,
lastKey=null,
avgKeyLen=0,
avgValueLen=0,
entries=0,
length=491
Trailer:
fileinfoOffset=56,
loadOnOpenDataOffset=0,
dataIndexCount=0,
metaIndexCount=0,
totalUncomressedBytes=489,
entryCount=0,
compressionCodec=SNAPPY,
uncompressedDataIndexSize=0,
numDataIndexLevels=1,
firstDataBlockOffset=-1,
lastDataBlockOffset=0,
comparatorClassName=org.apache.hadoop.hbase.KeyValue$KeyComparator,
majorVersion=2,
minorVersion=0
Fileinfo:
DATA_BLOCK_ENCODING = NONE
DELETE_FAMILY_COUNT = \x00\x00\x00\x00\x00\x00\x00\x00
EARLIEST_PUT_TS = \x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF
MAJOR_COMPACTION_KEY = \x00
MAX_SEQ_ID_KEY = 19978535453
TIMERANGE = -1-1
hfile.AVG_KEY_LEN = 0
hfile.AVG_VALUE_LEN = 0
Unable to retrieve the midkey
Bloom filter:
Not present
Delete Family Bloom filter:
Not present
Stats:
no data available for statistics
Scanned kv count -> 0
{code}
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776905#comment-13776905 ] Sergey Shelukhin commented on HBASE-9648: - I think in the reported case there was more than one file in store (and thus the candidate list), it's that it was selecting one expired file. Judging by HBASE-6059 description and comments, we actually don't have to generate a new file unless we are talking about the highest-seqid file in the region (which also implies not removing the last file). So it may make sense to allow lazy-writer code to run if there're files remaining in store (which would presumably mean for ever minor compactions). In that case not running expired check on single last file would be enough. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776913#comment-13776913
]
Jean-Marc Spaggiari commented on HBASE-9648:
Indeed. You're right. There was way more files than just one. I don't know why
I ignored that.
I'm not sure to get how it's related to HBASE-6059. Are you suggesting to
"simply" remove the test and do something like that? So if any other file need
to be compacted, it will be. I'm not sure we need to add any check for the last
store.
{code}
private ArrayList selectExpiredStoreFiles(
ArrayList candidates, long maxExpiredTimeStamp) {
if (candidates == null || candidates.size() == 0) return null;
ArrayList expiredStoreFiles = null;
for (StoreFile storeFile : candidates) {
// If the storeFile is empty, then there is no need to compact it.
// Empty files don't have time range information.
if ((storeFile.getReader().getEntries() > 0)
&& (storeFile.getReader().getMaxTimestamp() < maxExpiredTimeStamp)) {
LOG.info("Deleting the expired store file by compaction: " +
storeFile.getPath()
+ " whose maxTimeStamp is " +
storeFile.getReader().getMaxTimestamp()
+ " while the max expired timestamp is " + maxExpiredTimeStamp);
if (expiredStoreFiles == null) {
expiredStoreFiles = new ArrayList();
}
expiredStoreFiles.add(storeFile);
}
}
return expiredStoreFiles;
}
{code}
Also here is the list of files for reference.
{code}
-rw--- 1 hadoop supergroup 2194 2013-09-21 14:32
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/014ead47a9484d67b55205be16802ff1
-rw--- 1 hadoop supergroup 31321 2013-09-24 05:49
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/1305d625bd4a4be39a98ae4d91a66140
-rw--- 1 hadoop supergroup 1350 2013-09-24 10:31
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/1352e0828f974f08b1f3d7a9dff04abd
-rw--- 1 hadoop supergroup 4194 2013-09-21 10:38
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/17a546064bd840619816809ae0fc4c49
-rw--- 1 hadoop supergroup 1061 2013-09-20 22:55
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/1cb3df115da244288bd076968ab4ccf6
-rw--- 1 hadoop supergroup 1375 2013-08-24 10:17
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/1e41a96c49fc4e5ab59392d26935978d
-rw--- 1 hadoop supergroup 96296 2013-08-26 15:48
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/22d72fd897e34424b5420a96483a571e
-rw--- 1 hadoop supergroup 1356 2013-08-26 15:23
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/25fee1ffadbe42549bd0b7b13d782b72
-rw--- 1 hadoop supergroup 6229 2013-09-21 11:14
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/26289c777ec14dc5b7021b4d6b1050c5
-rw--- 1 hadoop supergroup 1223 2013-09-21 02:42
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/2757d7ba9c8448d6a3d5d46bd4d59758
-rw--- 1 hadoop supergroup5302248 2013-08-24 02:22
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/2ec40943787246ea983608dd6591db24
-rw--- 1 hadoop supergroup 1596 2013-08-24 03:37
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/3157fd1cabe4483aaa4d9a21f75e4d88
-rw--- 1 hadoop supergroup 1338 2013-09-22 04:25
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/36b0f80a4a7b492f97358b64d879a2df
-rw--- 1 hadoop supergroup 3264 2013-09-21 12:05
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/39e249fcb532400daed73aed6689ceeb
-rw--- 1 hadoop supergroup 4549 2013-09-21 08:56
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/3bc9e2a566ad460a9b0ed336b2fb5ed9
-rw--- 1 hadoop supergroup 1630 2013-09-22 03:22
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/48026d08aae748f08aad59e4eea903be
-rw--- 1 hadoop supergroup 105395 2013-09-20 21:12
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/53198825f085401cbbd4322faa0e3aae
-rw--- 1 hadoop supergroup 3859 2013-09-21 09:09
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/71c2f9b2a8ff4c049fcc5a9a22af5cfe
-rw--- 1 hadoop supergroup 311688 2013-09-20 21:12
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/97ff16d6da974c30835c6e0acc7c737a
-rw--- 1 hadoop supergroup 1897 2013-08-24 08:43
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/a172d7577641434d82abcce88a433213
-rw--- 1 hadoop supergroup 3380 2013-09-21 13:04
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/be678e5c60534c65a012a798fbc7e284
-rw--- 1 hadoop supergroup 43710 2013-09-22 02:15
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/e2508a23acf1491f9d38b9a8594e41e8
-rw--- 1 hadoop supergroup 5409 2013-09-21 10:10
/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/f432846182714b93a1c3d
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777081#comment-13777081 ] Sergey Shelukhin commented on HBASE-9648: - If you look at the code now (at least in 94 the archeological record survives) it has provisions for creating the writer for compaction output lazily (even a comment about it iirc), so that if there's no data, there's no file; but then below it goes and says, we need to create writer anyway because blah-blah see that jira. So when the original problem on the thread picks one file to remove, this code says oh no I need to create the empty file anyway, and creates it; after which the expired-file-selection code picks the new file, as a single file, to remove, and it creates another empty file, forever. Expired file compaction is supposed to be fast so it pre-empts any real compaction. What I'm saying is that this empty-file-creation might not always be necessary, judging by that jira. All we want to avoid is losing the latest-used seqNum in the store, which means we can have no-output compaction as long as we are not dropping the file with the last seqNum. So the original problem from the thread will go away if that was done, deletion of some random expired file would just nuke it. It will also be generally useful to not create these blank files as often. The only edge case remaining is when this file is the file with the last seqNum (I incorrectly said above that it is when it's the only file), in which case expired file thing should not pick it. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777092#comment-13777092
]
Hadoop QA commented on HBASE-9648:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604906/HBASE-9648-v0-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7364//console
This message is automatically generated.
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch
>
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777147#comment-13777147
]
Jean-Marc Spaggiari commented on HBASE-9648:
Ok. I see your point.
In trunk, the code moved to DefaultCompactor
{code}
// Create the writer even if no kv(Empty store file is also ok),
// because we need record the max seq id for the store file, see
HBASE-6059
writer = store.createWriterInTmp(fd.maxKeyCount,
this.compactionCompression, true,
fd.maxMVCCReadpoint >= smallestReadPoint, fd.maxTagsLength > 0);
{code}
So basically, we need to avoid creating this empty file if it doesn't have the
max seq id. The thing is, we don't have this information there.
However, the modification to RatioBasedcompactionPolicy.selectExpiredStoreFiles
should not be difficult.
... Ok. I think I found something. I will draft that and post it tomorrow
morning for your review.
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch
>
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1325#comment-1325
]
Hadoop QA commented on HBASE-9648:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12605019/HBASE-9648-v1-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7372//console
This message is automatically generated.
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch,
> HBASE-9648-v1-trunk.patch
>
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778235#comment-13778235 ] Sergey Shelukhin commented on HBASE-9648: - For the expired selection check, candidate list is not the same as list of files in store. For the other check logic for last file is ok, presence of data in input doesn't mean that there'd be anything in the output... I wonder if simpler check is possible when selecting. I will be in some meetings, let me thing about it. >From code perspective looks ok. Can you create a test? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778369#comment-13778369 ] Lars Hofhansl commented on HBASE-9648: -- Can we just not do the immediate deletion of the HFile when getMaxTimestamp() returns -1? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778470#comment-13778470
]
Jean-Marc Spaggiari commented on HBASE-9648:
[~sershe]:
{quote}
For the expired selection check, candidate list is not the same as list of
files in store.
{quote}
I will change to comment to say "If the last candidate" instead...
{quote}
presence of data in input doesn't mean that there'd be anything in the output
{quote}
I agree. Maybe everything will be deleted by the compaction, TTL, etc. But
worst case, that will write an empty file wich will be avoided at the next
iteration.
I will take a look how we can implement a test for that.
{code}
Can we just not do the immediate deletion of the HFile when getMaxTimestamp()
returns -1?
{code}
We can, but are they not going to stay there for ever if we do that? Or is
another process going to clean them later?
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch,
> HBASE-9648-v1-trunk.patch
>
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778480#comment-13778480 ] Lars Hofhansl commented on HBASE-9648: -- I'd have to study the code a bit more, but I thought it just means that they'll be removed after the compaction, rather than before (as part of the compaction selection). On the mailing list it was reported that disabling this feature fixed the problem. (See hbase.store.delete.expired.storefile in Store.java in 0.94) > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779054#comment-13779054 ] Sergey Shelukhin commented on HBASE-9648: - Yeah, that is what we recommended :) Deleting the file definitely makes sense, we can shortcut and archive it immediately without any compaction. At that point there's also no need to do convoluted checks because we also have full view of all files. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779314#comment-13779314 ] Lars Hofhansl commented on HBASE-9648: -- [~sershe], are you agreeing? Can't quite parse that out :) Basically what I was saying was that we skip the early delete optimization for empty storefiles. It just means they'll get deleted after the compaction, but apparently without the issue we're seeing here. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782332#comment-13782332 ] Sergey Shelukhin commented on HBASE-9648: - Oh. I meant we can just nuke them immediately without going thru the entire compaction pipeline. This would solve the problem much simpler (either your approach or mine), we don't even have to scan them. The last file cannot be deleted though, even if expired, to preserve seqNum > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782333#comment-13782333 ] Sergey Shelukhin commented on HBASE-9648: - Or do you mean the code to delete after compaction already exists? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782406#comment-13782406 ] Lars Hofhansl commented on HBASE-9648: -- I might be misunderstanding the issue... When hbase.store.delete.expired.storefile is true an old storefile is nuked immediately as part of the selection process (rather than being compacted and then deleted). The issue is when we only have a single store file left and it is expired, in that case we nuke, but write a new (empty) one to keep the sequenceId; this file is then nuked (because we return -1 from getMaxTimestamp() for empty files), and from here we repeat the same cycle. Another apparent issue is the ratio based selection with empty files. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782415#comment-13782415 ] Sergey Shelukhin commented on HBASE-9648: - Yeah, that is the issue. However, it's not just deleted, it goes thru compaction with a request for one file, then it's at least nominally scanned (probably it skips the entire file in scanner), committed, etc. etc. We could just archive it straight away, no compaction or anything. The fixes previously discussed here were fixing the special case handling for the creation of new file, as we don't really need to do it as often as we do. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783155#comment-13783155 ] Sergey Shelukhin commented on HBASE-9648: - let me try to make example patch > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783157#comment-13783157 ] Jean-Marc Spaggiari commented on HBASE-9648: Will be nice also to be able to reproduce this issue, to test a potential fix. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783563#comment-13783563 ] Sergey Shelukhin commented on HBASE-9648: - Actually we kind of already do, in testDeleteExpiredStoreFiles in TestStore. It expires 4 files and verifies there are still 4 file :) > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783564#comment-13783564 ] Sergey Shelukhin commented on HBASE-9648: - *on first loop iteration > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784040#comment-13784040 ] Jean-Marc Spaggiari commented on HBASE-9648: Ok. For the test, I see. We are testing it, but it's working as expected. We creates 4 files which are not empty and we try to compact them. We don't have the corner case where those files are empty, and when there is only one single file in the store, etc. So if I take the test from your patch but not the rest, it should be failing, right? Overall the patch looks good. There is some small things but we will keep that for the final version ;) > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784187#comment-13784187 ] Sergey Shelukhin commented on HBASE-9648: - Well, the test would fail but not just due to logical reasons, also because the flow changes, there's no longer real compaction when this happens, it just archives the files. [~eclark] wdyt? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784197#comment-13784197 ] Jean-Marc Spaggiari commented on HBASE-9648: But a good test should not really worry about the flow, right? That's the goal of it. I will try to reproduce that without the patch, and try it with and see... > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784202#comment-13784202 ] Sergey Shelukhin commented on HBASE-9648: - Well, I mean the logic changed. The test was checking that compactions happen, and they no longer do > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788393#comment-13788393 ] Sergey Shelukhin commented on HBASE-9648: - ping? any other opinion on this? We can also file a bug to limit empty file creation as per [~jmspaggi] patch, or do it the other way around. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800658#comment-13800658 ] Jean-Marc Spaggiari commented on HBASE-9648: Ping again ;) Should we move forward with what I proposed initially and open another JIRA if we want to go further? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800859#comment-13800859 ] Sergey Shelukhin commented on HBASE-9648: - [~jmspaggi] you were one of the people I was pinging :) Any opinion, incl. on the above patch? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13807297#comment-13807297 ] Sergey Shelukhin commented on HBASE-9648: - la la la > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13807398#comment-13807398 ] Jean-Marc Spaggiari commented on HBASE-9648: Sorry busy because of Strata :( I will have may more time next week (starting Thursday/Friday this week)... > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648.patch, HBASE-9648-v0-0.94.patch, > HBASE-9648-v0-trunk.patch, HBASE-9648-v1-trunk.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13839201#comment-13839201 ] Sergey Shelukhin commented on HBASE-9648: - stumbled upon this jira (not bug :)) again... do you want to go with either patch > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13842202#comment-13842202 ] Jean-Marc Spaggiari commented on HBASE-9648: So. Back on this JIRA ;) I think patch HBASE-9648-v1-trunk.patch can fix this issue the user faced on the mailing list. It basically do what need to be done to avoid this from the first level. It passed Hadoop QA, but I can also port it to 0.94 and give it a bigger try on my own cluster... > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845948#comment-13845948 ] Sergey Shelukhin commented on HBASE-9648: - Makes sense... I can move the other patch for different jira. Technically, in compaction policy, last candidate is not necessarily the latest file in store as some files might not be candidates (for example, already compacting if there are multiple compaction threads). But it should be ok. Can you resubmit the patch for QA to kick in again? Thanks. > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846494#comment-13846494 ] Jean-Marc Spaggiari commented on HBASE-9648: Sure. I will rebase (if required) and re-submit later today (this evening probably). > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846566#comment-13846566 ] Sergey Shelukhin commented on HBASE-9648: - Rebased the other patch in HBASE-10141 > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847560#comment-13847560
]
Hadoop QA commented on HBASE-9648:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618595/HBASE-9648-v2-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8157//console
This message is automatically generated.
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch,
> HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, HBASE-9648.patch
>
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847797#comment-13847797
]
Sergey Shelukhin commented on HBASE-9648:
-
{code}
+if ((requestMaxSequenceId == storeMaxSequenceId) || (requestEntries >
0)) writer =
+store.createWriterInTmp(fd.maxKeyCount,
this.compactionCompression, true,
+ fd.maxMVCCReadpoint >= smallestReadPoint, fd.maxTagsLength > 0);
{code}
Nit: please add braces.
Other than that, seeing if inputs have entries is not technically speaking
valid because coproc can replace scanners.
Should it rather create writer either if last seqId is being compacted there,
or on first record from the scanner, if needed?
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch,
> HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, HBASE-9648.patch
>
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847827#comment-13847827 ] Jean-Marc Spaggiari commented on HBASE-9648: Or can we look at the entries before we trigger the coprocessors? That way we are sure they will not modify the scanner? and we keep the writer creation the way it is in the patch? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847934#comment-13847934 ] Sergey Shelukhin commented on HBASE-9648: - coproc takes a scanner (StoreScanner of all files) and produces a scanner. What do you mean? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847947#comment-13847947 ] Jean-Marc Spaggiari commented on HBASE-9648: Hum. Looking at it again, I might be missing something. Even if coprocessors can replace the scanner, that doesn't change the list of store files, right? Or can the scanner make the compactor looking at another set of store files? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847970#comment-13847970 ] Sergey Shelukhin commented on HBASE-9648: - The scanner created by coproc can hypothetically do anything. Actually I am kind of ambivalent on that one. Maybe we should just document it, I suspect coprocs usually don't generate new KVs out of no input KVs. But, is it difficult to create writer on first entry? If it's easy, any reason to not do it? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853550#comment-13853550 ] Sergey Shelukhin commented on HBASE-9648: - [~jmspaggi] ping? > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[
https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853690#comment-13853690
]
Hadoop QA commented on HBASE-9648:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619721/HBASE-9648-v3-trunk.patch
against trunk revision .
ATTACHMENT ID: 12619721
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8238//console
This message is automatically generated.
> collection one expired storefile causes it to be replaced by another expired
> storefile
> --
>
> Key: HBASE-9648
> URL: https://issues.apache.org/jira/browse/HBASE-9648
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Jean-Marc Spaggiari
> Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch,
> HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch,
> HBASE-9648-v3-trunk.patch, HBASE-9648.patch
>
>
> There's a shortcut in compaction selection that causes the selection of
> expired store files to quickly delete.
> However, there's also the code that ensures we write at least one file to
> preserve seqnum. This new empty file is "expired", because it has no data,
> presumably.
> So it's collected again, etc.
> This affects 94, probably also 96.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9648) collection one expired storefile causes it to be replaced by another expired storefile
[ https://issues.apache.org/jira/browse/HBASE-9648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13854374#comment-13854374 ] Sergey Shelukhin commented on HBASE-9648: - Just clarifying, why is it hard to create writer is needed? For the case when there are seemingly no KVs when you were creating the writer. I think coprocs cannot screw up the seqIds, because set of files is already chosen, so that should be ok > collection one expired storefile causes it to be replaced by another expired > storefile > -- > > Key: HBASE-9648 > URL: https://issues.apache.org/jira/browse/HBASE-9648 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-9648-v0-0.94.patch, HBASE-9648-v0-trunk.patch, > HBASE-9648-v1-trunk.patch, HBASE-9648-v2-trunk.patch, > HBASE-9648-v3-trunk.patch, HBASE-9648.patch > > > There's a shortcut in compaction selection that causes the selection of > expired store files to quickly delete. > However, there's also the code that ensures we write at least one file to > preserve seqnum. This new empty file is "expired", because it has no data, > presumably. > So it's collected again, etc. > This affects 94, probably also 96. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
