Josh Elser created HBASE-26605:
----------------------------------
Summary: TestHStore#testRefreshStoreFiles broken due to
unqualified and qualified paths
Key: HBASE-26605
URL: https://issues.apache.org/jira/browse/HBASE-26605
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 2.4.9
Reporter: Josh Elser
Assignee: Josh Elser
Was looking at a failures of this method where
{noformat}
[ERROR] org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles
Time elapsed: 4.2 s <<< ERROR!
java.util.NoSuchElementException
at
org.apache.hbase.thirdparty.com.google.common.collect.AbstractIndexedListIterator.next(AbstractIndexedListIterator.java:75)
at
org.apache.hadoop.hbase.regionserver.TestHStore.closeCompactedFile(TestHStore.java:962)
at
org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:1000)
{noformat}
This was on a branch where I had some HBASE-26067 changes backported, so I
thought the problem _was_ those changes. After a bit of digging, I believe the
test case itself is "broken" (the test passes, but for the wrong reasons).
This test methods adds some files to a Store (via memstore flush or direct
addition of a file) and eventually tries to get the first file which is
candidate to be removed. The test {*}never compacted any files{*}. This was the
first sign that the test itself was wrong.
After lots of comparison with the HBASE-26067 logging to compare against, I
found that the Store was listing a file which was created by the memstore flush
as a file to retain AND a file to remove. Second warning. Upon closer
inspection, I finally noticed that one of the files was qualified with the
filesystem URI and the other was not.
{noformat}
2021-12-18 16:57:10,903 INFO [Time-limited test] regionserver.HStore(675):
toBeAddedFiles=[file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/dd8a430b391546d8b9bdc39bb77d447b,
file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73],
toBeRemovedFiles=[/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73]
{noformat}
{{d4c5442b772c43fd9ebdfed1a11c0e73}} how are we both adding and removing this
file! Turns out, this is because one of them is "/..." and the other is
"file:/...". Either the problem is in TestHStore in how it is creating/adding
these files behind the scenes or we should be qualifying the Path inside of
StoreFileInfo with the filesystem that we're using.
I remember too vividly the problems when trying to separate the rootdir and
waldir from each other and am cautious against adding something to
StoreFileInfo to {{{}fs.qualifyPath(p){}}}. Need to look some more, but will
get a patch up to fix.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
