[ https://issues.apache.org/jira/browse/HADOOP-16836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ctest updated HADOOP-16836: --------------------------- Attachment: HADOOP-16836-000.patch > Bug in widely-used helper function caused valid configuration value to fail > on multiple tests, causing build failure > -------------------------------------------------------------------------------------------------------------------- > > Key: HADOOP-16836 > URL: https://issues.apache.org/jira/browse/HADOOP-16836 > Project: Hadoop Common > Issue Type: Bug > Components: common > Affects Versions: 3.2.1 > Reporter: Ctest > Priority: Blocker > Labels: configuration, easyfix, patch, test > Attachments: HADOOP-16836-000.patch > > > Test helper function > `org.apache.hadoop.io.file.tfile.TestTFileByteArrays#readRecords(org.apache.hadoop.fs.FileSystem, > org.apache.hadoop.fs.Path, int, org.apache.hadoop.conf.Configuration)` > (abbreviate as `readRecords()` below) are called in 4 actively-used tests > below: > > {code:java} > org.apache.hadoop.io.file.tfile.TestTFileStreams#testOneEntryMixedLengths1 > org.apache.hadoop.io.file.tfile.TestTFileStreams#testOneEntryUnknownLength > org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams#testOneEntryMixedLengths1 > org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams#testOneEntryUnknownLength{code} > > These tests first call > `org.apache.hadoop.io.file.tfile.TestTFileStreams#writeRecords(int count, > boolean knownKeyLength, boolean knownValueLength, boolean close)` to write > `key-value` pair records in a `TFile` object, then call the helper function > `readRecords()` to assert the `key` part and the `value` part of `key-value` > pair records stored matched with what they wrote perviously. The `value` > parts of `key-value` pairs from these tests are hardcode strings with a > length of 6. > Assertions in `readRecords()` are directly related to the value of the > configuration parameter `tfile.io.chunk.size`. The formal definition of > `tfile.io.chunk.size` is "Value chunk size in bytes. Default to 1MB. Values > of the length less than the chunk size is guaranteed to have known value > length in read time (See also > TFile.Reader.Scanner.Entry.isValueLengthKnown())". > When `tfile.io.chunk.size` is configured to a value less than the length of > the `value` part of the `key-value` pairs from these 4 tests, these tests > will fail, even though the configured value for `tfile.io.chunk.size` is > correct in semantic. > > *Consequence* > At least 4 actively-used tests failed on correctly configured parameters. > Tests used `readRecords()` could fail if the length of the hardcoded `value` > part they tested is larger than the configured value of > `tfile.io.chunk.size`. This caused build failure of Hadoop-Common if these > tests are not skipped. > > *Root Cause* > `readRecords()` used > `org.apache.hadoop.io.file.tfile.TFile.Reader.Scanner.Entry#getValueLength()` > (abbreviate as `getValueLength()` below) to get the full length of the > `value` part in the `key-value` pair. But `getValueLength()` can only get the > full length of the `value` part when the full length is less than > `tfile.io.chunk.size`, otherwise, `getValueLength()` throws an exception, > causing `readRecords()` to fail, and thus resulting in failures in the > aforementioned 4 tests. This is because `getValueLength()` do not know the > full length of the `value` part when `value` part's size is larger than > `tfile.io.chunk.size`. > > *Fixes* > `readRecords()` should instead call > `org.apache.hadoop.io.file.tfile.TFile.Reader.Scanner.Entry#getValue(byte[])` > (abbreviate as `getValue()` below), which returns the correct full length of > the `value` part despite whether the `value` length is larger than > `tfile.io.chunk.size`. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org