[
https://issues.apache.org/jira/browse/HADOOP-16836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028343#comment-17028343
]
Hadoop QA commented on HADOOP-16836:
------------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HADOOP-16836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992438/HADOOP-16836-000.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 321410062d1f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1e3a0b0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16748/testReport/ |
| Max. process+thread count | 1376 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16748/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Bug in widely-used helper function caused valid configuration value to fail
> on multiple tests, causing build failure
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-16836
> URL: https://issues.apache.org/jira/browse/HADOOP-16836
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.2.1
> Reporter: Ctest
> Priority: Blocker
> Labels: configuration, easyfix, patch, test
> Attachments: HADOOP-16836-000.patch, HADOOP-16836-000.patch
>
>
> Test helper function
> `org.apache.hadoop.io.file.tfile.TestTFileByteArrays#readRecords(org.apache.hadoop.fs.FileSystem,
> org.apache.hadoop.fs.Path, int, org.apache.hadoop.conf.Configuration)`
> (abbreviate as `readRecords()` below) are called in 4 actively-used tests
> below:
>
> {code:java}
> org.apache.hadoop.io.file.tfile.TestTFileStreams#testOneEntryMixedLengths1
> org.apache.hadoop.io.file.tfile.TestTFileStreams#testOneEntryUnknownLength
> org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams#testOneEntryMixedLengths1
> org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams#testOneEntryUnknownLength{code}
>
> These tests first call
> `org.apache.hadoop.io.file.tfile.TestTFileStreams#writeRecords(int count,
> boolean knownKeyLength, boolean knownValueLength, boolean close)` to write
> `key-value` pair records in a `TFile` object, then call the helper function
> `readRecords()` to assert the `key` part and the `value` part of `key-value`
> pair records stored matched with what they wrote perviously. The `value`
> parts of `key-value` pairs from these tests are hardcode strings with a
> length of 6.
> Assertions in `readRecords()` are directly related to the value of the
> configuration parameter `tfile.io.chunk.size`. The formal definition of
> `tfile.io.chunk.size` is "Value chunk size in bytes. Default to 1MB. Values
> of the length less than the chunk size is guaranteed to have known value
> length in read time (See also
> TFile.Reader.Scanner.Entry.isValueLengthKnown())".
> When `tfile.io.chunk.size` is configured to a value less than the length of
> the `value` part of the `key-value` pairs from these 4 tests, these tests
> will fail, even though the configured value for `tfile.io.chunk.size` is
> correct in semantic.
>
> *Consequence*
> At least 4 actively-used tests failed on correctly configured parameters.
> Tests used `readRecords()` could fail if the length of the hardcoded `value`
> part they tested is larger than the configured value of
> `tfile.io.chunk.size`. This caused build failure of Hadoop-Common if these
> tests are not skipped.
>
> *Root Cause*
> `readRecords()` used
> `org.apache.hadoop.io.file.tfile.TFile.Reader.Scanner.Entry#getValueLength()`
> (abbreviate as `getValueLength()` below) to get the full length of the
> `value` part in the `key-value` pair. But `getValueLength()` can only get the
> full length of the `value` part when the full length is less than
> `tfile.io.chunk.size`, otherwise, `getValueLength()` throws an exception,
> causing `readRecords()` to fail, and thus resulting in failures in the
> aforementioned 4 tests. This is because `getValueLength()` do not know the
> full length of the `value` part when `value` part's size is larger than
> `tfile.io.chunk.size`.
>
> *Fixes*
> `readRecords()` should instead call
> `org.apache.hadoop.io.file.tfile.TFile.Reader.Scanner.Entry#getValue(byte[])`
> (abbreviate as `getValue()` below), which returns the correct full length of
> the `value` part despite whether the `value` length is larger than
> `tfile.io.chunk.size`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
