[jira] [Commented] (HADOOP-15502) Rolling upgrade from HADOOP 2.x to 3.x breaks due to backward in-compatible change in MetricsPlugin interface
[ https://issues.apache.org/jira/browse/HADOOP-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494642#comment-16494642 ] Suma Shivaprasad commented on HADOOP-15502: --- Thanks [~rohithsharma] for finding this issue > Rolling upgrade from HADOOP 2.x to 3.x breaks due to backward in-compatible > change in MetricsPlugin interface > - > > Key: HADOOP-15502 > URL: https://issues.apache.org/jira/browse/HADOOP-15502 > Project: Hadoop Common > Issue Type: Task >Reporter: Suma Shivaprasad >Priority: Major > > HADOOP-13660 breaks backward compatibility due to API changes in > MetricsPlugin. This also breaks rolling upgrades since any client > implementing this - like HadoopTimelineMetricsSink in Ambari is affected - > see https://issues.apache.org/jira/browse/AMBARI-20156. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494608#comment-16494608
]
Hudson commented on HADOOP-15480:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14313 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14313/])
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when (fabbri:
rev 5f6769f7964ff002b6c04a95893b5baeb424b6db)
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolLocal.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch, HADOOP-15480.005.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
[jira] [Updated] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-15480:
--
Resolution: Fixed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks for fixing this [~gabor.bota]!
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch, HADOOP-15480.005.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
>
[jira] [Commented] (HADOOP-15502) Rolling upgrade from HADOOP 2.x to 3.x breaks due to backward in-compatible change in MetricsPlugin interface
[ https://issues.apache.org/jira/browse/HADOOP-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494539#comment-16494539 ] Suma Shivaprasad commented on HADOOP-15502: --- This issue is seen when NameNode/DN etc are configured to write to HadoopTimelineMetricsSink by external tools like Ambari and this fails starting up NN/DN in the upgraded version. > Rolling upgrade from HADOOP 2.x to 3.x breaks due to backward in-compatible > change in MetricsPlugin interface > - > > Key: HADOOP-15502 > URL: https://issues.apache.org/jira/browse/HADOOP-15502 > Project: Hadoop Common > Issue Type: Task >Reporter: Suma Shivaprasad >Priority: Major > > HADOOP-13660 breaks backward compatibility due to API changes in > MetricsPlugin. This also breaks rolling upgrades since any client > implementing this - like HadoopTimelineMetricsSink in Ambari is affected - > see https://issues.apache.org/jira/browse/AMBARI-20156. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494509#comment-16494509
]
genericqa commented on HADOOP-15480:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 43m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
34s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15480 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925647/HADOOP-15480.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e30cbd9ee1a3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 135941e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14705/testReport/ |
| Max. process+thread count | 362 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14705/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
>
[jira] [Updated] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15499: --- Affects Version/s: 3.1.1 > Performance severe drop when running RawErasureCoderBenchmark with > NativeRSRawErasureCoder > -- > > Key: HADOOP-15499 > URL: https://issues.apache.org/jira/browse/HADOOP-15499 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 3.0.1, 3.0.2, 3.1.1 >Reporter: SammiChen >Assignee: SammiChen >Priority: Major > Attachments: HADOOP-15499.001.patch > > > Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec > encoding/decoding performance. > 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency > Native ISA-L case. It's abnormal. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 > 1024 1024 > Using 126MB buffer. > ISA-L coder encode 1008MB data, with chunk size 1024KB > Total time: 0.19 s. > Total throughput: 5390.37 MB/s > Threads statistics: > 1 threads in total. > Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 > 50 1024 10240 > Using 120MB buffer. > ISA-L coder encode 54000MB data, with chunk size 10240KB > Total time: 11.58 s. > Total throughput: 4662 MB/s > Threads statistics: > 50 threads in total. > Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. > > RawErasureCoderBenchmark shares a single coder between all concurrent > threads. While > NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on > doDecode and doEncode function. So 50 concurrent threads are forced to use > the shared coder encode/decode function one by one. > > To resolve the issue, there are two approaches. > # Refactor RawErasureCoderBenchmark to use dedicated coder for each > concurrent thread. > # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better > concurrency. Since the synchronized key work is to try to protect the > private variable nativeCoder from being checked in doEncode/doDecode and > being modified in release. We can use reentrantReadWriteLock to increase the > concurrency since doEncode/doDecode can be called multiple times without > change the nativeCoder state. > I prefer approach 2 and will upload a patch later. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15502) Rolling upgrade from HADOOP 2.x to 3.x breaks due to backward in-compatible change in MetricsPlugin interface
Suma Shivaprasad created HADOOP-15502: - Summary: Rolling upgrade from HADOOP 2.x to 3.x breaks due to backward in-compatible change in MetricsPlugin interface Key: HADOOP-15502 URL: https://issues.apache.org/jira/browse/HADOOP-15502 Project: Hadoop Common Issue Type: Task Reporter: Suma Shivaprasad HADOOP-13660 breaks backward compatibility due to API changes in MetricsPlugin. This also breaks rolling upgrades since any client implementing this - like HadoopTimelineMetricsSink in Ambari is affected - see https://issues.apache.org/jira/browse/AMBARI-20156. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15501) [Umbrella] Upgrade efforts to Hadoop 3.x
[ https://issues.apache.org/jira/browse/HADOOP-15501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli moved YARN-8347 to HADOOP-15501: Target Version/s: 3.2.0, 3.1.1 (was: 3.1.1) Issue Type: Task (was: Bug) Key: HADOOP-15501 (was: YARN-8347) Project: Hadoop Common (was: Hadoop YARN) > [Umbrella] Upgrade efforts to Hadoop 3.x > > > Key: HADOOP-15501 > URL: https://issues.apache.org/jira/browse/HADOOP-15501 > Project: Hadoop Common > Issue Type: Task >Reporter: Sunil Govindan >Priority: Major > > This is an umbrella ticket to manage all similar efforts to close gaps for > upgrade efforts to 3.x. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494342#comment-16494342
]
Hudson commented on HADOOP-14946:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14312 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14312/])
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor
(fabbri: rev 30284d020d36c502dad5bdbae61ec48e9dfe9f8c)
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL: https://issues.apache.org/jira/browse/HADOOP-14946
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14946.001.patch, HADOOP-14946.002.patch,
> HADOOP-14946.003.patch
>
>
> The test of the S3Guard CLI prune can sometimes fail on parallel test runs.
> Assumption: it is the parallelism which is causing the problem
> {code}
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
> testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 10.765 sec <<< FAILURE!
> java.lang.AssertionError: Pruned children count [] expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494331#comment-16494331
]
Aaron Fabbri commented on HADOOP-15480:
---
Thanks [~gabor.bota], and thank you for letting me make some small
modifications. I spent a bunch of time debugging / testing so being able to
add in my changes saved me some time.
v5 added: just rebased on latest trunk.
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch, HADOOP-15480.005.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
>
[jira] [Updated] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-15480:
--
Attachment: HADOOP-15480.005.patch
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch, HADOOP-15480.005.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Updated] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-15480:
--
Attachment: (was: HADOOP-15480.005.patch)
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch, HADOOP-15480.005.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Updated] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-15480:
--
Attachment: HADOOP-15480.005.patch
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch, HADOOP-15480.005.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Commented] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494317#comment-16494317
]
Gabor Bota commented on HADOOP-15480:
-
+1 on the latest patch (v004). This way a new test could use a raw
S3AFileSystem if needed later.
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
>
[jira] [Commented] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494316#comment-16494316
]
genericqa commented on HADOOP-15480:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-15480 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15480 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925637/HADOOP-15480.004.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14704/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Updated] (HADOOP-15480) AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
[
https://issues.apache.org/jira/browse/HADOOP-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-15480:
--
Attachment: HADOOP-15480.004.patch
> AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo
> ---
>
> Key: HADOOP-15480
> URL: https://issues.apache.org/jira/browse/HADOOP-15480
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-15480.001.patch, HADOOP-15480.002.patch,
> HADOOP-15480.003.patch, HADOOP-15480.004.patch
>
>
> When running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB, the
> testDiffCommand test fails with the following:
> {noformat}
> testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 8.059 s <<< FAILURE!
> java.lang.AssertionError:
> Mismatched metadata store outputs: MS D 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2
> MSF 100
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3
> S3F 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> MSF 0
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4
> expected:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4]>
> but was:<[
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-0,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-1,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-2,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-3,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/ms_only/file-4,
> s3a://cloudera-dev-gabor-ireland/test/test-diff/s3_only/file-4]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testDiffCommand(AbstractS3GuardToolTestBase.java:382)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
[jira] [Updated] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-14946:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thank you for the contribution [~gabor.bota].
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL: https://issues.apache.org/jira/browse/HADOOP-14946
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14946.001.patch, HADOOP-14946.002.patch,
> HADOOP-14946.003.patch
>
>
> The test of the S3Guard CLI prune can sometimes fail on parallel test runs.
> Assumption: it is the parallelism which is causing the problem
> {code}
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
> testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 10.765 sec <<< FAILURE!
> java.lang.AssertionError: Pruned children count [] expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494195#comment-16494195
]
genericqa commented on HADOOP-14946:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 41s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
25s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-14946 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925619/HADOOP-14946.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1106a914ff62 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e3236a9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14703/testReport/ |
| Max. process+thread count | 471 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14703/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL:
[jira] [Commented] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494136#comment-16494136
]
Gabor Bota commented on HADOOP-14946:
-
Thank you [~fabbri]!
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL: https://issues.apache.org/jira/browse/HADOOP-14946
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14946.001.patch, HADOOP-14946.002.patch,
> HADOOP-14946.003.patch
>
>
> The test of the S3Guard CLI prune can sometimes fail on parallel test runs.
> Assumption: it is the parallelism which is causing the problem
> {code}
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
> testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 10.765 sec <<< FAILURE!
> java.lang.AssertionError: Pruned children count [] expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494099#comment-16494099
]
Aaron Fabbri commented on HADOOP-14946:
---
I'll commit the v3 patch once yetus is clean. (same as v2, except use
max_prune_age + 2 instead of + 1 for sleep). This one has been stable in
testing for me in US West 2.
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL: https://issues.apache.org/jira/browse/HADOOP-14946
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14946.001.patch, HADOOP-14946.002.patch,
> HADOOP-14946.003.patch
>
>
> The test of the S3Guard CLI prune can sometimes fail on parallel test runs.
> Assumption: it is the parallelism which is causing the problem
> {code}
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
> testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 10.765 sec <<< FAILURE!
> java.lang.AssertionError: Pruned children count [] expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-14946:
--
Attachment: HADOOP-14946.003.patch
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL: https://issues.apache.org/jira/browse/HADOOP-14946
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14946.001.patch, HADOOP-14946.002.patch,
> HADOOP-14946.003.patch
>
>
> The test of the S3Guard CLI prune can sometimes fail on parallel test runs.
> Assumption: it is the parallelism which is causing the problem
> {code}
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
> testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 10.765 sec <<< FAILURE!
> java.lang.AssertionError: Pruned children count [] expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14201) Some 2.8.0 unit tests are failing on windows
[ https://issues.apache.org/jira/browse/HADOOP-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493914#comment-16493914 ] Íñigo Goiri commented on HADOOP-14201: -- We are now tracking failed unit tests in HADOOP-15475. I think there are some fixes in [^HADOOP-14201-001.patch] that still apply. [~huanbang1993] can you take a look? > Some 2.8.0 unit tests are failing on windows > > > Key: HADOOP-14201 > URL: https://issues.apache.org/jira/browse/HADOOP-14201 > Project: Hadoop Common > Issue Type: Task > Components: test >Affects Versions: 2.8.0 > Environment: Windows Server 2012. >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-14201-001.patch > > > Some of the 2.8.0 tests are failing locally, without much in the way of > diagnostics. They may be false alarms related to system, VM setup, > performance, or they may be a sign of a problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14199) TestFsShellList.testList fails on windows: illegal filenames
[
https://issues.apache.org/jira/browse/HADOOP-14199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493900#comment-16493900
]
Íñigo Goiri commented on HADOOP-14199:
--
[~huanbang1993] opened HADOOP-15496 for this.
Not sure what the right fix is here though.
We could just add an assume to exclude Windows from part of the test.
> TestFsShellList.testList fails on windows: illegal filenames
>
>
> Key: HADOOP-14199
> URL: https://issues.apache.org/jira/browse/HADOOP-14199
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0
> Environment: win64
>Reporter: Steve Loughran
>Priority: Minor
>
> {{TestFsShellList.testList}} fails setting up the files to test against
> {code}
> org.apache.hadoop.io.nativeio.NativeIOException: The filename, directory
> name, or volume label syntax is incorrect.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15496) TestFsShellList#testList fails on Windows
[
https://issues.apache.org/jira/browse/HADOOP-15496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HADOOP-15496:
-
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> TestFsShellList#testList fails on Windows
> -
>
> Key: HADOOP-15496
> URL: https://issues.apache.org/jira/browse/HADOOP-15496
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Anbang Hu
>Assignee: Anbang Hu
>Priority: Minor
> Labels: Windows
> Attachments: HADOOP-15496.000.patch
>
>
> [TestFsShellList#testList|https://builds.apache.org/job/hadoop-trunk-win/478/testReport/org.apache.hadoop.fs/TestFsShellList/testList/]
> fails on Windows because Windows filename does not accept "\", while in the
> test
> {code:java}
> createFile(new Path(testRootDir, "abc\bd\tef"));
> ...
> createFile(new Path(testRootDir, "qq\r123"));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15497) TestTrash should use proper test path to avoid failing on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493841#comment-16493841 ] Hudson commented on HADOOP-15497: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14308 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14308/]) HADOOP-15497. TestTrash should use proper test path to avoid failing on (inigoiri: rev 3c75f8e4933221fa60a87e86a3db5e4727530b6f) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestTrash.java > TestTrash should use proper test path to avoid failing on Windows > - > > Key: HADOOP-15497 > URL: https://issues.apache.org/jira/browse/HADOOP-15497 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 3.2.0, 3.0.3 > > Attachments: HADOOP-15497.000.patch, HDFS-13625.000.patch > > > The following fail on Windows due to improper path: > * > [TestHDFSTrash#testNonDefaultFS|https://builds.apache.org/job/hadoop-trunk-win/478/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testNonDefaultFS/] > * > [TestTrash|https://builds.apache.org/job/hadoop-trunk-win/478/testReport/org.apache.hadoop.fs/TestTrash/] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15406) hadoop-nfs dependencies for mockito and junit are not test scope
[ https://issues.apache.org/jira/browse/HADOOP-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493813#comment-16493813 ] Hudson commented on HADOOP-15406: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14307 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14307/]) YARN-8338. TimelineService V1.5 doesn't come up after HADOOP-15406. (jlowe: rev 31ab960f4f931df273481927b897388895d803ba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml * (edit) hadoop-project/pom.xml > hadoop-nfs dependencies for mockito and junit are not test scope > > > Key: HADOOP-15406 > URL: https://issues.apache.org/jira/browse/HADOOP-15406 > Project: Hadoop Common > Issue Type: Bug > Components: nfs >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.3 > > Attachments: HADOOP-15406.001.patch > > > hadoop-nfs asks for mockito-all and junit for its unit tests but it does not > mark the dependency as being required only for tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15475) Fix broken unit tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493803#comment-16493803 ] Íñigo Goiri commented on HADOOP-15475: -- A couple more builds: ||Run||Date||Total||Skipped||Failed||Passed|| |[459|https://builds.apache.org/job/hadoop-trunk-win/459/testReport/]|2018/05/06|19386|1465|846|17075| |[460|https://builds.apache.org/job/hadoop-trunk-win/460/testReport/]|2018/05/07|16906|1418|754|14734| |[463|https://builds.apache.org/job/hadoop-trunk-win/463/testReport/]|2018/05/10|19252|1472|540|17240| |[467|https://builds.apache.org/job/hadoop-trunk-win/467/testReport/]|2018/05/14|19317|1472|573|17272| |[469|https://builds.apache.org/job/hadoop-trunk-win/469/testReport/]|2018/05/16|19326|1472|516|17388| |[476|https://builds.apache.org/job/hadoop-trunk-win/476/testReport/]|2018/05/23|19355|1473|446|17436| |[479|https://builds.apache.org/job/hadoop-trunk-win/479/testReport/]|2018/05/26|19384|1473|375|17536| |[481|https://builds.apache.org/job/hadoop-trunk-win/481/testReport/]|2018/05/23|19409|1472|405|17532| Last week we resolved a couple JIRAs with a lot of failures in them so we reduced by more than 50. >From build #479 to #481 there was an increase in failed unit tests but I think >it's because of flaky unit tests that we should eventually fix too. We have been mostly focusing on HDFS so far and now we should try to target commons, YARN and MapReduce more. For YARN there are a lot of tests that we are Linux specific and we are thinking to disable in YARN-8359. > Fix broken unit tests on Windows > > > Key: HADOOP-15475 > URL: https://issues.apache.org/jira/browse/HADOOP-15475 > Project: Hadoop Common > Issue Type: Test >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > > There are hundreds of unit tests that fail on Windows. This JIRA tracks the > effort to fix them. > The main reasons for unit test failures on Windows are: > * Windows/Linux path formats (e.g., HDFS-10256). > * Line separator. > * Locked files: Windows locks files when opening them. > ** The typical trigger is not cleaning MiniDFSCluster leaves files locked > when a test times out; they need to be cleaned using After. > * Memory lock size. > * Slow DNS resolution (e.g., HDFS-13569). > * Locked ports (e.g., HDFS-11700) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15497) TestTrash should use proper test path to avoid failing on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HADOOP-15497: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.3 3.2.0 3.1.0 Status: Resolved (was: Patch Available) Thanks [~huanbang1993] for the patch and [~surmountian] for the review. Committed to trunk, branch-3.1, branch-3.0, branch-2, and branch-2.9. > TestTrash should use proper test path to avoid failing on Windows > - > > Key: HADOOP-15497 > URL: https://issues.apache.org/jira/browse/HADOOP-15497 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > Fix For: 3.1.0, 3.2.0, 3.0.3 > > Attachments: HADOOP-15497.000.patch, HDFS-13625.000.patch > > > The following fail on Windows due to improper path: > * > [TestHDFSTrash#testNonDefaultFS|https://builds.apache.org/job/hadoop-trunk-win/478/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testNonDefaultFS/] > * > [TestTrash|https://builds.apache.org/job/hadoop-trunk-win/478/testReport/org.apache.hadoop.fs/TestTrash/] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15217) org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
[ https://issues.apache.org/jira/browse/HADOOP-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493482#comment-16493482 ] Gabor Bota commented on HADOOP-15217: - +1 (non-binding) on the v001 patch > org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces > -- > > Key: HADOOP-15217 > URL: https://issues.apache.org/jira/browse/HADOOP-15217 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Joseph Fourny >Assignee: Zsolt Venczel >Priority: Major > Attachments: HADOOP-15217.01.patch, TestCase.java > > > When _FsUrlStreamHandlerFactory_ is registered with _java.net.URL_ (ex: when > Spark is initialized), it breaks URLs with spaces (even though they are > properly URI-encoded). I traced the problem down to > _FSUrlConnection.connect()_ method. It naively gets the path from the URL, > which contains encoded spaces, and pases it to > _org.apache.hadoop.fs.Path(String)_ constructor. This is not correct, because > the docs clearly say that the string must NOT be encoded. Doing so causes > double encoding within the Path class (ie: %20 becomes %2520). > See attached JUnit test. > This test case mimics an issue I ran into when trying to use Commons > Configuration 1.9 AFTER initializing Spark. Commons Configuration uses URL > class to load configuration files, but Spark installs > _FsUrlStreamHandlerFactory_, which hits this issue. For now, we are using an > AspectJ aspect to "patch" the bytecode at load time to work-around the issue. > The real fix is quite simple. All you need to do is replace this line in > _org.apache.hadoop.fs.FsUrlConnection.connect()_: > is = fs.open(new Path(url.getPath())); > with this line: > is = fs.open(new Path(url.*toUri()*.getPath())); > URI.getPath() will correctly decode the path, which is what is expected by > _org.apache.hadoop.fs.Path(String)_ constructor. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[
https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493406#comment-16493406
]
genericqa commented on HADOOP-15499:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 8 unchanged - 0 fixed = 10 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15499 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925517/HADOOP-15499.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9ed444bb38ca 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 438ef49 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14702/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14702/testReport/ |
| Max. process+thread count | 1501 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14702/console |
| Powered by | Apache Yetus
[jira] [Comment Edited] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493287#comment-16493287 ] SammiChen edited comment on HADOOP-15499 at 5/29/18 9:19 AM: - Performance data before the patch, bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 50 1024 64 Using 126MB buffer. ISA-L coder encode 50400MB data, with chunk size 64KB Total time: 0.98 s. Total throughput: 51639.34 MB/s Threads statistics: 50 threads in total. Performance data after the patch, bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 72 10240 4096 Using 120MB buffer. ISA-L coder encode 734400MB data, with chunk size 4096KB Total time: 8.11 s. Total throughput: 90521.39 MB/s Threads statistics: 72 threads in total. Min: 6.78 s, Max: 7.93 s, Avg: 7.36 s, 90th Percentile: 7.66 s. I also compared the performance data of two scenarios, one is remove all the synchronized key words, another is the current ReentrantReadWriteLock solution. The performance of ReentrantReadWriteLock solution is like less than 5% degrade than the remove synchronized key words case. It's acceptable for me. was (Author: sammi): Performance data before the patch, bin/hadoop jar ./share/hadoop/common/hadoop-common-3.0.0-alpha2.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 50 1024 64 Using 126MB buffer. ISA-L coder encode 50400MB data, with chunk size 64KB Total time: 0.98 s. Total throughput: 51639.34 MB/s Threads statistics: 50 threads in total. Performance data after the patch, bin/hadoop jar ./share/hadoop/common/hadoop-common-3.0.0-alpha2.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 72 10240 4096 Using 120MB buffer. ISA-L coder encode 734400MB data, with chunk size 4096KB Total time: 8.11 s. Total throughput: 90521.39 MB/s Threads statistics: 72 threads in total. Min: 6.78 s, Max: 7.93 s, Avg: 7.36 s, 90th Percentile: 7.66 s. I also compared the performance data of two scenarios, one is remove all the synchronized key words, another is the current ReentrantReadWriteLock solution. The performance of ReentrantReadWriteLock solution is like less than 5% degrade than the remove synchronized key words case. It's acceptable for me. > Performance severe drop when running RawErasureCoderBenchmark with > NativeRSRawErasureCoder > -- > > Key: HADOOP-15499 > URL: https://issues.apache.org/jira/browse/HADOOP-15499 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 3.0.1, 3.0.2 >Reporter: SammiChen >Assignee: SammiChen >Priority: Major > Attachments: HADOOP-15499.001.patch > > > Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec > encoding/decoding performance. > 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency > Native ISA-L case. It's abnormal. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 > 1024 1024 > Using 126MB buffer. > ISA-L coder encode 1008MB data, with chunk size 1024KB > Total time: 0.19 s. > Total throughput: 5390.37 MB/s > Threads statistics: > 1 threads in total. > Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 > 50 1024 10240 > Using 120MB buffer. > ISA-L coder encode 54000MB data, with chunk size 10240KB > Total time: 11.58 s. > Total throughput: 4662 MB/s > Threads statistics: > 50 threads in total. > Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. > > RawErasureCoderBenchmark shares a single coder between all concurrent > threads. While > NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on > doDecode and doEncode function. So 50 concurrent threads are forced to use > the shared coder encode/decode function one by one. > > To resolve the issue, there are two approaches. > # Refactor RawErasureCoderBenchmark to use dedicated coder for each > concurrent thread. > # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better > concurrency. Since the synchronized key work is to try to protect the > private variable nativeCoder from being checked in doEncode/doDecode and > being modified in release. We can use reentrantReadWriteLock to increase the > concurrency since doEncode/doDecode can be
[jira] [Commented] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493287#comment-16493287 ] SammiChen commented on HADOOP-15499: Performance data before the patch, bin/hadoop jar ./share/hadoop/common/hadoop-common-3.0.0-alpha2.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 50 1024 64 Using 126MB buffer. ISA-L coder encode 50400MB data, with chunk size 64KB Total time: 0.98 s. Total throughput: 51639.34 MB/s Threads statistics: 50 threads in total. Performance data after the patch, bin/hadoop jar ./share/hadoop/common/hadoop-common-3.0.0-alpha2.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 72 10240 4096 Using 120MB buffer. ISA-L coder encode 734400MB data, with chunk size 4096KB Total time: 8.11 s. Total throughput: 90521.39 MB/s Threads statistics: 72 threads in total. Min: 6.78 s, Max: 7.93 s, Avg: 7.36 s, 90th Percentile: 7.66 s. I also compared the performance data of two scenarios, one is remove all the synchronized key words, another is the current ReentrantReadWriteLock solution. The performance of ReentrantReadWriteLock solution is like less than 5% degrade than the remove synchronized key words case. It's acceptable for me. > Performance severe drop when running RawErasureCoderBenchmark with > NativeRSRawErasureCoder > -- > > Key: HADOOP-15499 > URL: https://issues.apache.org/jira/browse/HADOOP-15499 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 3.0.1, 3.0.2 >Reporter: SammiChen >Assignee: SammiChen >Priority: Major > Attachments: HADOOP-15499.001.patch > > > Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec > encoding/decoding performance. > 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency > Native ISA-L case. It's abnormal. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 > 1024 1024 > Using 126MB buffer. > ISA-L coder encode 1008MB data, with chunk size 1024KB > Total time: 0.19 s. > Total throughput: 5390.37 MB/s > Threads statistics: > 1 threads in total. > Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 > 50 1024 10240 > Using 120MB buffer. > ISA-L coder encode 54000MB data, with chunk size 10240KB > Total time: 11.58 s. > Total throughput: 4662 MB/s > Threads statistics: > 50 threads in total. > Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. > > RawErasureCoderBenchmark shares a single coder between all concurrent > threads. While > NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on > doDecode and doEncode function. So 50 concurrent threads are forced to use > the shared coder encode/decode function one by one. > > To resolve the issue, there are two approaches. > # Refactor RawErasureCoderBenchmark to use dedicated coder for each > concurrent thread. > # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better > concurrency. Since the synchronized key work is to try to protect the > private variable nativeCoder from being checked in doEncode/doDecode and > being modified in release. We can use reentrantReadWriteLock to increase the > concurrency since doEncode/doDecode can be called multiple times without > change the nativeCoder state. > I prefer approach 2 and will upload a patch later. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15499: --- Attachment: HADOOP-15499.001.patch > Performance severe drop when running RawErasureCoderBenchmark with > NativeRSRawErasureCoder > -- > > Key: HADOOP-15499 > URL: https://issues.apache.org/jira/browse/HADOOP-15499 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 3.0.1, 3.0.2 >Reporter: SammiChen >Assignee: SammiChen >Priority: Major > Attachments: HADOOP-15499.001.patch > > > Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec > encoding/decoding performance. > 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency > Native ISA-L case. It's abnormal. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 > 1024 1024 > Using 126MB buffer. > ISA-L coder encode 1008MB data, with chunk size 1024KB > Total time: 0.19 s. > Total throughput: 5390.37 MB/s > Threads statistics: > 1 threads in total. > Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 > 50 1024 10240 > Using 120MB buffer. > ISA-L coder encode 54000MB data, with chunk size 10240KB > Total time: 11.58 s. > Total throughput: 4662 MB/s > Threads statistics: > 50 threads in total. > Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. > > RawErasureCoderBenchmark shares a single coder between all concurrent > threads. While > NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on > doDecode and doEncode function. So 50 concurrent threads are forced to use > the shared coder encode/decode function one by one. > > To resolve the issue, there are two approaches. > # Refactor RawErasureCoderBenchmark to use dedicated coder for each > concurrent thread. > # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better > concurrency. Since the synchronized key work is to try to protect the > private variable nativeCoder from being checked in doEncode/doDecode and > being modified in release. We can use reentrantReadWriteLock to increase the > concurrency since doEncode/doDecode can be called multiple times without > change the nativeCoder state. > I prefer approach 2 and will upload a patch later. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15499: --- Status: Patch Available (was: Open) > Performance severe drop when running RawErasureCoderBenchmark with > NativeRSRawErasureCoder > -- > > Key: HADOOP-15499 > URL: https://issues.apache.org/jira/browse/HADOOP-15499 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.2, 3.0.1, 3.0.0 >Reporter: SammiChen >Assignee: SammiChen >Priority: Major > Attachments: HADOOP-15499.001.patch > > > Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec > encoding/decoding performance. > 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency > Native ISA-L case. It's abnormal. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 > 1024 1024 > Using 126MB buffer. > ISA-L coder encode 1008MB data, with chunk size 1024KB > Total time: 0.19 s. > Total throughput: 5390.37 MB/s > Threads statistics: > 1 threads in total. > Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 > 50 1024 10240 > Using 120MB buffer. > ISA-L coder encode 54000MB data, with chunk size 10240KB > Total time: 11.58 s. > Total throughput: 4662 MB/s > Threads statistics: > 50 threads in total. > Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. > > RawErasureCoderBenchmark shares a single coder between all concurrent > threads. While > NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on > doDecode and doEncode function. So 50 concurrent threads are forced to use > the shared coder encode/decode function one by one. > > To resolve the issue, there are two approaches. > # Refactor RawErasureCoderBenchmark to use dedicated coder for each > concurrent thread. > # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better > concurrency. Since the synchronized key work is to try to protect the > private variable nativeCoder from being checked in doEncode/doDecode and > being modified in release. We can use reentrantReadWriteLock to increase the > concurrency since doEncode/doDecode can be called multiple times without > change the nativeCoder state. > I prefer approach 2 and will upload a patch later. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15497) TestTrash should use proper test path to avoid failing on Windows
[ https://issues.apache.org/jira/browse/HADOOP-15497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493274#comment-16493274 ] Xiao Liang commented on HADOOP-15497: - The fix looks good to me, +1 for [^HADOOP-15497.000.patch] > TestTrash should use proper test path to avoid failing on Windows > - > > Key: HADOOP-15497 > URL: https://issues.apache.org/jira/browse/HADOOP-15497 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Anbang Hu >Assignee: Anbang Hu >Priority: Minor > Labels: Windows > Attachments: HADOOP-15497.000.patch, HDFS-13625.000.patch > > > The following fail on Windows due to improper path: > * > [TestHDFSTrash#testNonDefaultFS|https://builds.apache.org/job/hadoop-trunk-win/478/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testNonDefaultFS/] > * > [TestTrash|https://builds.apache.org/job/hadoop-trunk-win/478/testReport/org.apache.hadoop.fs/TestTrash/] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15500) Apache Nutch should be included in the releated project list
Cihad Guzel created HADOOP-15500: Summary: Apache Nutch should be included in the releated project list Key: HADOOP-15500 URL: https://issues.apache.org/jira/browse/HADOOP-15500 Project: Hadoop Common Issue Type: Task Components: documentation Reporter: Cihad Guzel [Apache Hadoop welcome page|http://hadoop.apache.org/] contains list of related projects. [Apache Nutch|http://nutch.apache.org/] is a well matured, production ready Web crawler. [Nutch 1.x|https://github.com/apache/nutch/tree/master] enables fine grained configuration, relying on Apache Hadoop data structures, which are great for batch processing. So, Apache Nutch also should be included in the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15499) Performance several drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
SammiChen created HADOOP-15499: -- Summary: Performance several drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder Key: HADOOP-15499 URL: https://issues.apache.org/jira/browse/HADOOP-15499 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.2, 3.0.1, 3.0.0 Reporter: SammiChen Assignee: SammiChen Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec encoding/decoding performance. 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency Native ISA-L case. It's abnormal. bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 1024 1024 Using 126MB buffer. ISA-L coder encode 1008MB data, with chunk size 1024KB Total time: 0.19 s. Total throughput: 5390.37 MB/s Threads statistics: 1 threads in total. Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 50 1024 10240 Using 120MB buffer. ISA-L coder encode 54000MB data, with chunk size 10240KB Total time: 11.58 s. Total throughput: 4662 MB/s Threads statistics: 50 threads in total. Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. RawErasureCoderBenchmark shares a single coder between all concurrent threads. While NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on doDecode and doEncode function. So 50 concurrent threads are forced to use the shared coder encode/decode function one by one. To resolve the issue, there are two approaches. # Refactor RawErasureCoderBenchmark to use dedicated coder for each concurrent thread. # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better concurrency. Since the synchronized key work is to try to protect the private variable nativeCoder from being checked in doEncode/doDecode and being modified in release. We can use reentrantReadWriteLock to increase the concurrency since doEncode/doDecode can be called multiple times without change the nativeCoder state. I prefer approach 2 and will upload a patch later. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15499: --- Summary: Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder (was: Performance several drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder) > Performance severe drop when running RawErasureCoderBenchmark with > NativeRSRawErasureCoder > -- > > Key: HADOOP-15499 > URL: https://issues.apache.org/jira/browse/HADOOP-15499 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 3.0.1, 3.0.2 >Reporter: SammiChen >Assignee: SammiChen >Priority: Major > > Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec > encoding/decoding performance. > 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency > Native ISA-L case. It's abnormal. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 > 1024 1024 > Using 126MB buffer. > ISA-L coder encode 1008MB data, with chunk size 1024KB > Total time: 0.19 s. > Total throughput: 5390.37 MB/s > Threads statistics: > 1 threads in total. > Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 > 50 1024 10240 > Using 120MB buffer. > ISA-L coder encode 54000MB data, with chunk size 10240KB > Total time: 11.58 s. > Total throughput: 4662 MB/s > Threads statistics: > 50 threads in total. > Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. > > RawErasureCoderBenchmark shares a single coder between all concurrent > threads. While > NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on > doDecode and doEncode function. So 50 concurrent threads are forced to use > the shared coder encode/decode function one by one. > > To resolve the issue, there are two approaches. > # Refactor RawErasureCoderBenchmark to use dedicated coder for each > concurrent thread. > # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better > concurrency. Since the synchronized key work is to try to protect the > private variable nativeCoder from being checked in doEncode/doDecode and > being modified in release. We can use reentrantReadWriteLock to increase the > concurrency since doEncode/doDecode can be called multiple times without > change the nativeCoder state. > I prefer approach 2 and will upload a patch later. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
