[jira] [Commented] (HDFS-13016) globStatus javadoc refers to glob pattern as "regular expression"
[
https://issues.apache.org/jira/browse/HDFS-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16325022#comment-16325022
]
genericqa commented on HDFS-13016:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
35s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
35s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13016 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905985/HDFS-13016.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux bf07133cdcb0 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9afb802 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22663/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/22663/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1347 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22663/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT
[jira] [Updated] (HDFS-12636) Ozone: OzoneFileSystem: Implement seek functionality for rpc client
[ https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12636: --- Attachment: HDFS-12636-HDFS-7240.004.patch v4 patch fixes checkstyle and findbug issues. > Ozone: OzoneFileSystem: Implement seek functionality for rpc client > --- > > Key: HDFS-12636 > URL: https://issues.apache.org/jira/browse/HDFS-12636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain > Fix For: HDFS-7240 > > Attachments: HDFS-12636-HDFS-7240.001.patch, > HDFS-12636-HDFS-7240.002.patch, HDFS-12636-HDFS-7240.003.patch, > HDFS-12636-HDFS-7240.004.patch > > > OzoneClient library provides a method to invoke both RPC as well as REST > based methods to ozone. This api will help in the improving both the > performance as well as the interface management in OzoneFileSystem. > This jira will be used to convert the REST based calls to use this new > unified client. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13016) globStatus javadoc refers to glob pattern as "regular expression"
[ https://issues.apache.org/jira/browse/HDFS-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324995#comment-16324995 ] Arpit Agarwal commented on HDFS-13016: -- +1 pending Jenkins. > globStatus javadoc refers to glob pattern as "regular expression" > - > > Key: HDFS-13016 > URL: https://issues.apache.org/jira/browse/HDFS-13016 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, hdfs >Reporter: Ryanne Dolan >Assignee: Mukul Kumar Singh >Priority: Trivial > Attachments: HDFS-13016.001.patch > > > Glob patterns are not regular expressions. Both are well-defined and > universally understood concepts. The term "regular expression" is misapplied > here. > The method name "globStatus" indicates that pathPattern should be a glob, but > the javadoc says pathPattern is a regular expression. The documentation goes > on to describe the accepted format of pathPattern and clearly describes globs. > The term "regular expression" should not be associated with pathPattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test file
[
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324989#comment-16324989
]
Mukul Kumar Singh commented on HDFS-13004:
--
Thanks for updating the patch [~zvenczel].
+1, the latest patch looks good to me.
> TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB
> or larger than the test file
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
> Labels: flaky-test
> Fix For: 3.0.1
>
> Attachments: HDFS-13004.01.patch, HDFS-13004.02.patch,
> HDFS-13004.03.patch
>
>
> andre{code}
> Error:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStr
[jira] [Updated] (HDFS-13016) globStatus javadoc refers to glob pattern as "regular expression"
[ https://issues.apache.org/jira/browse/HDFS-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-13016: - Attachment: HDFS-13016.001.patch > globStatus javadoc refers to glob pattern as "regular expression" > - > > Key: HDFS-13016 > URL: https://issues.apache.org/jira/browse/HDFS-13016 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, hdfs >Reporter: Ryanne Dolan >Assignee: Mukul Kumar Singh >Priority: Trivial > Attachments: HDFS-13016.001.patch > > > Glob patterns are not regular expressions. Both are well-defined and > universally understood concepts. The term "regular expression" is misapplied > here. > The method name "globStatus" indicates that pathPattern should be a glob, but > the javadoc says pathPattern is a regular expression. The documentation goes > on to describe the accepted format of pathPattern and clearly describes globs. > The term "regular expression" should not be associated with pathPattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13016) globStatus javadoc refers to glob pattern as "regular expression"
[ https://issues.apache.org/jira/browse/HDFS-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-13016: - Status: Patch Available (was: Open) > globStatus javadoc refers to glob pattern as "regular expression" > - > > Key: HDFS-13016 > URL: https://issues.apache.org/jira/browse/HDFS-13016 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, hdfs >Reporter: Ryanne Dolan >Assignee: Mukul Kumar Singh >Priority: Trivial > Attachments: HDFS-13016.001.patch > > > Glob patterns are not regular expressions. Both are well-defined and > universally understood concepts. The term "regular expression" is misapplied > here. > The method name "globStatus" indicates that pathPattern should be a glob, but > the javadoc says pathPattern is a regular expression. The documentation goes > on to describe the accepted format of pathPattern and clearly describes globs. > The term "regular expression" should not be associated with pathPattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12972) RBF: Display mount table quota info in Web UI and admin command
[ https://issues.apache.org/jira/browse/HDFS-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324982#comment-16324982 ] Hudson commented on HDFS-12972: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13494 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13494/]) HDFS-12972. RBF: Display mount table quota info in Web UI and admin (yqlin: rev 9afb8025d6549f0ade0ae7d36f5e67cd20c500f4) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUsage.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/MountTable.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/router/federationhealth.html * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/store/records/impl/pb/MountTablePBImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestFederationMetrics.java > RBF: Display mount table quota info in Web UI and admin command > --- > > Key: HDFS-12972 > URL: https://issues.apache.org/jira/browse/HDFS-12972 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: RBF > Fix For: 3.1.0, 2.10.0 > > Attachments: HDFS-12972.001.patch, HDFS-12972.002.patch > > > Display mount table quota info in Web UI and admin command -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12972) RBF: Display mount table quota info in Web UI and admin command
[ https://issues.apache.org/jira/browse/HDFS-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12972: - Labels: RBF (was: ) > RBF: Display mount table quota info in Web UI and admin command > --- > > Key: HDFS-12972 > URL: https://issues.apache.org/jira/browse/HDFS-12972 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: RBF > Fix For: 3.1.0, 2.10.0 > > Attachments: HDFS-12972.001.patch, HDFS-12972.002.patch > > > Display mount table quota info in Web UI and admin command -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12972) RBF: Display mount table quota info in Web UI and admin command
[ https://issues.apache.org/jira/browse/HDFS-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12972: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.10.0 3.1.0 Target Version/s: 3.1.0 (was: 2.9.1, 3.0.1) Status: Resolved (was: Patch Available) > RBF: Display mount table quota info in Web UI and admin command > --- > > Key: HDFS-12972 > URL: https://issues.apache.org/jira/browse/HDFS-12972 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Fix For: 3.1.0, 2.10.0 > > Attachments: HDFS-12972.001.patch, HDFS-12972.002.patch > > > Display mount table quota info in Web UI and admin command -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12972) RBF: Display mount table quota info in Web UI and admin command
[ https://issues.apache.org/jira/browse/HDFS-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324975#comment-16324975 ] Yiqun Lin commented on HDFS-12972: -- Committed this to trunk and branch-2. Thanks [~elgoiri] for the review. > RBF: Display mount table quota info in Web UI and admin command > --- > > Key: HDFS-12972 > URL: https://issues.apache.org/jira/browse/HDFS-12972 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Fix For: 3.1.0, 2.10.0 > > Attachments: HDFS-12972.001.patch, HDFS-12972.002.patch > > > Display mount table quota info in Web UI and admin command -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6804) Add test for race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"
[
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324875#comment-16324875
]
Brahma Reddy Battula commented on HDFS-6804:
[~jojochuang] thanks for review and commit. [~wangg23] thanks for reporting
this.
> Add test for race condition between transferring block and appending block
> causes "Unexpected checksum mismatch exception"
> ---
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.4
>
> Attachments: HDFS-6804-branch-2.8-002.patch,
> HDFS-6804-branch-2.8-003.patch, HDFS-6804-branch-2.8.patch,
> Testcase_append_transfer_block.patch
>
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException:
> Unexpected checksum mismatch while writing
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from
> /192.168.2.101:39495
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Da
> taTransfer: Transmitted
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad
> block to NameNode and NameNode marks the replica on the source datanode as
> corrupt. But actually, the replica on the source datanode is valid. Because
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13013) Fix closeContainer API with the right container state change
[
https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324842#comment-16324842
]
genericqa commented on HDFS-13013:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
3 unchanged - 1 fixed = 4 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
22s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}200m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Dead store to info in
org.apache.hadoop.ozone.scm.StorageContainerManager.notifyObjectStageChange(StorageContainerLocationProtocolProtos$ObjectStageChangeRequestProto$Type,
String,
StorageContainerLocationProtocolProtos$ObjectStageChangeRequestProto$Op,
StorageContainerLocationProtocolProtos$ObjectStageChangeRequestProto$Stage) At
StorageContainerManager.java:org.apache.hadoop.ozone.
[jira] [Updated] (HDFS-6804) Add test for race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"
[
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-6804:
--
Resolution: Fixed
Fix Version/s: 2.8.4
Status: Resolved (was: Patch Available)
Thanks [~brahmareddy] for the patch!
Committed it to branch-2.8 (2.8.4) I updated the summary to reflect that this
is a test-only change.
> Add test for race condition between transferring block and appending block
> causes "Unexpected checksum mismatch exception"
> ---
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.4
>
> Attachments: HDFS-6804-branch-2.8-002.patch,
> HDFS-6804-branch-2.8-003.patch, HDFS-6804-branch-2.8.patch,
> Testcase_append_transfer_block.patch
>
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException:
> Unexpected checksum mismatch while writing
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from
> /192.168.2.101:39495
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Da
> taTransfer: Transmitted
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad
> block to NameNode and NameNode marks the replica on the source datanode as
> corrupt. But actually, the replica on the source datanode is valid. Because
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6804) Add test for race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"
[
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-6804:
--
Summary: Add test for race condition between transferring block and
appending block causes "Unexpected checksum mismatch exception" (was: race
condition between transferring block and appending block causes "Unexpected
checksum mismatch exception" )
> Add test for race condition between transferring block and appending block
> causes "Unexpected checksum mismatch exception"
> ---
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-6804-branch-2.8-002.patch,
> HDFS-6804-branch-2.8-003.patch, HDFS-6804-branch-2.8.patch,
> Testcase_append_transfer_block.patch
>
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException:
> Unexpected checksum mismatch while writing
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from
> /192.168.2.101:39495
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Da
> taTransfer: Transmitted
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad
> block to NameNode and NameNode marks the replica on the source datanode as
> corrupt. But actually, the replica on the source datanode is valid. Because
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12934) RBF: Federation supports global quota
[
https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-12934:
---
Attachment: HDFS-12919-branch-3.001.patch
Thanks [~linyiqun] and [~eddyxu] for the review.
I committed to trunk but I also want to commit to branch-3 to fix the issues
submitting jobs in 3.0.
No point on branch-2 as it doesn't support EC.
> RBF: Federation supports global quota
> -
>
> Key: HDFS-12934
> URL: https://issues.apache.org/jira/browse/HDFS-12934
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: RBF
> Fix For: 3.1.0, 2.10.0
>
> Attachments: HDFS-12919-branch-3.001.patch,
> HDFS-12934-branch-2.001.patch, HDFS-12934-branch-2.002.patch,
> HDFS-12934.001.patch, HDFS-12934.002.patch, HDFS-12934.003.patch,
> HDFS-12934.004.patch, HDFS-12934.005.patch, HDFS-12934.006.patch,
> HDFS-12934.007.patch, HDFS-12934.008.patch, RBF support global quota.pdf
>
>
> Now federation doesn't support set the global quota for each folder.
> Currently the quota will be applied for each subcluster under the specified
> folder via RPC call.
> It will be very useful for users that federation can support setting global
> quota and exposing the command of this.
> In a federated environment, a folder can be spread across multiple
> subclusters. For this reason, we plan to solve this by following way:
> # Set global quota across each subcluster. We don't allow each subcluster can
> exceed maximun quota value.
> # We need to construct one cache map for storing the sum
> quota usage of these subclusters under federation folder. Every time we want
> to do WRITE operation under specified folder, we will get its quota usage
> from cache and verify its quota. If quota exceeded, throw exception,
> otherwise update its quota usage in cache when finishing operations.
> The quota will be set to mount table and as a new field in mount table. The
> set/unset command will be like:
> {noformat}
> hdfs dfsrouteradmin -setQuota -ns -ss
> hdfs dfsrouteradmin -clrQuota
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer
[
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324825#comment-16324825
]
Hudson commented on HDFS-12919:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13492 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13492/])
HDFS-12919. RBF: Support erasure coding methods in RouterRpcServer. (inigoiri:
rev d5d6a0353bb85b882cc4ef60e3a12d63243d34ba)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/AddErasureCodingPolicyResponse.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/RouterDFSCluster.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ErasureCoding.java
> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
> Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch,
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch,
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch,
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch,
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch,
> HDFS-12919.013.patch, HDFS-12919.014.patch, HDFS-12919.015.patch,
> HDFS-12919.016.patch, HDFS-12919.017.patch, HDFS-12919.018.patch,
> HDFS-12919.019.patch, HDFS-12919.020.patch, HDFS-12919.021.patch,
> HDFS-12919.022.patch, HDFS-12919.023.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files.
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "setErasureCodingPolicy" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[
https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324781#comment-16324781
]
genericqa commented on HDFS-13012:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
37m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
46s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
21s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 157 new + 1
unchanged - 0 fixed = 158 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-13012 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905950/HDFS-13012-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 4055d55eedea 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 6ce5ec6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22662/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22662/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/22662/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/22662/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/22662/artifact/out/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22662/artifact/out
[jira] [Commented] (HDFS-12996) DataNode Replica Trash
[ https://issues.apache.org/jira/browse/HDFS-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324775#comment-16324775 ] Hanisha Koneru commented on HDFS-12996: --- Thanks [~virajith] for the review. bq. In Section 13, Step 5, which component receives the restore-replica-trash command? From my understanding this should be the Namenode. However, in step 5 the Namenode is not running. How is this going to work? You are right. The Namenode would first need to be restarted and then the restore-replica-trash command needs to be issued. Steps 5 and 6 got interchanged. Will fix this in the next version of the document. Thanks for catching this. bq. Have you looked at "undoing" the delete operation? I think this would need to handle conflicts (for example, /foo/bar was deleted but a new /foo/bar/ is created) but could avoid having to stop and restart the Namenode. Our current proposal handles a simpler use case of recovery via rolling back everything to an earlier point in time. Something similar to what you proposed could be built on top of the replica trash though. bq. Can the replica purge daemon functionality be implemented in the VolumeScanner? Yes that may be possible. We will look into it when we get to the purge implementation. > DataNode Replica Trash > -- > > Key: HDFS-12996 > URL: https://issues.apache.org/jira/browse/HDFS-12996 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Attachments: DataNode_Replica_Trash_Design_Doc.pdf > > > DataNode Replica Trash will allow administrators to recover from a recent > delete request that resulted in catastrophic loss of user data. This is > achieved by placing all invalidated blocks in a replica trash on the datanode > before completely purging them from the system. The design doc is attached > here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13020) Add JMX metrics for computeReconstruction and computeInvalidation work
[ https://issues.apache.org/jira/browse/HDFS-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-13020: Attachment: (was: HDFS_13020.patch) > Add JMX metrics for computeReconstruction and computeInvalidation work > -- > > Key: HDFS-13020 > URL: https://issues.apache.org/jira/browse/HDFS-13020 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Minor > Attachments: HDFS_13020.patch > > > HDFS configuration allows tweaking of NameNode replication settings. However, > optimal settings may be tricky to determine and usually rely on observation > of key metrics in order to determine. > In one particular instance, I was trying to find the optimal value of > 'dfs.namenode.replication.work.multiplier.per.iteration' but was unable to > find any metrics around the ReconstructionMonitor's replication and > invalidation cycles that is mostly affected by changing that property. > If we expose some trivial JMX metrics for how long these cycles take we can > better understand what effects tweaking the work multiplier will have. > Of course, I am also open to suggestions around what else we should track > here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13020) Add JMX metrics for computeReconstruction and computeInvalidation work
[ https://issues.apache.org/jira/browse/HDFS-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-13020: Attachment: HDFS_13020.patch Attaching basic patch. It has no unit tests but I will add some soon. Just showcasing what I'd like to add. > Add JMX metrics for computeReconstruction and computeInvalidation work > -- > > Key: HDFS-13020 > URL: https://issues.apache.org/jira/browse/HDFS-13020 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Minor > Attachments: HDFS_13020.patch > > > HDFS configuration allows tweaking of NameNode replication settings. However, > optimal settings may be tricky to determine and usually rely on observation > of key metrics in order to determine. > In one particular instance, I was trying to find the optimal value of > 'dfs.namenode.replication.work.multiplier.per.iteration' but was unable to > find any metrics around the ReconstructionMonitor's replication and > invalidation cycles that is mostly affected by changing that property. > If we expose some trivial JMX metrics for how long these cycles take we can > better understand what effects tweaking the work multiplier will have. > Of course, I am also open to suggestions around what else we should track > here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13020) Add JMX metrics for computeReconstruction and computeInvalidation work
[ https://issues.apache.org/jira/browse/HDFS-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-13020 started by Plamen Jeliazkov. --- > Add JMX metrics for computeReconstruction and computeInvalidation work > -- > > Key: HDFS-13020 > URL: https://issues.apache.org/jira/browse/HDFS-13020 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Minor > > HDFS configuration allows tweaking of NameNode replication settings. However, > optimal settings may be tricky to determine and usually rely on observation > of key metrics in order to determine. > In one particular instance, I was trying to find the optimal value of > 'dfs.namenode.replication.work.multiplier.per.iteration' but was unable to > find any metrics around the ReconstructionMonitor's replication and > invalidation cycles that is mostly affected by changing that property. > If we expose some trivial JMX metrics for how long these cycles take we can > better understand what effects tweaking the work multiplier will have. > Of course, I am also open to suggestions around what else we should track > here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13020) Add JMX metrics for computeReconstruction and computeInvalidation work
Plamen Jeliazkov created HDFS-13020: --- Summary: Add JMX metrics for computeReconstruction and computeInvalidation work Key: HDFS-13020 URL: https://issues.apache.org/jira/browse/HDFS-13020 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor HDFS configuration allows tweaking of NameNode replication settings. However, optimal settings may be tricky to determine and usually rely on observation of key metrics in order to determine. In one particular instance, I was trying to find the optimal value of 'dfs.namenode.replication.work.multiplier.per.iteration' but was unable to find any metrics around the ReconstructionMonitor's replication and invalidation cycles that is mostly affected by changing that property. If we expose some trivial JMX metrics for how long these cycles take we can better understand what effects tweaking the work multiplier will have. Of course, I am also open to suggestions around what else we should track here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12968) Ozone : TestSCMCli and TestContainerStateManager tests are failing consistently while updating the container state info.
[
https://issues.apache.org/jira/browse/HDFS-12968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDFS-12968.
---
Resolution: Duplicate
Resolve as a dup of HDFS-13013.
> Ozone : TestSCMCli and TestContainerStateManager tests are failing
> consistently while updating the container state info.
>
>
> Key: HDFS-12968
> URL: https://issues.apache.org/jira/browse/HDFS-12968
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Fix For: HDFS-7240
>
>
> TestContainerStateManager#testUpdateContainerState is failing with the
> following exception:
> {code}
> org.apache.hadoop.ozone.scm.exceptions.SCMException: Failed to update
> container state container28655, reason: invalid state transition from state:
> OPEN upon event: CLOSE.
> at
> org.apache.hadoop.ozone.scm.container.ContainerStateManager.updateContainerState(ContainerStateManager.java:355)
> at
> org.apache.hadoop.ozone.scm.container.ContainerMapping.updateContainerState(ContainerMapping.java:336)
> at
> org.apache.hadoop.ozone.scm.container.TestContainerStateManager.testUpdateContainerState(TestContainerStateManager.java:244)
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> {code}
> Similarly, TestSCMCli#testDeleteContainer and TestSCMCli#testInfoContainer
> are failing with the same exception.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13013) Fix closeContainer API with the right container state change
[ https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13013: -- Summary: Fix closeContainer API with the right container state change (was: Fix closeContainer after HDFS-12980) > Fix closeContainer API with the right container state change > > > Key: HDFS-13013 > URL: https://issues.apache.org/jira/browse/HDFS-13013 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-13013-HDFS-7240.001.patch > > > SCMCLI close container command is based on ContainerMapping#closeContainer, > which is based on the state machine (open->close) before HDFS-12980. > HDFS-12980 changes the container state machine. A container has to be > finalized into closing state before closed. (open->closing->closed). This > ticket is opened to fix ContainerMapping#closeContainer to match the new > state machine. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[ https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13012: -- Status: Patch Available (was: Open) > Fix TestOzoneCOnfigurationFields > > > Key: HDFS-13012 > URL: https://issues.apache.org/jira/browse/HDFS-13012 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-13012-HDFS-7240.001.patch > > > "dfs.container.ratis.num.write.chunk.threads" and > "dfs.container.ratis.segment.size" were added with HDFS-12853, they need to > be added to ozone-default.xml to unblock test failures in > TestOzoneConfigurationFields. cc: [~msingh] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[ https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDFS-13012: - Assignee: Xiaoyu Yao (was: Mukul Kumar Singh) > Fix TestOzoneCOnfigurationFields > > > Key: HDFS-13012 > URL: https://issues.apache.org/jira/browse/HDFS-13012 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-13012-HDFS-7240.001.patch > > > "dfs.container.ratis.num.write.chunk.threads" and > "dfs.container.ratis.segment.size" were added with HDFS-12853, they need to > be added to ozone-default.xml to unblock test failures in > TestOzoneConfigurationFields. cc: [~msingh] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[ https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13012: -- Attachment: HDFS-13012-HDFS-7240.001.patch > Fix TestOzoneCOnfigurationFields > > > Key: HDFS-13012 > URL: https://issues.apache.org/jira/browse/HDFS-13012 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-13012-HDFS-7240.001.patch > > > "dfs.container.ratis.num.write.chunk.threads" and > "dfs.container.ratis.segment.size" were added with HDFS-12853, they need to > be added to ozone-default.xml to unblock test failures in > TestOzoneConfigurationFields. cc: [~msingh] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13012) Fix TestOzoneCOnfigurationFields
[ https://issues.apache.org/jira/browse/HDFS-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13012: -- Summary: Fix TestOzoneCOnfigurationFields (was: Fix TestOzoneCOnfigurationFields after HDFS-12853) > Fix TestOzoneCOnfigurationFields > > > Key: HDFS-13012 > URL: https://issues.apache.org/jira/browse/HDFS-13012 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Mukul Kumar Singh >Priority: Minor > > "dfs.container.ratis.num.write.chunk.threads" and > "dfs.container.ratis.segment.size" were added with HDFS-12853, they need to > be added to ozone-default.xml to unblock test failures in > TestOzoneConfigurationFields. cc: [~msingh] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10285: --- Attachment: SPS Modularization.pdf > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-SPS-TestReport-20170708.pdf, SPS Modularization.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer
[
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324613#comment-16324613
]
Lei (Eddy) Xu commented on HDFS-12919:
--
[~elgoiri] Thanks for the contribution! The code looks good to me. It mostly
pass the calls via the RBF proxy.
+1
> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
> Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch,
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch,
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch,
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch,
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch,
> HDFS-12919.013.patch, HDFS-12919.014.patch, HDFS-12919.015.patch,
> HDFS-12919.016.patch, HDFS-12919.017.patch, HDFS-12919.018.patch,
> HDFS-12919.019.patch, HDFS-12919.020.patch, HDFS-12919.021.patch,
> HDFS-12919.022.patch, HDFS-12919.023.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files.
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "setErasureCodingPolicy" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324607#comment-16324607
]
Ajay Kumar commented on HDFS-12942:
---
{quote}While cleaning up the new replica on failure, we call
volume.onBlockFileDeletion which decrements the space used for the block on the
new target volume. However I am not sure that the space used was incremented in
the first place by copyReplicaToVolume. This may be a pre-existing bug
actually.{quote}
[~arpitagarwal],[~virajith] good catch.
{code} try (AutoCloseableLock lock = datasetLock.acquire()) {
// Increment numBlocks here as this block moved without knowing to BPS
FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
volume.incrNumBlocks(block.getBlockPoolId());
}{code}
Seems in case of success we should increment the dfs used and no of blocks for
new volume and decrement the same for old block. Currently we only increment no
of blocks. Is this bug or intentional? [~arpitagarwal], thanks for offline
discussion on this.
In case of failure we should free up memory on disk. (will update patch for
same)
{quote}Adding to this, in patch v4, the generation stamp check in
finalizeReplica is done after a call to v.addFinalizedBlock which seems wasted
work if the check doesn't pass. Can the generation stamp check be done before
the call to v.addFinalizedBlock?{quote}
Agree,Will move check in start of finalize call.
> Synchronization issue in FSDataSetImpl#moveBlock
>
>
> Key: HDFS-12942
> URL: https://issues.apache.org/jira/browse/HDFS-12942
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch,
> HDFS-12942.003.patch, HDFS-12942.004.patch
>
>
> FSDataSetImpl#moveBlock works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324607#comment-16324607
]
Ajay Kumar edited comment on HDFS-12942 at 1/12/18 9:58 PM:
{quote}While cleaning up the new replica on failure, we call
volume.onBlockFileDeletion which decrements the space used for the block on the
new target volume. However I am not sure that the space used was incremented in
the first place by copyReplicaToVolume. This may be a pre-existing bug
actually.{quote}
[~arpitagarwal],[~virajith] good catch.
{code} try (AutoCloseableLock lock = datasetLock.acquire()) {
// Increment numBlocks here as this block moved without knowing to BPS
FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
volume.incrNumBlocks(block.getBlockPoolId());
}{code}
Seems in case of success we should increment the "dfs used" and "no of blocks"
for new volume and decrement the same for old volume. Currently we only
increment no of blocks. Is this bug or intentional? [~arpitagarwal], thanks for
offline discussion on this.
In case of failure we should free up memory on disk. (will update patch for
same)
{quote}Adding to this, in patch v4, the generation stamp check in
finalizeReplica is done after a call to v.addFinalizedBlock which seems wasted
work if the check doesn't pass. Can the generation stamp check be done before
the call to v.addFinalizedBlock?{quote}
Agree,Will move check in start of finalize call.
was (Author: ajayydv):
{quote}While cleaning up the new replica on failure, we call
volume.onBlockFileDeletion which decrements the space used for the block on the
new target volume. However I am not sure that the space used was incremented in
the first place by copyReplicaToVolume. This may be a pre-existing bug
actually.{quote}
[~arpitagarwal],[~virajith] good catch.
{code} try (AutoCloseableLock lock = datasetLock.acquire()) {
// Increment numBlocks here as this block moved without knowing to BPS
FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
volume.incrNumBlocks(block.getBlockPoolId());
}{code}
Seems in case of success we should increment the dfs used and no of blocks for
new volume and decrement the same for old block. Currently we only increment no
of blocks. Is this bug or intentional? [~arpitagarwal], thanks for offline
discussion on this.
In case of failure we should free up memory on disk. (will update patch for
same)
{quote}Adding to this, in patch v4, the generation stamp check in
finalizeReplica is done after a call to v.addFinalizedBlock which seems wasted
work if the check doesn't pass. Can the generation stamp check be done before
the call to v.addFinalizedBlock?{quote}
Agree,Will move check in start of finalize call.
> Synchronization issue in FSDataSetImpl#moveBlock
>
>
> Key: HDFS-12942
> URL: https://issues.apache.org/jira/browse/HDFS-12942
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch,
> HDFS-12942.003.patch, HDFS-12942.004.patch
>
>
> FSDataSetImpl#moveBlock works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324606#comment-16324606
]
Manoj Govindassamy commented on HDFS-11847:
---
Given that HDFS-10480 is available in 3.0, back ported both HDFS-11847 and
HDFS-11848 to branch-3.
> Enhance dfsadmin listOpenFiles command to list files blocking datanode
> decommissioning
> --
>
> Key: HDFS-11847
> URL: https://issues.apache.org/jira/browse/HDFS-11847
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-11847.01.patch, HDFS-11847.02.patch,
> HDFS-11847.03.patch, HDFS-11847.04.patch, HDFS-11847.05.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> Additionally, it would be very useful to only list open files that are
> blocking the DataNode decommissioning. With thousand+ node clusters, where
> there might be machines added and removed regularly for maintenance, any
> option to monitor and debug decommissioning status is very helpful. Proposal
> here is to add suboptions to {{listOpenFiles}} for the above case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324604#comment-16324604
]
Manoj Govindassamy commented on HDFS-11848:
---
[~linyiqun],
Given that HDFS-10480 is available in 3.0, back ported both HDFS-11847 and
HDFS-11848 to branch-3. Hope this is ok with you. Please let me know if
otherwise.
> Enhance dfsadmin listOpenFiles command to list files under a given path
> ---
>
> Key: HDFS-11848
> URL: https://issues.apache.org/jira/browse/HDFS-11848
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Yiqun Lin
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-11848.001.patch, HDFS-11848.002.patch,
> HDFS-11848.003.patch, HDFS-11848.004.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> One more thing that would be nice here is to filter the output on a passed
> path or DataNode. Usecases: An admin might already know a stale file by path
> (perhaps from fsck's -openforwrite), and wants to figure out who the lease
> holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files
> filtered by path.
> {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the
> open file list for any given ancestor directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[
https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11848:
--
Fix Version/s: 3.0.1
> Enhance dfsadmin listOpenFiles command to list files under a given path
> ---
>
> Key: HDFS-11848
> URL: https://issues.apache.org/jira/browse/HDFS-11848
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Yiqun Lin
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-11848.001.patch, HDFS-11848.002.patch,
> HDFS-11848.003.patch, HDFS-11848.004.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> One more thing that would be nice here is to filter the output on a passed
> path or DataNode. Usecases: An admin might already know a stale file by path
> (perhaps from fsck's -openforwrite), and wants to figure out who the lease
> holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files
> filtered by path.
> {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the
> open file list for any given ancestor directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13013) Fix closeContainer after HDFS-12980
[ https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13013: -- Attachment: (was: HDFS-13013-HDFS-7240.001.patch) > Fix closeContainer after HDFS-12980 > --- > > Key: HDFS-13013 > URL: https://issues.apache.org/jira/browse/HDFS-13013 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-13013-HDFS-7240.001.patch > > > SCMCLI close container command is based on ContainerMapping#closeContainer, > which is based on the state machine (open->close) before HDFS-12980. > HDFS-12980 changes the container state machine. A container has to be > finalized into closing state before closed. (open->closing->closed). This > ticket is opened to fix ContainerMapping#closeContainer to match the new > state machine. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13013) Fix closeContainer after HDFS-12980
[ https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13013: -- Attachment: HDFS-13013-HDFS-7240.001.patch > Fix closeContainer after HDFS-12980 > --- > > Key: HDFS-13013 > URL: https://issues.apache.org/jira/browse/HDFS-13013 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-13013-HDFS-7240.001.patch > > > SCMCLI close container command is based on ContainerMapping#closeContainer, > which is based on the state machine (open->close) before HDFS-12980. > HDFS-12980 changes the container state machine. A container has to be > finalized into closing state before closed. (open->closing->closed). This > ticket is opened to fix ContainerMapping#closeContainer to match the new > state machine. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12990) Change default NameNode RPC port back to 8020
[ https://issues.apache.org/jira/browse/HDFS-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324589#comment-16324589 ] Chris Douglas commented on HDFS-12990: -- bq. Can we retract Hadoop 3.0.0 release and restore NN port to 8020 that we made a bad release? Can this be an alternate approach to produce Hadoop 3.0.1? No, that's not available. bq. why not fixing it in 4.0.0 and declare 3.x as dead? bq. Are there bugs in our current release procedure? bq. This is a good lesson to avoid time driven release for major version change. Nah. Even if 3.x were "time driven", [~vinodkv] [predicted|https://s.apache.org/KnCL] that incompatible changes would require incompatible fixes, and of course it happened. It _always_ happens, because we don't control when others try out the release. We don't know how people are using and extending Hadoop, so we curate compatibility guidelines and specifications to signal what's intended so applications avoid relying on incidental behavior. But our guidelines are no more a contract than they are a suicide pact. bq. Are we going to have other incompatible changes in the future minor and dot releases? What is the criteria to decide which incompatible changes are allowed? I'm sure there are more of these, and we'll remember this issue fondly for having such a straightforward fix. When an incompatible change is a mistake, it's a sunk cost. From there, we find the least harmful way to address it and move on. We created compatibility guidelines to prioritize users' needs over development expedience. Reverting the NN port change is consistent with that intent. We failed to follow our own policy when this change was committed, and absent a technical justification for this incompatible change, it is a bug. The current patch restores compatibility. This needs to converge. Absent a technical justification for the original change or the veto, and without any alternative approach, the current patch is the best remedy we have. +1 > Change default NameNode RPC port back to 8020 > - > > Key: HDFS-12990 > URL: https://issues.apache.org/jira/browse/HDFS-12990 > Project: Hadoop HDFS > Issue Type: Task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Critical > Attachments: HDFS-12990.01.patch > > > In HDFS-9427 (HDFS should not default to ephemeral ports), we changed all > default ports to ephemeral ports, which is very appreciated by admin. As part > of that change, we also modified the NN RPC port from the famous 8020 to > 9820, to be closer to other ports changed there. > With more integration going on, it appears that all the other ephemeral port > changes are fine, but the NN RPC port change is painful for downstream on > migrating to Hadoop 3. Some examples include: > # Hive table locations pointing to hdfs://nn:port/dir > # Downstream minicluster unit tests that assumed 8020 > # Oozie workflows / downstream scripts that used 8020 > This isn't a problem for HA URLs, since that does not include the port > number. But considering the downstream impact, instead of requiring all of > them change their stuff, it would be a way better experience to leave the NN > port unchanged. This will benefit Hadoop 3 adoption and ease unnecessary > upgrade burdens. > It is of course incompatible, but giving 3.0.0 is just out, IMO it worths to > switch the port back. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13008) Ozone: Add DN container open/close state to container report
[ https://issues.apache.org/jira/browse/HDFS-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13008: -- Attachment: HDFS-13013-HDFS-7240.001.patch > Ozone: Add DN container open/close state to container report > > > Key: HDFS-13008 > URL: https://issues.apache.org/jira/browse/HDFS-13008 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > > HDFS-12799 added support to allow SCM send closeContainerCommand to DNs. This > ticket is opened to add the DN container close state to container report so > that SCM container state manager can update state from closing to closed when > DN side container is fully closed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13013) Fix closeContainer after HDFS-12980
[ https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13013: -- Status: Patch Available (was: Open) > Fix closeContainer after HDFS-12980 > --- > > Key: HDFS-13013 > URL: https://issues.apache.org/jira/browse/HDFS-13013 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-13013-HDFS-7240.001.patch > > > SCMCLI close container command is based on ContainerMapping#closeContainer, > which is based on the state machine (open->close) before HDFS-12980. > HDFS-12980 changes the container state machine. A container has to be > finalized into closing state before closed. (open->closing->closed). This > ticket is opened to fix ContainerMapping#closeContainer to match the new > state machine. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13013) Fix closeContainer after HDFS-12980
[ https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13013: -- Attachment: HDFS-13013-HDFS-7240.001.patch > Fix closeContainer after HDFS-12980 > --- > > Key: HDFS-13013 > URL: https://issues.apache.org/jira/browse/HDFS-13013 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-13013-HDFS-7240.001.patch > > > SCMCLI close container command is based on ContainerMapping#closeContainer, > which is based on the state machine (open->close) before HDFS-12980. > HDFS-12980 changes the container state machine. A container has to be > finalized into closing state before closed. (open->closing->closed). This > ticket is opened to fix ContainerMapping#closeContainer to match the new > state machine. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13008) Ozone: Add DN container open/close state to container report
[ https://issues.apache.org/jira/browse/HDFS-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13008: -- Attachment: (was: HDFS-13013-HDFS-7240.001.patch) > Ozone: Add DN container open/close state to container report > > > Key: HDFS-13008 > URL: https://issues.apache.org/jira/browse/HDFS-13008 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > > HDFS-12799 added support to allow SCM send closeContainerCommand to DNs. This > ticket is opened to add the DN container close state to container report so > that SCM container state manager can update state from closing to closed when > DN side container is fully closed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11847:
--
Fix Version/s: 3.0.1
> Enhance dfsadmin listOpenFiles command to list files blocking datanode
> decommissioning
> --
>
> Key: HDFS-11847
> URL: https://issues.apache.org/jira/browse/HDFS-11847
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-11847.01.patch, HDFS-11847.02.patch,
> HDFS-11847.03.patch, HDFS-11847.04.patch, HDFS-11847.05.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> Additionally, it would be very useful to only list open files that are
> blocking the DataNode decommissioning. With thousand+ node clusters, where
> there might be machines added and removed regularly for maintenance, any
> option to monitor and debug decommissioning status is very helpful. Proposal
> here is to add suboptions to {{listOpenFiles}} for the above case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13013) Fix closeContainer after HDFS-12980
[ https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13013: -- Affects Version/s: HDFS-7240 > Fix closeContainer after HDFS-12980 > --- > > Key: HDFS-13013 > URL: https://issues.apache.org/jira/browse/HDFS-13013 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > > SCMCLI close container command is based on ContainerMapping#closeContainer, > which is based on the state machine (open->close) before HDFS-12980. > HDFS-12980 changes the container state machine. A container has to be > finalized into closing state before closed. (open->closing->closed). This > ticket is opened to fix ContainerMapping#closeContainer to match the new > state machine. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13013) Fix closeContainer after HDFS-12980
[ https://issues.apache.org/jira/browse/HDFS-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13013: -- Summary: Fix closeContainer after HDFS-12980 (was: Fix ContainerMapping#closeContainer after HDFS-12980) > Fix closeContainer after HDFS-12980 > --- > > Key: HDFS-13013 > URL: https://issues.apache.org/jira/browse/HDFS-13013 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > > SCMCLI close container command is based on ContainerMapping#closeContainer, > which is based on the state machine (open->close) before HDFS-12980. > HDFS-12980 changes the container state machine. A container has to be > finalized into closing state before closed. (open->closing->closed). This > ticket is opened to fix ContainerMapping#closeContainer to match the new > state machine. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13019) dfs put with -f to dir with existing file in dest should return 0, not -1
[ https://issues.apache.org/jira/browse/HDFS-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HDFS-13019: - Assignee: Bharat Viswanadham > dfs put with -f to dir with existing file in dest should return 0, not -1 > - > > Key: HDFS-13019 > URL: https://issues.apache.org/jira/browse/HDFS-13019 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.3 >Reporter: BRYAN T VOLD >Assignee: Bharat Viswanadham > > When doing an hdfs dfs -put and there are existing > files, the return code will be -1, which is expected. > When you do an hdfs dfs -put -f (force), the error code > still comes back as -1, which is unexpected. > If you use hdfs dfs -copyFromLocal using the same directories as above, the > -copyFromLocal stills gives the error which is expected and when you pass -f > to this version of the command, the error code is 0, which I think is the > correct behavior and I think the hdfs dfs -put should match this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13019) dfs put with -f to dir with existing file in dest should return 0, not -1
BRYAN T VOLD created HDFS-13019: --- Summary: dfs put with -f to dir with existing file in dest should return 0, not -1 Key: HDFS-13019 URL: https://issues.apache.org/jira/browse/HDFS-13019 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 2.7.3 Reporter: BRYAN T VOLD When doing an hdfs dfs -put and there are existing files, the return code will be -1, which is expected. When you do an hdfs dfs -put -f (force), the error code still comes back as -1, which is unexpected. If you use hdfs dfs -copyFromLocal using the same directories as above, the -copyFromLocal stills gives the error which is expected and when you pass -f to this version of the command, the error code is 0, which I think is the correct behavior and I think the hdfs dfs -put should match this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12636) Ozone: OzoneFileSystem: Implement seek functionality for rpc client
[
https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324479#comment-16324479
]
genericqa commented on HDFS-12636:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
4s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 20s{color} | {color:orange} root: The patch generated 6 new + 2 unchanged -
0 fixed = 8 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
17s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 2 new
+ 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}173m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-ozone in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
38s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}296m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Inconsistent synchronization of
org.apache.hadoop.scm.storage.ChunkInputStream.bufferIndex; locked 75% of time
Unsynchronized access at ChunkInputStream.java:75% of time Unsynchronized
access at ChunkInputStream.java:[line 240] |
| | Inconsistent synchronization of
org.apache.hadoop.scm.storage.ChunkInputStream.buffers; locked 66% of time
Unsynchronized access at ChunkInputStream.java:66% of time Unsynchronized
access at ChunkInputStream.java:[line 232] |
| Failed junit tests | hadoop.
[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer
[
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324455#comment-16324455
]
genericqa commented on HDFS-12919:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.TestStartup |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12919 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905893/HDFS-12919.023.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e1e6051147d5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchpro
[jira] [Commented] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test file
[
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324449#comment-16324449
]
genericqa commented on HDFS-13004:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 55s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}136m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}199m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13004 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905885/HDFS-13004.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux df8004e531a5 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b278f7b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22657/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22657/testReport/ |
| Max. process+thread count | 3560 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop
[jira] [Commented] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[
https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324421#comment-16324421
]
genericqa commented on HDFS-13017:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}146m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}207m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.ozone.tools.TestCorona |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.web.client.Tes
[jira] [Comment Edited] (HDFS-6804) race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"
[
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311238#comment-16311238
]
Brahma Reddy Battula edited comment on HDFS-6804 at 1/12/18 6:48 PM:
-
[~jojochuang]if you get chance,can you check once..? Hope it's ready for commit.
was (Author: brahmareddy):
[~jojochuang] can you please look once.Hope it's ready for commit.
> race condition between transferring block and appending block causes
> "Unexpected checksum mismatch exception"
> --
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-6804-branch-2.8-002.patch,
> HDFS-6804-branch-2.8-003.patch, HDFS-6804-branch-2.8.patch,
> Testcase_append_transfer_block.patch
>
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException:
> Unexpected checksum mismatch while writing
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from
> /192.168.2.101:39495
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Da
> taTransfer: Transmitted
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad
> block to NameNode and NameNode marks the replica on the source datanode as
> corrupt. But actually, the replica on the source datanode is valid. Because
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11915) Sync rbw dir on the first hsync() to avoid file lost on power failure
[ https://issues.apache.org/jira/browse/HDFS-11915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11915: --- Fix Version/s: 3.0.1 > Sync rbw dir on the first hsync() to avoid file lost on power failure > - > > Key: HDFS-11915 > URL: https://issues.apache.org/jira/browse/HDFS-11915 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kanaka Kumar Avvaru >Assignee: Vinayakumar B >Priority: Critical > Fix For: 3.1.0, 2.9.1, 3.0.1 > > Attachments: HDFS-11915-01.patch, HDFS-11915-branch-2-01.patch, > HDFS-11915-branch-2-02.patch > > > As discussed in HDFS-5042, there is a chance to lose blocks on power failure > if rbw file creation entry is not yet sync to device. Then the block created > is nowhere exists on disk. Neither in rbw nor in finalized. > As suggested by [~kihwal], will discuss and track it in this JIRA. > As suggested by [~vinayrpet], May be first hsync() request on block file can > call fsync on its parent directory (rbw) directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations.
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Description: One of the key comment from the discussions was to modularize the SPS code, so we can easily plug in the external/internal implementations. This JIRA for doing the necessary refactoring. So other comments to handle Daryn: # Lock should not kept while executing placement policy. - handled by HDFS-12982 # While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. was: One of the key comment from the discussions was to modularize the SPS code, so we can easily plug in the external/internal implementations. So other comments to handle Daryn: # Lock should not kept while executing placement policy. - already handled by # While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. > [SPS]: Modularize the SPS code and expose necessary interfaces for > external/internal implementations. > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911-HDFS-10285-03.patch, > HDFS-12911.00.patch > > > One of the key comment from the discussions was to modularize the SPS code, > so we can easily plug in the external/internal implementations. This JIRA for > doing the necessary refactoring. > So other comments to handle > Daryn: > # Lock should not kept while executing placement policy. > - handled by HDFS-12982 > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations.
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Description: One of the key comment from the discussions was to modularize the SPS code, so we can easily plug in the external/internal implementations. So other comments to handle Daryn: # Lock should not kept while executing placement policy. - already handled by # While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. was: This is the JIRA for tracking the possible improvements or issues discussed in main JIRA So far comments to handle Daryn: # Lock should not kept while executing placement policy. # While starting up the NN, SPS Xattrs checks happen even if feature disabled. This could potentially impact the startup speed. UMA: # I am adding one more possible improvement to reduce Xattr objects significantly. SPS Xattr is constant object. So, we create one Xattr deduplication object once statically and use the same object reference when required to add SPS Xattr to Inode. So, here additional bytes required for storing SPS Xattr would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr overhead should come down significantly IMO. Lets explore the feasibility on this option. Xattr list Future will not be specially created for SPS, that list would have been created by SetStoragePolicy already on the same directory. So, no extra Feature creation because of SPS alone. # Currently SPS putting long id objects in Q for tracking SPS called Inodes. So, it is additional created and size of it would be (obj ref + value) = (8 + 8) bytes [ ignoring alignment for time being] So, the possible improvement here is, instead of creating new Long obj, we can keep existing inode object for tracking. Advantage is, Inode object already maintained in NN, so no new object creation is needed. So, we just need to maintain one obj ref. Above two points should significantly reduce the memory requirements of SPS. So, for SPS call: 8bytes for called inode tracking + 8 bytes for Xattr ref. # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will reduce unnecessary Node creations inside LinkedList. > [SPS]: Modularize the SPS code and expose necessary interfaces for > external/internal implementations. > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911-HDFS-10285-03.patch, > HDFS-12911.00.patch > > > One of the key comment from the discussions was to modularize the SPS code, > so we can easily plug in the external/internal implementations. > So other comments to handle > Daryn: > # Lock should not kept while executing placement policy. > - already handled by > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12911) [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations.
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Summary: [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations. (was: [SPS]: Fix review comments from discussions in HDFS-10285) > [SPS]: Modularize the SPS code and expose necessary interfaces for > external/internal implementations. > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911-HDFS-10285-03.patch, > HDFS-12911.00.patch > > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. > # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will > reduce unnecessary Node creations inside LinkedList. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11915) Sync rbw dir on the first hsync() to avoid file lost on power failure
[ https://issues.apache.org/jira/browse/HDFS-11915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11915: --- Resolution: Fixed Fix Version/s: 2.9.1 Status: Resolved (was: Patch Available) Committed the patch to branch-2 for 2.9.1 release. Thanks [~vinayrpet]! > Sync rbw dir on the first hsync() to avoid file lost on power failure > - > > Key: HDFS-11915 > URL: https://issues.apache.org/jira/browse/HDFS-11915 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kanaka Kumar Avvaru >Assignee: Vinayakumar B >Priority: Critical > Fix For: 3.1.0, 2.9.1 > > Attachments: HDFS-11915-01.patch, HDFS-11915-branch-2-01.patch, > HDFS-11915-branch-2-02.patch > > > As discussed in HDFS-5042, there is a chance to lose blocks on power failure > if rbw file creation entry is not yet sync to device. Then the block created > is nowhere exists on disk. Neither in rbw nor in finalized. > As suggested by [~kihwal], will discuss and track it in this JIRA. > As suggested by [~vinayrpet], May be first hsync() request on block file can > call fsync on its parent directory (rbw) directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[
https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324270#comment-16324270
]
genericqa commented on HDFS-12911:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} root in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
20s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in HDFS-10285 failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 21 new + 334 unchanged - 0 fixed = 355 total (was 334) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
9s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 5m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12911 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905895/HDFS-12911-HDFS-10285-03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4f5b3d18093b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / ed60479 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/22659/artifact/out/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/22659/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/22659/artifact/out/branch-mvnsite-
[jira] [Updated] (HDFS-12911) [SPS]: Fix review comments from discussions in HDFS-10285
[ https://issues.apache.org/jira/browse/HDFS-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-12911: --- Attachment: HDFS-12911-HDFS-10285-03.patch Attached another version of path. Introduced following Interfaces: SPSService:- Contains sps life cycle APIs and some of processing related APIs FileIdCollector:- An interface for plugging in the directory scanning functionality. BlockMoveTaskHandler:- An interface for implementing the core block movement logic. When users called SPS, BM adds the sps invoked pathids(a path id can be file/directoty) into SPSPathIds class. Now, SPSPathIdProcessor will pick the path ids and uses FileIdCollector to scan that paths. The scanned file ids will be added to SPSService for further processing. SPSService will take file ids and analyzes the block states and finds if any mismatches. If it finds mismatches, then it will use BlockMoveTaskHandler to move the blocks. > [SPS]: Fix review comments from discussions in HDFS-10285 > - > > Key: HDFS-12911 > URL: https://issues.apache.org/jira/browse/HDFS-12911 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R > Attachments: HDFS-12911-HDFS-10285-01.patch, > HDFS-12911-HDFS-10285-02.patch, HDFS-12911-HDFS-10285-03.patch, > HDFS-12911.00.patch > > > This is the JIRA for tracking the possible improvements or issues discussed > in main JIRA > So far comments to handle > Daryn: > # Lock should not kept while executing placement policy. > # While starting up the NN, SPS Xattrs checks happen even if feature > disabled. This could potentially impact the startup speed. > UMA: > # I am adding one more possible improvement to reduce Xattr objects > significantly. > SPS Xattr is constant object. So, we create one Xattr deduplication object > once statically and use the same object reference when required to add SPS > Xattr to Inode. So, here additional bytes required for storing SPS Xattr > would turn to same as single object ref ( i.e 4 bytes in 32 bit). So Xattr > overhead should come down significantly IMO. Lets explore the feasibility on > this option. > Xattr list Future will not be specially created for SPS, that list would have > been created by SetStoragePolicy already on the same directory. So, no extra > Feature creation because of SPS alone. > # Currently SPS putting long id objects in Q for tracking SPS called Inodes. > So, it is additional created and size of it would be (obj ref + value) = (8 + > 8) bytes [ ignoring alignment for time being] > So, the possible improvement here is, instead of creating new Long obj, we > can keep existing inode object for tracking. Advantage is, Inode object > already maintained in NN, so no new object creation is needed. So, we just > need to maintain one obj ref. Above two points should significantly reduce > the memory requirements of SPS. So, for SPS call: 8bytes for called inode > tracking + 8 bytes for Xattr ref. > # Use LightWeightLinkedSet instead of using LinkedList for from Q. This will > reduce unnecessary Node creations inside LinkedList. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12972) RBF: Display mount table quota info in Web UI and admin command
[ https://issues.apache.org/jira/browse/HDFS-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324253#comment-16324253 ] Íñigo Goiri commented on HDFS-12972: Tested [^HDFS-12972.002.patch] in a test cluster. The unit tests are unrelated (actually, I've seen these two failing lately more than usual). +1. > RBF: Display mount table quota info in Web UI and admin command > --- > > Key: HDFS-12972 > URL: https://issues.apache.org/jira/browse/HDFS-12972 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-12972.001.patch, HDFS-12972.002.patch > > > Display mount table quota info in Web UI and admin command -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12984) BlockPoolSlice can leak in a mini dfs cluster
[ https://issues.apache.org/jira/browse/HDFS-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324245#comment-16324245 ] Ajay Kumar commented on HDFS-12984: --- [~revans2] thanks for reporting and review. [~arpitagarwal] thanks for review and commit. > BlockPoolSlice can leak in a mini dfs cluster > - > > Key: HDFS-12984 > URL: https://issues.apache.org/jira/browse/HDFS-12984 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5 >Reporter: Robert Joseph Evans >Assignee: Ajay Kumar > Fix For: 3.1.0 > > Attachments: HDFS-12984.001.patch, Screen Shot 2018-01-05 at 4.38.06 > PM.png, Screen Shot 2018-01-05 at 5.26.54 PM.png, Screen Shot 2018-01-05 at > 5.31.52 PM.png > > > When running some unit tests for storm we found that we would occasionally > get out of memory errors on the HDFS integration tests. > When I got a heap dump I found that the ShutdownHookManager was full of > BlockPoolSlice$1 instances. Which hold a reference to the BlockPoolSlice > which then in turn holds a reference to the DataNode etc > It looks like when shutdown is called on the BlockPoolSlice there is no way > to remove the shut down hook in because no reference to it is saved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer
[
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-12919:
---
Attachment: HDFS-12919.023.patch
Thanks for the review [~linyiqun], I tackled your comments in
[^HDFS-12919.023.patch] and tweaked {{testProxyGetStats()}} a little to check
what's up with those spurious errors.
[~eddyxu], for the EC part, I just added {{equals()}} and {{hash()}} to one of
the responses and the unit test should cover:
* Listing policies
* Listing codecs
* Getting and setting EC policies
* Creating new EC policies
* Get EC stats
Let me know if you think there's anything that can be improved.
> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
> Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch,
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch,
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch,
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch,
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch,
> HDFS-12919.013.patch, HDFS-12919.014.patch, HDFS-12919.015.patch,
> HDFS-12919.016.patch, HDFS-12919.017.patch, HDFS-12919.018.patch,
> HDFS-12919.019.patch, HDFS-12919.020.patch, HDFS-12919.021.patch,
> HDFS-12919.022.patch, HDFS-12919.023.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files.
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "setErasureCodingPolicy" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12984) BlockPoolSlice can leak in a mini dfs cluster
[ https://issues.apache.org/jira/browse/HDFS-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324189#comment-16324189 ] Hudson commented on HDFS-12984: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13485 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13485/]) HDFS-12984. BlockPoolSlice can leak in a mini dfs cluster. Contributed (arp: rev b278f7b29305cb67d22ef0bb08b067c422381f48) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java > BlockPoolSlice can leak in a mini dfs cluster > - > > Key: HDFS-12984 > URL: https://issues.apache.org/jira/browse/HDFS-12984 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5 >Reporter: Robert Joseph Evans >Assignee: Ajay Kumar > Fix For: 3.1.0 > > Attachments: HDFS-12984.001.patch, Screen Shot 2018-01-05 at 4.38.06 > PM.png, Screen Shot 2018-01-05 at 5.26.54 PM.png, Screen Shot 2018-01-05 at > 5.31.52 PM.png > > > When running some unit tests for storm we found that we would occasionally > get out of memory errors on the HDFS integration tests. > When I got a heap dump I found that the ShutdownHookManager was full of > BlockPoolSlice$1 instances. Which hold a reference to the BlockPoolSlice > which then in turn holds a reference to the DataNode etc > It looks like when shutdown is called on the BlockPoolSlice there is no way > to remove the shut down hook in because no reference to it is saved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test file
[
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324186#comment-16324186
]
Zsolt Venczel commented on HDFS-13004:
--
Thanks for taking the time and reviewing my patch [~msingh]!
I updated the patch keeping in mind your suggestion.
> TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB
> or larger than the test file
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
> Labels: flaky-test
> Fix For: 3.0.1
>
> Attachments: HDFS-13004.01.patch, HDFS-13004.02.patch,
> HDFS-13004.03.patch
>
>
> andre{code}
> Error:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hd
[jira] [Assigned] (HDFS-13016) globStatus javadoc refers to glob pattern as "regular expression"
[ https://issues.apache.org/jira/browse/HDFS-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDFS-13016: Assignee: Mukul Kumar Singh > globStatus javadoc refers to glob pattern as "regular expression" > - > > Key: HDFS-13016 > URL: https://issues.apache.org/jira/browse/HDFS-13016 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, hdfs >Reporter: Ryanne Dolan >Assignee: Mukul Kumar Singh >Priority: Trivial > > Glob patterns are not regular expressions. Both are well-defined and > universally understood concepts. The term "regular expression" is misapplied > here. > The method name "globStatus" indicates that pathPattern should be a glob, but > the javadoc says pathPattern is a regular expression. The documentation goes > on to describe the accepted format of pathPattern and clearly describes globs. > The term "regular expression" should not be associated with pathPattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13004) TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB or larger than the test file
[
https://issues.apache.org/jira/browse/HDFS-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zsolt Venczel updated HDFS-13004:
-
Attachment: HDFS-13004.03.patch
> TestLeaseRecoveryStriped#testLeaseRecovery is failing when safeLength is 0MB
> or larger than the test file
> -
>
> Key: HDFS-13004
> URL: https://issues.apache.org/jira/browse/HDFS-13004
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Zsolt Venczel
>Assignee: Zsolt Venczel
> Labels: flaky-test
> Fix For: 3.0.1
>
> Attachments: HDFS-13004.01.patch, HDFS-13004.02.patch,
> HDFS-13004.03.patch
>
>
> andre{code}
> Error:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths=
> {4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304},safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
> Stack:
> java.lang.AssertionError:
> failed testCase at i=1,
> blockLengths=org.apache.hadoop.hdfs.TestLeaseRecoveryStriped$BlockLengths@5a4c638d[blockLengths={4194304,4194304,4194304,1048576,4194304,4194304,2097152,1048576,4194304}
> ,safeLength=25165824]
> java.lang.AssertionError: File length should be the same expected:<25165824>
> but was:<18874368>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.verifyLength(StripedFileTestUtil.java:79)
> at
> org.apache.hadoop.hdfs.StripedFileTestUtil.checkData(StripedFileTestUtil.java:362)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:198)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ
[jira] [Updated] (HDFS-12984) BlockPoolSlice can leak in a mini dfs cluster
[ https://issues.apache.org/jira/browse/HDFS-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-12984: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) I've committed this. Thanks for reporting and reviewing this Robert. Thanks for the fix Ajay. > BlockPoolSlice can leak in a mini dfs cluster > - > > Key: HDFS-12984 > URL: https://issues.apache.org/jira/browse/HDFS-12984 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5 >Reporter: Robert Joseph Evans >Assignee: Ajay Kumar > Fix For: 3.1.0 > > Attachments: HDFS-12984.001.patch, Screen Shot 2018-01-05 at 4.38.06 > PM.png, Screen Shot 2018-01-05 at 5.26.54 PM.png, Screen Shot 2018-01-05 at > 5.31.52 PM.png > > > When running some unit tests for storm we found that we would occasionally > get out of memory errors on the HDFS integration tests. > When I got a heap dump I found that the ShutdownHookManager was full of > BlockPoolSlice$1 instances. Which hold a reference to the BlockPoolSlice > which then in turn holds a reference to the DataNode etc > It looks like when shutdown is called on the BlockPoolSlice there is no way > to remove the shut down hook in because no reference to it is saved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13017: Status: Patch Available (was: Open) > Block Storage: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-13017-HDFS-7240.001.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[
https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324141#comment-16324141
]
Elek, Marton edited comment on HDFS-13017 at 1/12/18 3:43 PM:
--
The first version is uploaded. It just extends the existing protocolt to get
the list of the available jscsi services.
The easiest way to test is using docker-compose file from HDFS-12983, but you
can also startup a new cblock server.
After the creation of a cblock, you can test the discovery:
{code}
sudo iscsiadm -m discovery -t st -p 127.0.0.1:3260
{code}
iscsi volumes should be detected and visible under the next call:
{code}
sudo iscsiadm -m node
{code}
was (Author: elek):
The first version is uploaded. It just extends the existing protocolt to get
the list of the available jscsi services.
The easiest way to test is using docker-compose file from HDFS-12983, but you
can also startup a new cblock server.
After the creation of a cblock, you can test the discovery:
```
sudo iscsiadm -m discovery -t st -p 127.0.0.1:3260
```
iscsi volumes should be detected and visible under the next call:
```
sudo iscsiadm -m node
```
> Block Storage: implement simple iscsi discovery in jscsi server
> ---
>
> Key: HDFS-13017
> URL: https://issues.apache.org/jira/browse/HDFS-13017
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-13017-HDFS-7240.001.patch
>
>
> The current jscsi server doesn't support iscsi discovery.
> To use jscsi server as a kubernetes storage backend we need the discovery.
> jScsi supports it we need just override a method and add an additional call
> to the server protocl to get the list of the available cblocks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324141#comment-16324141 ] Elek, Marton commented on HDFS-13017: - The first version is uploaded. It just extends the existing protocolt to get the list of the available jscsi services. The easiest way to test is using docker-compose file from HDFS-12983, but you can also startup a new cblock server. After the creation of a cblock, you can test the discovery: ``` sudo iscsiadm -m discovery -t st -p 127.0.0.1:3260 ``` iscsi volumes should be detected and visible under the next call: ``` sudo iscsiadm -m node ``` > Block Storage: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-13017-HDFS-7240.001.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13017) Block Manager: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13017: Attachment: HDFS-13017-HDFS-7240.001.patch > Block Manager: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-13017-HDFS-7240.001.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13017) Block Sorage: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13017: Summary: Block Sorage: implement simple iscsi discovery in jscsi server (was: Block Manager: implement simple iscsi discovery in jscsi server) > Block Sorage: implement simple iscsi discovery in jscsi server > -- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-13017-HDFS-7240.001.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13017) Block Storage: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13017: Summary: Block Storage: implement simple iscsi discovery in jscsi server (was: Block Sorage: implement simple iscsi discovery in jscsi server) > Block Storage: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-13017-HDFS-7240.001.patch > > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13018) Block Storage: make the iscsi target addres configurable for discovery
[ https://issues.apache.org/jira/browse/HDFS-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDFS-13018: --- Assignee: Elek, Marton > Block Storage: make the iscsi target addres configurable for discovery > -- > > Key: HDFS-13018 > URL: https://issues.apache.org/jira/browse/HDFS-13018 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > > Current jscsi server returns with the targetAddress (as ip) and 3260 (as > port) during the iscsi discovery. But in some cases we need to configure > these values. > For example in kubernetes the iscsi server could run behind a service where > the address (where the jscsi server is available from the cluster) could be > different from the targetAddress where the server is listening. > I propose to add two more configuration key to override the default > address/port for configuration but it also requires modification in the jscsi > project. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13017) Block Manager: implement simple iscsi discovery in jscsi server
[ https://issues.apache.org/jira/browse/HDFS-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDFS-13017: --- Assignee: Elek, Marton > Block Manager: implement simple iscsi discovery in jscsi server > --- > > Key: HDFS-13017 > URL: https://issues.apache.org/jira/browse/HDFS-13017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > > The current jscsi server doesn't support iscsi discovery. > To use jscsi server as a kubernetes storage backend we need the discovery. > jScsi supports it we need just override a method and add an additional call > to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13018) Block Storage: make the iscsi target addres configurable for discovery
Elek, Marton created HDFS-13018: --- Summary: Block Storage: make the iscsi target addres configurable for discovery Key: HDFS-13018 URL: https://issues.apache.org/jira/browse/HDFS-13018 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Elek, Marton Current jscsi server returns with the targetAddress (as ip) and 3260 (as port) during the iscsi discovery. But in some cases we need to configure these values. For example in kubernetes the iscsi server could run behind a service where the address (where the jscsi server is available from the cluster) could be different from the targetAddress where the server is listening. I propose to add two more configuration key to override the default address/port for configuration but it also requires modification in the jscsi project. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13016) globStatus javadoc refers to glob pattern as "regular expression"
Ryanne Dolan created HDFS-13016: --- Summary: globStatus javadoc refers to glob pattern as "regular expression" Key: HDFS-13016 URL: https://issues.apache.org/jira/browse/HDFS-13016 Project: Hadoop HDFS Issue Type: Bug Components: documentation, hdfs Reporter: Ryanne Dolan Priority: Trivial Glob patterns are not regular expressions. Both are well-defined and universally understood concepts. The term "regular expression" is misapplied here. The method name "globStatus" indicates that pathPattern should be a glob, but the javadoc says pathPattern is a regular expression. The documentation goes on to describe the accepted format of pathPattern and clearly describes globs. The term "regular expression" should not be associated with pathPattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13017) Block Manager: implement simple iscsi discovery in jscsi server
Elek, Marton created HDFS-13017: --- Summary: Block Manager: implement simple iscsi discovery in jscsi server Key: HDFS-13017 URL: https://issues.apache.org/jira/browse/HDFS-13017 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Elek, Marton The current jscsi server doesn't support iscsi discovery. To use jscsi server as a kubernetes storage backend we need the discovery. jScsi supports it we need just override a method and add an additional call to the server protocl to get the list of the available cblocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12983) Block Storage: provide docker-compose file for cblock clusters
[ https://issues.apache.org/jira/browse/HDFS-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-12983: Summary: Block Storage: provide docker-compose file for cblock clusters (was: Block Manager: provide docker-compose file for cblock clusters) > Block Storage: provide docker-compose file for cblock clusters > -- > > Key: HDFS-12983 > URL: https://issues.apache.org/jira/browse/HDFS-12983 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: ozone >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12983-HDFS-7240.001.patch > > > Since HDFS-12469 we have a docker compose file at dev-support/compose/ozone > which makes it easy to start local ozone clusers with multiple datanodes. > In this patch I propose similar config file for the cblock/iscsi servers > (jscsi + cblock + scm + namenode + datanode) to make it easier to check the > latest state. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12983) Block Manager: provide docker-compose file for cblock clusters
[ https://issues.apache.org/jira/browse/HDFS-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-12983: Summary: Block Manager: provide docker-compose file for cblock clusters (was: Ozone: dozone: provide docker-compose file for cblock clusters) > Block Manager: provide docker-compose file for cblock clusters > -- > > Key: HDFS-12983 > URL: https://issues.apache.org/jira/browse/HDFS-12983 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: ozone >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12983-HDFS-7240.001.patch > > > Since HDFS-12469 we have a docker compose file at dev-support/compose/ozone > which makes it easy to start local ozone clusers with multiple datanodes. > In this patch I propose similar config file for the cblock/iscsi servers > (jscsi + cblock + scm + namenode + datanode) to make it easier to check the > latest state. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12718) Block Storage: fix thread number calculation in CBlockManager
[
https://issues.apache.org/jira/browse/HDFS-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDFS-12718:
Summary: Block Storage: fix thread number calculation in CBlockManager
(was: Ozone: fix thread number calculation in CBlockManager)
> Block Storage: fix thread number calculation in CBlockManager
> -
>
> Key: HDFS-12718
> URL: https://issues.apache.org/jira/browse/HDFS-12718
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Fix For: HDFS-7240
>
> Attachments: HDFS-12718-HDFS-7240.001.patch,
> HDFS-12718-HDFS-7240.002.patch
>
>
> When starting cblock server or during the unit tests I got many
> IllegalArgumentException:
> {code}
> testCliInfoVolume(org.apache.hadoop.cblock.TestCBlockCLI) Time elapsed:
> 0.004 sec <<< ERROR!
> org.apache.hadoop.ipc.RemoteException: java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1307)
> at
> java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1265)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolumeContainers(StorageManager.java:212)
> at
> org.apache.hadoop.cblock.storage.StorageManager.createVolume(StorageManager.java:304)
> at
> org.apache.hadoop.cblock.CBlockManager.createVolume(CBlockManager.java:257)
> at
> org.apache.hadoop.cblock.protocolPB.CBlockServiceProtocolServerSideTranslatorPB.createVolume(CBlockServiceProtocolServerSideTranslatorPB.java:57)
> at
> org.apache.hadoop.cblock.protocol.proto.CBlockServiceProtocolProtos$CBlockServiceProtocolService$2.callBlockingMethod(CBlockServiceProtocolProtos.java:6056)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy10.createVolume(Unknown Source)
> at
> org.apache.hadoop.cblock.client.CBlockServiceProtocolClientSideTranslatorPB.createVolume(CBlockServiceProtocolClientSideTranslatorPB.java:64)
> at
> org.apache.hadoop.cblock.client.CBlockVolumeClient.createVolume(CBlockVolumeClient.java:64)
> at
> org.apache.hadoop.cblock.cli.CBlockCli.createVolume(CBlockCli.java:239)
> at org.apache.hadoop.cblock.cli.CBlockCli.run(CBlockCli.java:173)
> at
> org.apache.hadoop.cblock.TestCBlockCLI.testCliInfoVolume(TestCBlockCLI.java:232)
> {code}
> The root cause is that in CBlock Manager we create ThreadGroups:
> {code}
> ThreadPoolExecutor executor = new ThreadPoolExecutor(numThreads,
> MAX_THREADS, 1, TimeUnit.SECONDS,
> new ArrayBlockingQueue<>(MAX_QUEUE_CAPACITY),
> new ThreadPoolExecutor.CallerRunsPolicy());
> {code}
> Where numThreads (the number of always active threads) comes from config and
> 16 by default MAX_THREADS is `Runtime.getRuntime().availableProcessors() * 2`.
> My problem was that MAX_THREAD was lower than numThreads (as I have only 2
> processors, shame on me), so I got IllegalArgumentException.
> In the fix I suggest:
> * Limit the maximum number of threads not the always active threads, as
> ususally this is the number which is needed to adjust
> * Use the core dependent numbers as the number of active threas (but if
> numThreads is smaller, that should be used).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12984) BlockPoolSlice can leak in a mini dfs cluster
[ https://issues.apache.org/jira/browse/HDFS-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324092#comment-16324092 ] Robert Joseph Evans commented on HDFS-12984: +1 for committing it. > BlockPoolSlice can leak in a mini dfs cluster > - > > Key: HDFS-12984 > URL: https://issues.apache.org/jira/browse/HDFS-12984 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.5 >Reporter: Robert Joseph Evans >Assignee: Ajay Kumar > Attachments: HDFS-12984.001.patch, Screen Shot 2018-01-05 at 4.38.06 > PM.png, Screen Shot 2018-01-05 at 5.26.54 PM.png, Screen Shot 2018-01-05 at > 5.31.52 PM.png > > > When running some unit tests for storm we found that we would occasionally > get out of memory errors on the HDFS integration tests. > When I got a heap dump I found that the ShutdownHookManager was full of > BlockPoolSlice$1 instances. Which hold a reference to the BlockPoolSlice > which then in turn holds a reference to the DataNode etc > It looks like when shutdown is called on the BlockPoolSlice there is no way > to remove the shut down hook in because no reference to it is saved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12636) Ozone: OzoneFileSystem: Implement seek functionality for rpc client
[ https://issues.apache.org/jira/browse/HDFS-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-12636: --- Attachment: HDFS-12636-HDFS-7240.003.patch v3 patch uses the configured replication type and replication factor for filesystem operations. > Ozone: OzoneFileSystem: Implement seek functionality for rpc client > --- > > Key: HDFS-12636 > URL: https://issues.apache.org/jira/browse/HDFS-12636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Lokesh Jain > Fix For: HDFS-7240 > > Attachments: HDFS-12636-HDFS-7240.001.patch, > HDFS-12636-HDFS-7240.002.patch, HDFS-12636-HDFS-7240.003.patch > > > OzoneClient library provides a method to invoke both RPC as well as REST > based methods to ozone. This api will help in the improving both the > performance as well as the interface management in OzoneFileSystem. > This jira will be used to convert the REST based calls to use this new > unified client. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323879#comment-16323879
]
genericqa commented on HDFS-12897:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDistributedFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12897 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905833/HDFS-12897.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 14b0b2220b70 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / addbcd8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22654/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22654/testReport/ |
| Max. process+thread count | 3481 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-p
[jira] [Commented] (HDFS-13011) Support replacing multiple nodes during pipeline recovery and append
[
https://issues.apache.org/jira/browse/HDFS-13011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323870#comment-16323870
]
genericqa commented on HDFS-13011:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 279
unchanged - 1 fixed = 279 total (was 280) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13011 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905829/HDFS-13011-02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2593b043e775 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / addbcd8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
|
[jira] [Commented] (HDFS-12972) RBF: Display mount table quota info in Web UI and admin command
[
https://issues.apache.org/jira/browse/HDFS-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323758#comment-16323758
]
genericqa commented on HDFS-12972:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestSecureNameNode |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12972 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905814/HDFS-12972.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 98467b6c1b1b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / addbcd8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22652/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22652/testReport/ |
| Max. process+thread count | 3079 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22652/co
[jira] [Commented] (HDFS-9049) Make Datanode Netty reverse proxy port to be configurable
[
https://issues.apache.org/jira/browse/HDFS-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323750#comment-16323750
]
Brahma Reddy Battula commented on HDFS-9049:
LGTM, thanks for updating the patch.
> Make Datanode Netty reverse proxy port to be configurable
> -
>
> Key: HDFS-9049
> URL: https://issues.apache.org/jira/browse/HDFS-9049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-9049-01.patch, HDFS-9049-02.patch,
> HDFS-9049-03.patch, HDFS-9049-04.patch
>
>
> In DatanodeHttpServer.java Netty is used as reverse proxy. But uses random
> port to start with binding to localhost. This port can be made configurable
> for better deployments.
> {code}
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323735#comment-16323735
]
LiXin Ge commented on HDFS-12897:
-
It seems [~andrew.wang] doesn't have different opinion with this issue after a
week of waiting.I have update the patch according to our discussion result.
[~xiaochen] could you please help to review the latest patch, thanks!
I will create new issues to trace the similar issues in {{getPolicy}} and
{{encryption}} soon.
> Path not found when we get the ec policy for a .snapshot dir
>
>
> Key: HDFS-12897
> URL: https://issues.apache.org/jira/browse/HDFS-12897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs, snapshots
>Affects Versions: 3.0.0-alpha1, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: LiXin Ge
> Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch,
> HDFS-12897.003.patch, HDFS-12897.004.patch
>
>
> Scenario:-
> ---
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
> replicationFallbacks=[]}
> *Which is the correct behavior ?*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9049) Make Datanode Netty reverse proxy port to be configurable
[
https://issues.apache.org/jira/browse/HDFS-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323731#comment-16323731
]
Vinayakumar B commented on HDFS-9049:
-
Test failures are unrelated.
> Make Datanode Netty reverse proxy port to be configurable
> -
>
> Key: HDFS-9049
> URL: https://issues.apache.org/jira/browse/HDFS-9049
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-9049-01.patch, HDFS-9049-02.patch,
> HDFS-9049-03.patch, HDFS-9049-04.patch
>
>
> In DatanodeHttpServer.java Netty is used as reverse proxy. But uses random
> port to start with binding to localhost. This port can be made configurable
> for better deployments.
> {code}
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDFS-12897:
Status: Patch Available (was: Open)
> Path not found when we get the ec policy for a .snapshot dir
>
>
> Key: HDFS-12897
> URL: https://issues.apache.org/jira/browse/HDFS-12897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs, snapshots
>Affects Versions: 3.0.0-alpha1, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: LiXin Ge
> Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch,
> HDFS-12897.003.patch, HDFS-12897.004.patch
>
>
> Scenario:-
> ---
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
> replicationFallbacks=[]}
> *Which is the correct behavior ?*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDFS-12897:
Attachment: HDFS-12897.004.patch
> Path not found when we get the ec policy for a .snapshot dir
>
>
> Key: HDFS-12897
> URL: https://issues.apache.org/jira/browse/HDFS-12897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs, snapshots
>Affects Versions: 3.0.0-alpha1, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: LiXin Ge
> Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch,
> HDFS-12897.003.patch, HDFS-12897.004.patch
>
>
> Scenario:-
> ---
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
> replicationFallbacks=[]}
> *Which is the correct behavior ?*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
[
https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDFS-12897:
Status: Open (was: Patch Available)
> Path not found when we get the ec policy for a .snapshot dir
>
>
> Key: HDFS-12897
> URL: https://issues.apache.org/jira/browse/HDFS-12897
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs, snapshots
>Affects Versions: 3.0.0-alpha1, 3.1.0
>Reporter: Harshakiran Reddy
>Assignee: LiXin Ge
> Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch,
> HDFS-12897.003.patch, HDFS-12897.004.patch
>
>
> Scenario:-
> ---
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x - user group 0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x - user group 0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
> replicationFallbacks=[]}
> *Which is the correct behavior ?*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13011) Support replacing multiple nodes during pipeline recovery and append
[ https://issues.apache.org/jira/browse/HDFS-13011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-13011: - Attachment: HDFS-13011-02.patch Updated the patch to fix checkstyles Test failure is unrelated. > Support replacing multiple nodes during pipeline recovery and append > > > Key: HDFS-13011 > URL: https://issues.apache.org/jira/browse/HDFS-13011 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-13011-01.patch, HDFS-13011-02.patch > > > During pipeline recovery only one additional node will be asked and will be > replaced with failed node. > But if initial pipeline size is less than replication, then extra nodes could > be added during pipeline recovery to satisfy the replication during write > itself. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer
[
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323719#comment-16323719
]
Yiqun Lin commented on HDFS-12919:
--
Overall looks good. Two minor comments related to method
{{TestRouterRpc#getFileDFSClient}}.
* Would you use LOG instance to print track info as well?
* The parameter passed in {{getFileDFSClient}} seems not correct.
{code}
assertTrue(verifyFileExists(routerFS, filePath3));
+DFSClient file3Protocol = getFileDFSClient(filePath1);
{code}
I think here should be filePath3, the same problem to filePath2.
After this, It would be better to let [~eddyxu] have a look. He will be more
familiar than me with HDFS EC.
> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
> Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch,
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch,
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch,
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch,
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch,
> HDFS-12919.013.patch, HDFS-12919.014.patch, HDFS-12919.015.patch,
> HDFS-12919.016.patch, HDFS-12919.017.patch, HDFS-12919.018.patch,
> HDFS-12919.019.patch, HDFS-12919.020.patch, HDFS-12919.021.patch,
> HDFS-12919.022.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files.
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "setErasureCodingPolicy" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12919) RBF: Support erasure coding methods in RouterRpcServer
[
https://issues.apache.org/jira/browse/HDFS-12919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323719#comment-16323719
]
Yiqun Lin edited comment on HDFS-12919 at 1/12/18 8:54 AM:
---
Overall looks good. Two minor comments related to method
{{TestRouterRpc#getFileDFSClient}}.
* Would you use LOG instance to print track info as well?
* The parameter passed in {{getFileDFSClient}} seems not correct.
{code}
assertTrue(verifyFileExists(routerFS, filePath3));
+DFSClient file3Protocol = getFileDFSClient(filePath1);
{code}
I think here should be filePath3, the same problem to filePath2.
After this, It would be better to let [~eddyxu] have a look. He will be more
familiar than me with HDFS EC.
was (Author: linyiqun):
Overall looks good. Two minor comments related to method
{{TestRouterRpc#getFileDFSClient}}.
* Would you use LOG instance to print track info as well?
* The parameter passed in {{getFileDFSClient}} seems not correct.
{code}
assertTrue(verifyFileExists(routerFS, filePath3));
+DFSClient file3Protocol = getFileDFSClient(filePath1);
{code}
I think here should be filePath3, the same problem to filePath2.
After this, It would be better to let [~eddyxu] have a look. He will be more
familiar than me with HDFS EC.
> RBF: Support erasure coding methods in RouterRpcServer
> --
>
> Key: HDFS-12919
> URL: https://issues.apache.org/jira/browse/HDFS-12919
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Critical
> Labels: RBF
> Attachments: HDFS-12919.000.patch, HDFS-12919.001.patch,
> HDFS-12919.002.patch, HDFS-12919.003.patch, HDFS-12919.004.patch,
> HDFS-12919.005.patch, HDFS-12919.006.patch, HDFS-12919.007.patch,
> HDFS-12919.008.patch, HDFS-12919.009.patch, HDFS-12919.010.patch,
> HDFS-12919.011.patch, HDFS-12919.012.patch, HDFS-12919.013.patch,
> HDFS-12919.013.patch, HDFS-12919.014.patch, HDFS-12919.015.patch,
> HDFS-12919.016.patch, HDFS-12919.017.patch, HDFS-12919.018.patch,
> HDFS-12919.019.patch, HDFS-12919.020.patch, HDFS-12919.021.patch,
> HDFS-12919.022.patch
>
>
> MAPREDUCE-6954 started to tune the erasure coding settings for staging files.
> However, the {{Router}} does not support this operation and throws:
> {code}
> 17/12/12 14:36:07 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/hadoop/.staging/job_1513116010218_0002
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "setErasureCodingPolicy" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:368)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setErasureCodingPolicy(RouterRpcServer.java:1805)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
