[jira] [Created] (HDFS-12426) Ozone: BlockManager MBean unregister failure upon shutdown
Xiaoyu Yao created HDFS-12426: - Summary: Ozone: BlockManager MBean unregister failure upon shutdown Key: HDFS-12426 URL: https://issues.apache.org/jira/browse/HDFS-12426 Project: Hadoop HDFS Issue Type: Bug Affects Versions: HDFS-7240 Reporter: Xiaoyu Yao This was found during unit test, seems introduced by HDFS-12005. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12424) Datatable sorting on the Datanode Information page in the Namenode UI is broken
[ https://issues.apache.org/jira/browse/HDFS-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shawna Martell updated HDFS-12424: -- Attachment: HDFS-12424-branch-2.8.1.patch HDFS-12424.1.patch > Datatable sorting on the Datanode Information page in the Namenode UI is > broken > --- > > Key: HDFS-12424 > URL: https://issues.apache.org/jira/browse/HDFS-12424 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Shawna Martell >Assignee: Shawna Martell > Attachments: HDFS-12424.1.patch, HDFS-12424-branch-2.8.1.patch > > > Attempting to sort the "In operation" table by Last contact, Capacity, > Blocks, or Version can result in unexpected behavior. Sorting by Blocks and > Version actually sort by entirely different columns, and Last contact and > Capacity are not being sorted numerically but rather alphabetically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11294) libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails
[ https://issues.apache.org/jira/browse/HDFS-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11294: --- Attachment: HDFS-11294.HDFS-8707.002.patch New patch with test fixes. Top level domain needs to exist or else the dns resolver will exit before it gets to the bad dereference so I'm using a non-existent subdomain of apache.org. Test also explicitly sets the nameservice to the one in the config and verifies the connection failure error message is the one that should be coming out of this fix. > libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails > - > > Key: HDFS-11294 > URL: https://issues.apache.org/jira/browse/HDFS-11294 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-11294.HDFS-8707.001.patch, > HDFS-11294.HDFS-8707.002.patch, HDFS-8707.HDFS-11294.000.patch > > > Hit while doing more manual testing on HDFS-11028. > The HANamenodeTracker takes an asio endpoint to figure out what endpoint on > the other node to try next during a failover. This is done by passing the > element at index 0 from endpoints (a std::vector). > When DNS fails the endpoints vector for that node will be empty so the > iterator returned by endpoints\[0\] is just a null pointer that gets > dereferenced and causes a segfault. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12409) Add metrics of execution time of different stages in EC recovery task
[ https://issues.apache.org/jira/browse/HDFS-12409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163482#comment-16163482 ] Andrew Wang commented on HDFS-12409: I retriggered this since I just committed HDFS-12417 and the host had bind exceptions. > Add metrics of execution time of different stages in EC recovery task > - > > Key: HDFS-12409 > URL: https://issues.apache.org/jira/browse/HDFS-12409 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu >Priority: Minor > Attachments: HDFS-12409.00.patch, HDFS-12409.01.patch > > > Admin can use more metrics to monitor EC recovery tasks, to get insights to > tune recovery performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[ https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163481#comment-16163481 ] Hudson commented on HDFS-12417: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12849 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12849/]) HDFS-12417. Disable flaky TestDFSStripedOutputStreamWithFailure. (wang: rev 80ee89b2871ce2b4b7387de6e410f53d798dfd0b) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java > Disable flaky TestDFSStripedOutputStreamWithFailure > --- > > Key: HDFS-12417 > URL: https://issues.apache.org/jira/browse/HDFS-12417 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Chris Douglas >Assignee: Andrew Wang > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12417.000.patch, HDFS-12417.001.patch, > HDFS-12417.002.patch > > > Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail > in test-patch runs. Since its failure is no longer seen as a blocker for > commit, it should be disabled until it is more reliable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
[ https://issues.apache.org/jira/browse/HDFS-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-12323: --- Attachment: HDFS-12323.002.patch Alright, seems fair. Attaching v002 patch; only difference is the addition of a no-arg constructor. > NameNode terminates after full GC thinking QJM unresponsive if full GC is > much longer than timeout > -- > > Key: HDFS-12323 > URL: https://issues.apache.org/jira/browse/HDFS-12323 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, qjm >Affects Versions: 2.7.4 >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: HDFS-12323.000.patch, HDFS-12323.001.patch, > HDFS-12323.002.patch > > > HDFS-10733 attempted to fix the issue where the Namenode process would > terminate itself if it had a GC pause which lasted longer than the QJM > timeout, since it would think that the QJM had taken too long to respond. > However, it only bumps up the timeout expiration by one timeout length, so if > the GC pause was e.g. 2x the length of the timeout, a TimeoutException will > be thrown and the NN will still terminate itself. > Thanks to [~yangjiandan] for noting this issue as a comment on HDFS-10733; we > have also seen this issue on a real cluster even after HDFS-10733 is applied. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[
https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163462#comment-16163462
]
Íñigo Goiri commented on HDFS-12381:
The errors are caused by modifying {{hdfs-default.xml}}.
I'll ignore the output from jenkins.
> [Documentation] Adding configuration keys for the Router
>
>
> Key: HDFS-12381
> URL: https://issues.apache.org/jira/browse/HDFS-12381
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Fix For: HDFS-10467
>
> Attachments: HDFS-12381-HDFS-10467.000.patch,
> HDFS-12381-HDFS-10467.001.patch
>
>
> Adding configuration options in tabular format.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
[ https://issues.apache.org/jira/browse/HDFS-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163427#comment-16163427 ] Konstantin Shvachko commented on HDFS-12323: You are right about impossibility of subclassing and about reflections. I just think adding private no-arg constructor will make our intentions with the class usage very explicit and save people thinking about Java rules on constructors etc. > NameNode terminates after full GC thinking QJM unresponsive if full GC is > much longer than timeout > -- > > Key: HDFS-12323 > URL: https://issues.apache.org/jira/browse/HDFS-12323 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, qjm >Affects Versions: 2.7.4 >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: HDFS-12323.000.patch, HDFS-12323.001.patch > > > HDFS-10733 attempted to fix the issue where the Namenode process would > terminate itself if it had a GC pause which lasted longer than the QJM > timeout, since it would think that the QJM had taken too long to respond. > However, it only bumps up the timeout expiration by one timeout length, so if > the GC pause was e.g. 2x the length of the timeout, a TimeoutException will > be thrown and the NN will still terminate itself. > Thanks to [~yangjiandan] for noting this issue as a comment on HDFS-10733; we > have also seen this issue on a real cluster even after HDFS-10733 is applied. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[ https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12417: --- Resolution: Fixed Fix Version/s: 3.0.0-beta1 Status: Resolved (was: Patch Available) Pushed to trunk and branch-3.0, thanks again all. > Disable flaky TestDFSStripedOutputStreamWithFailure > --- > > Key: HDFS-12417 > URL: https://issues.apache.org/jira/browse/HDFS-12417 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Chris Douglas >Assignee: Andrew Wang > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12417.000.patch, HDFS-12417.001.patch, > HDFS-12417.002.patch > > > Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail > in test-patch runs. Since its failure is no longer seen as a blocker for > commit, it should be disabled until it is more reliable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12417) Disable flaky TestDFSStripedOutputStreamWithFailure
[ https://issues.apache.org/jira/browse/HDFS-12417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163412#comment-16163412 ] Andrew Wang commented on HDFS-12417: This seems to be working since the DFSStripedWithFailure tests aren't showing up anymore, though there are still tons of other failures. I'll commit this as an incremental improvement. Thanks for the review Chris! > Disable flaky TestDFSStripedOutputStreamWithFailure > --- > > Key: HDFS-12417 > URL: https://issues.apache.org/jira/browse/HDFS-12417 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Chris Douglas >Assignee: Andrew Wang > Attachments: HDFS-12417.000.patch, HDFS-12417.001.patch, > HDFS-12417.002.patch > > > Some subset of TestDFSStripedOutputStreamWithFailure tests almost always fail > in test-patch runs. Since its failure is no longer seen as a blocker for > commit, it should be disabled until it is more reliable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[
https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163393#comment-16163393
]
Hadoop QA commented on HDFS-12381:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-10467 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 4m
22s{color} | {color:red} root in HDFS-10467 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs in HDFS-10467 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs in HDFS-10467 failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} HDFS-10467 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
28s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12381 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886681/HDFS-12381-HDFS-10467.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux ad85c1e10760 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10467 / 1a4ced3 |
| Default Java | 1.8.0_144 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21098/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-12019) Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created from mvn package.
[
https://issues.apache.org/jira/browse/HDFS-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163382#comment-16163382
]
Hadoop QA commented on HDFS-12019:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12019 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886655/HDFS-12019-HDFS-7240.001.patch
|
| Optional Tests | asflicense mvnsite unit xml |
| uname | Linux 626388db789f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 830474f |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21097/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21097/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created
> from mvn package.
>
>
> Key: HDFS-12019
> URL: https://issues.apache.org/jira/browse/HDFS-12019
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Mukul Kumar Singh
>Priority: Trivial
> Labels: ozoneMerge
> Attachments: HDFS-12019-HDFS-7240.001.patch
>
>
> Right now the tarball that is created by `mvn package` command does not carry
> ozone-defaults.xml. We need to make sure this is packaged as part of the
> tarball. Thanks to [~xyao] for pointing this out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12425) Ozone: OzoneFileSystem: OzoneFileystem read/write/create/open APIs
Mukul Kumar Singh created HDFS-12425: Summary: Ozone: OzoneFileSystem: OzoneFileystem read/write/create/open APIs Key: HDFS-12425 URL: https://issues.apache.org/jira/browse/HDFS-12425 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Fix For: HDFS-7240 This jira will add create/open and read/write APIs for OzoneFileSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163372#comment-16163372 ] Íñigo Goiri commented on HDFS-12381: Thanks [~brahmareddy] for the comments. I pushed a new patch with most of them. Not super proud of the best-practices part so I'm open to suggestions. > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch, > HDFS-12381-HDFS-10467.001.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12381: --- Attachment: HDFS-12381-HDFS-10467.001.patch > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch, > HDFS-12381-HDFS-10467.001.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163342#comment-16163342
]
Chen Liang commented on HDFS-12420:
---
Thanks for taking care of this [~ajayydv]!
Some of the failed tests seem related. For example seems that
{{TestSaveNamespace#testTxIdPersistence}} fails because it tries to formatName
but the previous test has data leftover in test dir, so the format aborted.
Consequently the following txid assertion in this test also fail. We will need
to delete test directory content for certain tests.
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage,
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12019) Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created from mvn package.
[ https://issues.apache.org/jira/browse/HDFS-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12019: - Status: Patch Available (was: Open) > Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created > from mvn package. > > > Key: HDFS-12019 > URL: https://issues.apache.org/jira/browse/HDFS-12019 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Trivial > Labels: ozoneMerge > Attachments: HDFS-12019-HDFS-7240.001.patch > > > Right now the tarball that is created by `mvn package` command does not carry > ozone-defaults.xml. We need to make sure this is packaged as part of the > tarball. Thanks to [~xyao] for pointing this out. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12019) Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created from mvn package.
[ https://issues.apache.org/jira/browse/HDFS-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar reassigned HDFS-12019: - Assignee: Mukul Kumar Singh (was: Nandakumar) > Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created > from mvn package. > > > Key: HDFS-12019 > URL: https://issues.apache.org/jira/browse/HDFS-12019 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Trivial > Labels: ozoneMerge > Attachments: HDFS-12019-HDFS-7240.001.patch > > > Right now the tarball that is created by `mvn package` command does not carry > ozone-defaults.xml. We need to make sure this is packaged as part of the > tarball. Thanks to [~xyao] for pointing this out. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163311#comment-16163311
]
Hadoop QA commented on HDFS-5040:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 274 unchanged - 8 fixed = 274 total (was 282) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-5040 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886651/HDFS-5040.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e189f8772393 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Updated] (HDFS-11294) libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails
[ https://issues.apache.org/jira/browse/HDFS-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11294: --- Attachment: HDFS-11294.HDFS-8707.001.patch New patch with test, should be ready for review. > libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails > - > > Key: HDFS-11294 > URL: https://issues.apache.org/jira/browse/HDFS-11294 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-11294.HDFS-8707.001.patch, > HDFS-8707.HDFS-11294.000.patch > > > Hit while doing more manual testing on HDFS-11028. > The HANamenodeTracker takes an asio endpoint to figure out what endpoint on > the other node to try next during a failover. This is done by passing the > element at index 0 from endpoints (a std::vector). > When DNS fails the endpoints vector for that node will be empty so the > iterator returned by endpoints\[0\] is just a null pointer that gets > dereferenced and causes a segfault. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163283#comment-16163283
]
Arpit Agarwal edited comment on HDFS-12420 at 9/12/17 4:59 PM:
---
Thanks for this improvement [~ajayydv]. Couple of comments, in addition to the
test case as suggested by Rushabh:
# Let's also make the -force option a no-op. We can continue to accept it but
it should have no effect and we should print a warning saying that the force
option is being ignored.
# Same with the -nonInteractive option.
# Also let's update the site documentation with the new behavior:
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#namenode
The site docs are in
[HDFScommands.md|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md]
was (Author: arpitagarwal):
Thanks for this improvement [~ajayydv]. Couple of comments, in addition to the
test case as suggested by Rushabh:
# Let's also make the -force option a no-op. We can continue to accept it but
it should have no effect and we should print a warning saying that the force
option is being ignored.
# Same with the -nonInteractive option.
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage,
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163283#comment-16163283
]
Arpit Agarwal commented on HDFS-12420:
--
Thanks for this improvement [~ajayydv]. Couple of comments, in addition to the
test case as suggested by Rushabh:
# Let's also make the -force option a no-op. We can continue to accept it but
it should have no effect and we should print a warning saying that the force
option is being ignored.
# Same with the -nonInteractive option.
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage,
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163275#comment-16163275 ] Lukas Majercak commented on HDFS-11576: --- Kindly ping, any reviewers? > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.009.patch, HDFS-11576.010.patch, HDFS-11576.011.patch, > HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-11576: -- Attachment: HDFS-11576.011.patch > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.009.patch, HDFS-11576.010.patch, HDFS-11576.011.patch, > HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11921) Ozone: KSM: Unable to put keys with zero length
[ https://issues.apache.org/jira/browse/HDFS-11921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDFS-11921. - Resolution: Not A Problem > Ozone: KSM: Unable to put keys with zero length > --- > > Key: HDFS-11921 > URL: https://issues.apache.org/jira/browse/HDFS-11921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Minor > Labels: OzonePostMerge > Attachments: HDFS-11921-HDFS-7240.001.patch > > > As part of working on HDFS-11909, I was trying to put zero length keys. I > found that put key refuses to do that. Here is the call trace, > bq. at ScmBlockLocationProtocolClientSideTranslatorPB.allocateBlock > we check if the block size is greater than 0, which makes sense since we > should not call into SCM to allocate a block of zero size. > However these 2 calls are invoked for creating the key, so that metadata for > key can be created, we should probably take care of this behavior here. > bq. ksm.KeyManagerImpl.allocateKey > bq. ksm.KeySpaceManager.allocateKey(KeySpaceManager.java:428) > Another way to fix this might be to just allocate a block with at least 1 > byte always, which might be easier than special casing code. > [~vagarychen] Would you like to fix this in the next patch you are working on > ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11921) Ozone: KSM: Unable to put keys with zero length
[ https://issues.apache.org/jira/browse/HDFS-11921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163194#comment-16163194 ] Anu Engineer commented on HDFS-11921: - bq. This is needed for OzoneFileSystem as well, where this API is needed for directories. Directories will have keys with no data. good to know. So you are saying that the behaviour specified by [~vagarychen] is all we need. bq.Right now, the behaviour is that if a length of 0 is given, key allocation will succeed but following writes will throw exception due to writing out of range. I am going to resolve this JIRA as fixed. > Ozone: KSM: Unable to put keys with zero length > --- > > Key: HDFS-11921 > URL: https://issues.apache.org/jira/browse/HDFS-11921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Minor > Labels: OzonePostMerge > Attachments: HDFS-11921-HDFS-7240.001.patch > > > As part of working on HDFS-11909, I was trying to put zero length keys. I > found that put key refuses to do that. Here is the call trace, > bq. at ScmBlockLocationProtocolClientSideTranslatorPB.allocateBlock > we check if the block size is greater than 0, which makes sense since we > should not call into SCM to allocate a block of zero size. > However these 2 calls are invoked for creating the key, so that metadata for > key can be created, we should probably take care of this behavior here. > bq. ksm.KeyManagerImpl.allocateKey > bq. ksm.KeySpaceManager.allocateKey(KeySpaceManager.java:428) > Another way to fix this might be to just allocate a block with at least 1 > byte always, which might be easier than special casing code. > [~vagarychen] Would you like to fix this in the next patch you are working on > ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11982) Add markdown documentation about Ozone
[ https://issues.apache.org/jira/browse/HDFS-11982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163172#comment-16163172 ] Mukul Kumar Singh commented on HDFS-11982: -- Hi [~anu], This has already been fixed with HDFS-11963. Duping this. This was added before any documentation was added. > Add markdown documentation about Ozone > -- > > Key: HDFS-11982 > URL: https://issues.apache.org/jira/browse/HDFS-11982 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Labels: documentation, ozoneMerge, tocheck > Fix For: HDFS-7240 > > > Ozone should be documented in detail using markdown. > This documentation should include information about setting up various Ozone > components, usage and configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11909) Ozone: KSM : Support for simulated file system operations
[ https://issues.apache.org/jira/browse/HDFS-11909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163166#comment-16163166 ] Mukul Kumar Singh commented on HDFS-11909: -- Adding rename to the list as well apart from directory and file list operations, With KSM supporting renames, rename operation in OzoneFS can be atomic as well. > Ozone: KSM : Support for simulated file system operations > -- > > Key: HDFS-11909 > URL: https://issues.apache.org/jira/browse/HDFS-11909 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer > Labels: OzonePostMerge > Attachments: simulation-file-system.pdf > > > This JIRA adds a proposal that makes it easy to implement OzoneFileSystem. > This allows the directory and file list operations simpler. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11921) Ozone: KSM: Unable to put keys with zero length
[ https://issues.apache.org/jira/browse/HDFS-11921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDFS-11921: Assignee: Mukul Kumar Singh > Ozone: KSM: Unable to put keys with zero length > --- > > Key: HDFS-11921 > URL: https://issues.apache.org/jira/browse/HDFS-11921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Minor > Labels: OzonePostMerge > Attachments: HDFS-11921-HDFS-7240.001.patch > > > As part of working on HDFS-11909, I was trying to put zero length keys. I > found that put key refuses to do that. Here is the call trace, > bq. at ScmBlockLocationProtocolClientSideTranslatorPB.allocateBlock > we check if the block size is greater than 0, which makes sense since we > should not call into SCM to allocate a block of zero size. > However these 2 calls are invoked for creating the key, so that metadata for > key can be created, we should probably take care of this behavior here. > bq. ksm.KeyManagerImpl.allocateKey > bq. ksm.KeySpaceManager.allocateKey(KeySpaceManager.java:428) > Another way to fix this might be to just allocate a block with at least 1 > byte always, which might be easier than special casing code. > [~vagarychen] Would you like to fix this in the next patch you are working on > ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11921) Ozone: KSM: Unable to put keys with zero length
[ https://issues.apache.org/jira/browse/HDFS-11921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163163#comment-16163163 ] Mukul Kumar Singh commented on HDFS-11921: -- This is needed for OzoneFileSystem as well, where this API is needed for directories. Directories will have keys with no data. > Ozone: KSM: Unable to put keys with zero length > --- > > Key: HDFS-11921 > URL: https://issues.apache.org/jira/browse/HDFS-11921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Priority: Minor > Labels: OzonePostMerge > Attachments: HDFS-11921-HDFS-7240.001.patch > > > As part of working on HDFS-11909, I was trying to put zero length keys. I > found that put key refuses to do that. Here is the call trace, > bq. at ScmBlockLocationProtocolClientSideTranslatorPB.allocateBlock > we check if the block size is greater than 0, which makes sense since we > should not call into SCM to allocate a block of zero size. > However these 2 calls are invoked for creating the key, so that metadata for > key can be created, we should probably take care of this behavior here. > bq. ksm.KeyManagerImpl.allocateKey > bq. ksm.KeySpaceManager.allocateKey(KeySpaceManager.java:428) > Another way to fix this might be to just allocate a block with at least 1 > byte always, which might be easier than special casing code. > [~vagarychen] Would you like to fix this in the next patch you are working on > ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
[
https://issues.apache.org/jira/browse/HDFS-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163146#comment-16163146
]
Erik Krogen commented on HDFS-12323:
Thanks for the review [~shv]! Actually it is not possible to subclass
{{QuorumCall}} (with or without this patch) since all of its constructors are
private (effectively a {{final}} class), so that part of your concern does not
apply. The point about reflection is interesting but I am hesitant to add in
dead code just to support a usage pattern on a private API that probably
shouldn't be encouraged anyhow. If you have strong feelings about this concern
let me know and I'll add the no-arg constructor.
> NameNode terminates after full GC thinking QJM unresponsive if full GC is
> much longer than timeout
> --
>
> Key: HDFS-12323
> URL: https://issues.apache.org/jira/browse/HDFS-12323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.4
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: HDFS-12323.000.patch, HDFS-12323.001.patch
>
>
> HDFS-10733 attempted to fix the issue where the Namenode process would
> terminate itself if it had a GC pause which lasted longer than the QJM
> timeout, since it would think that the QJM had taken too long to respond.
> However, it only bumps up the timeout expiration by one timeout length, so if
> the GC pause was e.g. 2x the length of the timeout, a TimeoutException will
> be thrown and the NN will still terminate itself.
> Thanks to [~yangjiandan] for noting this issue as a comment on HDFS-10733; we
> have also seen this issue on a real cluster even after HDFS-10733 is applied.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12019) Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created from mvn package.
[ https://issues.apache.org/jira/browse/HDFS-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12019: - Attachment: HDFS-12019-HDFS-7240.001.patch > Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created > from mvn package. > > > Key: HDFS-12019 > URL: https://issues.apache.org/jira/browse/HDFS-12019 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar >Priority: Trivial > Labels: ozoneMerge > Attachments: HDFS-12019-HDFS-7240.001.patch > > > Right now the tarball that is created by `mvn package` command does not carry > ozone-defaults.xml. We need to make sure this is packaged as part of the > tarball. Thanks to [~xyao] for pointing this out. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12019) Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created from mvn package.
[
https://issues.apache.org/jira/browse/HDFS-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163140#comment-16163140
]
Mukul Kumar Singh commented on HDFS-12019:
--
ozone-default.xml is already added as part of
{code}
{code}
However, ozone-site.xml is still not added, adding that with this patch.
> Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created
> from mvn package.
>
>
> Key: HDFS-12019
> URL: https://issues.apache.org/jira/browse/HDFS-12019
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Nandakumar
>Priority: Trivial
> Labels: ozoneMerge
>
> Right now the tarball that is created by `mvn package` command does not carry
> ozone-defaults.xml. We need to make sure this is packaged as part of the
> tarball. Thanks to [~xyao] for pointing this out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11909) Ozone: KSM : Support for simulated file system operations
[ https://issues.apache.org/jira/browse/HDFS-11909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-11909: --- Assignee: (was: Anu Engineer) > Ozone: KSM : Support for simulated file system operations > -- > > Key: HDFS-11909 > URL: https://issues.apache.org/jira/browse/HDFS-11909 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer > Labels: OzonePostMerge > Attachments: simulation-file-system.pdf > > > This JIRA adds a proposal that makes it easy to implement OzoneFileSystem. > This allows the directory and file list operations simpler. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11715) Ozone: SCM : Add priority for datanode commands
[ https://issues.apache.org/jira/browse/HDFS-11715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-11715: --- Assignee: (was: Anu Engineer) > Ozone: SCM : Add priority for datanode commands > --- > > Key: HDFS-11715 > URL: https://issues.apache.org/jira/browse/HDFS-11715 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer > Labels: OzonePostMerge, tocheck > > While reviewing HDFS-11493, [~cheersyang] commented that it would be a good > idea to support priority for datanode commands send from SCM. > bq. The queue seems to be time ordered, I think it will be better to support > priority as well. Commands may have different priority, for example, > replicate a container priority is usually higher than delete a container > replica; replicate a container also may have different priorities according > to the number of replicas > This jira tracks that work item. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11921) Ozone: KSM: Unable to put keys with zero length
[ https://issues.apache.org/jira/browse/HDFS-11921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-11921: --- Assignee: (was: Anu Engineer) > Ozone: KSM: Unable to put keys with zero length > --- > > Key: HDFS-11921 > URL: https://issues.apache.org/jira/browse/HDFS-11921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Priority: Minor > Labels: OzonePostMerge > Attachments: HDFS-11921-HDFS-7240.001.patch > > > As part of working on HDFS-11909, I was trying to put zero length keys. I > found that put key refuses to do that. Here is the call trace, > bq. at ScmBlockLocationProtocolClientSideTranslatorPB.allocateBlock > we check if the block size is greater than 0, which makes sense since we > should not call into SCM to allocate a block of zero size. > However these 2 calls are invoked for creating the key, so that metadata for > key can be created, we should probably take care of this behavior here. > bq. ksm.KeyManagerImpl.allocateKey > bq. ksm.KeySpaceManager.allocateKey(KeySpaceManager.java:428) > Another way to fix this might be to just allocate a block with at least 1 > byte always, which might be easier than special casing code. > [~vagarychen] Would you like to fix this in the next patch you are working on > ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11937) Ozone: Support range in getKey operation
[ https://issues.apache.org/jira/browse/HDFS-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-11937: --- Assignee: (was: Anu Engineer) > Ozone: Support range in getKey operation > > > Key: HDFS-11937 > URL: https://issues.apache.org/jira/browse/HDFS-11937 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer > Labels: OzonePostMerge > > We need to support HTTP ranges so that users can get a key by ranges. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12331) Ozone: Mini cluster can't start up on Windows after HDFS-12159
[
https://issues.apache.org/jira/browse/HDFS-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163109#comment-16163109
]
Anu Engineer commented on HDFS-12331:
-
[~linyiqun] Thanks for the comment, I have also flipped this to a ozoneMerge
tracking. so that we can verify this issue indeed goes away when Ratis merge
happens.
> Ozone: Mini cluster can't start up on Windows after HDFS-12159
> --
>
> Key: HDFS-12331
> URL: https://issues.apache.org/jira/browse/HDFS-12331
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge, tocheck
> Attachments: HDFS-12331-HDFS-7240.001.patch
>
>
> ozone mini cluster can't start up on Windows after HDFS-12159.
> The error log:
> {noformat}
> java.net.URISyntaxException: Illegal character in opaque part at index 2:
> D:\work-project\hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\dn0_data-1\3d3d5718-4219-4ec3-a9c5-e594801a1430
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3058)
> at java.net.URI.(URI.java:588)
> at org.apache.ratis.util.FileUtils.stringAsURI(FileUtils.java:133)
> at
> org.apache.ratis.server.storage.RaftStorage.(RaftStorage.java:49)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:85)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:94)
> at
> org.apache.ratis.server.impl.RaftServerProxy.initImpl(RaftServerProxy.java:67)
> at
> org.apache.ratis.server.impl.RaftServerProxy.(RaftServerProxy.java:62)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:43)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:35)
> at org.apache.ratis.server.RaftServer$Builder.build(RaftServer.java:70)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.(XceiverServerRatis.java:68)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.newXceiverServerRatis(XceiverServerRatis.java:130)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:113)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> {noformat}
> The root cause is that RatiServer instance was newly created in
> {{OzoneContainer}} after HDFS-12159 but it can't recognize the path under
> Windows.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11590) Nodemanagers have DDoS our namenode due to HDFS_DELEGATION_TOKEN expired or not in the cache
[
https://issues.apache.org/jira/browse/HDFS-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163096#comment-16163096
]
Hadoop QA commented on HDFS-11590:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project: The patch generated 8 new +
81 unchanged - 1 fixed = 89 total (was 82) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 2s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestEncryptedTransfer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-11590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861062/HDFS-11590.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6bfa5df9a4a6 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool
[jira] [Commented] (HDFS-6489) DFS Used space is not correct computed on frequent append operations
[
https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163094#comment-16163094
]
Brahma Reddy Battula commented on HDFS-6489:
bq.Where is the code you posted last? I wasn't able to find it in trunk or
branch-2
it's from the patch
[HDFS-6489.007.patch|https://issues.apache.org/jira/secure/attachment/12803016/HDFS-6489.007.patch]
you uploaded.
> DFS Used space is not correct computed on frequent append operations
>
>
> Key: HDFS-6489
> URL: https://issues.apache.org/jira/browse/HDFS-6489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0, 2.7.1, 2.7.2
>Reporter: stanley shi
> Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch,
> HDFS-6489.003.patch, HDFS-6489.004.patch, HDFS-6489.005.patch,
> HDFS-6489.006.patch, HDFS-6489.007.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space
> on each block write operation. This is correct in most scenario (create new
> file), but sometimes it will behave in-correct(append small data to a large
> block).
> For example, I have a file with only one block(say, 60M). Then I try to
> append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the
> block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large
> number of files (1000+), assume the block size is 32M (half of the default
> value), then the dfs used will be increased 1000*32M = 32G on each append to
> the files; but actually I only write 10K bytes; this will cause the datanode
> to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
> BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received
> exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException:
> Insufficient space for appending to FinalizedReplica, blk_1073742834_45306,
> FINALIZED{quote}
> But the actual disk usage:
> {quote}
> [root@hdsh143 ~]# df -h
> FilesystemSize Used Avail Use% Mounted on
> /dev/sda3 16G 2.9G 13G 20% /
> tmpfs 1.9G 72K 1.9G 1% /dev/shm
> /dev/sda1 97M 32M 61M 35% /boot
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12331) Ozone: Mini cluster can't start up on Windows after HDFS-12159
[
https://issues.apache.org/jira/browse/HDFS-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12331:
Labels: ozoneMerge tocheck (was: OzonePostMerge tocheck)
> Ozone: Mini cluster can't start up on Windows after HDFS-12159
> --
>
> Key: HDFS-12331
> URL: https://issues.apache.org/jira/browse/HDFS-12331
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge, tocheck
> Attachments: HDFS-12331-HDFS-7240.001.patch
>
>
> ozone mini cluster can't start up on Windows after HDFS-12159.
> The error log:
> {noformat}
> java.net.URISyntaxException: Illegal character in opaque part at index 2:
> D:\work-project\hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\dn0_data-1\3d3d5718-4219-4ec3-a9c5-e594801a1430
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3058)
> at java.net.URI.(URI.java:588)
> at org.apache.ratis.util.FileUtils.stringAsURI(FileUtils.java:133)
> at
> org.apache.ratis.server.storage.RaftStorage.(RaftStorage.java:49)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:85)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:94)
> at
> org.apache.ratis.server.impl.RaftServerProxy.initImpl(RaftServerProxy.java:67)
> at
> org.apache.ratis.server.impl.RaftServerProxy.(RaftServerProxy.java:62)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:43)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:35)
> at org.apache.ratis.server.RaftServer$Builder.build(RaftServer.java:70)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.(XceiverServerRatis.java:68)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.newXceiverServerRatis(XceiverServerRatis.java:130)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:113)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> {noformat}
> The root cause is that RatiServer instance was newly created in
> {{OzoneContainer}} after HDFS-12159 but it can't recognize the path under
> Windows.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated HDFS-5040: -- Attachment: HDFS-5040.008.patch Apologies for the delay [~brahmareddy], I somehow missed the last ping. Attaching revised patch with test changes. > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch, > HDFS-5040.008.patch, HDFS-5040.patch, HDFS-5040.patch, HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163071#comment-16163071
]
Rushabh S Shah commented on HDFS-12420:
---
[~ajaykumar] Can you please write a test case for the new behavior ?
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage,
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12331) Ozone: Mini cluster can't start up on Windows after HDFS-12159
[
https://issues.apache.org/jira/browse/HDFS-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163056#comment-16163056
]
Anu Engineer commented on HDFS-12331:
-
[~linyiqun] Please ignore these tags. I am going to send out an invite so we
can all discuss these. We are using this to create a set of bugs that we must
absolutely work on with priority. This issue will automatically go away when
the next version of Ratis is updated. So the post merge allows us not to worry
about this in the work schedules. That is all. Thanks for asking though.
> Ozone: Mini cluster can't start up on Windows after HDFS-12159
> --
>
> Key: HDFS-12331
> URL: https://issues.apache.org/jira/browse/HDFS-12331
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-12331-HDFS-7240.001.patch
>
>
> ozone mini cluster can't start up on Windows after HDFS-12159.
> The error log:
> {noformat}
> java.net.URISyntaxException: Illegal character in opaque part at index 2:
> D:\work-project\hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\dn0_data-1\3d3d5718-4219-4ec3-a9c5-e594801a1430
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3058)
> at java.net.URI.(URI.java:588)
> at org.apache.ratis.util.FileUtils.stringAsURI(FileUtils.java:133)
> at
> org.apache.ratis.server.storage.RaftStorage.(RaftStorage.java:49)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:85)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:94)
> at
> org.apache.ratis.server.impl.RaftServerProxy.initImpl(RaftServerProxy.java:67)
> at
> org.apache.ratis.server.impl.RaftServerProxy.(RaftServerProxy.java:62)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:43)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:35)
> at org.apache.ratis.server.RaftServer$Builder.build(RaftServer.java:70)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.(XceiverServerRatis.java:68)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.newXceiverServerRatis(XceiverServerRatis.java:130)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:113)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> {noformat}
> The root cause is that RatiServer instance was newly created in
> {{OzoneContainer}} after HDFS-12159 but it can't recognize the path under
> Windows.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12424) Datatable sorting on the Datanode Information page in the Namenode UI is broken
[ https://issues.apache.org/jira/browse/HDFS-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-12424: -- Target Version/s: 2.8.3 > Datatable sorting on the Datanode Information page in the Namenode UI is > broken > --- > > Key: HDFS-12424 > URL: https://issues.apache.org/jira/browse/HDFS-12424 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Shawna Martell >Assignee: Shawna Martell > > Attempting to sort the "In operation" table by Last contact, Capacity, > Blocks, or Version can result in unexpected behavior. Sorting by Blocks and > Version actually sort by entirely different columns, and Last contact and > Capacity are not being sorted numerically but rather alphabetically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12424) Datatable sorting on the Datanode Information page in the Namenode UI is broken
[ https://issues.apache.org/jira/browse/HDFS-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shawna Martell reassigned HDFS-12424: - Assignee: Shawna Martell > Datatable sorting on the Datanode Information page in the Namenode UI is > broken > --- > > Key: HDFS-12424 > URL: https://issues.apache.org/jira/browse/HDFS-12424 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Shawna Martell >Assignee: Shawna Martell > > Attempting to sort the "In operation" table by Last contact, Capacity, > Blocks, or Version can result in unexpected behavior. Sorting by Blocks and > Version actually sort by entirely different columns, and Last contact and > Capacity are not being sorted numerically but rather alphabetically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12424) Datatable sorting on the Datanode Information page in the Namenode UI is broken
Shawna Martell created HDFS-12424: - Summary: Datatable sorting on the Datanode Information page in the Namenode UI is broken Key: HDFS-12424 URL: https://issues.apache.org/jira/browse/HDFS-12424 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.1 Reporter: Shawna Martell Attempting to sort the "In operation" table by Last contact, Capacity, Blocks, or Version can result in unexpected behavior. Sorting by Blocks and Version actually sort by entirely different columns, and Last contact and Capacity are not being sorted numerically but rather alphabetically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162976#comment-16162976
]
Hadoop QA commented on HDFS-12414:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 44 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 1116 unchanged - 1 fixed = 1116 total (was 1117) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.TestSmallBlock |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.TestEncryptedTransfer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12414 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886618/HDFS-12414.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux c5430eed57e2 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e74d1be |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| mvnsite |
[jira] [Commented] (HDFS-10701) TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162941#comment-16162941
]
Hadoop QA commented on HDFS-10701:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-10701 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886614/HDFS-10701.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f9c272ef9682 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e74d1be |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21090/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21090/testReport/ |
|
[jira] [Commented] (HDFS-11590) Nodemanagers have DDoS our namenode due to HDFS_DELEGATION_TOKEN expired or not in the cache
[
https://issues.apache.org/jira/browse/HDFS-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162875#comment-16162875
]
Nicolas Fraison commented on HDFS-11590:
[~jojochuang] let me know if you need more informationor if you think that I
miss something here.
> Nodemanagers have DDoS our namenode due to HDFS_DELEGATION_TOKEN expired or
> not in the cache
>
>
> Key: HDFS-11590
> URL: https://issues.apache.org/jira/browse/HDFS-11590
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
> Environment: Releases:
> cloudera release cdh-5.5.0
> openjdk version "1.8.0_91"
> linux centos6 servers
> Cluster info:
> Namenode and resourcemanager in HA with kerberos authentication
> More than 1300 datanodes/nodemanagers
>Reporter: Nicolas Fraison
>Priority: Minor
> Attachments: HDFS-11590.patch
>
>
> We have faced some huge slowdowns on our namenode due to all our nodemanagers
> continuing to retry to renew a lease and reconnecting to the namenode every
> second during 1 hour due to some HDFS_DELEGATION_TOKEN being expired or not
> in the cache.
> The number of time_wait connection on our namenode was stuck to the maximum
> configured of 250k during this period due to the reconnections each time.
> {code}
> 2017-03-02 11:51:42,817 INFO
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization successful for appattempt_1488396860014_156103_01
> (auth:TOKEN) for protocol=interface
> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB
> 2017-03-02 11:51:43,414 INFO
> SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
> Authorization successful for appattempt_1488396860014_156120_01
> (auth:TOKEN) for protocol=interface
> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB
> 2017-03-02 11:51:51,994 WARN
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:prediction (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 111018676 for prediction) is expired
> 2017-03-02 11:51:51,995 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 111018676 for prediction) is expired
> 2017-03-02 11:51:51,995 WARN
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:prediction (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 111018676 for prediction) is expired
> 2017-03-02 11:51:51,995 WARN org.apache.hadoop.hdfs.LeaseRenewer: Failed to
> renew lease for [DFSClient_NONMAPREDUCE_1560141256_4187204] for 30 seconds.
> Will retry shortly ...
> token (HDFS_DELEGATION_TOKEN token 111018676 for prediction) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1472)
> at org.apache.hadoop.ipc.Client.call(Client.java:1403)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy20.renewLease(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:571)
> at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy21.renewLease(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:921)
> at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
> at java.lang.Thread.run(Thread.java:745)
> 2017-03-02 12:51:22,032 WARN
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:prediction (auth:SIMPLE)
> cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 111018676 for prediction) can't be found
[jira] [Commented] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[
https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162858#comment-16162858
]
Hadoop QA commented on HDFS-12353:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
52s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
48s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 183 unchanged - 0 fixed = 187 total (was 183) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
46s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12353 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886605/HDFS-12353-2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6ddaae742175 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e74d1be |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/21092/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/21092/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/21092/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/21092/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/21092/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/21092/artifact/patchprocess/whitespace-eol.txt
|
| findbugs |
[jira] [Commented] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162849#comment-16162849
]
Hadoop QA commented on HDFS-12423:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.ozone.scm.TestXceiverClientManager |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.ozone.scm.TestAllocateContainer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12423 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886604/HDFS-12423-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bb5cd4c5b4b7 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 830474f |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Status: Patch Available (was: Open) > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353-2.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12331) Ozone: Mini cluster can't start up on Windows after HDFS-12159
[
https://issues.apache.org/jira/browse/HDFS-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162818#comment-16162818
]
Yiqun Lin edited comment on HDFS-12331 at 9/12/17 11:31 AM:
Hi [~anu], I am confused why we tagged this as {{OzonePostMerge}}, do you mean
we won't fix this when merging to trunk?
Even if the latest Ratis version is not released before merging, we can also
fix this issue by other way like [~szetszwo] commented.
was (Author: linyiqun):
Hi [~anu], I am confused why we tagged this as {{OzonePostMerge}}, did you
mean we won't fix this when merging to trunk?
Even if the latest Ratis version is not released before merging, we can also
fix this issue by other way like [~szetszwo] commented.
> Ozone: Mini cluster can't start up on Windows after HDFS-12159
> --
>
> Key: HDFS-12331
> URL: https://issues.apache.org/jira/browse/HDFS-12331
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-12331-HDFS-7240.001.patch
>
>
> ozone mini cluster can't start up on Windows after HDFS-12159.
> The error log:
> {noformat}
> java.net.URISyntaxException: Illegal character in opaque part at index 2:
> D:\work-project\hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\dn0_data-1\3d3d5718-4219-4ec3-a9c5-e594801a1430
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3058)
> at java.net.URI.(URI.java:588)
> at org.apache.ratis.util.FileUtils.stringAsURI(FileUtils.java:133)
> at
> org.apache.ratis.server.storage.RaftStorage.(RaftStorage.java:49)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:85)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:94)
> at
> org.apache.ratis.server.impl.RaftServerProxy.initImpl(RaftServerProxy.java:67)
> at
> org.apache.ratis.server.impl.RaftServerProxy.(RaftServerProxy.java:62)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:43)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:35)
> at org.apache.ratis.server.RaftServer$Builder.build(RaftServer.java:70)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.(XceiverServerRatis.java:68)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.newXceiverServerRatis(XceiverServerRatis.java:130)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:113)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> {noformat}
> The root cause is that RatiServer instance was newly created in
> {{OzoneContainer}} after HDFS-12159 but it can't recognize the path under
> Windows.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12331) Ozone: Mini cluster can't start up on Windows after HDFS-12159
[
https://issues.apache.org/jira/browse/HDFS-12331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162818#comment-16162818
]
Yiqun Lin commented on HDFS-12331:
--
Hi [~anu], I am confused why we tagged this as {{OzonePostMerge}}, did you
mean we won't fix this when merging to trunk?
Even if the latest Ratis version is not released before merging, we can also
fix this issue by other way like [~szetszwo] commented.
> Ozone: Mini cluster can't start up on Windows after HDFS-12159
> --
>
> Key: HDFS-12331
> URL: https://issues.apache.org/jira/browse/HDFS-12331
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: OzonePostMerge, tocheck
> Attachments: HDFS-12331-HDFS-7240.001.patch
>
>
> ozone mini cluster can't start up on Windows after HDFS-12159.
> The error log:
> {noformat}
> java.net.URISyntaxException: Illegal character in opaque part at index 2:
> D:\work-project\hadoop\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\data\dn0_data-1\3d3d5718-4219-4ec3-a9c5-e594801a1430
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3058)
> at java.net.URI.(URI.java:588)
> at org.apache.ratis.util.FileUtils.stringAsURI(FileUtils.java:133)
> at
> org.apache.ratis.server.storage.RaftStorage.(RaftStorage.java:49)
> at org.apache.ratis.server.impl.ServerState.(ServerState.java:85)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:94)
> at
> org.apache.ratis.server.impl.RaftServerProxy.initImpl(RaftServerProxy.java:67)
> at
> org.apache.ratis.server.impl.RaftServerProxy.(RaftServerProxy.java:62)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:43)
> at
> org.apache.ratis.server.impl.ServerImplUtils.newRaftServer(ServerImplUtils.java:35)
> at org.apache.ratis.server.RaftServer$Builder.build(RaftServer.java:70)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.(XceiverServerRatis.java:68)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.newXceiverServerRatis(XceiverServerRatis.java:130)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:113)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> {noformat}
> The root cause is that RatiServer instance was newly created in
> {{OzoneContainer}} after HDFS-12159 but it can't recognize the path under
> Windows.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12414) Ensure to use CLI command to enable/disable erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-12414: - Attachment: HDFS-12414.003.patch Refine the HDFSErasureCoding.md to try to fix the mvnsite issue > Ensure to use CLI command to enable/disable erasure coding policy > - > > Key: HDFS-12414 > URL: https://issues.apache.org/jira/browse/HDFS-12414 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12414.001.patch, HDFS-12414.002.patch, > HDFS-12414.003.patch > > > Currently, there are two methods for user to enable/disable a erasure coding > policy. One is through "dfs.namenode.ec.policies.enabled" property which is a > static way to configure the enabled erasure coding policies. Another is > through "enableErasureCodingPolicy" or "disabledErasureCodingPolicy" API > which can enabled or disable erasure coding policy at runtime. > When Namenode restart, there is potential state conflicts between the policy > defined in "dfs.namenode.ec.policies.enabled" and policy saved in fsImage. To > resolve the conflict and simplify the operation, it's better to use just one > way and remove the old method configuring the property. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10701) TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails
[ https://issues.apache.org/jira/browse/HDFS-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-10701: - Attachment: HDFS-10701.000.patch > TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails > -- > > Key: HDFS-10701 > URL: https://issues.apache.org/jira/browse/HDFS-10701 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang > Attachments: HDFS-10701.000.patch > > > I noticed this test failure in a recent precommit build, and I also found > this test had failed for a few times in Hadoop-Hdfs-trunk build in the past. > But I do not have sufficient knowledge to tell if it's a flaky test or a bug > in the code. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10701) TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails
[ https://issues.apache.org/jira/browse/HDFS-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-10701: - Status: Patch Available (was: Open) > TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails > -- > > Key: HDFS-10701 > URL: https://issues.apache.org/jira/browse/HDFS-10701 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang > Attachments: HDFS-10701.000.patch > > > I noticed this test failure in a recent precommit build, and I also found > this test had failed for a few times in Hadoop-Hdfs-trunk build in the past. > But I do not have sufficient knowledge to tell if it's a flaky test or a bug > in the code. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10701) TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162792#comment-16162792
]
SammiChen commented on HDFS-10701:
--
I have noticed 2 frequently happened {{testBlockTokenExpired}} failure cases.
1.expected:<1001> but was:<1002>
2.java.lang.AssertionError: If an internal block is smaller than parity block,
then its last cell should be small than last parity cell
(1) it expects there is no block timestamp bump when no Datanode failure.
But actuallly there is write streamer failure due to token expiration even
there is no Datanode failure. It's caused by the cell size change, from 64K to
1M, while the block token life time is still 1s.
(2) when there is more than parity block streamer failure, assertion will be
triggered.
It's also because the token expiration period is too short, so there is more
than expected write streamer failures.
There is maybe other failure scenarios not covered here.
Upload a patch to address case (1) and (2)
> TestDFSStripedOutputStreamWithFailure#testBlockTokenExpired occasionally fails
> --
>
> Key: HDFS-10701
> URL: https://issues.apache.org/jira/browse/HDFS-10701
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>
> I noticed this test failure in a recent precommit build, and I also found
> this test had failed for a few times in Hadoop-Hdfs-trunk build in the past.
> But I do not have sufficient knowledge to tell if it's a flaky test or a bug
> in the code.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162770#comment-16162770 ] Brahma Reddy Battula commented on HDFS-12381: - [~goirix] thanks for reporting and working on this. *Minor nits:* 1) Please correct the typo's bq.182 | Propery | Description| 2) How about chage like *"If true ,the Router should monitor the namenode in the local machine" which will be sync with existing conf* .If you agree, please change hdfs-default.xml and remaining configs also. bq.| dfs.federation.router.monitor.localnamenode.enable | If the Router should monitor the namenode in the local machine. | 3) Are you going update with this some more details..? bq.recommendations/best-practices about setting up the mount table, I think I can extend this JIRA to cover this a little more. > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12222) Add EC information to BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162765#comment-16162765
]
Hadoop QA commented on HDFS-1:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} root: The patch generated 0 new + 354 unchanged - 2
fixed = 354 total (was 356) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 14s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestEncryptedTransfer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Attachment: HDFS-12353-2.patch > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353-2.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Status: Open (was: Patch Available) > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162755#comment-16162755 ] Brahma Reddy Battula commented on HDFS-5040: Would any one else like to chime in? [~kshukla] do update the patch..? > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.007.patch, > HDFS-5040.patch, HDFS-5040.patch, HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[
https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162750#comment-16162750
]
Hadoop QA commented on HDFS-12353:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 183 unchanged - 0 fixed = 187 total (was 183) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
37s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 9
unchanged - 0 fixed = 10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12353 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886600/HDFS-12353-1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 589b52b628ca 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e74d1be |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/21088/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/21088/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/21088/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/21088/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/21088/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/21088/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javadoc
[jira] [Updated] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12423:
-
Attachment: HDFS-12423-HDFS-7240.002.patch
> Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge
> Attachments: HDFS-12423-HDFS-7240.001.patch,
> HDFS-12423-HDFS-7240.002.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162747#comment-16162747
]
Yiqun Lin commented on HDFS-12423:
--
I agree with you suggestion, [~cheersyang].
The failed UT {{TestBlockDeletingService}} is related. That's because container
manager instance was initialized before writing deletion blocks into metastore.
Actually we don't need to use TopN policy for testing, random policy way is
enough.
Attach the updated patch.
> Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge
> Attachments: HDFS-12423-HDFS-7240.001.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Attachment: HDFS-12353-1.patch > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Status: Patch Available (was: Open) > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Status: Open (was: Patch Available) > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Attachment: (was: HDFS-12353-1.patch) > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Attachment: HDFS-12353-1.patch > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162717#comment-16162717
]
Hadoop QA commented on HDFS-12423:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.ozone.container.common.TestBlockDeletingService |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12423 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886587/HDFS-12423-HDFS-7240.001.patch
|
| Optional Tests
[jira] [Commented] (HDFS-12405) Clean up removed erasure coding policies from namenode
[ https://issues.apache.org/jira/browse/HDFS-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162668#comment-16162668 ] Huafeng Wang commented on HDFS-12405: - I got few questions about this issue. Why do we have to clean up the removed policies? I think NameNode's restart is not frequently enough so clean up at that time can only cover a little portion of policies, so clean up them when NameNode restart would suffice? > Clean up removed erasure coding policies from namenode > -- > > Key: HDFS-12405 > URL: https://issues.apache.org/jira/browse/HDFS-12405 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Reporter: SammiChen >Assignee: Huafeng Wang > Labels: hdfs-ec-3.0-nice-to-have > > Currently, when an erasure coding policy is removed, it's been transited to > "removed" state. User cannot apply policy with "removed" state to > file/directory anymore. The policy cannot be safely removed from the system > unless we know there are no existing files or directories that use this > "removed" policy. To find out whether there are files or directories which > are using the policy is time consuming in runtime and might impact the > Namenode performance. So a better choice is doing the work when NameNode > restarts and loads Inodes. Collecting the information at that time will not > introduce much extra overhead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162656#comment-16162656
]
Weiwei Yang commented on HDFS-12423:
Hi [~linyiqun]
Thanks for the patch, it looks good to me. Just one suggestion, since
{{ContainerStatus}} are sorted, can we stop looking for next entry if we met a
container has 0 number of pending deletion blocks? So topN policy returns us at
most N containers, but it could be any size Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge
> Attachments: HDFS-12423-HDFS-7240.001.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12423:
---
Labels: ozoneMerge (was: )
> Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge
> Attachments: HDFS-12423-HDFS-7240.001.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12222) Add EC information to BlockLocation
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huafeng Wang updated HDFS-1:
Attachment: HDFS-1.006.patch
> Add EC information to BlockLocation
> ---
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-1.001.patch, HDFS-1.002.patch,
> HDFS-1.003.patch, HDFS-1.004.patch, HDFS-1.005.patch,
> HDFS-1.006.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12423:
-
Status: Patch Available (was: Open)
> Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12423-HDFS-7240.001.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12423) Ozone: TopN container choosing policy should ignore unnecessary containers
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12423:
-
Attachment: HDFS-12423-HDFS-7240.001.patch
Thanks for the comments, [~cheersyang]. Attach the initial patch.
> Ozone: TopN container choosing policy should ignore unnecessary containers
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12423-HDFS-7240.001.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5086) Support RPCSEC_GSS authentication in NFSv3 gateway
[ https://issues.apache.org/jira/browse/HDFS-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162563#comment-16162563 ] Manjunath B S commented on HDFS-5086: - When do you plan to push this into the main stream Jing Zhao? > Support RPCSEC_GSS authentication in NFSv3 gateway > -- > > Key: HDFS-5086 > URL: https://issues.apache.org/jira/browse/HDFS-5086 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0-alpha1 >Reporter: Brandon Li >Assignee: Jing Zhao > Attachments: HDFS-5086.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12205) Ozone: List Key on an empty ozone bucket fails with command failed error
[
https://issues.apache.org/jira/browse/HDFS-12205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162538#comment-16162538
]
Lokesh Jain commented on HDFS-12205:
It is still an issue. I have to add a test for the fix. I will upload a patch
soon.
> Ozone: List Key on an empty ozone bucket fails with command failed error
>
>
> Key: HDFS-12205
> URL: https://issues.apache.org/jira/browse/HDFS-12205
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Lokesh Jain
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
>
> Tried listing an empty bucket and it fails with
> {code}
> [root@88970a014980 opt]# hadoop-3.0.0-alpha4-SNAPSHOT/bin/hdfs oz -listKey
> http://localhost:9864/vol1/bucket1
> Command Failed :
> {"httpCode":400,"shortMessage":"invalidResourceName","resource":"vol1/bucket1","message":"Invalid
> volume, bucket or key
> name.","requestID":"a38471bb-3fbf-416f-b89d-758506185244","hostName":"88970a014980"}
> {code}
> However after adding one key the same command works.
> {code}
> [root@88970a014980 opt]# hadoop-3.0.0-alpha4-SNAPSHOT/bin/hdfs oz -listKey
> http://localhost:9864/vol1/bucket1
> {
> "version" : 0,
> "md5hash" : "d41d8cd98f00b204e9800998ecf8427e",
> "createdOn" : "Thu, 27 Jul 2017 11:43:55 +",
> "size" : 0,
> "keyName" : "key1"
> }
> {code}
> I feel that for an empty bucket, an empty json should be returned.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
