[jira] [Commented] (HDFS-12573) Divide the total block metrics into replica and ec
[
https://issues.apache.org/jira/browse/HDFS-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187705#comment-16187705
]
Hadoop QA commented on HDFS-12573:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 306 unchanged - 0 fixed = 307 total (was 306) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 42s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| |
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.decrementBlockStat(Block)
uses the same code for two branches At BlocksMap.java:for two branches At
BlocksMap.java:[line 219] |
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12573 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889908/HDFS-12573.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8c6fb4271b5b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 06df6ab |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://bu
[jira] [Commented] (HDFS-12539) Ozone: refactor some functions in KSMMetadataManagerImpl to be more readable and reusable
[
https://issues.apache.org/jira/browse/HDFS-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187685#comment-16187685
]
Hadoop QA commented on HDFS-12539:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12539 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889907/HDFS-12539-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6ce21c4bc6c5 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b5f1926 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21473/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/21473/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdf
[jira] [Updated] (HDFS-12572) Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
[ https://issues.apache.org/jira/browse/HDFS-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12572: - Attachment: HDFS-12572-HDFS-7240.002.patch > Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs > > > Key: HDFS-12572 > URL: https://issues.apache.org/jira/browse/HDFS-12572 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12572-HDFS-7240.001.patch, > HDFS-12572-HDFS-7240.002.patch > > > This jira will add the delete/list status/rename/mkdir APIs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12572) Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
[
https://issues.apache.org/jira/browse/HDFS-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187672#comment-16187672
]
Hadoop QA commented on HDFS-12572:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 20s{color} | {color:orange} root: The patch generated 1 new + 6 unchanged -
0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
42s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-ozone in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
37s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}210m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.TestMetadataStore |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12572 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889884/HDFS-12572-HDFS-7240.001.patch
|
| Optional Tests | asflicense compil
[jira] [Commented] (HDFS-8865) Improve quota initialization performance
[
https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187671#comment-16187671
]
Xiao Chen commented on HDFS-8865:
-
I gave up on pre-commits and went on to run hadoop-hdfs tests in internal
infra. Results below:
{noformat}
branch-2.7:
Test Result (12 failures / +12)
org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten
org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner.testThrottling
org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode.testNonDFSUsedONDeadNodeReReg
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testSnapshotWithAppendTruncate
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testCopyOnTruncateWithDataNodesRestart
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testSnapshotWithTruncates
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateRecovery
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateShellCommandOnBlockBoundary
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateWithDataNodesRestartImmediately
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncate4Symlink
branch-2.6:
Test Result (7 failures / -5)
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testBasicFunctionality
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner.testDataBlockScanner
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner.testBlockScannerAfterRefresh
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner.testBlockScannerAfterRestart
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles.testFallbackToDiskFull
org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache.testDataXceiverCleansUpSlotsOnFailure
{noformat}
All passed locally, except {{TestDeadDatanode#testNonDFSUsedONDeadNodeReReg}}
for branch-2.7 and {{TestLazyPersistFiles#testFallbackToDiskFull}} for
branch-2.6 are failing with or without this patch.
Will file a jira for branch-2.7, and backport the following (in the exact
order, left is earlier) HDFS-9072->HDFS-9740->HDFS-10270->HDFS-9743 to fix
branch-2.6.
Plan to commit this by end of Monday if no objections.
Thanks.
> Improve quota initialization performance
>
>
> Key: HDFS-8865
> URL: https://issues.apache.org/jira/browse/HDFS-8865
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HDFS-8865_branch-2.6.patch, HDFS-8865.branch-2.6.patch,
> HDFS-8865_branch-2.7.patch, HDFS-8865.branch-2.7.patch, HDFS-8865.patch,
> HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch, HDFS-8865.v3.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in
> order to initialize the quota. For big name space, this can take a very long
> time. Since this is done during namenode failover, it also affects failover
> latency.
> By using the Fork-Join framework, I was able to greatly reduce the
> initialization time. The following is the test result using the fsimage from
> one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12573) Divide the total block metrics into replica and ec
[ https://issues.apache.org/jira/browse/HDFS-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-12573: Summary: Divide the total block metrics into replica and ec (was: Add new total blocks metrics which is divided into replica and ec) > Divide the total block metrics into replica and ec > -- > > Key: HDFS-12573 > URL: https://issues.apache.org/jira/browse/HDFS-12573 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, metrics, namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Attachments: HDFS-12573.1.patch > > > Following HDFS-10999. let's separate total blocks metrics. It would be useful > for administrators. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12573) Add new total blocks metrics which is divided into replica and ec
[ https://issues.apache.org/jira/browse/HDFS-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-12573: Status: Patch Available (was: Open) > Add new total blocks metrics which is divided into replica and ec > - > > Key: HDFS-12573 > URL: https://issues.apache.org/jira/browse/HDFS-12573 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, metrics, namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Attachments: HDFS-12573.1.patch > > > Following HDFS-10999. let's separate total blocks metrics. It would be useful > for administrators. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12573) Add new total blocks metrics which is divided into replica and ec
[ https://issues.apache.org/jira/browse/HDFS-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-12573: Attachment: HDFS-12573.1.patch Uploaded the 1st patch. > Add new total blocks metrics which is divided into replica and ec > - > > Key: HDFS-12573 > URL: https://issues.apache.org/jira/browse/HDFS-12573 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, metrics, namenode >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Attachments: HDFS-12573.1.patch > > > Following HDFS-10999. let's separate total blocks metrics. It would be useful > for administrators. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12573) Add new total blocks metrics which is divided into replica and ec
Takanobu Asanuma created HDFS-12573: --- Summary: Add new total blocks metrics which is divided into replica and ec Key: HDFS-12573 URL: https://issues.apache.org/jira/browse/HDFS-12573 Project: Hadoop HDFS Issue Type: Improvement Components: erasure-coding, metrics, namenode Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma Following HDFS-10999. let's separate total blocks metrics. It would be useful for administrators. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12539) Ozone: refactor some functions in KSMMetadataManagerImpl to be more readable and reusable
[ https://issues.apache.org/jira/browse/HDFS-12539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-12539: -- Attachment: HDFS-12539-HDFS-7240.002.patch Thanks Anu and Weiwei's review. Upload v2 patch > Ozone: refactor some functions in KSMMetadataManagerImpl to be more readable > and reusable > - > > Key: HDFS-12539 > URL: https://issues.apache.org/jira/browse/HDFS-12539 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yuanbo Liu >Priority: Minor > Attachments: HDFS-12539-HDFS-7240.001.patch, > HDFS-12539-HDFS-7240.002.patch > > > This is from [~anu]'s review comment in HDFS-12506, > [https://issues.apache.org/jira/browse/HDFS-12506?focusedCommentId=16178356&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16178356]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[
https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187647#comment-16187647
]
Hadoop QA commented on HDFS-11575:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} root: The patch generated 0 new + 165 unchanged - 32
fixed = 165 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 55s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
5s{color} | {color:green} hadoop-hdfs-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-11575 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889900/HDFS-11575.010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d14adc7c7969 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 06df6ab |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21471/testReport/ |
| modules | C: hadoop-common-project/hadoop-nfs
hadoop-hdfs-project/hadoop-h
[jira] [Commented] (HDFS-12562) Ozone: Add ASF license to docker files
[
https://issues.apache.org/jira/browse/HDFS-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187639#comment-16187639
]
Hadoop QA commented on HDFS-12562:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12562 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889886/HDFS-12562-HDFS-7240.001.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux 9fad24779aa0 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b5f1926 |
| shellcheck | v0.4.6 |
| modules | C: U: |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21470/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Add ASF license to docker files
> --
>
> Key: HDFS-12562
> URL: https://issues.apache.org/jira/browse/HDFS-12562
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Anu Engineer
>Assignee: Mukul Kumar Singh
>Priority: Blocker
> Labels: ozoneMerge
> Attachments: HDFS-12562-HDFS-7240.001.patch
>
>
> HDFS-12469 was checked in without valid ASF headers for these files. Not sure
> if we can add them, if not we should exclude them in RAT check.
> cc : [~elek]
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header:
> !? dev-support/compose/ozone/docker-compose.yaml
> !? dev-support/compose/ozone/docker-config
> !? dev-support/compose/ozone/.env
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12572) Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
[ https://issues.apache.org/jira/browse/HDFS-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12572: - Status: Patch Available (was: Open) > Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs > > > Key: HDFS-12572 > URL: https://issues.apache.org/jira/browse/HDFS-12572 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12572-HDFS-7240.001.patch > > > This jira will add the delete/list status/rename/mkdir APIs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12562) Ozone: Add ASF license to docker files
[ https://issues.apache.org/jira/browse/HDFS-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12562: - Status: Patch Available (was: Open) > Ozone: Add ASF license to docker files > -- > > Key: HDFS-12562 > URL: https://issues.apache.org/jira/browse/HDFS-12562 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: ozoneMerge > Attachments: HDFS-12562-HDFS-7240.001.patch > > > HDFS-12469 was checked in without valid ASF headers for these files. Not sure > if we can add them, if not we should exclude them in RAT check. > cc : [~elek] > Lines that start with ? in the ASF License report indicate files that do > not have an Apache license header: > !? dev-support/compose/ozone/docker-compose.yaml > !? dev-support/compose/ozone/docker-config > !? dev-support/compose/ozone/.env -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[ https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-11575: - Attachment: HDFS-11575.010.patch > Supporting HDFS NFS gateway with Federated HDFS > --- > > Key: HDFS-11575 > URL: https://issues.apache.org/jira/browse/HDFS-11575 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11575.001.patch, HDFS-11575.002.patch, > HDFS-11575.003.patch, HDFS-11575.004.patch, HDFS-11575.005.patch, > HDFS-11575.006.patch, HDFS-11575.007.patch, HDFS-11575.008.patch, > HDFS-11575.009.patch, HDFS-11575.010.patch, SupportingNFSwithFederatedHDFS.pdf > > > Currently HDFS NFS gateway only supports HDFS as the underlying filesystem. > Federated HDFS with ViewFS helps in improving the scalability of the name > nodes. However NFS is not supported with ViewFS. > With this change, ViewFS using HDFS as the underlying filesystem can be > exported using NFS. ViewFS mount table will be used to determine the exports > which needs to be supported. > Some important points > 1) This patch only supports HDFS as the underlying filesystem for ViewFS. > 2) This patch add support to add more than one export point in the NFS gateway > 3) Root filesystem of the ViewFS will not be mountable for NFS gateway with > ViewFS, >however this will not be the case for NFS gateway with HDFS > 4) A filehandle now apart from the field will also contain an identifier to > identify the name node, this will be used to map to correct name node for > file operations. > Please find the attached pdf document which helps in explaining the design > and the solution. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12564) Add the documents of swebhdfs configurations on the client side
[ https://issues.apache.org/jira/browse/HDFS-12564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187607#comment-16187607 ] Takanobu Asanuma commented on HDFS-12564: - Hello, [~jojochuang]. Could you review this jira? The latest patch reuses some of the sentences which removed from Hadoop-3.0 by HDFS-9640. :) > Add the documents of swebhdfs configurations on the client side > --- > > Key: HDFS-12564 > URL: https://issues.apache.org/jira/browse/HDFS-12564 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, webhdfs >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > Attachments: HDFS-12564.1.patch, HDFS-12564.2.patch > > > Documentation does not cover the swebhdfs configurations on the client side. > We can reuse the hftp/hsftp documents which was removed from Hadoop-3.0 in > HDFS-5570, HDFS-9640. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[
https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187515#comment-16187515
]
Hadoop QA commented on HDFS-11575:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 57s{color}
| {color:red} root generated 6 new + 1289 unchanged - 0 fixed = 1295 total (was
1289) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 2 new + 167 unchanged
- 30 fixed = 169 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 35s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
3s{color} | {color:green} hadoop-hdfs-nfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-11575 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889890/HDFS-11575.009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9947135b8a2f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 06df6ab |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/21469/artifact/patchprocess/diff-compile-javac-root.tx
[jira] [Comment Edited] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[
https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187492#comment-16187492
]
Mukul Kumar Singh edited comment on HDFS-11575 at 10/1/17 6:47 PM:
---
The latest patch fixes the issue with {{resolvePath}} in the last patch, in the
new patch for {{ViewFileSystem}} the path conversion is done by reading the
entries in the ViewFileSystem.
[~jnp] Please have a look at the latest patch.
was (Author: msingh):
The latest patch fixes the issue with {{resolvePath}} in the last patch, in the
new patch for {{ViewFileSystem}} the path conversion is done by reading the
entries in the ViewFileSystem.
> Supporting HDFS NFS gateway with Federated HDFS
> ---
>
> Key: HDFS-11575
> URL: https://issues.apache.org/jira/browse/HDFS-11575
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11575.001.patch, HDFS-11575.002.patch,
> HDFS-11575.003.patch, HDFS-11575.004.patch, HDFS-11575.005.patch,
> HDFS-11575.006.patch, HDFS-11575.007.patch, HDFS-11575.008.patch,
> HDFS-11575.009.patch, SupportingNFSwithFederatedHDFS.pdf
>
>
> Currently HDFS NFS gateway only supports HDFS as the underlying filesystem.
> Federated HDFS with ViewFS helps in improving the scalability of the name
> nodes. However NFS is not supported with ViewFS.
> With this change, ViewFS using HDFS as the underlying filesystem can be
> exported using NFS. ViewFS mount table will be used to determine the exports
> which needs to be supported.
> Some important points
> 1) This patch only supports HDFS as the underlying filesystem for ViewFS.
> 2) This patch add support to add more than one export point in the NFS gateway
> 3) Root filesystem of the ViewFS will not be mountable for NFS gateway with
> ViewFS,
>however this will not be the case for NFS gateway with HDFS
> 4) A filehandle now apart from the field will also contain an identifier to
> identify the name node, this will be used to map to correct name node for
> file operations.
> Please find the attached pdf document which helps in explaining the design
> and the solution.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[
https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-11575:
-
Attachment: HDFS-11575.009.patch
The latest patch fixes the issue with {{resolvePath}} in the last patch, in the
new patch for {{ViewFileSystem}} the path conversion is done by reading the
entries in the ViewFileSystem.
> Supporting HDFS NFS gateway with Federated HDFS
> ---
>
> Key: HDFS-11575
> URL: https://issues.apache.org/jira/browse/HDFS-11575
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11575.001.patch, HDFS-11575.002.patch,
> HDFS-11575.003.patch, HDFS-11575.004.patch, HDFS-11575.005.patch,
> HDFS-11575.006.patch, HDFS-11575.007.patch, HDFS-11575.008.patch,
> HDFS-11575.009.patch, SupportingNFSwithFederatedHDFS.pdf
>
>
> Currently HDFS NFS gateway only supports HDFS as the underlying filesystem.
> Federated HDFS with ViewFS helps in improving the scalability of the name
> nodes. However NFS is not supported with ViewFS.
> With this change, ViewFS using HDFS as the underlying filesystem can be
> exported using NFS. ViewFS mount table will be used to determine the exports
> which needs to be supported.
> Some important points
> 1) This patch only supports HDFS as the underlying filesystem for ViewFS.
> 2) This patch add support to add more than one export point in the NFS gateway
> 3) Root filesystem of the ViewFS will not be mountable for NFS gateway with
> ViewFS,
>however this will not be the case for NFS gateway with HDFS
> 4) A filehandle now apart from the field will also contain an identifier to
> identify the name node, this will be used to map to correct name node for
> file operations.
> Please find the attached pdf document which helps in explaining the design
> and the solution.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12537) Ozone: Reduce key creation overhead in Corona
[ https://issues.apache.org/jira/browse/HDFS-12537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187480#comment-16187480 ] Lokesh Jain commented on HDFS-12537: [~nandakumar131] [~anu] Thanks for reviewing the patch. Currently the default behavior adds a random UUID in each key value. I think the optimizedWrites option can be removed. If it would be removed there would be two writes made to the outputStream, the option does it in a single write. But it is not much of an overhead. So it can be removed. > Ozone: Reduce key creation overhead in Corona > - > > Key: HDFS-12537 > URL: https://issues.apache.org/jira/browse/HDFS-12537 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain > Attachments: HDFS-12537-HDFS-7240.001.patch, > HDFS-12537-HDFS-7240.002.patch > > > Currently Corona creates random key values for each key. This creates a lot > of overhead. An option should be provided to use a single key value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12467) Ozone: SCM: NodeManager should log when it comes out of chill mode
[
https://issues.apache.org/jira/browse/HDFS-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187456#comment-16187456
]
Nandakumar commented on HDFS-12467:
---
Thanks [~vagarychen] for the detailed description. I agree with whatever you
have said.
Previously there was no way to bring SCM out of startup chill mode with any API
calls, it happened only through datanode registration. I used
{{forceExitChillMode}} for this, now with forceExitChillMode() the system will
come out of both the chill modes.
I would like to propose the following changes for chillMode (refactor the whole
chill mode logic)
The following methods will be supported to manipulate chill mode status of SCM
* {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and
manual)
* {{enterChillMode}} - will set manual chill mode flag to true
* {{exitChillMode}} - will set manual chill mode flag to false
* {{getChillModeStatus}} - will return chill mode status string
* {{inChillMode}} - will return true if SCM is in chill mode, else false
The following methods will be removed
* {{clearChillModeFlag}}
* {{forceEnterChillMode}}
* {{isInManualChillMode}}
* {{isOutOfNodeChillMode}}
The getter and setter for MinimumChillModeNodes will remain as it is
* {{setMinimumChillModeNodes}}
* {{getMinimumChillModeNodes}}
This will make it easy to understand and use.
> Ozone: SCM: NodeManager should log when it comes out of chill mode
> --
>
> Key: HDFS-12467
> URL: https://issues.apache.org/jira/browse/HDFS-12467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
>Priority: Minor
> Attachments: HDFS-12467-HDFS-7240.000.patch,
> HDFS-12467-HDFS-7240.001.patch, HDFS-12467-HDFS-7240.002.patch,
> HDFS-12467-HDFS-7240.003.patch, HDFS-12467-HDFS-7240.004.patch
>
>
> {{NodeManager}} should add a log message when it comes out of chill mode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12476) TestCopyFromLocal.testCopyFromLocalWithThreads fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-12476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh resolved HDFS-12476.
--
Resolution: Duplicate
HADOOP-14752 fixes this issue, duping this.
> TestCopyFromLocal.testCopyFromLocalWithThreads fails intermittently
> ---
>
> Key: HDFS-12476
> URL: https://issues.apache.org/jira/browse/HDFS-12476
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>
> TestCopyFromLocal.testCopyFromLocalWithThreads fails intermittently.
> {code}
> java.lang.AssertionError: expected:<0> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.shell.TestCopyFromLocal$TestMultiThreadedCopy.processArguments(TestCopyFromLocal.java:167)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at
> org.apache.hadoop.fs.shell.TestCopyFromLocal.run(TestCopyFromLocal.java:101)
> at
> org.apache.hadoop.fs.shell.TestCopyFromLocal.testCopyFromLocalWithThreads(TestCopyFromLocal.java:121)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12562) Ozone: Add ASF license to docker files
[ https://issues.apache.org/jira/browse/HDFS-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12562: - Attachment: HDFS-12562-HDFS-7240.001.patch > Ozone: Add ASF license to docker files > -- > > Key: HDFS-12562 > URL: https://issues.apache.org/jira/browse/HDFS-12562 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Priority: Blocker > Labels: ozoneMerge > Attachments: HDFS-12562-HDFS-7240.001.patch > > > HDFS-12469 was checked in without valid ASF headers for these files. Not sure > if we can add them, if not we should exclude them in RAT check. > cc : [~elek] > Lines that start with ? in the ASF License report indicate files that do > not have an Apache license header: > !? dev-support/compose/ozone/docker-compose.yaml > !? dev-support/compose/ozone/docker-config > !? dev-support/compose/ozone/.env -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12562) Ozone: Add ASF license to docker files
[ https://issues.apache.org/jira/browse/HDFS-12562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDFS-12562: Assignee: Mukul Kumar Singh > Ozone: Add ASF license to docker files > -- > > Key: HDFS-12562 > URL: https://issues.apache.org/jira/browse/HDFS-12562 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: ozoneMerge > Attachments: HDFS-12562-HDFS-7240.001.patch > > > HDFS-12469 was checked in without valid ASF headers for these files. Not sure > if we can add them, if not we should exclude them in RAT check. > cc : [~elek] > Lines that start with ? in the ASF License report indicate files that do > not have an Apache license header: > !? dev-support/compose/ozone/docker-compose.yaml > !? dev-support/compose/ozone/docker-config > !? dev-support/compose/ozone/.env -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12556) [SPS] : Block movement analysis should be done in read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187395#comment-16187395
]
Hadoop QA commented on HDFS-12556:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
14s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} HDFS-10285 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 10m
59s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 10m
23s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12556 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889882/HDFS-12556-HDFS-10285-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Updated] (HDFS-12572) Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
[ https://issues.apache.org/jira/browse/HDFS-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12572: - Attachment: HDFS-12572-HDFS-7240.001.patch [~xyao], this jira will introduces the final list of APIs of OzoneFS, please have a look. Also I have rename ozfs to "o3" > Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs > > > Key: HDFS-12572 > URL: https://issues.apache.org/jira/browse/HDFS-12572 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12572-HDFS-7240.001.patch > > > This jira will add the delete/list status/rename/mkdir APIs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12572) Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
Mukul Kumar Singh created HDFS-12572: Summary: Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs Key: HDFS-12572 URL: https://issues.apache.org/jira/browse/HDFS-12572 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Fix For: HDFS-7240 This jira will add the delete/list status/rename/mkdir APIs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12556) [SPS] : Block movement analysis should be done in read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187347#comment-16187347
]
Surendra Singh Lilhore commented on HDFS-12556:
---
Some time {{TestPersistentStoragePolicySatisfier#testWithRestarts}} failing
because of {{ArrayIndexOutOfBoundsException}}. This is happening because when
block movement analysis is going on, same time NN receive block report from one
DN and it will update the block storage list.
{{BlockManager#getStorages(BlockInfo)}} API create internal array {{storages}}
based on current number of node for a block but later number of node is
increased while filling the array and it cause
{{ArrayIndexOutOfBoundsException}}.
{code}
public DatanodeStorageInfo[] getStorages(BlockInfo block) {
final DatanodeStorageInfo[] storages = new
DatanodeStorageInfo[block.numNodes()];
int i = 0;
for(DatanodeStorageInfo s : blocksMap.getStorages(block)) {
storages[i++] = s;
}
return storages;
}
{code}
Block analysis should be done in read lock, so we can avoid in-between block
location storage update.
Attached initial patch. Please review...
> [SPS] : Block movement analysis should be done in read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done on read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Status: Patch Available (was: Open)
> [SPS] : Block movement analysis should be done on read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done in read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Summary: [SPS] : Block movement analysis should be done in read lock.
(was: [SPS] : Block movement analysis should be done on read lock.)
> [SPS] : Block movement analysis should be done in read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done on read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Attachment: HDFS-12556-HDFS-10285-01.patch
> [SPS] : Block movement analysis should be done on read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done on read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Summary: [SPS] : Block movement analysis should be done on read lock.
(was: [SPS] : Fix TestPersistentStoragePolicySatisfier#testWithRestarts )
> [SPS] : Block movement analysis should be done on read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12568) Ozone: Cleanup the ozone-default.xml
[
https://issues.apache.org/jira/browse/HDFS-12568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16187318#comment-16187318
]
Hadoop QA commented on HDFS-12568:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 26s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
39s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}192m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
| | hadoop.ozone.TestOzoneConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12568 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889870/HDFS-12568-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 9ad19fce0c98 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_
