[jira] [Updated] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[ https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12069: --- Attachment: HDFS-12069-HDFS-7240.006.patch > Ozone: Create a general abstraction for metadata store > -- > > Key: HDFS-12069 > URL: https://issues.apache.org/jira/browse/HDFS-12069 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12069-HDFS-7240.001.patch, > HDFS-12069-HDFS-7240.002.patch, HDFS-12069-HDFS-7240.003.patch, > HDFS-12069-HDFS-7240.004.patch, HDFS-12069-HDFS-7240.005.patch, > HDFS-12069-HDFS-7240.006.patch > > > Create a general abstraction for metadata store so that we can plug other key > value store to host ozone metadata. Currently only levelDB is implemented, we > want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11704) OzoneFileSystem: A Hadoop file system implementation for Ozone
[ https://issues.apache.org/jira/browse/HDFS-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDFS-11704: Assignee: Mukul Kumar Singh (was: Mingliang Liu) > OzoneFileSystem: A Hadoop file system implementation for Ozone > -- > > Key: HDFS-11704 > URL: https://issues.apache.org/jira/browse/HDFS-11704 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs/ozone, ozone >Reporter: Mingliang Liu >Assignee: Mukul Kumar Singh > Attachments: HDFS-11704-HDFS-7240.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11679) Ozone: SCM CLI: Implement list container command
[
https://issues.apache.org/jira/browse/HDFS-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077591#comment-16077591
]
Hadoop QA commented on HDFS-11679:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 14m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} cc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11679 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12876021/HDFS-11679-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 335e97af6a0b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 8efb1fa |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20181/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HD
[jira] [Comment Edited] (HDFS-11679) Ozone: SCM CLI: Implement list container command
[
https://issues.apache.org/jira/browse/HDFS-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077516#comment-16077516
]
Yuanbo Liu edited comment on HDFS-11679 at 7/7/17 3:16 AM:
---
[~cheersyang]
Thanks for your comments. Upload v5 patch.
was (Author: yuanbo):
[~Weiwei Yang]
Thanks for your comments. Upload v5 patch.
> Ozone: SCM CLI: Implement list container command
>
>
> Key: HDFS-11679
> URL: https://issues.apache.org/jira/browse/HDFS-11679
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Labels: command-line
> Attachments: HDFS-11679-HDFS-7240.001.patch,
> HDFS-11679-HDFS-7240.002.patch, HDFS-11679-HDFS-7240.003.patch,
> HDFS-11679-HDFS-7240.004.patch, HDFS-11679-HDFS-7240.005.patch
>
>
> Implement the command to list containers
> {code}
> hdfs scm -container list -start [-count <100> | -end
> ]{code}
> Lists all containers known to SCM. The option -start allows the listing to
> start from a specified container and -count controls the number of entries
> returned but it is mutually exclusive with the -end option which returns keys
> from the -start to -end range.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11679) Ozone: SCM CLI: Implement list container command
[
https://issues.apache.org/jira/browse/HDFS-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-11679:
--
Attachment: HDFS-11679-HDFS-7240.005.patch
[~Weiwei Yang]
Thanks for your comments. Upload v5 patch.
> Ozone: SCM CLI: Implement list container command
>
>
> Key: HDFS-11679
> URL: https://issues.apache.org/jira/browse/HDFS-11679
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Labels: command-line
> Attachments: HDFS-11679-HDFS-7240.001.patch,
> HDFS-11679-HDFS-7240.002.patch, HDFS-11679-HDFS-7240.003.patch,
> HDFS-11679-HDFS-7240.004.patch, HDFS-11679-HDFS-7240.005.patch
>
>
> Implement the command to list containers
> {code}
> hdfs scm -container list -start [-count <100> | -end
> ]{code}
> Lists all containers known to SCM. The option -start allows the listing to
> start from a specified container and -count controls the number of entries
> returned but it is mutually exclusive with the -end option which returns keys
> from the -start to -end range.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[ https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077504#comment-16077504 ] Brahma Reddy Battula commented on HDFS-12084: - Testfailures are unrelated.Kindly review. > Scheduled Count will not decrement when file is deleted before all IBR's > received > - > > Key: HDFS-12084 > URL: https://issues.apache.org/jira/browse/HDFS-12084 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch > > > When small files creation && deletion happens so frequently and DN's did not > report blocks to NN before deletion, then scheduled count will keep on > increment and which will not deleted as blocks are deleted. > *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be > more as so many operations. > when batchIBR enabled with committed allowed=1 this will be observed more. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[ https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077472#comment-16077472 ] Weiwei Yang commented on HDFS-12069: Hi [~xyao] Thanks for helping to review this, please see my comments below bq. Should we change the KeyUtils.getDB here to use the abstracted interface instead of the hardcoded LevelDBStore? ... Similar question to SCM components I was trying not to fix all LevelDBStore in this patch, that will make this patch a very big one. But I actually have done some more work so I can add them in one patch if you like. I was thinking that is not easy for review so I split the work to 3 steps: 1) create general abstraction, the interface 2) Replace existing levelDB references 3) implement rocksDB; I can combine 1 & 2 in this JIRA, if you prefer. I will fix rest of issues in the comments. Thank you! > Ozone: Create a general abstraction for metadata store > -- > > Key: HDFS-12069 > URL: https://issues.apache.org/jira/browse/HDFS-12069 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12069-HDFS-7240.001.patch, > HDFS-12069-HDFS-7240.002.patch, HDFS-12069-HDFS-7240.003.patch, > HDFS-12069-HDFS-7240.004.patch, HDFS-12069-HDFS-7240.005.patch > > > Create a general abstraction for metadata store so that we can plug other key > value store to host ozone metadata. Currently only levelDB is implemented, we > want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12000) Ozone: Container : Add key versioning support
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077462#comment-16077462 ] Weiwei Yang commented on HDFS-12000: Hi [~linyiqun] Thank you for reviewing this. bq. A better way is that only when we detect the change between current key data and uploaded file, then we put this file. Do you mean if user puts a same file twice to a same key, we do not store the file twice and just update each key version reference to the same file? That sounds a good idea but how we can tell if two files are same? Even if we have a neat way to, this will need to be done in every PUT operation and compare with every previous versions, that's a lot of overhead. I think we can live with an assumption with versioning: user generally does not upload same object multiple times to a single key, they upload different versions. Similarly here is the document from s3 : _Each version of an object is the entire object; it is not just a diff from the previous version. Thus, if you have three versions of an object stored, you are charged for three objects._ bq. I think it will still be a hard work for KSM to delete all the version keys/files even if we use a async way. True, this is the missing piece right now in ozone. I have added some comments in HDFS-11922, please take a look. I think we need to support async delete of keys, once that is implemented, it will be easy to support delete any version of objects from KSM. bq. How about using an expire policy for old key versions as that have mentioned in the doc? This should be a good way for this case. Expire policy is one part of the object life cycle management, that would be another new feature. I don't think that will be a near time target. bq. If we need to complete this work in the first phase work of Ozone? I agree. I think this is not a MUST for the 1st phase. Lets listen to the opinions from other folks. Anyway it would be good to study this and see how much effort is there if we need to implement this. Thanks for sharing your idea! > Ozone: Container : Add key versioning support > - > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11704) OzoneFileSystem: A Hadoop file system implementation for Ozone
[ https://issues.apache.org/jira/browse/HDFS-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HDFS-11704: - Attachment: HDFS-11704-HDFS-7240.wip.patch Post a wip patch. [~msingh], I know you are also working on this. Please pick this up and assign the JIRA to yourself if you plan to take over. Thanks, > OzoneFileSystem: A Hadoop file system implementation for Ozone > -- > > Key: HDFS-11704 > URL: https://issues.apache.org/jira/browse/HDFS-11704 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs/ozone, ozone >Reporter: Mingliang Liu >Assignee: Mingliang Liu > Attachments: HDFS-11704-HDFS-7240.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11912) Add a snapshot unit test with randomized file IO operations
[
https://issues.apache.org/jira/browse/HDFS-11912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077407#comment-16077407
]
Hadoop QA commented on HDFS-11912:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 66 new 0 unchanged - 0 fixed = 66 total (was 0) {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestMaintenanceState |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11912 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875998/HDFS-11912.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c6813b96be37 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7576a68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20180/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20180/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20180/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20180/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20180/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-9388) Refactor decommission related code to support maintenance state for datanodes
[ https://issues.apache.org/jira/browse/HDFS-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077367#comment-16077367 ] Ming Ma commented on HDFS-9388: --- 1. Thanks [~manojg]. > Refactor decommission related code to support maintenance state for datanodes > - > > Key: HDFS-9388 > URL: https://issues.apache.org/jira/browse/HDFS-9388 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Ming Ma >Assignee: Manoj Govindassamy > Attachments: HDFS-9388.01.patch, HDFS-9388.02.patch > > > Lots of code can be shared between the existing decommission functionality > and to-be-added maintenance state support for datanodes. To make it easier to > add maintenance state support, let us first modify the existing code to make > it more general. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11912) Add a snapshot unit test with randomized file IO operations
[ https://issues.apache.org/jira/browse/HDFS-11912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Huang updated HDFS-11912: Attachment: HDFS-11912.003.patch Addressing code review comments. > Add a snapshot unit test with randomized file IO operations > --- > > Key: HDFS-11912 > URL: https://issues.apache.org/jira/browse/HDFS-11912 > Project: Hadoop HDFS > Issue Type: Test > Components: hdfs >Reporter: George Huang >Assignee: George Huang >Priority: Minor > Labels: TestGap > Attachments: HDFS-11912.001.patch, HDFS-11912.002.patch, > HDFS-11912.003.patch > > > Adding a snapshot unit test with randomized file IO operations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11912) Add a snapshot unit test with randomized file IO operations
[ https://issues.apache.org/jira/browse/HDFS-11912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077305#comment-16077305 ] George Huang commented on HDFS-11912: - Hi Manoj, Thank you for the code review and great feedback! Please see below, 1. Operation type, just like weight can be part of the constructor and we can avoid checking the value string starting with to identify the operation and get random operations. ==> Fixed 2. Random operation initialization in line 153 and line 157 are not needed. ==> Fixed 3. In createFile, line 616, the loop limit says TOTAL_BLOCKS, but it is infact treated like max file count. ==> Changed to TOTAL_FILECOUNT 4. In createFile, line 607, some top level directory might have been set as snapshotable earlier. Later when you enable snapshot for a nested directory under it, the call can fail with below exception. Are you ignoring the exception? ==> Great catch. In this case, we'll swallow the exception. This should be fine. 5. Line 280, we can assert here. Otherwise, there is an issue with creating random operation. 6. Line 304, same here. ==> Sure. This should not happen. We'll assert the option is null, which is the initial value. Any none null value would be an error if it fall into here. 7. Line 318, we can add a unique number suffix to the created directory so as to differentiate where the failure happened 8. Line 366, rename dir suffix can also take a unique number for differentiation. ==> Fixed 9. Line 551, 552: Aren't FileStatus[] from listStatus already sorted ? can you please check? ==> Trying to find if anywhere document whether the return set is sorted. For example apache link at here: https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#listStatus(org.apache.hadoop.fs.Path) Does not see if's mentioning the return set is sorted. It is implementation specific I assume. Therefore, to be implementation independent, we''ll keep the sorting as is. Many thanks, George > Add a snapshot unit test with randomized file IO operations > --- > > Key: HDFS-11912 > URL: https://issues.apache.org/jira/browse/HDFS-11912 > Project: Hadoop HDFS > Issue Type: Test > Components: hdfs >Reporter: George Huang >Assignee: George Huang >Priority: Minor > Labels: TestGap > Attachments: HDFS-11912.001.patch, HDFS-11912.002.patch > > > Adding a snapshot unit test with randomized file IO operations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077281#comment-16077281
]
Hadoop QA commented on HDFS-12051:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new 627 unchanged - 17 fixed = 635 total (was 644) {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12051 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875976/HDFS-12051.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5294f01f06cd 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7576a68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20179/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20179/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20179/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20179/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20179/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> --
[jira] [Commented] (HDFS-11696) Fix warnings from Spotbugs in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077218#comment-16077218 ] Andrew Wang commented on HDFS-11696: Looks good, thanks Yiqun, do you mind also adding a regression test for the flags behavior? 1 pending that. > Fix warnings from Spotbugs in hadoop-hdfs > - > > Key: HDFS-11696 > URL: https://issues.apache.org/jira/browse/HDFS-11696 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: findbugsHtml.html, HADOOP-14337.001.patch, > HADOOP-14337.002.patch, HADOOP-14337.003.patch, HDFS-11696.004.patch, > HDFS-11696.005.patch, HDFS-11696.006.patch, HDFS-11696.007.patch, > HDFS-11696.008.patch > > > There are totally 12 findbugs issues generated after switching from Findbugs > to Spotbugs across the project in HADOOP-14316. This JIRA focus on cleaning > up the part of warnings under scope of HDFS(mainly contained in hadoop-hdfs > and hadoop-hdfs-client). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12097) libhdfs++: add Clang build and tests to the CI system
[ https://issues.apache.org/jira/browse/HDFS-12097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077209#comment-16077209 ] James Clampffer commented on HDFS-12097: I'd like to make sure that this is committed at the same time or very shortly after HDFS-12026 so there isn't a gap for the build to break again. The library used to compile fine with clang. > libhdfs++: add Clang build and tests to the CI system > - > > Key: HDFS-12097 > URL: https://issues.apache.org/jira/browse/HDFS-12097 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein > > For better portability we should add a Clang build and tests of libhdfs++ > library to the CI system. To accomplish that the Dockerfile will need to be > updated with the environment setup, and the maven files should be updated to > build libhdfs++ using Clang and then run the tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Status: Patch Available (was: Open)
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12051.01.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
>
> {code}
> To eliminate th
[jira] [Updated] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Attachment: HDFS-12051.01.patch
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12051.01.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
>
> {code}
> To eliminate this du
[jira] [Commented] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[
https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077155#comment-16077155
]
Hadoop QA commented on HDFS-12026:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
49s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 7m
32s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 7m
4s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_131 {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_131 {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 6m
53s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green} 1{color} | {color:green} cc {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green} 1{color} | {color:green} cc {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green} 9m
45s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with
JDK v1.7.0_131. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5ae34ac |
| JIRA Issue | HDFS-12026 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875968/HDFS-12026.HDFS-8707.003.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 0460100daa40 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / 0d2d073 |
| Default Java | 1.7.0_131 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_131
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/20178/artifact/patchprocess/whitespace-eol.txt
|
| JDK v1.7.0_131 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20178/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20178/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> libhdfs++: Fix compilation errors and warnings when compiling with Clang
> -
>
> Key: HDFS-12026
> URL: https://issues.apache.org/jira/browse/HDFS-12026
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-12026.HDFS-8707.0
[jira] [Updated] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-12026: - Attachment: HDFS-12026.HDFS-8707.003.patch In the new patch I added the the CheckCXXSourceCompiles to cmake in order to test whether the installed protoc library is compatible with the compiler. I also created a JIRA HDFS-12097 to add Clang build and tests to the CI system. Please review. > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flags: > -std=c++11 -stdlib=libc++ > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-12026: - Summary: libhdfs++: Fix compilation errors and warnings when compiling with Clang (was: libhdfspp: Fix compilation errors and warnings when compiling with Clang ) > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flags: > -std=c++11 -stdlib=libc++ > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12097) libhdfs++: add Clang build and tests to the CI system
Anatoli Shein created HDFS-12097: Summary: libhdfs++: add Clang build and tests to the CI system Key: HDFS-12097 URL: https://issues.apache.org/jira/browse/HDFS-12097 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anatoli Shein For better portability we should add a Clang build and tests of libhdfs++ library to the CI system. To accomplish that the Dockerfile will need to be updated with the environment setup, and the maven files should be updated to build libhdfs++ using Clang and then run the tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[ https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077063#comment-16077063 ] Xiaoyu Yao commented on HDFS-12069: --- Thanks [~cheersyang] for working on this. Patch v5 looks pretty good to me. Few comments below. 1. KeyManagerImpl.java Line 34/89/114/148/182: Should we change the KeyUtils.getDB here to use the abstracted interface instead of the hardcoded LevelDBStore? Similar question to SCM components that are hard coded to LevelDBStore today? Should we switch that to use the MetadataStore interface like we've done here for some of the ops in MetadataManagerImpl for KSM? 2. MetadataManagerImpl.java Line 213/232: should we update to use the MetadataStore interface methods like writeBatch()? 3. OzoneConfigKeys.java Line 67: NIT: can we define the const as "LevelDB" to be consistent with the official name. Line 69: NIT: can we define the const as "RocksDB" to be consistent with the official name. 4. ozone-default.xml Line 570: NIT: "LevelDB" Line 575: NIT: "LevelDB" and "RocksDB" 5. TestMetadataStore.java Line 65-66: NIT: remove the unused comments. > Ozone: Create a general abstraction for metadata store > -- > > Key: HDFS-12069 > URL: https://issues.apache.org/jira/browse/HDFS-12069 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12069-HDFS-7240.001.patch, > HDFS-12069-HDFS-7240.002.patch, HDFS-12069-HDFS-7240.003.patch, > HDFS-12069-HDFS-7240.004.patch, HDFS-12069-HDFS-7240.005.patch > > > Create a general abstraction for metadata store so that we can plug other key > value store to host ozone metadata. Currently only levelDB is implemented, we > want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077000#comment-16077000
]
Hadoop QA commented on HDFS-11975:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 17m
8s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
31s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
45s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} cc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green} 1m
13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11975 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875950/HDFS-11975-007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 98662bf6ff45 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7576a68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20177/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
|
| findbugs |
https://builds.apache.org/j
[jira] [Commented] (HDFS-9388) Refactor decommission related code to support maintenance state for datanodes
[ https://issues.apache.org/jira/browse/HDFS-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076990#comment-16076990 ] Manoj Govindassamy commented on HDFS-9388: -- Above unit test failures are not related to patch. > Refactor decommission related code to support maintenance state for datanodes > - > > Key: HDFS-9388 > URL: https://issues.apache.org/jira/browse/HDFS-9388 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Ming Ma >Assignee: Manoj Govindassamy > Attachments: HDFS-9388.01.patch, HDFS-9388.02.patch > > > Lots of code can be shared between the existing decommission functionality > and to-be-added maintenance state support for datanodes. To make it easier to > add maintenance state support, let us first modify the existing code to make > it more general. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12000) Ozone: Container : Add key versioning support
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076872#comment-16076872 ] Chen Liang commented on HDFS-12000: --- Thanks [~cheersyang] for putting up the doc, I really appreciate it! Ping [~anu] who I think is also working on writing up some design doc for versioning. Anu do you want to share it with Weiwei? I will look into both docs in more detail also for sure and come back if got any feedbacks. > Ozone: Container : Add key versioning support > - > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luhuichun updated HDFS-11975:
-
Attachment: HDFS-11975-007.patch
update with some small changes
> Provide a system-default EC policy
> --
>
> Key: HDFS-11975
> URL: https://issues.apache.org/jira/browse/HDFS-11975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: luhuichun
> Attachments: HDFS-11975-001.patch, HDFS-11975-002.patch,
> HDFS-11975-003.patch, HDFS-11975-004.patch, HDFS-11975-005.patch,
> HDFS-11975-006.patch, HDFS-11975-007.patch
>
>
> From the usability point of view, it'd be nice to be able to specify a
> system-wide EC policy, i.e., in {{hdfs-site.xml}}. For most of users / admin
> / downstream projects, it is not necessary to know the tradeoffs of the EC
> policy, considering that it requires the knowledge of EC, the actual physical
> topology of the clusters, and many other factors (i.e., network, cluster size
> and etc).
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12000) Ozone: Container : Add key versioning support
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076819#comment-16076819 ] Yiqun Lin edited comment on HDFS-12000 at 7/6/17 4:19 PM: -- Hi [~cheersyang], thanks for uploading the desgin doc for the oznoe key versioning support. It looks really good. Just some comments from me: If Ozone support multiple key versions, I think how to manager different key version datas should be a major problem we should deal with. The multiple key versions mean that we will store a lot of datas in ozone. Actually we don't really need to store some file datas every time users put in ozone when key version is enabled. A better way is that only when we detect the change between current key data and uploaded file, then we put this file. Otherwise we can just update the reference info, generate a new key version, but the actual data is current file. This is a little similar to HDFS snapshot. This way can save some spaces for ozone. I have a different opition on the way of deleteing key versions when switch enable multiple version to disabled state. I think it will still be a hard work for KSM to delete all the version keys/files even if we use a async way. How about using an expire policy for old key versions as that have mentioned in the doc? This should be a good way for this case. Finally, I have a question: if we need to complete this work in the first phase work of Ozone? I am just curious of this. This feature looks realy usefule for the users. But seems this should need some more work for complte this. There will be lots of places to have a adjustment for supporting mutilple key versions. was (Author: linyiqun): Hi [~cheersyang], thanks for the desgin doc for the oznoe key versioning support. It looks really good. Just some comments from me: If Ozone support multiple key versions, I think how to manager different key version datas should be a major problem we should deal with. The multiple key versions mean that we will store a lot of datas in ozone. Actually we don't really need to store some file datas every time users put in ozone when key version is enabled. A better way is that only when we detect the change between current key data and uploaded file, then we put this file. Otherwise we can just update the reference info, generate a new key version, but the actual data is current file. This is a little similar to HDFS snapshot. This way can save some spaces for ozone. I have a different opition on the way of deleteing key versions when switch enable multiple version to disabled state. I think it will still be a hard work for KSM to delete all the version keys/files even we use a async way. How about using an expire policy for old key versions as that have mentioned in the doc? This should be a good way for this case. Finally, I have a question: if we need to complete this work in the first phase work of Ozone? I am just curious of this. This feature looks realy usefule for the users. But seems this should need some more work for complte this. There will be lots of places to have a adjustment for supporting mutilple key versions. > Ozone: Container : Add key versioning support > - > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12000) Ozone: Container : Add key versioning support
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076819#comment-16076819 ] Yiqun Lin commented on HDFS-12000: -- Hi [~cheersyang], thanks for the desgin doc for the oznoe key versioning support. It looks really good. Just some comments from me: If Ozone support multiple key versions, I think how to manager different key version datas should be a major problem we should deal with. The multiple key versions mean that we will store a lot of datas in ozone. Actually we don't really need to store some file datas every time users put in ozone when key version is enabled. A better way is that only when we detect the change between current key data and uploaded file, then we put this file. Otherwise we can just update the reference info, generate a new key version, but the actual data is current file. This is a little similar to HDFS snapshot. This way can save some spaces for ozone. I have a different opition on the way of deleteing key versions when switch enable multiple version to disabled state. I think it will still be a hard work for KSM to delete all the version keys/files even we use a async way. How about using an expire policy for old key versions as that have mentioned in the doc? This should be a good way for this case. Finally, I have a question: if we need to complete this work in the first phase work of Ozone? I am just curious of this. This feature looks realy usefule for the users. But seems this should need some more work for complte this. There will be lots of places to have a adjustment for supporting mutilple key versions. > Ozone: Container : Add key versioning support > - > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[
https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076639#comment-16076639
]
Hadoop QA commented on HDFS-12069:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
31s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new + 9
unchanged - 1 fixed = 9 total (was 10) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12069 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875923/HDFS-12069-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 31342c323e72 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 8efb1fa |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20176/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20176/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Create a general abstraction for metadata store
> --
>
> Key: HDFS-12069
> URL: https://issues.apache.org/jira/browse/HDFS-12069
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Ya
[jira] [Commented] (HDFS-11696) Fix warnings from Spotbugs in hadoop-hdfs
[
https://issues.apache.org/jira/browse/HDFS-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076556#comment-16076556
]
Hadoop QA commented on HDFS-11696:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
38s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 318
unchanged - 2 fixed = 318 total (was 320) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs-client generated 0
new + 0 unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs generated 0 new + 0
unchanged - 10 fixed = 0 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
10s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11696 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875917/HDFS-11696.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 70ef8ba8d94b 3.13
[jira] [Commented] (HDFS-12000) Ozone: Container : Add key versioning support
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076547#comment-16076547 ] Weiwei Yang commented on HDFS-12000: Hi [~vagarychen] Sorry I did not notice this one and created a dup one today in HDFS-12096. I spent sometime thinking about this problem today and uploaded a doc there :P. Would you please take a look? Hope that can help to enrich your idea. I will move the discussion here and continue to discuss if you want to upload a design doc. Thanks a lot. > Ozone: Container : Add key versioning support > - > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[
https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076507#comment-16076507
]
Hadoop QA commented on HDFS-12084:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12084 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875908/HDFS-12084-002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2e28ec4c2384 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 946dd25 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20174/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20174/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20174/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20174/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Scheduled Count will not decrement when file is deleted before all IBR's
> received
> -
>
> Key: HDFS-12084
> URL: https://issues.apache.org/jira/browse/HDFS-12084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>
[jira] [Commented] (HDFS-12096) Ozone: Bucket versioning design document
[ https://issues.apache.org/jira/browse/HDFS-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076505#comment-16076505 ] Weiwei Yang commented on HDFS-12096: I just realized there is already one JIRA tracking the similar issue. Duplicate this one with that one. > Ozone: Bucket versioning design document > > > Key: HDFS-12096 > URL: https://issues.apache.org/jira/browse/HDFS-12096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: Ozone Bucket Versioning v1.pdf > > > This JIRA is opened for the discussion of the bucket versioning design. > The bucket versioning is the ability to hold multiple versions objects of a > key in a bucket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12096) Ozone: Bucket versioning design document
[ https://issues.apache.org/jira/browse/HDFS-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang resolved HDFS-12096. Resolution: Duplicate > Ozone: Bucket versioning design document > > > Key: HDFS-12096 > URL: https://issues.apache.org/jira/browse/HDFS-12096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: Ozone Bucket Versioning v1.pdf > > > This JIRA is opened for the discussion of the bucket versioning design. > The bucket versioning is the ability to hold multiple versions objects of a > key in a bucket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12096) Ozone: Bucket versioning design document
[ https://issues.apache.org/jira/browse/HDFS-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076473#comment-16076473 ] Weiwei Yang edited comment on HDFS-12096 at 7/6/17 1:13 PM: Uploaded an initial design doc, please feel free to comment and advise. Appreciate! was (Author: cheersyang): Uploaded an init design doc, please feel free to comment and advise. Appreciate! > Ozone: Bucket versioning design document > > > Key: HDFS-12096 > URL: https://issues.apache.org/jira/browse/HDFS-12096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: Ozone Bucket Versioning v1.pdf > > > This JIRA is opened for the discussion of the bucket versioning design. > The bucket versioning is the ability to hold multiple versions objects of a > key in a bucket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12096) Ozone: Bucket versioning design document
[ https://issues.apache.org/jira/browse/HDFS-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076473#comment-16076473 ] Weiwei Yang commented on HDFS-12096: Uploaded an init design doc, please feel free to comment and advise. Appreciate! > Ozone: Bucket versioning design document > > > Key: HDFS-12096 > URL: https://issues.apache.org/jira/browse/HDFS-12096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: Ozone Bucket Versioning v1.pdf > > > This JIRA is opened for the discussion of the bucket versioning design. > The bucket versioning is the ability to hold multiple versions objects of a > key in a bucket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[ https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12069: --- Attachment: HDFS-12069-HDFS-7240.005.patch Oh, right, it now checks the completeness of default site xml, thanks [~msingh] to point this out [~msingh]. I just fixed this in v5 patch. > Ozone: Create a general abstraction for metadata store > -- > > Key: HDFS-12069 > URL: https://issues.apache.org/jira/browse/HDFS-12069 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12069-HDFS-7240.001.patch, > HDFS-12069-HDFS-7240.002.patch, HDFS-12069-HDFS-7240.003.patch, > HDFS-12069-HDFS-7240.004.patch, HDFS-12069-HDFS-7240.005.patch > > > Create a general abstraction for metadata store so that we can plug other key > value store to host ozone metadata. Currently only levelDB is implemented, we > want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11696) Fix warnings from Spotbugs in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11696: - Attachment: HDFS-11696.008.patch bg. Is there a way to address the spotbugs issues without changing the option parsing behavior? I think we can just add some new entries in findbug exclude file to ignore these warnings. Attach the new patch. [~andrew.wang], please have a check. Thanks. > Fix warnings from Spotbugs in hadoop-hdfs > - > > Key: HDFS-11696 > URL: https://issues.apache.org/jira/browse/HDFS-11696 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: findbugsHtml.html, HADOOP-14337.001.patch, > HADOOP-14337.002.patch, HADOOP-14337.003.patch, HDFS-11696.004.patch, > HDFS-11696.005.patch, HDFS-11696.006.patch, HDFS-11696.007.patch, > HDFS-11696.008.patch > > > There are totally 12 findbugs issues generated after switching from Findbugs > to Spotbugs across the project in HADOOP-14316. This JIRA focus on cleaning > up the part of warnings under scope of HDFS(mainly contained in hadoop-hdfs > and hadoop-hdfs-client). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11696) Fix warnings from Spotbugs in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076422#comment-16076422 ] Yiqun Lin edited comment on HDFS-11696 at 7/6/17 12:24 PM: --- bq. Is there a way to address the spotbugs issues without changing the option parsing behavior? I think we can just add some new entries in findbug exclude file to ignore these warnings. Attach the new patch. [~andrew.wang], please have a check. Thanks. was (Author: linyiqun): bg. Is there a way to address the spotbugs issues without changing the option parsing behavior? I think we can just add some new entries in findbug exclude file to ignore these warnings. Attach the new patch. [~andrew.wang], please have a check. Thanks. > Fix warnings from Spotbugs in hadoop-hdfs > - > > Key: HDFS-11696 > URL: https://issues.apache.org/jira/browse/HDFS-11696 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: findbugsHtml.html, HADOOP-14337.001.patch, > HADOOP-14337.002.patch, HADOOP-14337.003.patch, HDFS-11696.004.patch, > HDFS-11696.005.patch, HDFS-11696.006.patch, HDFS-11696.007.patch, > HDFS-11696.008.patch > > > There are totally 12 findbugs issues generated after switching from Findbugs > to Spotbugs across the project in HADOOP-14316. This JIRA focus on cleaning > up the part of warnings under scope of HDFS(mainly contained in hadoop-hdfs > and hadoop-hdfs-client). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11696) Fix warnings from Spotbugs in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11696: - Status: Patch Available (was: Reopened) > Fix warnings from Spotbugs in hadoop-hdfs > - > > Key: HDFS-11696 > URL: https://issues.apache.org/jira/browse/HDFS-11696 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: findbugsHtml.html, HADOOP-14337.001.patch, > HADOOP-14337.002.patch, HADOOP-14337.003.patch, HDFS-11696.004.patch, > HDFS-11696.005.patch, HDFS-11696.006.patch, HDFS-11696.007.patch, > HDFS-11696.008.patch > > > There are totally 12 findbugs issues generated after switching from Findbugs > to Spotbugs across the project in HADOOP-14316. This JIRA focus on cleaning > up the part of warnings under scope of HDFS(mainly contained in hadoop-hdfs > and hadoop-hdfs-client). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[ https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076408#comment-16076408 ] Mukul Kumar Singh commented on HDFS-12069: -- Hi [~cheersyang], the jenkins failure for TestOzoneConfigurationFields seems to be related to this change. Could you please take a look, Thanks > Ozone: Create a general abstraction for metadata store > -- > > Key: HDFS-12069 > URL: https://issues.apache.org/jira/browse/HDFS-12069 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12069-HDFS-7240.001.patch, > HDFS-12069-HDFS-7240.002.patch, HDFS-12069-HDFS-7240.003.patch, > HDFS-12069-HDFS-7240.004.patch > > > Create a general abstraction for metadata store so that we can plug other key > value store to host ozone metadata. Currently only levelDB is implemented, we > want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[
https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076397#comment-16076397
]
Hadoop QA commented on HDFS-12069:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new + 9
unchanged - 1 fixed = 9 total (was 10) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.ozone.TestOzoneConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12069 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875897/HDFS-12069-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2d249efe89b9 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 8efb1fa |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20173/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20173/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20173/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Create a general abstraction for metadata store
> --
>
> Key: HDFS-12069
> URL: https://issues.apache.org/jira/browse/HDFS-12069
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>
[jira] [Updated] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[ https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12084: Summary: Scheduled Count will not decrement when file is deleted before all IBR's received (was: Scheduled will not decrement when file is deleted before all IBR's received) > Scheduled Count will not decrement when file is deleted before all IBR's > received > - > > Key: HDFS-12084 > URL: https://issues.apache.org/jira/browse/HDFS-12084 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch > > > When small files creation && deletion happens so frequently and DN's did not > report blocks to NN before deletion, then scheduled count will keep on > increment and which will not deleted as blocks are deleted. > *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be > more as so many operations. > when batchIBR enabled with committed allowed=1 this will be observed more. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12084) Scheduled will not decrement when file is deleted before all IBR's received
[ https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12084: Attachment: HDFS-12084-002.patch Updated the patch to address the checkystyle issues. > Scheduled will not decrement when file is deleted before all IBR's received > --- > > Key: HDFS-12084 > URL: https://issues.apache.org/jira/browse/HDFS-12084 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch > > > When small files creation && deletion happens so frequently and DN's did not > report blocks to NN before deletion, then scheduled count will keep on > increment and which will not deleted as blocks are deleted. > *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be > more as so many operations. > when batchIBR enabled with committed allowed=1 this will be observed more. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11965) [SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr
[
https://issues.apache.org/jira/browse/HDFS-11965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076345#comment-16076345
]
Hadoop QA commented on HDFS-11965:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
41s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}117m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11965 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875875/HDFS-11965-HDFS-10285.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 93d20d5a683f 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10285 / 258fdc6 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20171/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20171/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20171/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [SPS]: Should give chance to satisfy the low redundant blocks before removing
> the xattr
> ---
>
> Key: HDFS-11965
> URL: https://issues.apache.org/jira/browse/HDFS-11965
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attac
[jira] [Commented] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[ https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076299#comment-16076299 ] Weiwei Yang commented on HDFS-12069: Hi [~msingh] Thanks for helping to review this. bq. Can the config changes in OzoneConfigKeys.java be move to SCMConfig keys or another common location ? The properties I intend to add are not just for SCM, so it is inappropriate to move them to SCMConfigKeys. If cblock classes have no access to ozone package and you think it's necessary to share this to cblocks, we need to think another way around, I would love to know your suggestion. bq. LevelDBStore.java:81, Can we log the DB Path and the options if open fails ? I just removed that line, I don't think it will ever get a null value. JniDBFactory.factory.open always returns a DB instance unless there is some exception creating it. So either it throws exception, or it is a non-null value. So I removed that line. bq. but I feel that if there is only one operation inside the batch then the operation can be called directly without creating a batch. The batch operation is only used when there is a batch of atomic operations, so we don't really expect a single operation here. Thank you > Ozone: Create a general abstraction for metadata store > -- > > Key: HDFS-12069 > URL: https://issues.apache.org/jira/browse/HDFS-12069 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12069-HDFS-7240.001.patch, > HDFS-12069-HDFS-7240.002.patch, HDFS-12069-HDFS-7240.003.patch, > HDFS-12069-HDFS-7240.004.patch > > > Create a general abstraction for metadata store so that we can plug other key > value store to host ozone metadata. Currently only levelDB is implemented, we > want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12069) Ozone: Create a general abstraction for metadata store
[ https://issues.apache.org/jira/browse/HDFS-12069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12069: --- Attachment: HDFS-12069-HDFS-7240.004.patch > Ozone: Create a general abstraction for metadata store > -- > > Key: HDFS-12069 > URL: https://issues.apache.org/jira/browse/HDFS-12069 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12069-HDFS-7240.001.patch, > HDFS-12069-HDFS-7240.002.patch, HDFS-12069-HDFS-7240.003.patch, > HDFS-12069-HDFS-7240.004.patch > > > Create a general abstraction for metadata store so that we can plug other key > value store to host ozone metadata. Currently only levelDB is implemented, we > want to support RocksDB as it provides more production ready features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11922) Ozone: KSM: Garbage collect deleted blocks
[ https://issues.apache.org/jira/browse/HDFS-11922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076288#comment-16076288 ] Weiwei Yang commented on HDFS-11922: There seems to have more and more desires to have this feature ready. I just think about this again. What could do might be add a few states to keys, ||State||Description|| |CREATING|A key is under creation, data not fully flushed to disk| |CREATED|A key is created, it is visible in namespace and ready for read/write| |MISSINGREPLICA|One or more replica is missing, but it still has at least 1 replica live available| |CORRUPTED|All replicas are lost, the key is unable to read.| |STALE|A key is no longer valid, it needs to be purged from namespace, its data needs to be removed from datanodes.| Deletes a key can be implemented like following # KSM looks up the key from DB to gets the container where this key is stored # KSM looks up SCM to get the pipeline where the container is replicated to # KSM connects to each container server, updates the container metadata to mark the key is "stale" (the actual delete is done in an async thread on each datanode) # KSM removes the key from its namespace (the key is no longer visible for clients) # Container server on each datanode scans the "stale" keys in each container, and deletes the files at background. This can be done either in certain interval, or optimally by listening to the key update events. We just need to find a good place to store key - keystate or keystate - keys mapping. This will help the putkey as well, to support the commit key phase. Thoughts? > Ozone: KSM: Garbage collect deleted blocks > -- > > Key: HDFS-11922 > URL: https://issues.apache.org/jira/browse/HDFS-11922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Priority: Critical > > We need to garbage collect deleted blocks from the Datanodes. There are two > cases where we will have orphaned blocks. One is like the classical HDFS, > where someone deletes a key and we need to delete the corresponding blocks. > Another case, is when someone overwrites a key -- an overwrite can be treated > as a delete and a new put -- that means that older blocks need to be GC-ed at > some point of time. > Couple of JIRAs has discussed this in one form or another -- so consolidating > all those discussions in this JIRA. > HDFS-11796 -- needs to fix this issue for some tests to pass > HDFS-11780 -- changed the old overwriting behavior to not supporting this > feature for time being. > HDFS-11920 - Once again runs into this issue when user tries to put an > existing key. > HDFS-11781 - delete key API in KSM only deletes the metadata -- and relies on > GC for Datanodes. > When we solve this issue, we should also consider 2 more aspects. > One, we support versioning in the buckets, tracking which blocks are really > orphaned is something that KSM will do. So delete and overwrite at some point > needs to decide how to handle versioning of buckets. > Two, If a key exists in a closed container, then it is immutable, hence the > strategy of removing the key might be more complex than just talking to an > open container. > cc : [~xyao], [~cheersyang], [~vagarychen], [~msingh], [~yuanbo], > [~szetszwo], [~nandakumar131] > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12094) Log torrent when none isa-l EC is used.
[
https://issues.apache.org/jira/browse/HDFS-12094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076282#comment-16076282
]
Hadoop QA commented on HDFS-12094:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12094 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875883/HDFS-12094.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 01888c7ad10a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 946dd25 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20172/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20172/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Log torrent when none isa-l EC is used.
> ---
>
> Key: HDFS-12094
> URL: https://issues.apache.org/jira/browse/HDFS-12094
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-beta1
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Minor
> Attachments: HDFS-12094.001.patch
>
>
> My hadoop is built without isa-l support, after the EC policy is enabled,
> whenever I get/put directory which contains many files, the log of
> warnings(see below) spam on the screen!
> This is unfriendly and depress the performance. Since we come to the beta
> version now,
> these
[jira] [Updated] (HDFS-12080) Ozone: Fix UT failure in TestOzoneConfigurationFields
[
https://issues.apache.org/jira/browse/HDFS-12080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12080:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
Just committed to the feature branch, thanks for the review [~vagarychen],
[~anu]!
> Ozone: Fix UT failure in TestOzoneConfigurationFields
> -
>
> Key: HDFS-12080
> URL: https://issues.apache.org/jira/browse/HDFS-12080
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Mukul Kumar Singh
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-12080-HDFS-7240.001.patch
>
>
> HDFS-12023 added a test case {{TestOzoneConfigurationFields}} to make sure
> ozone configuration properties is fully documented in ozone-default.xml. This
> is currently failing because
> 1. ozone-default.xml has 1 property not used anywhere
> {code}
> ozone.scm.internal.bind.host
> {code}
> 2. Some cblock properties are missing in ozone-default.xml
> {code}
> dfs.cblock.scm.ipaddress
> dfs.cblock.scm.port
> dfs.cblock.jscsi-address
> dfs.cblock.service.rpc-bind-host
> dfs.cblock.jscsi.rpc-bind-host
> {code}
> this needs to be fixed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12080) Ozone: Fix UT failure in TestOzoneConfigurationFields
[
https://issues.apache.org/jira/browse/HDFS-12080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076243#comment-16076243
]
Weiwei Yang commented on HDFS-12080:
None of those UT failures seem to be related to this patch, +1. I am going to
commit this shortly.
> Ozone: Fix UT failure in TestOzoneConfigurationFields
> -
>
> Key: HDFS-12080
> URL: https://issues.apache.org/jira/browse/HDFS-12080
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Mukul Kumar Singh
>Priority: Minor
> Attachments: HDFS-12080-HDFS-7240.001.patch
>
>
> HDFS-12023 added a test case {{TestOzoneConfigurationFields}} to make sure
> ozone configuration properties is fully documented in ozone-default.xml. This
> is currently failing because
> 1. ozone-default.xml has 1 property not used anywhere
> {code}
> ozone.scm.internal.bind.host
> {code}
> 2. Some cblock properties are missing in ozone-default.xml
> {code}
> dfs.cblock.scm.ipaddress
> dfs.cblock.scm.port
> dfs.cblock.jscsi-address
> dfs.cblock.service.rpc-bind-host
> dfs.cblock.jscsi.rpc-bind-host
> {code}
> this needs to be fixed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12096) Ozone: Bucket versioning design document
[ https://issues.apache.org/jira/browse/HDFS-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12096 started by Weiwei Yang. -- > Ozone: Bucket versioning design document > > > Key: HDFS-12096 > URL: https://issues.apache.org/jira/browse/HDFS-12096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: Ozone Bucket Versioning v1.pdf > > > This JIRA is opened for the discussion of the bucket versioning design. > The bucket versioning is the ability to hold multiple versions objects of a > key in a bucket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12096) Ozone: Bucket versioning design document
[ https://issues.apache.org/jira/browse/HDFS-12096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12096: --- Attachment: Ozone Bucket Versioning v1.pdf > Ozone: Bucket versioning design document > > > Key: HDFS-12096 > URL: https://issues.apache.org/jira/browse/HDFS-12096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: Ozone Bucket Versioning v1.pdf > > > This JIRA is opened for the discussion of the bucket versioning design. > The bucket versioning is the ability to hold multiple versions objects of a > key in a bucket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12096) Ozone: Bucket versioning design document
Weiwei Yang created HDFS-12096: -- Summary: Ozone: Bucket versioning design document Key: HDFS-12096 URL: https://issues.apache.org/jira/browse/HDFS-12096 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Reporter: Weiwei Yang Assignee: Weiwei Yang This JIRA is opened for the discussion of the bucket versioning design. The bucket versioning is the ability to hold multiple versions objects of a key in a bucket. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Description: Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in
{{OzoneClient}}
> Ozone: OzoneClient: Add list calls
> --
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{OzoneClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12060) Ozone: OzoneClient: Add list calls
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12060:
--
Environment: (was: Support for {{listVolumes}}, {{listBuckets}},
{{listKeys}} in {{OzoneClient}})
> Ozone: OzoneClient: Add list calls
> --
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12095) Block Storage: Fix cblock unit test failures in TestLocalBlockCache, TestCBlockReadWrite and TestBufferManager
Mukul Kumar Singh created HDFS-12095:
Summary: Block Storage: Fix cblock unit test failures in
TestLocalBlockCache, TestCBlockReadWrite and TestBufferManager
Key: HDFS-12095
URL: https://issues.apache.org/jira/browse/HDFS-12095
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Fix For: HDFS-7240
Cblock test TestLocalBlockCache, TestCBlockReadWrite and TestBufferManager are
failing quite frequently.
Please find the analysis for each of these failure as following
*TestCBlockReadWrite* - Times out in the Jenkins run however it passes
successfully in local runs
*TestCBlockReadWrite* - This test fails because of invalid assertion, as there
can be time triggered flushes which are triggered and got finished, hence
number of completed flushes need not be '1'
{code}
java.lang.AssertionError: expected:<1> but was:<13>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.cblock.TestCBlockReadWrite.testContainerWrites(TestCBlockReadWrite.java:245)
{code}
*TestBufferManager* - This assertion failed because there might be flushes
which are in progress. A sleep need to be introduced to ensure that the entire
dirty buffer is flused
{code}
java.lang.AssertionError: expected:<16384> but was:<15976>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.cblock.TestBufferManager.testMultipleBuffersFlush(TestBufferManager.java:346)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12094) Log torrent when none isa-l EC is used.
[
https://issues.apache.org/jira/browse/HDFS-12094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDFS-12094:
Status: Patch Available (was: Open)
> Log torrent when none isa-l EC is used.
> ---
>
> Key: HDFS-12094
> URL: https://issues.apache.org/jira/browse/HDFS-12094
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-beta1
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Minor
> Attachments: HDFS-12094.001.patch
>
>
> My hadoop is built without isa-l support, after the EC policy is enabled,
> whenever I get/put directory which contains many files, the log of
> warnings(see below) spam on the screen!
> This is unfriendly and depress the performance. Since we come to the beta
> version now,
> these logs should be deprecated and a one-time warning log instead of
> exception may be much better.
> {quote}
> 2017-07-06 15:42:41,398 WARN erasurecode.CodecUtil: Failed to create raw
> erasure encoder xor_native, fallback to next codec if possible
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder
> at
> org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.createEncoder(NativeXORRawErasureCoderFactory.java:35)
> at
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:177)
> at
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:129)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.(DFSStripedOutputStream.java:302)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:309)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1216)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1133)
> ...
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
>
> Caused by: java.lang.RuntimeException: libhadoop was built without ISA-L
> support
> at
> org.apache.hadoop.io.erasurecode.ErasureCodeNative.checkNativeCodeLoaded(ErasureCodeNative.java:69)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.(NativeXORRawDecoder.java:33)
> ... 25 more
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12094) Log torrent when none isa-l EC is used.
[
https://issues.apache.org/jira/browse/HDFS-12094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDFS-12094:
Attachment: HDFS-12094.001.patch
> Log torrent when none isa-l EC is used.
> ---
>
> Key: HDFS-12094
> URL: https://issues.apache.org/jira/browse/HDFS-12094
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-beta1
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Minor
> Attachments: HDFS-12094.001.patch
>
>
> My hadoop is built without isa-l support, after the EC policy is enabled,
> whenever I get/put directory which contains many files, the log of
> warnings(see below) spam on the screen!
> This is unfriendly and depress the performance. Since we come to the beta
> version now,
> these logs should be deprecated and a one-time warning log instead of
> exception may be much better.
> {quote}
> 2017-07-06 15:42:41,398 WARN erasurecode.CodecUtil: Failed to create raw
> erasure encoder xor_native, fallback to next codec if possible
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder
> at
> org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.createEncoder(NativeXORRawErasureCoderFactory.java:35)
> at
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:177)
> at
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:129)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.(DFSStripedOutputStream.java:302)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:309)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1216)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1133)
> ...
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
>
> Caused by: java.lang.RuntimeException: libhadoop was built without ISA-L
> support
> at
> org.apache.hadoop.io.erasurecode.ErasureCodeNative.checkNativeCodeLoaded(ErasureCodeNative.java:69)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.(NativeXORRawDecoder.java:33)
> ... 25 more
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12094) Log torrent when none isa-l EC is used.
LiXin Ge created HDFS-12094:
---
Summary: Log torrent when none isa-l EC is used.
Key: HDFS-12094
URL: https://issues.apache.org/jira/browse/HDFS-12094
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding
Affects Versions: 3.0.0-beta1
Reporter: LiXin Ge
Assignee: LiXin Ge
Priority: Minor
My hadoop is built without isa-l support, after the EC policy is enabled,
whenever I get/put directory which contains many files, the log of warnings(see
below) spam on the screen!
This is unfriendly and depress the performance. Since we come to the beta
version now,
these logs should be deprecated and a one-time warning log instead of exception
may be much better.
{quote}
2017-07-06 15:42:41,398 WARN erasurecode.CodecUtil: Failed to create raw
erasure encoder xor_native, fallback to next codec if possible
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawEncoder
at
org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.createEncoder(NativeXORRawErasureCoderFactory.java:35)
at
org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:177)
at
org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:129)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.(DFSStripedOutputStream.java:302)
at
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:309)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1216)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1133)
...
at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
Caused by: java.lang.RuntimeException: libhadoop was built without ISA-L support
at
org.apache.hadoop.io.erasurecode.ErasureCodeNative.checkNativeCodeLoaded(ErasureCodeNative.java:69)
at
org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawDecoder.(NativeXORRawDecoder.java:33)
... 25 more
{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11965) [SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr
[
https://issues.apache.org/jira/browse/HDFS-11965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076158#comment-16076158
]
Surendra Singh Lilhore commented on HDFS-11965:
---
Attached v6 patch..
1. Added FEW_LOW_REDUNDENCY_BLOCKS state. FEW_LOW_REDUNDENCY_BLOCKS case will
come only if all the reported blocks are satisfied and some block are low
redundant.
2. Removed retry count. We will handle in another jira.
bq. If the status is FEW_LOW_REDUNDENCY_BLOCKS, you could call
this.storageMovementsMonitor.add(blockCollectionID, false);
This is not required I think. If block is low redundant, mean nothing to move
so we will add it back to {{storageMovementNeeded}}
Please review...
> [SPS]: Should give chance to satisfy the low redundant blocks before removing
> the xattr
> ---
>
> Key: HDFS-11965
> URL: https://issues.apache.org/jira/browse/HDFS-11965
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11965-HDFS-10285.001.patch,
> HDFS-11965-HDFS-10285.002.patch, HDFS-11965-HDFS-10285.003.patch,
> HDFS-11965-HDFS-10285.004.patch, HDFS-11965-HDFS-10285.005.patch,
> HDFS-11965-HDFS-10285.006.patch
>
>
> The test case is failing because all the required replicas are not moved in
> expected storage. This is happened because of delay in datanode registration
> after cluster restart.
> Scenario :
> 1. Start cluster with 3 DataNodes.
> 2. Create file and set storage policy to WARM.
> 3. Restart the cluster.
> 4. Now Namenode and two DataNodes started first and got registered with
> NameNode. (one datanode not yet registered)
> 5. SPS scheduled block movement based on available DataNodes (It will move
> one replica in ARCHIVE based on policy).
> 6. Block movement also success and Xattr removed from the file because this
> condition is true {{itemInfo.isAllBlockLocsAttemptedToSatisfy()}}.
> {code}
> if (itemInfo != null
> && !itemInfo.isAllBlockLocsAttemptedToSatisfy()) {
> blockStorageMovementNeeded
> .add(storageMovementAttemptedResult.getTrackId());
>
> ..
> } else {
>
> ..
> this.sps.postBlkStorageMovementCleanup(
> storageMovementAttemptedResult.getTrackId());
> }
> {code}
> 7. Now third DN registered with namenode and its reported one more DISK
> replica. Now Namenode has two DISK and one ARCHIVE replica.
> In test case we have condition to check the number of DISK replica..
> {code} DFSTestUtil.waitExpectedStorageType(testFileName, StorageType.DISK, 1,
> timeout, fs);{code}
> This condition never became true and test case will be timed out.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11965) [SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr
[
https://issues.apache.org/jira/browse/HDFS-11965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-11965:
--
Attachment: HDFS-11965-HDFS-10285.006.patch
> [SPS]: Should give chance to satisfy the low redundant blocks before removing
> the xattr
> ---
>
> Key: HDFS-11965
> URL: https://issues.apache.org/jira/browse/HDFS-11965
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11965-HDFS-10285.001.patch,
> HDFS-11965-HDFS-10285.002.patch, HDFS-11965-HDFS-10285.003.patch,
> HDFS-11965-HDFS-10285.004.patch, HDFS-11965-HDFS-10285.005.patch,
> HDFS-11965-HDFS-10285.006.patch
>
>
> The test case is failing because all the required replicas are not moved in
> expected storage. This is happened because of delay in datanode registration
> after cluster restart.
> Scenario :
> 1. Start cluster with 3 DataNodes.
> 2. Create file and set storage policy to WARM.
> 3. Restart the cluster.
> 4. Now Namenode and two DataNodes started first and got registered with
> NameNode. (one datanode not yet registered)
> 5. SPS scheduled block movement based on available DataNodes (It will move
> one replica in ARCHIVE based on policy).
> 6. Block movement also success and Xattr removed from the file because this
> condition is true {{itemInfo.isAllBlockLocsAttemptedToSatisfy()}}.
> {code}
> if (itemInfo != null
> && !itemInfo.isAllBlockLocsAttemptedToSatisfy()) {
> blockStorageMovementNeeded
> .add(storageMovementAttemptedResult.getTrackId());
>
> ..
> } else {
>
> ..
> this.sps.postBlkStorageMovementCleanup(
> storageMovementAttemptedResult.getTrackId());
> }
> {code}
> 7. Now third DN registered with namenode and its reported one more DISK
> replica. Now Namenode has two DISK and one ARCHIVE replica.
> In test case we have condition to check the number of DISK replica..
> {code} DFSTestUtil.waitExpectedStorageType(testFileName, StorageType.DISK, 1,
> timeout, fs);{code}
> This condition never became true and test case will be timed out.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12093) [READ] Share remoteFS between ProvidedReplica instances.
Ewan Higgs created HDFS-12093:
-
Summary: [READ] Share remoteFS between ProvidedReplica instances.
Key: HDFS-12093
URL: https://issues.apache.org/jira/browse/HDFS-12093
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Ewan Higgs
Then a Datanode comes online using Provided storage, it fills the
{{ReplicaMap}} with the known replicas. With Provided Storage, this includes
{{ProvidedReplica}} instances. Each of these objects, in their constructor,
will construct an FileSystem using the Service Provider. This can result in
contacting the remote file system and checking that the credentials are correct
and that the data is there. For large systems this is a prohibitively expensive
operation to perform per replica.
Instead, the {{ProvidedVolumeImpl}} should own the reference to the
{{remoteFS}} and should share it with the {{ProvidedReplica}} objects on their
creation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12080) Ozone: Fix UT failure in TestOzoneConfigurationFields
[
https://issues.apache.org/jira/browse/HDFS-12080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076110#comment-16076110
]
Hadoop QA commented on HDFS-12080:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis
|
| | hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| Timed out junit tests | org.apache.hadoop.ozone.scm.node.TestNodeManager |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12080 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875772/HDFS-12080-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 2442b485f3fe 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 4d3993b |
| Default Java | 1.8.0_131 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20170/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20170/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20170/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Fix UT failure in TestOzoneConfigurationFields
> -
>
> Key: HDFS-12080
> URL: https://issues.apache.org/jira/browse/HDFS-12080
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter:
[jira] [Commented] (HDFS-12057) Ozone: OzoneClient: Implementation of OzoneClient
[
https://issues.apache.org/jira/browse/HDFS-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076089#comment-16076089
]
Nandakumar commented on HDFS-12057:
---
Thanks [~anu] for the review and commit.
bq. This is also the first time we are using UGI, since OzoneFS might use it, I
am ok with keeping it. Let us revisit that choice later.
Sure.
> Ozone: OzoneClient: Implementation of OzoneClient
> -
>
> Key: HDFS-12057
> URL: https://issues.apache.org/jira/browse/HDFS-12057
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12057-HDFS-7240.000.patch,
> HDFS-12057-HDFS-7240.001.patch
>
>
> OzoneClient brings in support for client facing Java APIs in Ozone to access
> ObjectStore. It should supports all the calls which are supported by
> {{OzoneHandler}} through REST.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11975) Provide a system-default EC policy
[
https://issues.apache.org/jira/browse/HDFS-11975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076080#comment-16076080
]
Hadoop QA commented on HDFS-11975:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
7s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
30s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Load of known null value in
org.apache.hadoop.hdfs.DFSClient.setErasureCodingPolicy(String, String) At
DFSClient.java:in
org.apache.hadoop.hdfs.DFSClient.setErasureCodingPolicy(String, String) At
DFSClient.java:[line 2649] |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11975 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875853/HDFS-11975-006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 7d46336aa1f3 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /
[jira] [Commented] (HDFS-9388) Refactor decommission related code to support maintenance state for datanodes
[
https://issues.apache.org/jira/browse/HDFS-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076060#comment-16076060
]
Hadoop QA commented on HDFS-9388:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
48s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 167 unchanged - 12 fixed = 167 total (was 179) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-9388 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875855/HDFS-9388.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 3b083973d746 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 946dd25 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20169/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20169/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20169/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20169/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor decommission related code to support maintenance state for datanodes
> ---
