[jira] [Commented] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144784#comment-16144784
]
Hadoop QA commented on HDFS-12354:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 56s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
30s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12354 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884176/HDFS-12354-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux cea89e61acce 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b06f4f6 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-12191) Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done
[
https://issues.apache.org/jira/browse/HDFS-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144762#comment-16144762
]
Yongjun Zhang commented on HDFS-12191:
--
Uploaded rev5 to correct the config name in javadoc of the test code:
{quote}
TestSnapshotDiffReport:732 still refers to the the old property name
{quote}
> Provide option to not capture the accessTime change of a file to snapshot if
> no other modification has been done
>
>
> Key: HDFS-12191
> URL: https://issues.apache.org/jira/browse/HDFS-12191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 3.0.0-beta1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-12191.001.patch, HDFS-12191.002.patch,
> HDFS-12191.003.patch, HDFS-12191.004.patch, HDFS-12191.005.patch
>
>
> Currently, if the accessTime of a file changed before a snapshot is taken,
> this accessTime will be captured in the snapshot, even if there is no other
> modifications made to this file.
> Because of this, when we calculate snapshotDiff, more work need to be done
> for this file, e,g,, metadataEquals method will be called, even if there is
> no modification is made (thus not recorded to snapshotDiff). This can cause
> snapshotDiff to slow down quite a lot when there are a lot of files to be
> examined.
> This jira is to provide an option to skip capturing accessTime only change to
> snapshot. Thus snapshotDiff can be done faster.
> When accessTime of a file changed, if there is other modification to the
> file, the access time will still be captured in snapshot.
> Sometimes we want accessTime be captured to snapshot, such that when
> restoring from the snapshot, we know the accessTime of this snapshot. So this
> new feature is optional, and is controlled by a config property.
> Worth to mention is, how accurately the acessTime is captured is dependent on
> the following config that has default value of 1 hour, which means new access
> within an hour of previous access will not be captured.
> {code}
> public static final String DFS_NAMENODE_ACCESSTIME_PRECISION_KEY =
>
> HdfsClientConfigKeys.DeprecatedKeys.DFS_NAMENODE_ACCESSTIME_PRECISION_KEY;
> public static final longDFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT =
> 360;
> {code}
> .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12191) Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done
[
https://issues.apache.org/jira/browse/HDFS-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-12191:
-
Attachment: HDFS-12191.005.patch
> Provide option to not capture the accessTime change of a file to snapshot if
> no other modification has been done
>
>
> Key: HDFS-12191
> URL: https://issues.apache.org/jira/browse/HDFS-12191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 3.0.0-beta1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-12191.001.patch, HDFS-12191.002.patch,
> HDFS-12191.003.patch, HDFS-12191.004.patch, HDFS-12191.005.patch
>
>
> Currently, if the accessTime of a file changed before a snapshot is taken,
> this accessTime will be captured in the snapshot, even if there is no other
> modifications made to this file.
> Because of this, when we calculate snapshotDiff, more work need to be done
> for this file, e,g,, metadataEquals method will be called, even if there is
> no modification is made (thus not recorded to snapshotDiff). This can cause
> snapshotDiff to slow down quite a lot when there are a lot of files to be
> examined.
> This jira is to provide an option to skip capturing accessTime only change to
> snapshot. Thus snapshotDiff can be done faster.
> When accessTime of a file changed, if there is other modification to the
> file, the access time will still be captured in snapshot.
> Sometimes we want accessTime be captured to snapshot, such that when
> restoring from the snapshot, we know the accessTime of this snapshot. So this
> new feature is optional, and is controlled by a config property.
> Worth to mention is, how accurately the acessTime is captured is dependent on
> the following config that has default value of 1 hour, which means new access
> within an hour of previous access will not be captured.
> {code}
> public static final String DFS_NAMENODE_ACCESSTIME_PRECISION_KEY =
>
> HdfsClientConfigKeys.DeprecatedKeys.DFS_NAMENODE_ACCESSTIME_PRECISION_KEY;
> public static final longDFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT =
> 360;
> {code}
> .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12191) Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done
[
https://issues.apache.org/jira/browse/HDFS-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144748#comment-16144748
]
Yongjun Zhang commented on HDFS-12191:
--
Thanks [~manojg].
{quote}
There are quite a few thread sleep for 3sec. You can always call the setTimes()
with a constructed time instead of the actual waiting.
{quote}
The sleep is there because the new operation takes the real time. If we use
setTimes() to set the time to a later time, the real operation that records
real time could move the time back which would be wrong.
> Provide option to not capture the accessTime change of a file to snapshot if
> no other modification has been done
>
>
> Key: HDFS-12191
> URL: https://issues.apache.org/jira/browse/HDFS-12191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 3.0.0-beta1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-12191.001.patch, HDFS-12191.002.patch,
> HDFS-12191.003.patch, HDFS-12191.004.patch
>
>
> Currently, if the accessTime of a file changed before a snapshot is taken,
> this accessTime will be captured in the snapshot, even if there is no other
> modifications made to this file.
> Because of this, when we calculate snapshotDiff, more work need to be done
> for this file, e,g,, metadataEquals method will be called, even if there is
> no modification is made (thus not recorded to snapshotDiff). This can cause
> snapshotDiff to slow down quite a lot when there are a lot of files to be
> examined.
> This jira is to provide an option to skip capturing accessTime only change to
> snapshot. Thus snapshotDiff can be done faster.
> When accessTime of a file changed, if there is other modification to the
> file, the access time will still be captured in snapshot.
> Sometimes we want accessTime be captured to snapshot, such that when
> restoring from the snapshot, we know the accessTime of this snapshot. So this
> new feature is optional, and is controlled by a config property.
> Worth to mention is, how accurately the acessTime is captured is dependent on
> the following config that has default value of 1 hour, which means new access
> within an hour of previous access will not be captured.
> {code}
> public static final String DFS_NAMENODE_ACCESSTIME_PRECISION_KEY =
>
> HdfsClientConfigKeys.DeprecatedKeys.DFS_NAMENODE_ACCESSTIME_PRECISION_KEY;
> public static final longDFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT =
> 360;
> {code}
> .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12354:
-
Attachment: HDFS-12354-HDFS-7240.002.patch
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Attachments: HDFS-12354-HDFS-7240.001.patch,
> HDFS-12354-HDFS-7240.002.patch
>
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12354:
-
Attachment: (was: HDFS-12354-HDFS-7240.002.patch)
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Attachments: HDFS-12354-HDFS-7240.001.patch,
> HDFS-12354-HDFS-7240.002.patch
>
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12369) Edit log corruption due to hard lease recovery of not-closed file
[
https://issues.apache.org/jira/browse/HDFS-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12369:
-
Attachment: HDFS-12369.01.patch
After further looking I think the different exception is due to block state
being different. From the logs, the close op is due to block completed, and
hence {{finalizeINodeFileUnderConstruction}} getting executed.
The fix then, is along HDFS-6257 and HDFS-7707's lines, to prevent the wrongly
{{CloseOp}}. Patch 1 tries to do that.
I'm not entirely familiar with this area, will consult [~yzhangal] offline. Any
comments welcome!
> Edit log corruption due to hard lease recovery of not-closed file
> -
>
> Key: HDFS-12369
> URL: https://issues.apache.org/jira/browse/HDFS-12369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12369.01.patch, HDFS-12369.test.patch
>
>
> HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations
> of client operations.
> Recently, we have observed NN not able to start with the following exception:
> {noformat}
> 2017-08-17 14:32:18,418 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.io.FileNotFoundException: File does not exist:
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
> {noformat}
> Quoting a nicely analysed edits:
> {quote}
> In the edits logged about 1 hour later, we see this failing OP_CLOSE. The
> sequence in the edits shows the file going through:
> OPEN
> ADD_BLOCK
> CLOSE
> ADD_BLOCK # perhaps this was an append
> DELETE
> (about 1 hour later) CLOSE
> It is interesting that there was no CLOSE logged before the delete.
> {quote}
> Grepping that file name, it turns out the close was triggered by
> {{LeaseManager}}, when the lease reaches hard limit.
> {noformat}
> 2017-08-16 15:05:45,927 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> Recovering [Lease. Holder: DFSClient_NONMAPREDUCE_-1997177597_28, pending
> creates: 75],
> src=/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> 2017-08-16 15:05:45,927 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M closed.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12369) Edit log corruption due to hard lease recovery of not-closed file
[
https://issues.apache.org/jira/browse/HDFS-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12369:
-
Status: Patch Available (was: Open)
> Edit log corruption due to hard lease recovery of not-closed file
> -
>
> Key: HDFS-12369
> URL: https://issues.apache.org/jira/browse/HDFS-12369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12369.01.patch, HDFS-12369.test.patch
>
>
> HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations
> of client operations.
> Recently, we have observed NN not able to start with the following exception:
> {noformat}
> 2017-08-17 14:32:18,418 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.io.FileNotFoundException: File does not exist:
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
> {noformat}
> Quoting a nicely analysed edits:
> {quote}
> In the edits logged about 1 hour later, we see this failing OP_CLOSE. The
> sequence in the edits shows the file going through:
> OPEN
> ADD_BLOCK
> CLOSE
> ADD_BLOCK # perhaps this was an append
> DELETE
> (about 1 hour later) CLOSE
> It is interesting that there was no CLOSE logged before the delete.
> {quote}
> Grepping that file name, it turns out the close was triggered by
> {{LeaseManager}}, when the lease reaches hard limit.
> {noformat}
> 2017-08-16 15:05:45,927 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> Recovering [Lease. Holder: DFSClient_NONMAPREDUCE_-1997177597_28, pending
> creates: 75],
> src=/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> 2017-08-16 15:05:45,927 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M closed.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12354:
-
Attachment: HDFS-12354-HDFS-7240.002.patch
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Attachments: HDFS-12354-HDFS-7240.001.patch,
> HDFS-12354-HDFS-7240.002.patch
>
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144688#comment-16144688
]
Yiqun Lin commented on HDFS-12354:
--
Thanks for the review, [~cheersyang]!
bq. Can we just maintain prefix and prevKey internally in an implementation
class (if necessary) instead of exposing them via arguments? Want to keep the
method simple to call.
I agree with this. Addressed.
bq. Can we use int type count?
Addressed.
bq. in ozone-default.xml, can we add some more doc here
Addressed.
Attach the updated patch.
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Attachments: HDFS-12354-HDFS-7240.001.patch,
> HDFS-12354-HDFS-7240.002.patch
>
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
[ https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144651#comment-16144651 ] Weiwei Yang commented on HDFS-12235: Hi [~yuanbo] Thanks for uploading your patch, I will review it today. > Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions > --- > > Key: HDFS-12235 > URL: https://issues.apache.org/jira/browse/HDFS-12235 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Yuanbo Liu > Attachments: HDFS-12235-HDFS-7240.001.patch, > HDFS-12235-HDFS-7240.002.patch > > > KSM and SCM interaction for delete key operation, both KSM and SCM stores key > state info in a backlog, KSM needs to scan this log and send block-deletion > command to SCM, once SCM is fully aware of the message, KSM removes the key > completely from namespace. See more from the design doc under HDFS-11922, > this is task break down 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12363) Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages
[
https://issues.apache.org/jira/browse/HDFS-12363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144642#comment-16144642
]
Hadoop QA commented on HDFS-12363:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
|
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12363 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-12367) Ozone: Too many open files error while running corona
[
https://issues.apache.org/jira/browse/HDFS-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144641#comment-16144641
]
Weiwei Yang commented on HDFS-12367:
Hi [~msingh]
Thanks for picking this up, are you able to reproduce this and any assessment
how this issue happens? Wondering when can this be fixed as I want to leverage
corona for testing. Please let me know, thanks.
> Ozone: Too many open files error while running corona
> -
>
> Key: HDFS-12367
> URL: https://issues.apache.org/jira/browse/HDFS-12367
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, tools
>Reporter: Weiwei Yang
>Assignee: Mukul Kumar Singh
>
> Too many open files error keeps happening to me while using corona, I have
> simply setup a single node cluster and run corona to generate 1000 keys, but
> I keep getting following error
> {noformat}
> ./bin/hdfs corona -numOfThreads 1 -numOfVolumes 1 -numOfBuckets 1 -numOfKeys
> 1000
> 17/08/28 00:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 17/08/28 00:47:42 INFO tools.Corona: Number of Threads: 1
> 17/08/28 00:47:42 INFO tools.Corona: Mode: offline
> 17/08/28 00:47:42 INFO tools.Corona: Number of Volumes: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Buckets per Volume: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Keys per Bucket: 1000.
> 17/08/28 00:47:42 INFO rpc.OzoneRpcClient: Creating Volume: vol-0-05000, with
> wwei as owner and quota set to 1152921504606846976 bytes.
> 17/08/28 00:47:42 INFO tools.Corona: Starting progress bar Thread.
> ...
> ERROR tools.Corona: Exception while adding key: key-251-19293 in bucket:
> bucket-0-34960 of volume: vol-0-05000.
> java.io.IOException: Exception getting XceiverClient.
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:156)
> at
> org.apache.hadoop.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:122)
> at
> org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.getFromKsmKeyInfo(ChunkGroupOutputStream.java:289)
> at
> org.apache.hadoop.ozone.client.rpc.OzoneRpcClient.createKey(OzoneRpcClient.java:487)
> at
> org.apache.hadoop.ozone.tools.Corona$OfflineProcessor.run(Corona.java:352)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: failed to create a child event loop
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:144)
> ... 9 more
> Caused by: java.lang.IllegalStateException: failed to create a child event
> loop
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
> at
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
> at org.apache.hadoop.scm.XceiverClient.connect(XceiverClient.java:76)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:151)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:145)
> at
> com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> ... 12 more
> Caused by: io.netty.channel.ChannelException: failed to open a new selector
> at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
> at
[jira] [Commented] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144634#comment-16144634
]
Weiwei Yang commented on HDFS-12354:
Hi [~linyiqun]
Thanks for posting the first patch, it looks very good overall. However, I have
a different opinion on the interface signature. Currently it is
{code}
List chooseContainerForBlockDeletion(String prefix, long count,
String prevKey, Map candidateContainers) throws
StorageContainerException;
{code}
Few comments
# Can we just maintain {{prefix}} and {{prevKey}} internally in an
implementation class (if necessary) instead of exposing them via arguments?
Want to keep the method simple to call.
# Can we use {{int}} type count?
NIT in ozone-default.xml, can we add some more doc here
Datanode selects a number of containers to process block deletion in a certain
interval defined by _ozone.block.deleting.service.interval.ms_, the number of
containers to process in each interval is defined by
_ozone.block.deleting.container.limit.per.interval_. This property is used to
configure the policy applied while selecting containers,
_org.apache.hadoop.ozone.container.common.impl.RandomContainerDeletionChoosingPolicy_
implements a simply random policy that to return a random list of containers.
Thanks
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Attachments: HDFS-12354-HDFS-7240.001.patch
>
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[ https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144632#comment-16144632 ] Junping Du commented on HDFS-11896: --- Thanks [~brahmareddy] for identify this not landing on branch-2.8.2. Drop 2.8.3 in fix version. > Non-dfsUsed will be doubled on dead node re-registration > > > Key: HDFS-11896 > URL: https://issues.apache.org/jira/browse/HDFS-11896 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2 > > Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch, > HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch, > HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch, > HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch, > HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch, > HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch, > HDFS-11896.patch > > > *Scenario:* > i)Make you sure you've non-dfs data. > ii) Stop Datanode > iii) wait it becomes dead > iv) now restart and check the non-dfs data -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[ https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-11896: -- Fix Version/s: (was: 2.8.3) > Non-dfsUsed will be doubled on dead node re-registration > > > Key: HDFS-11896 > URL: https://issues.apache.org/jira/browse/HDFS-11896 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2 > > Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch, > HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch, > HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch, > HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch, > HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch, > HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch, > HDFS-11896.patch > > > *Scenario:* > i)Make you sure you've non-dfs data. > ii) Stop Datanode > iii) wait it becomes dead > iv) now restart and check the non-dfs data -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12361) Ozone: SCM failed to start when a container metadata is empty
[
https://issues.apache.org/jira/browse/HDFS-12361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144600#comment-16144600
]
Weiwei Yang commented on HDFS-12361:
Thanks [~anu] for your quick response and commit :)
> Ozone: SCM failed to start when a container metadata is empty
> -
>
> Key: HDFS-12361
> URL: https://issues.apache.org/jira/browse/HDFS-12361
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12361-HDFS-7240.001.patch
>
>
> When I run tests to create keys via corona, sometimes it left some containers
> with empty metadata. This might also happen when SCM stopped at some point
> that metadata was not yet written. When this happens, we got following error
> and SCM could not be started
> {noformat}
> 17/08/27 20:10:57 WARN datanode.DataNode: Unexpected exception in block pool
> Block pool BP-821804790-172.16.165.133-1503887277256 (Datanode Uuid
> 7ee16a59-9604-406e-a0f8-6f44650a725b) service to
> ozone1.fyre.ibm.com/172.16.165.133:8111
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerData.getFromProtBuf(ContainerData.java:66)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.readContainerInfo(ContainerManagerImpl.java:210)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.init(ContainerManagerImpl.java:158)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:99)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:77)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should add a NPE check and mark such containers as inactive without
> failing the SCM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144599#comment-16144599
]
Weiwei Yang commented on HDFS-12365:
Thank you [~vagarychen]
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL: https://issues.apache.org/jira/browse/HDFS-12365
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12365-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a key in ozone with corona (this delegates the call to
> OzoneRpcClient), e.g
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs corona -numOfThreads 1
> -numOfVolumes 1 -numOfBuckets 1 -numOfKeys 1
> {code}
> 2. Run listVolume
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs oz -listVolume
> http://localhost:9864 -user wwei
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-31437",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-38900",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {code}
> Note, the time displayed in {{createdOn}} are both incorrect {{Thu, 01 Jan
> 1970 00:00:00 GMT}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12363) Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages
[
https://issues.apache.org/jira/browse/HDFS-12363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12363:
-
Attachment: HDFS-12363.02.patch
Thanks for the review [~liuml07] and [~jojochuang].
Initially changed the later part only to make the code style more consistent
(== null continue, v.s. !=null and indentation). But the least lines argument
also makes sense, and style isn't that different anyways. So doing a minimum in
patch 2.
It is legit for a datanode to be removed between the lock release-reacquire.
But as said in my last comment, unit test seems to need some non-trivial
effort, and the NPE fix seems straightforward. So I'm hoping to go without a
test. Easy testing ideas welcomed though :)
> Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages
> ---
>
> Key: HDFS-12363
> URL: https://issues.apache.org/jira/browse/HDFS-12363
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12363.01.patch, HDFS-12363.02.patch
>
>
> Saw NN going down with NPE below:
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Thread
> received Runtime exception.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$StorageInfoDefragmenter.scanAndCompactStorages(BlockManager.java:3897)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$StorageInfoDefragmenter.run(BlockManager.java:3852)
> at java.lang.Thread.run(Thread.java:745)
> 2017-08-21 22:14:05,303 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2017-08-21 22:14:05,313 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> {noformat}
> In that version, {{BlockManager}} code is:
> {code}
> 3896 try {
> 3897 DatanodeStorageInfo storage = datanodeManager.
> 3898 getDatanode(datanodesAndStorages.get(i)).
> 3899getStorageInfo(datanodesAndStorages.get(i + 1));
> 3900if (storage != null) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
[ https://issues.apache.org/jira/browse/HDFS-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-12368: - Resolution: Duplicate Status: Resolved (was: Patch Available) +1 thanks [~vagarychen]. I've committed this and updated HDFS-11998. > [branch-2] Enable DFSNetworkTopology as default > --- > > Key: HDFS-12368 > URL: https://issues.apache.org/jira/browse/HDFS-12368 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12368-branch-2.001.patch > > > This JIRA is to backport HDFS-11998 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11998) Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-11998:
-
Fix Version/s: 2.9.0
Committed to branch-2 via HDFS-12368. Thanks [~vagarychen].
> Enable DFSNetworkTopology as default
>
>
> Key: HDFS-11998
> URL: https://issues.apache.org/jira/browse/HDFS-11998
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11998.001.patch, HDFS-11998.002.patch
>
>
> HDFS-11530 has made it configurable to use {{DFSNetworkTopology}}, and still
> uses {{NetworkTopology}} as default.
> Given the stress testing in HDFS-11923 which shows the correctness of
> DFSNetworkTopology, and the performance testing in HDFS-11535 which shows how
> DFSNetworkTopology can outperform NetworkTopology. I think we are at the
> point where I can and should enable DFSNetworkTopology as default.
> Any comments/thoughts are more than welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12293) DataNode should log file name on disk error
[
https://issues.apache.org/jira/browse/HDFS-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144534#comment-16144534
]
Hudson commented on HDFS-12293:
---
ABORTED: Integrated in Jenkins build Hadoop-trunk-Commit #12256 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12256/])
HDFS-12293. DataNode should log file name on disk error. Contributed by (arp:
rev a1e3f84afe6c02cc642699634052d2fb60b30179)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
> DataNode should log file name on disk error
> ---
>
> Key: HDFS-12293
> URL: https://issues.apache.org/jira/browse/HDFS-12293
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Ajay Kumar
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12293.01.patch, HDFS-12293.02.patch
>
>
> Found the following error message in precommit build
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/488/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/
> {noformat}
> 2017-08-10 09:36:53,619 [DataXceiver for client
> DFSClient_NONMAPREDUCE_670847838_18 at /127.0.0.1:55851 [Receiving block
> BP-219227751-172.17.0.2-1502357801473:blk_1073741829_1005]] WARN
> datanode.DataNode (BlockReceiver.java:(287)) - IOException in
> BlockReceiver constructor. Cause is
> java.io.IOException: Not a directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.FileIoProvider.createFile(FileIoProvider.java:302)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createFileWithExistsCheck(DatanodeUtil.java:69)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:306)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:933)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbw(FsVolumeImpl.java:1202)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1356)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:215)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1291)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
> {noformat}
> It is not known what file was being created.
> What's interesting is that {{DatanodeUtil#createFileWithExistsCheck}} does
> carry file name in the exception message, but the exception handler at
> {{DataTransfer#run()}} and {{BlockReceiver#BlockReceiver}} ignores it:
> {code:title=BlockReceiver#BlockReceiver}
> // check if there is a disk error
> IOException cause = DatanodeUtil.getCauseIfDiskError(ioe);
> DataNode.LOG.warn("IOException in BlockReceiver constructor"
> + (cause == null ? "" : ". Cause is "), cause);
> if (cause != null) {
> ioe = cause;
> // Volume error check moved to FileIoProvider
> }
> {code}
> The logger should print the file name in addition to the cause.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12369) Edit log corruption due to hard lease recovery of not-closed file
[
https://issues.apache.org/jira/browse/HDFS-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12369:
-
Attachment: HDFS-12369.test.patch
> Edit log corruption due to hard lease recovery of not-closed file
> -
>
> Key: HDFS-12369
> URL: https://issues.apache.org/jira/browse/HDFS-12369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12369.test.patch
>
>
> HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations
> of client operations.
> Recently, we have observed NN not able to start with the following exception:
> {noformat}
> 2017-08-17 14:32:18,418 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.io.FileNotFoundException: File does not exist:
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
> {noformat}
> Quoting a nicely analysed edits:
> {quote}
> In the edits logged about 1 hour later, we see this failing OP_CLOSE. The
> sequence in the edits shows the file going through:
> OPEN
> ADD_BLOCK
> CLOSE
> ADD_BLOCK # perhaps this was an append
> DELETE
> (about 1 hour later) CLOSE
> It is interesting that there was no CLOSE logged before the delete.
> {quote}
> Grepping that file name, it turns out the close was triggered by
> {{LeaseManager}}, when the lease reaches hard limit.
> {noformat}
> 2017-08-16 15:05:45,927 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> Recovering [Lease. Holder: DFSClient_NONMAPREDUCE_-1997177597_28, pending
> creates: 75],
> src=/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> 2017-08-16 15:05:45,927 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M closed.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12369) Edit log corruption due to hard lease recovery of not-closed file
[
https://issues.apache.org/jira/browse/HDFS-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144529#comment-16144529
]
Xiao Chen edited comment on HDFS-12369 at 8/28/17 11:21 PM:
Attaching a unit test that sorta reproduces this - it ends up with an NPE when
loading {{ReassignLeaseOp}}, instead of the FNFE when loading the {{CloseOp}}.
I'm still looking into this because I thought deleting a file will always
remove the relase, but wanted to post here for early discussions.
was (Author: xiaochen):
Attaching a unit test that sorta reproduces this - it ends up with an NPE when
loading {{ReassignLeaseOp}}, instead of the FNFE when loading the {{CloseOp}}.
I'm still looking into this, but wanted to post here for early discussions.
> Edit log corruption due to hard lease recovery of not-closed file
> -
>
> Key: HDFS-12369
> URL: https://issues.apache.org/jira/browse/HDFS-12369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12369.test.patch
>
>
> HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations
> of client operations.
> Recently, we have observed NN not able to start with the following exception:
> {noformat}
> 2017-08-17 14:32:18,418 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.io.FileNotFoundException: File does not exist:
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
> {noformat}
> Quoting a nicely analysed edits:
> {quote}
> In the edits logged about 1 hour later, we see this failing OP_CLOSE. The
> sequence in the edits shows the file going through:
> OPEN
> ADD_BLOCK
> CLOSE
> ADD_BLOCK # perhaps this was an append
> DELETE
> (about 1 hour later) CLOSE
> It is interesting that there was no CLOSE logged before the delete.
> {quote}
> Grepping that file name, it turns out the close was triggered by
> {{LeaseManager}}, when the lease reaches hard limit.
> {noformat}
> 2017-08-16 15:05:45,927 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> Recovering [Lease. Holder: DFSClient_NONMAPREDUCE_-1997177597_28, pending
> creates: 75],
> src=/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> 2017-08-16 15:05:45,927 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M closed.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12369) Edit log corruption due to hard lease recovery of not-closed file
[
https://issues.apache.org/jira/browse/HDFS-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144529#comment-16144529
]
Xiao Chen commented on HDFS-12369:
--
Attaching a unit test that sorta reproduces this - it ends up with an NPE when
loading {{ReassignLeaseOp}}, instead of the FNFE when loading the {{CloseOp}}.
I'm still looking into this, but wanted to post here for early discussions.
> Edit log corruption due to hard lease recovery of not-closed file
> -
>
> Key: HDFS-12369
> URL: https://issues.apache.org/jira/browse/HDFS-12369
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12369.test.patch
>
>
> HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations
> of client operations.
> Recently, we have observed NN not able to start with the following exception:
> {noformat}
> 2017-08-17 14:32:18,418 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.io.FileNotFoundException: File does not exist:
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
> {noformat}
> Quoting a nicely analysed edits:
> {quote}
> In the edits logged about 1 hour later, we see this failing OP_CLOSE. The
> sequence in the edits shows the file going through:
> OPEN
> ADD_BLOCK
> CLOSE
> ADD_BLOCK # perhaps this was an append
> DELETE
> (about 1 hour later) CLOSE
> It is interesting that there was no CLOSE logged before the delete.
> {quote}
> Grepping that file name, it turns out the close was triggered by
> {{LeaseManager}}, when the lease reaches hard limit.
> {noformat}
> 2017-08-16 15:05:45,927 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> Recovering [Lease. Holder: DFSClient_NONMAPREDUCE_-1997177597_28, pending
> creates: 75],
> src=/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
> 2017-08-16 15:05:45,927 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
> /home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M closed.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12369) Edit log corruption due to hard lease recovery of not-closed file
[
https://issues.apache.org/jira/browse/HDFS-12369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12369:
-
Description:
HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations of
client operations.
Recently, we have observed NN not able to start with the following exception:
{noformat}
2017-08-17 14:32:18,418 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
Failed to start namenode.
java.io.FileNotFoundException: File does not exist:
/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
{noformat}
Quoting a nicely analysed edits:
{quote}
In the edits logged about 1 hour later, we see this failing OP_CLOSE. The
sequence in the edits shows the file going through:
OPEN
ADD_BLOCK
CLOSE
ADD_BLOCK # perhaps this was an append
DELETE
(about 1 hour later) CLOSE
It is interesting that there was no CLOSE logged before the delete.
{quote}
Grepping that file name, it turns out the close was triggered by
{{LeaseManager}}, when the lease reaches hard limit.
{noformat}
2017-08-16 15:05:45,927 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Recovering [Lease. Holder: DFSClient_NONMAPREDUCE_-1997177597_28, pending
creates: 75],
src=/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
2017-08-16 15:05:45,927 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M closed.
{noformat}
was:
HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations of
client operations.
Recently, we have observed NN not able to start with the following exception:
{noformat}
2017-08-17 14:32:18,418 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
Failed to start namenode.
java.io.FileNotFoundException: File does not exist:
/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
at
[jira] [Created] (HDFS-12369) Edit log corruption due to hard lease recovery of not-closed file
Xiao Chen created HDFS-12369:
Summary: Edit log corruption due to hard lease recovery of
not-closed file
Key: HDFS-12369
URL: https://issues.apache.org/jira/browse/HDFS-12369
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Xiao Chen
Assignee: Xiao Chen
HDFS-6257 and HDFS-7707 worked hard to prevent corruption from combinations of
client operations.
Recently, we have observed NN not able to start with the following exception:
{noformat}
2017-08-17 14:32:18,418 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
Failed to start namenode.
java.io.FileNotFoundException: File does not exist:
/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:897)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:750)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:318)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1125)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:789)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:844)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
{noformat}
Quoting a nicely analysed edits:
{quote}
In the edits logged about 1 hour later, we see this failing OP_CLOSE. The
sequence in the edits shows the file going through:
OPEN
ADD_BLOCK
CLOSE
ADD_BLOCK # perhaps this was an append
DELETE
(about 1 hour later) CLOSE
It is interesting that there was no CLOSE logged before the delete.
{quote}
Grepping that file name, it turns out the close was triggered by lease reaching
hard limit.
{noformat}
2017-08-16 15:05:45,927 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Recovering [Lease. Holder: DFSClient_NONMAPREDUCE_-1997177597_28, pending
creates: 75],
src=/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M
2017-08-16 15:05:45,927 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
/home/Events/CancellationSurvey_MySQL/2015/12/31/.part-0.9nlJ3M closed.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12293) DataNode should log file name on disk error
[
https://issues.apache.org/jira/browse/HDFS-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-12293:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
Thanks [~jojochuang]. I've committed this.
Thanks for the contribution [~ajayydv].
> DataNode should log file name on disk error
> ---
>
> Key: HDFS-12293
> URL: https://issues.apache.org/jira/browse/HDFS-12293
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Ajay Kumar
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12293.01.patch, HDFS-12293.02.patch
>
>
> Found the following error message in precommit build
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/488/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/
> {noformat}
> 2017-08-10 09:36:53,619 [DataXceiver for client
> DFSClient_NONMAPREDUCE_670847838_18 at /127.0.0.1:55851 [Receiving block
> BP-219227751-172.17.0.2-1502357801473:blk_1073741829_1005]] WARN
> datanode.DataNode (BlockReceiver.java:(287)) - IOException in
> BlockReceiver constructor. Cause is
> java.io.IOException: Not a directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.FileIoProvider.createFile(FileIoProvider.java:302)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createFileWithExistsCheck(DatanodeUtil.java:69)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:306)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:933)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbw(FsVolumeImpl.java:1202)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1356)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:215)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1291)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
> {noformat}
> It is not known what file was being created.
> What's interesting is that {{DatanodeUtil#createFileWithExistsCheck}} does
> carry file name in the exception message, but the exception handler at
> {{DataTransfer#run()}} and {{BlockReceiver#BlockReceiver}} ignores it:
> {code:title=BlockReceiver#BlockReceiver}
> // check if there is a disk error
> IOException cause = DatanodeUtil.getCauseIfDiskError(ioe);
> DataNode.LOG.warn("IOException in BlockReceiver constructor"
> + (cause == null ? "" : ". Cause is "), cause);
> if (cause != null) {
> ioe = cause;
> // Volume error check moved to FileIoProvider
> }
> {code}
> The logger should print the file name in addition to the cause.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12293) DataNode should log file name on disk error
[
https://issues.apache.org/jira/browse/HDFS-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144488#comment-16144488
]
Wei-Chiu Chuang commented on HDFS-12293:
Thanks [~arpitagarwal] and [~ajayydv]
+1 LGTM!
> DataNode should log file name on disk error
> ---
>
> Key: HDFS-12293
> URL: https://issues.apache.org/jira/browse/HDFS-12293
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Ajay Kumar
> Labels: newbie
> Attachments: HDFS-12293.01.patch, HDFS-12293.02.patch
>
>
> Found the following error message in precommit build
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/488/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/
> {noformat}
> 2017-08-10 09:36:53,619 [DataXceiver for client
> DFSClient_NONMAPREDUCE_670847838_18 at /127.0.0.1:55851 [Receiving block
> BP-219227751-172.17.0.2-1502357801473:blk_1073741829_1005]] WARN
> datanode.DataNode (BlockReceiver.java:(287)) - IOException in
> BlockReceiver constructor. Cause is
> java.io.IOException: Not a directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.FileIoProvider.createFile(FileIoProvider.java:302)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createFileWithExistsCheck(DatanodeUtil.java:69)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:306)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:933)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbw(FsVolumeImpl.java:1202)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1356)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:215)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1291)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
> {noformat}
> It is not known what file was being created.
> What's interesting is that {{DatanodeUtil#createFileWithExistsCheck}} does
> carry file name in the exception message, but the exception handler at
> {{DataTransfer#run()}} and {{BlockReceiver#BlockReceiver}} ignores it:
> {code:title=BlockReceiver#BlockReceiver}
> // check if there is a disk error
> IOException cause = DatanodeUtil.getCauseIfDiskError(ioe);
> DataNode.LOG.warn("IOException in BlockReceiver constructor"
> + (cause == null ? "" : ". Cause is "), cause);
> if (cause != null) {
> ioe = cause;
> // Volume error check moved to FileIoProvider
> }
> {code}
> The logger should print the file name in addition to the cause.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12293) DataNode should log file name on disk error
[
https://issues.apache.org/jira/browse/HDFS-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144481#comment-16144481
]
Arpit Agarwal commented on HDFS-12293:
--
+1 for the v2 patch.
[~jojochuang], does the patch look fine to you?
> DataNode should log file name on disk error
> ---
>
> Key: HDFS-12293
> URL: https://issues.apache.org/jira/browse/HDFS-12293
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Ajay Kumar
> Labels: newbie
> Attachments: HDFS-12293.01.patch, HDFS-12293.02.patch
>
>
> Found the following error message in precommit build
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/488/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailureReporting/testSuccessiveVolumeFailures/
> {noformat}
> 2017-08-10 09:36:53,619 [DataXceiver for client
> DFSClient_NONMAPREDUCE_670847838_18 at /127.0.0.1:55851 [Receiving block
> BP-219227751-172.17.0.2-1502357801473:blk_1073741829_1005]] WARN
> datanode.DataNode (BlockReceiver.java:(287)) - IOException in
> BlockReceiver constructor. Cause is
> java.io.IOException: Not a directory
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.FileIoProvider.createFile(FileIoProvider.java:302)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createFileWithExistsCheck(DatanodeUtil.java:69)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:306)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:933)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbw(FsVolumeImpl.java:1202)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1356)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:215)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1291)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:758)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
> {noformat}
> It is not known what file was being created.
> What's interesting is that {{DatanodeUtil#createFileWithExistsCheck}} does
> carry file name in the exception message, but the exception handler at
> {{DataTransfer#run()}} and {{BlockReceiver#BlockReceiver}} ignores it:
> {code:title=BlockReceiver#BlockReceiver}
> // check if there is a disk error
> IOException cause = DatanodeUtil.getCauseIfDiskError(ioe);
> DataNode.LOG.warn("IOException in BlockReceiver constructor"
> + (cause == null ? "" : ". Cause is "), cause);
> if (cause != null) {
> ioe = cause;
> // Volume error check moved to FileIoProvider
> }
> {code}
> The logger should print the file name in addition to the cause.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10391) Always enable NameNode service RPC port
[
https://issues.apache.org/jira/browse/HDFS-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144411#comment-16144411
]
Arpit Agarwal commented on HDFS-10391:
--
Calling NameNode.getServiceAddress from {{SecondaryNameNode.initialize}} looks
incorrect after this change, since it may return a logical nameservice in a
federated cluster.
Not sure we have a unit test for this configuration.
> Always enable NameNode service RPC port
> ---
>
> Key: HDFS-10391
> URL: https://issues.apache.org/jira/browse/HDFS-10391
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
>Reporter: Arpit Agarwal
>Assignee: Gergely Novák
> Labels: Incompatible
> Attachments: HDFS-10391.001.patch, HDFS-10391.002.patch,
> HDFS-10391.003.patch, HDFS-10391.004.patch, HDFS-10391.005.patch,
> HDFS-10391.006.patch, HDFS-10391.007.patch, HDFS-10391.008.patch,
> HDFS-10391.009.patch, HDFS-10391.010.patch, HDFS-10391.v5-v6-delta.patch
>
>
> The NameNode should always be setup with a service RPC port so that it does
> not have to be explicitly enabled by an administrator.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144379#comment-16144379
]
Hadoop QA commented on HDFS-12368:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} branch-2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 56s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests |
hadoop.hdfs.server.datanode.TestDirectoryScanner |
| JDK v1.7.0_131 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain |
| JDK v1.7.0_131 Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5e40efe |
| JIRA Issue | HDFS-12368 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884100/HDFS-12368-branch-2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 4f4732b16abc
[jira] [Updated] (HDFS-12361) Ozone: SCM failed to start when a container metadata is empty
[
https://issues.apache.org/jira/browse/HDFS-12361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12361:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
[~cheersyang] Thanks for the contribution. I have committed this to the feature
branch.
> Ozone: SCM failed to start when a container metadata is empty
> -
>
> Key: HDFS-12361
> URL: https://issues.apache.org/jira/browse/HDFS-12361
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12361-HDFS-7240.001.patch
>
>
> When I run tests to create keys via corona, sometimes it left some containers
> with empty metadata. This might also happen when SCM stopped at some point
> that metadata was not yet written. When this happens, we got following error
> and SCM could not be started
> {noformat}
> 17/08/27 20:10:57 WARN datanode.DataNode: Unexpected exception in block pool
> Block pool BP-821804790-172.16.165.133-1503887277256 (Datanode Uuid
> 7ee16a59-9604-406e-a0f8-6f44650a725b) service to
> ozone1.fyre.ibm.com/172.16.165.133:8111
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerData.getFromProtBuf(ContainerData.java:66)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.readContainerInfo(ContainerManagerImpl.java:210)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.init(ContainerManagerImpl.java:158)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:99)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:77)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should add a NPE check and mark such containers as inactive without
> failing the SCM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12361) Ozone: SCM failed to start when a container metadata is empty
[
https://issues.apache.org/jira/browse/HDFS-12361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144286#comment-16144286
]
Anu Engineer commented on HDFS-12361:
-
+1, thanks for the fix. I will commit this shortly.
> Ozone: SCM failed to start when a container metadata is empty
> -
>
> Key: HDFS-12361
> URL: https://issues.apache.org/jira/browse/HDFS-12361
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12361-HDFS-7240.001.patch
>
>
> When I run tests to create keys via corona, sometimes it left some containers
> with empty metadata. This might also happen when SCM stopped at some point
> that metadata was not yet written. When this happens, we got following error
> and SCM could not be started
> {noformat}
> 17/08/27 20:10:57 WARN datanode.DataNode: Unexpected exception in block pool
> Block pool BP-821804790-172.16.165.133-1503887277256 (Datanode Uuid
> 7ee16a59-9604-406e-a0f8-6f44650a725b) service to
> ozone1.fyre.ibm.com/172.16.165.133:8111
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerData.getFromProtBuf(ContainerData.java:66)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.readContainerInfo(ContainerManagerImpl.java:210)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.init(ContainerManagerImpl.java:158)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.(OzoneContainer.java:99)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.(DatanodeStateMachine.java:77)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1592)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:409)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:783)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:816)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We should add a NPE check and mark such containers as inactive without
> failing the SCM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[
https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144258#comment-16144258
]
Anu Engineer commented on HDFS-12340:
-
[~shashikant] Thank you for sharing the work in progress. Some minor comments
# nit: add comments for functions and rename the file extension to .c instead
of .C
# handleCreate* functions -- assert that info is not null.
# In function main -- vol and bucket max size is 64 bytes. key size max is 1024
--
# always use curly ?
{{if (rc != 0)
return -1;}}
# OzoneClient.c {{initOzoneClient}} -- init the vars, As a habit :
{code}
char user[64] = {0};
char version[64]= {0};
char url[256]= {0};
CURLcode res = 0;
{code}
# printf("invalid or no user name provided\n"); --> printf to stderror please.
# Missing goto exit
{code}
if (!isValidStr(userName))
{
printf("invalid or no user name provided\n");
rc = -1;
>> missing goto exit <<
}
{code}
# Check return values
{code}
ozoneInfo *info = (ozoneInfo *) malloc(sizeof(struct ozoneInfo));
info->url = (char *) malloc(strlen(url) +1);
info->version = (char *) malloc(strlen(ozoneVersion) +1);
check if the malloc failed, and bail if needed.
{code}
# Also memset or use calloc here, so these structs are all clean.
# use something which checks the array bounds while doing the copy in
getUrlHead or check them yourself, please.
# There is a memory leak here.
{code}
else
{
free(info);
return NULL;
}
{code}
You should free all the internally allocated variables before calling this free.
# resetOzoneInfo -- check if info is valid before accessing this variable.
# Init all the variables, please.
# replace strcat/strcpy with safer functions.
# Is there a memory leak in this function-- should we call curl_slist_free_all
before adding to this list ?
# curl_slist_append - check the return is not null -- at all places.
# Set info to NULL after destroy is called?
# check *chunk is valid in addDateHeader
# executeCurlPostOperation -- check info is valid before dereferencing ?
# if (rc == -1)
goto exit; -- put curly brackets?
# Missing "goto exit" in createVolume, createBucket, putKey after the rc = -1;
> Ozone: C/C++ implementation of ozone client using curl
> --
>
> Key: HDFS-12340
> URL: https://issues.apache.org/jira/browse/HDFS-12340
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Fix For: HDFS-7240
>
> Attachments: main.C, ozoneClient.C, ozoneClient.h
>
>
> This Jira is introduced for implementation of ozone client in C/C++ using
> curl library.
> All these calls will make use of HTTP protocol and would require libcurl. The
> libcurl API are referenced from here:
> https://curl.haxx.se/libcurl/
> Additional details would be posted along with the patches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11398) TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure still fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-11398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144225#comment-16144225
]
George Huang commented on HDFS-11398:
-
We recently see this in house, seems related?
java.lang.Exception: test timed out after 12 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.join(BPServiceActor.java:579)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.join(BPOfferService.java:314)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.shutDownAll(BlockPoolManager.java:118)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2082)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNode(MiniDFSCluster.java:2001)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1991)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1970)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
at
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure.tearDown(TestDataNodeVolumeFailure.java:142)
> TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure still fails
> intermittently
>
>
> Key: HDFS-11398
> URL: https://issues.apache.org/jira/browse/HDFS-11398
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: failure.log, HDFS-11398.001.patch,
> HDFS-11398-reproduce.patch
>
>
> The test {{TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure}}
> still fails intermittently in trunk after HDFS-11316. The stack infos:
> {code}
> testUnderReplicationAfterVolFailure(org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure)
> Time elapsed: 95.021 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for condition.
> Thread diagnostics:
> Timestamp: 2017-02-07 07:00:34,193
>
> java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native
> Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:511)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:276)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure.testUnderReplicationAfterVolFailure(TestDataNodeVolumeFailure.java:412)
> {code}
> I looked into this and found there is one chance that the vaule
> {{UnderReplicatedBlocksCount}} will be no longer > 0. The following is my
> analysation:
> In test {{TestDataNodeVolumeFailure.testUnderReplicationAfterVolFailure}}, it
> uses creating file to trigger the disk error checking. The related codes:
> {code}
> Path file1 = new Path("/test1");
> DFSTestUtil.createFile(fs, file1, 1024, (short)3, 1L);
> DFSTestUtil.waitReplication(fs, file1, (short)3);
> // Fail the first volume on both datanodes
> File dn1Vol1 = new File(dataDir, "data"+(2*0+1));
> File dn2Vol1 = new File(dataDir, "data"+(2*1+1));
> DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn2Vol1);
> Path file2 = new Path("/test2");
> DFSTestUtil.createFile(fs, file2, 1024, (short)3, 1L);
> DFSTestUtil.waitReplication(fs, file2, (short)3);
> {code}
> This will lead one problem: If the cluster is busy, and it costs long time to
> wait replication of file2 to be desired value. During this time, the under
> replication blocks of file1 can also be rereplication in cluster. If this is
> done, the condition {{underReplicatedBlocks > 0}} will never be satisfied.
> And this can be reproduced in my local env.
> Actually, we can use a easy way {{DataNodeTestUtils.waitForDiskError}} to
> replace this, it runs fast and be more reliable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
[ https://issues.apache.org/jira/browse/HDFS-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12368: -- Status: Patch Available (was: Open) > [branch-2] Enable DFSNetworkTopology as default > --- > > Key: HDFS-12368 > URL: https://issues.apache.org/jira/browse/HDFS-12368 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12368-branch-2.001.patch > > > This JIRA is to backport HDFS-11998 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
[ https://issues.apache.org/jira/browse/HDFS-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12368: -- Attachment: HDFS-12368-branch-2.001.patch > [branch-2] Enable DFSNetworkTopology as default > --- > > Key: HDFS-12368 > URL: https://issues.apache.org/jira/browse/HDFS-12368 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12368-branch-2.001.patch > > > This JIRA is to backport HDFS-11998 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
Chen Liang created HDFS-12368: - Summary: [branch-2] Enable DFSNetworkTopology as default Key: HDFS-12368 URL: https://issues.apache.org/jira/browse/HDFS-12368 Project: Hadoop HDFS Issue Type: Improvement Reporter: Chen Liang Assignee: Chen Liang This JIRA is to backport HDFS-11998 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12191) Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done
[
https://issues.apache.org/jira/browse/HDFS-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144199#comment-16144199
]
Manoj Govindassamy commented on HDFS-12191:
---
LGTM, +1. Thanks [~yzhangal].
Nits:
* TestSnapshotDiffReport:732 still refers to the the old property name
* There are quite a few thread sleep for 3sec. You can always call the
setTimes() with a constructed time instead of the actual waiting.
> Provide option to not capture the accessTime change of a file to snapshot if
> no other modification has been done
>
>
> Key: HDFS-12191
> URL: https://issues.apache.org/jira/browse/HDFS-12191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 3.0.0-beta1
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-12191.001.patch, HDFS-12191.002.patch,
> HDFS-12191.003.patch, HDFS-12191.004.patch
>
>
> Currently, if the accessTime of a file changed before a snapshot is taken,
> this accessTime will be captured in the snapshot, even if there is no other
> modifications made to this file.
> Because of this, when we calculate snapshotDiff, more work need to be done
> for this file, e,g,, metadataEquals method will be called, even if there is
> no modification is made (thus not recorded to snapshotDiff). This can cause
> snapshotDiff to slow down quite a lot when there are a lot of files to be
> examined.
> This jira is to provide an option to skip capturing accessTime only change to
> snapshot. Thus snapshotDiff can be done faster.
> When accessTime of a file changed, if there is other modification to the
> file, the access time will still be captured in snapshot.
> Sometimes we want accessTime be captured to snapshot, such that when
> restoring from the snapshot, we know the accessTime of this snapshot. So this
> new feature is optional, and is controlled by a config property.
> Worth to mention is, how accurately the acessTime is captured is dependent on
> the following config that has default value of 1 hour, which means new access
> within an hour of previous access will not be captured.
> {code}
> public static final String DFS_NAMENODE_ACCESSTIME_PRECISION_KEY =
>
> HdfsClientConfigKeys.DeprecatedKeys.DFS_NAMENODE_ACCESSTIME_PRECISION_KEY;
> public static final longDFS_NAMENODE_ACCESSTIME_PRECISION_DEFAULT =
> 360;
> {code}
> .
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10480) Add an admin command to list currently open files
[ https://issues.apache.org/jira/browse/HDFS-10480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-10480: Issue Type: New Feature (was: Improvement) > Add an admin command to list currently open files > - > > Key: HDFS-10480 > URL: https://issues.apache.org/jira/browse/HDFS-10480 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Kihwal Lee >Assignee: Manoj Govindassamy > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3 > > Attachments: HDFS-10480.02.patch, HDFS-10480.03.patch, > HDFS-10480.04.patch, HDFS-10480.05.patch, HDFS-10480.06.patch, > HDFS-10480.07.patch, HDFS-10480-branch-2.01.patch, > HDFS-10480-branch-2.8.01.patch, HDFS-10480-trunk-1.patch, > HDFS-10480-trunk.patch > > > Currently there is no easy way to obtain the list of active leases or files > being written. It will be nice if we have an admin command to list open files > and their lease holders. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-2538) option to disable fsck dots
[ https://issues.apache.org/jira/browse/HDFS-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2538: --- Component/s: namenode > option to disable fsck dots > > > Key: HDFS-2538 > URL: https://issues.apache.org/jira/browse/HDFS-2538 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.2.0 >Reporter: Allen Wittenauer >Assignee: Mohammad Kamrul Islam >Priority: Minor > Labels: newbie > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-2538.1.patch, HDFS-2538.2.patch, HDFS-2538.3.patch, > HDFS-2538-branch-0.20-security-204.patch, > HDFS-2538-branch-0.20-security-204.patch, HDFS-2538-branch-1.0.patch, > HDFS-2538-branch-2.7.patch > > > this patch turns the dots during fsck off by default and provides an option > to turn them back on if you have a fetish for millions and millions of dots > on your terminal. i haven't done any benchmarks, but i suspect fsck is now > 300% faster to boot. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144144#comment-16144144
]
Chen Liang commented on HDFS-12321:
---
The failed tests all passed in my local run. {{TestSQLQuery}} seems to complain
not able to find ozoneQuery.json file when Jenkins was running it. But
interestingly the other test {{TestKSMSQLCli}} that also uses this file seem to
have passed.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch,
> HDFS-12321-HDFS-7240.008.patch, HDFS-12321-HDFS-7240.009.patch,
> HDFS-12321-HDFS-7240.010.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12358) Handle IOException when transferring edit log to Journal current dir through JN sync
[ https://issues.apache.org/jira/browse/HDFS-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDFS-12358: Assignee: Arpit Agarwal (was: Hanisha Koneru) > Handle IOException when transferring edit log to Journal current dir through > JN sync > > > Key: HDFS-12358 > URL: https://issues.apache.org/jira/browse/HDFS-12358 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Arpit Agarwal > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12358.001.patch, HDFS-12358.002.patch, > HDFS-12358.003.patch > > > During JN sync, a missing edit log is first downloaded from a remote JN to a > tmp dir and then moved to the current directory (protected by the Journal's > synchronization). > Irrespective of whether the move succeeds or fails, we should delete the tmp > file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12358) Handle IOException when transferring edit log to Journal current dir through JN sync
[ https://issues.apache.org/jira/browse/HDFS-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDFS-12358: Assignee: Hanisha Koneru (was: Arpit Agarwal) > Handle IOException when transferring edit log to Journal current dir through > JN sync > > > Key: HDFS-12358 > URL: https://issues.apache.org/jira/browse/HDFS-12358 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12358.001.patch, HDFS-12358.002.patch, > HDFS-12358.003.patch > > > During JN sync, a missing edit log is first downloaded from a remote JN to a > tmp dir and then moved to the current directory (protected by the Journal's > synchronization). > Irrespective of whether the move succeeds or fails, we should delete the tmp > file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12358) Handle IOException when transferring edit log to Journal current dir through JN sync
[ https://issues.apache.org/jira/browse/HDFS-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144129#comment-16144129 ] Hanisha Koneru commented on HDFS-12358: --- Thanks for committing the patch, [~arpitagarwal] > Handle IOException when transferring edit log to Journal current dir through > JN sync > > > Key: HDFS-12358 > URL: https://issues.apache.org/jira/browse/HDFS-12358 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12358.001.patch, HDFS-12358.002.patch, > HDFS-12358.003.patch > > > During JN sync, a missing edit log is first downloaded from a remote JN to a > tmp dir and then moved to the current directory (protected by the Journal's > synchronization). > Irrespective of whether the move succeeds or fails, we should delete the tmp > file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8312) Trash does not descent into child directories to check for permissions
[
https://issues.apache.org/jira/browse/HDFS-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-8312:
---
Release Note: Permissions are now checked when moving a file to Trash.
> Trash does not descent into child directories to check for permissions
> --
>
> Key: HDFS-8312
> URL: https://issues.apache.org/jira/browse/HDFS-8312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, security
>Affects Versions: 2.2.0, 2.6.0, 2.7.2
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: 2.9.0, 2.7.4, 3.0.0-alpha1, 2.8.2
>
> Attachments: HDFS-8312-001.patch, HDFS-8312-002.patch,
> HDFS-8312-003.patch, HDFS-8312-004.patch, HDFS-8312-005.patch,
> HDFS-8312-branch-2.7.patch, HDFS-8312-branch-2.8.01.patch,
> HDFS-8312-branch-2.8.1.001.patch, HDFS-8312-testcase.patch
>
>
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example:
> Run the following command to initialize directory structure as super user:
> {code}
> hadoop fs -mkdir /BSS/level1
> hadoop fs -mkdir /BSS/level1/level2
> hadoop fs -mkdir /BSS/level1/level2/level3
> hadoop fs -put /tmp/appConfig.json /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown user1:users /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown -R user1:users /BSS/level1
> hadoop fs -chown -R 750 /BSS/level1
> hadoop fs -chmod -R 640 /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chmod 775 /BSS
> {code}
> Change to a normal user called user2.
> When trash is enabled:
> {code}
> sudo su user2 -
> hadoop fs -rm -r /BSS/level1
> 15/05/01 16:51:20 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 3600 minutes, Emptier interval = 0 minutes.
> Moved: 'hdfs://bdvs323.svl.ibm.com:9000/BSS/level1' to trash at:
> hdfs://bdvs323.svl.ibm.com:9000/user/user2/.Trash/Current
> {code}
> When trash is disabled:
> {code}
> /opt/ibm/biginsights/IHC/bin/hadoop fs -Dfs.trash.interval=0 -rm -r
> /BSS/level1
> 15/05/01 16:58:31 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 0 minutes, Emptier interval = 0 minutes.
> rm: Permission denied: user=user2, access=ALL,
> inode="/BSS/level1":user1:users:drwxr-x---
> {code}
> There is inconsistency between trash behavior and delete behavior. When
> trash is enabled, files owned by user1 is deleted by user2. It looks like
> trash does not recursively validate if the child directory files can be
> removed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8312) Trash does not descent into child directories to check for permissions
[
https://issues.apache.org/jira/browse/HDFS-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-8312:
---
Release Note: (was: HDFS-8312. Added permission check for moving file to
Trash. (Weiwei Yang via Eric Yang))
> Trash does not descent into child directories to check for permissions
> --
>
> Key: HDFS-8312
> URL: https://issues.apache.org/jira/browse/HDFS-8312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, security
>Affects Versions: 2.2.0, 2.6.0, 2.7.2
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: 2.9.0, 2.7.4, 3.0.0-alpha1, 2.8.2
>
> Attachments: HDFS-8312-001.patch, HDFS-8312-002.patch,
> HDFS-8312-003.patch, HDFS-8312-004.patch, HDFS-8312-005.patch,
> HDFS-8312-branch-2.7.patch, HDFS-8312-branch-2.8.01.patch,
> HDFS-8312-branch-2.8.1.001.patch, HDFS-8312-testcase.patch
>
>
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example:
> Run the following command to initialize directory structure as super user:
> {code}
> hadoop fs -mkdir /BSS/level1
> hadoop fs -mkdir /BSS/level1/level2
> hadoop fs -mkdir /BSS/level1/level2/level3
> hadoop fs -put /tmp/appConfig.json /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown user1:users /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown -R user1:users /BSS/level1
> hadoop fs -chown -R 750 /BSS/level1
> hadoop fs -chmod -R 640 /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chmod 775 /BSS
> {code}
> Change to a normal user called user2.
> When trash is enabled:
> {code}
> sudo su user2 -
> hadoop fs -rm -r /BSS/level1
> 15/05/01 16:51:20 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 3600 minutes, Emptier interval = 0 minutes.
> Moved: 'hdfs://bdvs323.svl.ibm.com:9000/BSS/level1' to trash at:
> hdfs://bdvs323.svl.ibm.com:9000/user/user2/.Trash/Current
> {code}
> When trash is disabled:
> {code}
> /opt/ibm/biginsights/IHC/bin/hadoop fs -Dfs.trash.interval=0 -rm -r
> /BSS/level1
> 15/05/01 16:58:31 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 0 minutes, Emptier interval = 0 minutes.
> rm: Permission denied: user=user2, access=ALL,
> inode="/BSS/level1":user1:users:drwxr-x---
> {code}
> There is inconsistency between trash behavior and delete behavior. When
> trash is enabled, files owned by user1 is deleted by user2. It looks like
> trash does not recursively validate if the child directory files can be
> removed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12340: --- Attachment: ozoneClient.C ozoneClient.h main.C The uploaded patch consists of 3 files: 1)ozoneClient.C - implements the following API's of ozoneClient library: a) CreateVolume b) CreateBucket c) PutKey d) GetKey 2) ozoneClient.h -- Contains the prototypes for the ozoneClient Library API's 3) main. C - Application using the API for the ozoneClient Library The implementation has been done using the libcurl API's . To Do: 1)There are issue with reusing the same curl easy handles in between calls to different API's of the ozoneClient from the application which need to be optimized.Currently , for every API the curl handle is cleaned up and recreated gain as otherwise, the subsequent operations using the same handle hang over HTTP. 2) Needs integeration with maven for building with Ozone source repository. 3) Needs further implementation of remaining API's for ozoneClient. > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: main.C, ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12363) Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages
[
https://issues.apache.org/jira/browse/HDFS-12363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144103#comment-16144103
]
Wei-Chiu Chuang commented on HDFS-12363:
Hi Xiao, thanks for reporting this bug.
Looks legit to me if a datanode is removed while this for loop releases
namenode lock.
Would it be possible to add a test?
> Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages
> ---
>
> Key: HDFS-12363
> URL: https://issues.apache.org/jira/browse/HDFS-12363
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-12363.01.patch
>
>
> Saw NN going down with NPE below:
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Thread
> received Runtime exception.
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$StorageInfoDefragmenter.scanAndCompactStorages(BlockManager.java:3897)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$StorageInfoDefragmenter.run(BlockManager.java:3852)
> at java.lang.Thread.run(Thread.java:745)
> 2017-08-21 22:14:05,303 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2017-08-21 22:14:05,313 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> {noformat}
> In that version, {{BlockManager}} code is:
> {code}
> 3896 try {
> 3897 DatanodeStorageInfo storage = datanodeManager.
> 3898 getDatanode(datanodesAndStorages.get(i)).
> 3899getStorageInfo(datanodesAndStorages.get(i + 1));
> 3900if (storage != null) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12333) Ozone: Extend Datanode web interface with SCM information
[
https://issues.apache.org/jira/browse/HDFS-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144096#comment-16144096
]
Hadoop QA commented on HDFS-12333:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.scm.TestAllocateContainer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12333 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884061/HDFS-12333-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6486e3ee70bd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 76a156e |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20900/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20900/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144093#comment-16144093
]
Chen Liang commented on HDFS-12365:
---
Thanks [~cheersyang] for the catch! The v001 patch LGTM, I've committed to the
feature branch.
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL: https://issues.apache.org/jira/browse/HDFS-12365
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12365-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a key in ozone with corona (this delegates the call to
> OzoneRpcClient), e.g
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs corona -numOfThreads 1
> -numOfVolumes 1 -numOfBuckets 1 -numOfKeys 1
> {code}
> 2. Run listVolume
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs oz -listVolume
> http://localhost:9864 -user wwei
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-31437",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-38900",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {code}
> Note, the time displayed in {{createdOn}} are both incorrect {{Thu, 01 Jan
> 1970 00:00:00 GMT}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12365:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL: https://issues.apache.org/jira/browse/HDFS-12365
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12365-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a key in ozone with corona (this delegates the call to
> OzoneRpcClient), e.g
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs corona -numOfThreads 1
> -numOfVolumes 1 -numOfBuckets 1 -numOfKeys 1
> {code}
> 2. Run listVolume
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs oz -listVolume
> http://localhost:9864 -user wwei
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-31437",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-38900",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {code}
> Note, the time displayed in {{createdOn}} are both incorrect {{Thu, 01 Jan
> 1970 00:00:00 GMT}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144094#comment-16144094
]
Zhe Zhang commented on HDFS-9079:
-
Unassigning myself from the JIRA as I won't be able to work on it in the next
couple of months. [~andrew.wang] [~drankye] If you see anyone with some cycles
working on this that'd be great. I'm still a little concerned about the
complexity on the write pipeline of EC, and I think this will help.
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
>
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9079.01.patch, HDFS-9079.02.patch,
> HDFS-9079.03.patch, HDFS-9079.04.patch, HDFS-9079.05.patch,
> HDFS-9079.06.patch, HDFS-9079.07.patch, HDFS-9079.08.patch,
> HDFS-9079.09.patch, HDFS-9079.10.patch, HDFS-9079.11.patch,
> HDFS-9079.12.patch, HDFS-9079.13.patch, HDFS-9079.14.patch,
> HDFS-9079.15.patch, HDFS-9079-HDFS-7285.00.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> With multiple streamer threads run in parallel, we need to correctly handle a
> large number of possible combinations of interleaved thread events. For
> example, {{streamer_B}} starts step 2 in between events {{streamer_A.2}} and
> {{streamer_A.3}}.
> HDFS-9040 moves steps 1, 2, 3, 6 from streamer to {{DFSStripedOutputStream}}.
> This JIRA proposes some further optimizations based on HDFS-9040:
> # We can preallocate GS when NN creates a new striped block group
> ({{FSN#createNewBlock}}). For each new striped block group we can reserve
> {{NUM_PARITY_BLOCKS}} GS's. If more than {{NUM_PARITY_BLOCKS}} errors have
> happened we shouldn't try to further recover anyway.
> # We can use a dedicated event processor to offload the error handling logic
> from {{DFSStripedOutputStream}}, which is not a long running daemon.
> # We can limit the lifespan of a streamer to be a single block. A streamer
> ends either after finishing the current block or when encountering a DN
> failure.
> With the proposed change, a {{StripedDataStreamer}}'s flow becomes:
> {code}
> 1) Finds DN error => 2) Notify coordinator (async, not waiting for response)
> => terminates
> 1) Finds external error => 2) Applies new GS to DN (createBlockOutputStream)
> => 3) Ack from DN => 4) Notify coordinator (async, not waiting for response)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang reassigned HDFS-9079:
---
Assignee: (was: Zhe Zhang)
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
>
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-9079.01.patch, HDFS-9079.02.patch,
> HDFS-9079.03.patch, HDFS-9079.04.patch, HDFS-9079.05.patch,
> HDFS-9079.06.patch, HDFS-9079.07.patch, HDFS-9079.08.patch,
> HDFS-9079.09.patch, HDFS-9079.10.patch, HDFS-9079.11.patch,
> HDFS-9079.12.patch, HDFS-9079.13.patch, HDFS-9079.14.patch,
> HDFS-9079.15.patch, HDFS-9079-HDFS-7285.00.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> With multiple streamer threads run in parallel, we need to correctly handle a
> large number of possible combinations of interleaved thread events. For
> example, {{streamer_B}} starts step 2 in between events {{streamer_A.2}} and
> {{streamer_A.3}}.
> HDFS-9040 moves steps 1, 2, 3, 6 from streamer to {{DFSStripedOutputStream}}.
> This JIRA proposes some further optimizations based on HDFS-9040:
> # We can preallocate GS when NN creates a new striped block group
> ({{FSN#createNewBlock}}). For each new striped block group we can reserve
> {{NUM_PARITY_BLOCKS}} GS's. If more than {{NUM_PARITY_BLOCKS}} errors have
> happened we shouldn't try to further recover anyway.
> # We can use a dedicated event processor to offload the error handling logic
> from {{DFSStripedOutputStream}}, which is not a long running daemon.
> # We can limit the lifespan of a streamer to be a single block. A streamer
> ends either after finishing the current block or when encountering a DN
> failure.
> With the proposed change, a {{StripedDataStreamer}}'s flow becomes:
> {code}
> 1) Finds DN error => 2) Notify coordinator (async, not waiting for response)
> => terminates
> 1) Finds external error => 2) Applies new GS to DN (createBlockOutputStream)
> => 3) Ack from DN => 4) Notify coordinator (async, not waiting for response)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
[
https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144047#comment-16144047
]
Hadoop QA commented on HDFS-11096:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
9s{color} | {color:orange} The patch generated 422 new + 0 unchanged - 0 fixed
= 422 total (was 0) {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
2s{color} | {color:red} The patch generated 291 new + 0 unchanged - 0 fixed =
291 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11096 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884080/HDFS-11096.001.patch |
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs pylint |
| uname | Linux 259c108a4a02 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 02599bd |
| shellcheck | v0.4.6 |
| pylint | v1.7.2 |
| pylint |
https://builds.apache.org/job/PreCommit-HDFS-Build/20901/artifact/patchprocess/diff-patch-pylint.txt
|
| shellcheck |
https://builds.apache.org/job/PreCommit-HDFS-Build/20901/artifact/patchprocess/diff-patch-shellcheck.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20901/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/20901/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: U: |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20901/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support rolling upgrade between 2.x and 3.x
> ---
>
> Key: HDFS-11096
> URL: https://issues.apache.org/jira/browse/HDFS-11096
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rolling upgrades
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Sean Mackrory
>Priority: Blocker
> Attachments: HDFS-11096.001.patch
>
>
> trunk has a minimum software version of 3.0.0-alpha1. This means we can't
> rolling upgrade between branch-2 and trunk.
> This is a showstopper for large deployments. Unless there are very compelling
> reasons to break compatibility, let's restore the ability to rolling upgrade
> to 3.x releases.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143965#comment-16143965 ] Sean Mackrory edited comment on HDFS-11096 at 8/28/17 4:40 PM: --- So Docker support has been added for the rolling-upgrade and pull-over-http test. They're using the same Docker image as Yetus builds, etc. And they've been really robust lately. I've corrected the copyright headers at the top of the files, and I think dev-support/compat is a good place for these tests to live - but I'm open to other ideas as well. I've also added to the README - now that the scripts spin up the clusters on Docker, it's *really* easy to run these. The Python tests are all still working, but they did not seem to catch the previous incompatibility that prevented older clients from writing to newer DataNodes. There's also still a few TODOs or thing that don't work and it's not clear why. So definitely more work to be done, but there's value in the existing CLI compatibility tests. I'd like to get this put in the codebase and get some Jenkins jobs running on it soon. was (Author: mackrorysd): So Docker support has been added for the rolling-upgrade and pull-over-http test. They're using the same Docker image as Yetus builds, etc. And they've been really robust lately. I've corrected the copyright headers at the top of the files, and I think dev-support/compat is a good place for these tests to live - but I'm open to other ideas as well. The Python tests are all still working, but they did not seem to catch the previous incompatibility that prevented older clients from writing to newer DataNodes. There's also still a few TODOs or thing that don't work and it's not clear why. So definitely more work to be done, but there's value in the existing CLI compatibility tests. I'd like to get this put in the codebase and get some Jenkins jobs running on it soon. > Support rolling upgrade between 2.x and 3.x > --- > > Key: HDFS-11096 > URL: https://issues.apache.org/jira/browse/HDFS-11096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rolling upgrades >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Sean Mackrory >Priority: Blocker > Attachments: HDFS-11096.001.patch > > > trunk has a minimum software version of 3.0.0-alpha1. This means we can't > rolling upgrade between branch-2 and trunk. > This is a showstopper for large deployments. Unless there are very compelling > reasons to break compatibility, let's restore the ability to rolling upgrade > to 3.x releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HDFS-11096: - Status: Patch Available (was: Open) > Support rolling upgrade between 2.x and 3.x > --- > > Key: HDFS-11096 > URL: https://issues.apache.org/jira/browse/HDFS-11096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rolling upgrades >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Sean Mackrory >Priority: Blocker > Attachments: HDFS-11096.001.patch > > > trunk has a minimum software version of 3.0.0-alpha1. This means we can't > rolling upgrade between branch-2 and trunk. > This is a showstopper for large deployments. Unless there are very compelling > reasons to break compatibility, let's restore the ability to rolling upgrade > to 3.x releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HDFS-11096: - Attachment: HDFS-11096.001.patch So Docker support has been added for the rolling-upgrade and pull-over-http test. They're using the same Docker image as Yetus builds, etc. And they've been really robust lately. I've corrected the copyright headers at the top of the files, and I think dev-support/compat is a good place for these tests to live - but I'm open to other ideas as well. The Python tests are all still working, but they did not seem to catch the previous incompatibility that prevented older clients from writing to newer DataNodes. There's also still a few TODOs or thing that don't work and it's not clear why. So definitely more work to be done, but there's value in the existing CLI compatibility tests. I'd like to get this put in the codebase and get some Jenkins jobs running on it soon. > Support rolling upgrade between 2.x and 3.x > --- > > Key: HDFS-11096 > URL: https://issues.apache.org/jira/browse/HDFS-11096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rolling upgrades >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Sean Mackrory >Priority: Blocker > Attachments: HDFS-11096.001.patch > > > trunk has a minimum software version of 3.0.0-alpha1. This means we can't > rolling upgrade between branch-2 and trunk. > This is a showstopper for large deployments. Unless there are very compelling > reasons to break compatibility, let's restore the ability to rolling upgrade > to 3.x releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12333) Ozone: Extend Datanode web interface with SCM information
[ https://issues.apache.org/jira/browse/HDFS-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143874#comment-16143874 ] Elek, Marton commented on HDFS-12333: - thx, the idea [~msingh], I improved the patch. Now the ozone section will be hidden if ozone is disabled. Sent from JIRA Mobile > Ozone: Extend Datanode web interface with SCM information > - > > Key: HDFS-12333 > URL: https://issues.apache.org/jira/browse/HDFS-12333 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12333-HDFS-7240.001.patch, > HDFS-12333-HDFS-7240.002.patch, HDFS-12333-HDFS-7240.003.patch, > ozonedatanode.png > > > Current Datanode web interface shows information about the Block Pools: > * Namenode Address > * BlockPoolID > * ActorState > * Last Heartbeat > * Last Block Report > I propose in this jira to add the same information about the SCM (if the > datanode is a member of ozone cluster). It would help to check the current > state of the datanode (is ozone enabled? Is there an active connection to the > SCM?) > 1. Suggested information to display: > * SCM hostname/address > * EndpointState (GETVERSION, REGISTER, HEARTBEAT, SHUTDOWN) > * version (from the VersoinResponse) > * missedCount > They could be displayed with publishing JMX information from > SCMConnectionManager or EndpointStateMachines. > 2. StorageLocationReport[] from the ContainerLocationManager also should be > exposed over JMX and displayed on the web interface: > * id > * failed (bool) > * capacity > * scmUsed > * remaining > 3. Possible report of Last Heartbeat/Container report should be investigated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143870#comment-16143870
]
Brahma Reddy Battula commented on HDFS-11896:
-
Pushed to {{branch-2.8.2}} as well. Compiled the ran the testcase locally.
{noformat}
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.49 sec - in
org.apache.hadoop.hdfs.server.namenode.Te
stDeadDatanode
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Results :
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Non-dfsUsed will be doubled on dead node re-registration
>
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2, 2.8.3
>
> Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch,
> HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch,
> HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch,
> HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch,
> HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch,
> HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[ https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11896: Fix Version/s: 2.8.2 > Non-dfsUsed will be doubled on dead node re-registration > > > Key: HDFS-11896 > URL: https://issues.apache.org/jira/browse/HDFS-11896 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.3 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Blocker > Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2, 2.8.3 > > Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch, > HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch, > HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch, > HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch, > HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch, > HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch, > HDFS-11896.patch > > > *Scenario:* > i)Make you sure you've non-dfs data. > ii) Stop Datanode > iii) wait it becomes dead > iv) now restart and check the non-dfs data -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12333) Ozone: Extend Datanode web interface with SCM information
[ https://issues.apache.org/jira/browse/HDFS-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-12333: Attachment: HDFS-12333-HDFS-7240.003.patch > Ozone: Extend Datanode web interface with SCM information > - > > Key: HDFS-12333 > URL: https://issues.apache.org/jira/browse/HDFS-12333 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12333-HDFS-7240.001.patch, > HDFS-12333-HDFS-7240.002.patch, HDFS-12333-HDFS-7240.003.patch, > ozonedatanode.png > > > Current Datanode web interface shows information about the Block Pools: > * Namenode Address > * BlockPoolID > * ActorState > * Last Heartbeat > * Last Block Report > I propose in this jira to add the same information about the SCM (if the > datanode is a member of ozone cluster). It would help to check the current > state of the datanode (is ozone enabled? Is there an active connection to the > SCM?) > 1. Suggested information to display: > * SCM hostname/address > * EndpointState (GETVERSION, REGISTER, HEARTBEAT, SHUTDOWN) > * version (from the VersoinResponse) > * missedCount > They could be displayed with publishing JMX information from > SCMConnectionManager or EndpointStateMachines. > 2. StorageLocationReport[] from the ContainerLocationManager also should be > exposed over JMX and displayed on the web interface: > * id > * failed (bool) > * capacity > * scmUsed > * remaining > 3. Possible report of Last Heartbeat/Container report should be investigated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10391) Always enable NameNode service RPC port
[ https://issues.apache.org/jira/browse/HDFS-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143858#comment-16143858 ] Gergely Novák commented on HDFS-10391: -- I cannot reproduce the last failing unit test (it times out on Jenkins). > Always enable NameNode service RPC port > --- > > Key: HDFS-10391 > URL: https://issues.apache.org/jira/browse/HDFS-10391 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Reporter: Arpit Agarwal >Assignee: Gergely Novák > Labels: Incompatible > Attachments: HDFS-10391.001.patch, HDFS-10391.002.patch, > HDFS-10391.003.patch, HDFS-10391.004.patch, HDFS-10391.005.patch, > HDFS-10391.006.patch, HDFS-10391.007.patch, HDFS-10391.008.patch, > HDFS-10391.009.patch, HDFS-10391.010.patch, HDFS-10391.v5-v6-delta.patch > > > The NameNode should always be setup with a service RPC port so that it does > not have to be explicitly enabled by an administrator. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143847#comment-16143847
]
Hadoop QA commented on HDFS-12354:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 39s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12354 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884040/HDFS-12354-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux d10d630598e2 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 76a156e |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20899/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| unit |
[jira] [Updated] (HDFS-12364) [branch-2.8.2] Fix the Compile Error after HDFS-12299
[
https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12364:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.2
Status: Resolved (was: Patch Available)
Committed to {{branch-2.8.2}},[~yangjiandan] thanks for your contribution.
> [branch-2.8.2] Fix the Compile Error after HDFS-12299
> -
>
> Key: HDFS-12364
> URL: https://issues.apache.org/jira/browse/HDFS-12364
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Blocker
> Fix For: 2.8.2
>
> Attachments: HDFS-12364-branch-2.8.2.001.patch
>
>
> error line ：dn1.setHeartbeatsDisabledForTests(true)
> use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true);
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12364) [branch-2.8.2] Fix the Compile Error after HDFS-12299
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12364: Summary: [branch-2.8.2] Fix the Compile Error after HDFS-12299 (was: [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg) > [branch-2.8.2] Fix the Compile Error after HDFS-12299 > - > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Blocker > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
[
https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143804#comment-16143804
]
Hadoop QA commented on HDFS-12235:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} root: The patch generated 0 new + 70 unchanged - 4
fixed = 70 total (was 74) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 10s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 46s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
| | hadoop.metrics2.sink.TestFileSink |
| | hadoop.scm.TestArchive |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.ksm.TestKSMSQLCli |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12235 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884031/HDFS-12235-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 031f75cde889 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017
[jira] [Commented] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143771#comment-16143771
]
Brahma Reddy Battula commented on HDFS-11896:
-
IMO, this should be backported to {{branch-2.8.2}} as well..?
> Non-dfsUsed will be doubled on dead node re-registration
>
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Blocker
> Fix For: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.3
>
> Attachments: HDFS-11896-002.patch, HDFS-11896-003.patch,
> HDFS-11896-004.patch, HDFS-11896-005.patch, HDFS-11896-006.patch,
> HDFS-11896-007.patch, HDFS-11896-008.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896-branch-2.7-003.patch,
> HDFS-11896-branch-2.7-004.patch, HDFS-11896-branch-2.7-005.patch,
> HDFS-11896-branch-2.7-006.patch, HDFS-11896-branch-2.7-008.patch,
> HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11738) Hedged pread takes more time when block moved from initial locations
[ https://issues.apache.org/jira/browse/HDFS-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-11738: - Fix Version/s: (was: 2.8.2) 2.8.3 > Hedged pread takes more time when block moved from initial locations > > > Key: HDFS-11738 > URL: https://issues.apache.org/jira/browse/HDFS-11738 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3 > > Attachments: HDFS-11738-01.patch, HDFS-11738-02.patch, > HDFS-11738-03.patch, HDFS-11738-04.patch, HDFS-11738-05.patch, > HDFS-11738-branch-2.8-committed.patch, HDFS-11738-branch-2-committed.patch > > > Scenario : > Same as HDFS-11708. > During Hedge read, > 1. First two locations fails to read the data in hedged mode. > 2. chooseData refetches locations and adds a future to read from DN3. > 3. after adding future to DN3, main thread goes for refetching locations in > loop and stucks there till all 3 retries to fetch locations exhausted, which > consumes ~20 seconds with exponential retry time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12352) [branch-2] Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-12352: - Resolution: Duplicate Status: Resolved (was: Patch Available) Had committed this in HDFS-11530. Close this JIRA. > [branch-2] Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > --- > > Key: HDFS-12352 > URL: https://issues.apache.org/jira/browse/HDFS-12352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12352-branch-2.001.patch > > > This JIRA is to backport HDFS-11530 to branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143736#comment-16143736
]
Yiqun Lin commented on HDFS-11530:
--
Committed to branch-2 via HDFS-12352. Thanks [~vagarychen]!
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch,
> HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch,
> HDFS-11530.012.patch, HDFS-11530.013.patch, HDFS-11530.014.patch,
> HDFS-11530.015.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11530) Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[
https://issues.apache.org/jira/browse/HDFS-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11530:
-
Fix Version/s: 2.9.0
> Use HDFS specific network topology to choose datanode in
> BlockPlacementPolicyDefault
>
>
> Key: HDFS-11530
> URL: https://issues.apache.org/jira/browse/HDFS-11530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HDFS-11530.001.patch, HDFS-11530.002.patch,
> HDFS-11530.003.patch, HDFS-11530.004.patch, HDFS-11530.005.patch,
> HDFS-11530.006.patch, HDFS-11530.007.patch, HDFS-11530.008.patch,
> HDFS-11530.009.patch, HDFS-11530.010.patch, HDFS-11530.011.patch,
> HDFS-11530.012.patch, HDFS-11530.013.patch, HDFS-11530.014.patch,
> HDFS-11530.015.patch
>
>
> The work for {{chooseRandomWithStorageType}} has been merged in HDFS-11482.
> But this method is contained in new topology {{DFSNetworkTopology}} which is
> specified for HDFS. We should update this and let
> {{BlockPlacementPolicyDefault}} use the new way since the original way is
> inefficient.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[ https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143735#comment-16143735 ] Brahma Reddy Battula commented on HDFS-12084: - [~kihwal] thanks for taking a look. bq. but I see reserve RBW space gets stuck and never going downon certain datanode. are you telling about DN side reserveSpaceForReplica..? > Scheduled Count will not decrement when file is deleted before all IBR's > received > - > > Key: HDFS-12084 > URL: https://issues.apache.org/jira/browse/HDFS-12084 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch, > HDFS-12084-003.patch, HDFS-12084-branch-2.patch > > > When small files creation && deletion happens so frequently and DN's did not > report blocks to NN before deletion, then scheduled count will keep on > increment and which will not deleted as blocks are deleted. > *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be > more as so many operations. > when batchIBR enabled with committed allowed=1 this will be observed more. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12354:
-
Attachment: HDFS-12354-HDFS-7240.001.patch
Attach the initial patch. Additionally seperating the class {{ContainerStatus}}
into the independent class so that it can be used by other class.
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Attachments: HDFS-12354-HDFS-7240.001.patch
>
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12354:
-
Status: Patch Available (was: Open)
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
> Attachments: HDFS-12354-HDFS-7240.001.patch
>
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12258) ec -listPolicies should list all policies in system, no matter it's enabled or disabled
[
https://issues.apache.org/jira/browse/HDFS-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143664#comment-16143664
]
Hadoop QA commented on HDFS-12258:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestReconstructStripedFile |
| Timed out junit tests |
[jira] [Updated] (HDFS-12364) [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12364: Priority: Blocker (was: Major) > [branch-2.8.2] Compile > Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg > --- > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Blocker > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
[ https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-12235: -- Attachment: HDFS-12235-HDFS-7240.002.patch upload v2 patch to rebase code and address checkstyle/findbug issues. > Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions > --- > > Key: HDFS-12235 > URL: https://issues.apache.org/jira/browse/HDFS-12235 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Yuanbo Liu > Attachments: HDFS-12235-HDFS-7240.001.patch, > HDFS-12235-HDFS-7240.002.patch > > > KSM and SCM interaction for delete key operation, both KSM and SCM stores key > state info in a backlog, KSM needs to scan this log and send block-deletion > command to SCM, once SCM is fully aware of the message, KSM removes the key > completely from namespace. See more from the design doc under HDFS-11922, > this is task break down 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12364) [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[ https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143657#comment-16143657 ] Brahma Reddy Battula commented on HDFS-12364: - Test failures are unrelated, will commit soon. > [branch-2.8.2] Compile > Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg > --- > > Key: HDFS-12364 > URL: https://issues.apache.org/jira/browse/HDFS-12364 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.2 >Reporter: Jiandan Yang >Assignee: Jiandan Yang > Attachments: HDFS-12364-branch-2.8.2.001.patch > > > error line ：dn1.setHeartbeatsDisabledForTests(true) > use DataNodeTestUtils.setHeartbeatsDisabledForTests(dn1, true); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12366) Ozone: Refactor KSM metadata class names to avoid confusion
[
https://issues.apache.org/jira/browse/HDFS-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143653#comment-16143653
]
Hadoop QA commented on HDFS-12366:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 2s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.ozone.scm.node.TestNodeManager |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12366 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884013/HDFS-12366-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3fa0c86b3b47 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 76a156e |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20896/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20896/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20896/console |
| Powered
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[
https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143632#comment-16143632
]
Hadoop QA commented on HDFS-11968:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 14 unchanged - 0 fixed = 19 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.TestDFSInputStream |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestFileCreationEmpty |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Assigned] (HDFS-12367) Ozone: Too many open files error while running corona
[
https://issues.apache.org/jira/browse/HDFS-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh reassigned HDFS-12367:
Assignee: Mukul Kumar Singh
> Ozone: Too many open files error while running corona
> -
>
> Key: HDFS-12367
> URL: https://issues.apache.org/jira/browse/HDFS-12367
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, tools
>Reporter: Weiwei Yang
>Assignee: Mukul Kumar Singh
>
> Too many open files error keeps happening to me while using corona, I have
> simply setup a single node cluster and run corona to generate 1000 keys, but
> I keep getting following error
> {noformat}
> ./bin/hdfs corona -numOfThreads 1 -numOfVolumes 1 -numOfBuckets 1 -numOfKeys
> 1000
> 17/08/28 00:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 17/08/28 00:47:42 INFO tools.Corona: Number of Threads: 1
> 17/08/28 00:47:42 INFO tools.Corona: Mode: offline
> 17/08/28 00:47:42 INFO tools.Corona: Number of Volumes: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Buckets per Volume: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Keys per Bucket: 1000.
> 17/08/28 00:47:42 INFO rpc.OzoneRpcClient: Creating Volume: vol-0-05000, with
> wwei as owner and quota set to 1152921504606846976 bytes.
> 17/08/28 00:47:42 INFO tools.Corona: Starting progress bar Thread.
> ...
> ERROR tools.Corona: Exception while adding key: key-251-19293 in bucket:
> bucket-0-34960 of volume: vol-0-05000.
> java.io.IOException: Exception getting XceiverClient.
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:156)
> at
> org.apache.hadoop.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:122)
> at
> org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.getFromKsmKeyInfo(ChunkGroupOutputStream.java:289)
> at
> org.apache.hadoop.ozone.client.rpc.OzoneRpcClient.createKey(OzoneRpcClient.java:487)
> at
> org.apache.hadoop.ozone.tools.Corona$OfflineProcessor.run(Corona.java:352)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: failed to create a child event loop
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:144)
> ... 9 more
> Caused by: java.lang.IllegalStateException: failed to create a child event
> loop
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
> at
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
> at org.apache.hadoop.scm.XceiverClient.connect(XceiverClient.java:76)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:151)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:145)
> at
> com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> ... 12 more
> Caused by: io.netty.channel.ChannelException: failed to open a new selector
> at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
> at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> at
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
>
[jira] [Commented] (HDFS-12210) Block Storage: volume creation times out while creating 3TB volume because of too many containers
[
https://issues.apache.org/jira/browse/HDFS-12210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143571#comment-16143571
]
Hadoop QA commented on HDFS-12210:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 51s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12210 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884010/HDFS-12210-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 58cfd3792ac3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 76a156e |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20893/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| unit |
[jira] [Commented] (HDFS-12329) Ozone: Ratis: Readonly calls in XceiverClientRatis should use sendReadOnly
[
https://issues.apache.org/jira/browse/HDFS-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143569#comment-16143569
]
Hadoop QA commented on HDFS-12329:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 45s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12329 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884008/HDFS-12329-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2d0a320e63aa 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 76a156e |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20892/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20892/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
[jira] [Commented] (HDFS-12364) [branch-2.8.2] Compile Error：TestClientProtocolForPipelineRecovery#testUpdatePipeLineAfterDNReg
[
https://issues.apache.org/jira/browse/HDFS-12364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143561#comment-16143561
]
Hadoop QA commented on HDFS-12364:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8.2 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 5m
58s{color} | {color:red} root in branch-2.8.2 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
50s{color} | {color:red} hadoop-hdfs in branch-2.8.2 failed with JDK
v1.8.0_144. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
52s{color} | {color:red} hadoop-hdfs in branch-2.8.2 failed with JDK
v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} branch-2.8.2 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
51s{color} | {color:red} hadoop-hdfs in branch-2.8.2 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} hadoop-hdfs in branch-2.8.2 failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} branch-2.8.2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} branch-2.8.2 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_144 with
JDK v1.8.0_144 generated 0 new + 47 unchanged - 2 fixed = 47 total (was 49)
{color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131 with
JDK v1.7.0_131 generated 0 new + 49 unchanged - 2 fixed = 49 total (was 51)
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 28s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_131. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| JDK v1.7.0_131 Failed junit tests |
hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.datanode.TestFsDatasetCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:d946387 |
| JIRA Issue | HDFS-12364 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884005/HDFS-12364-branch-2.8.2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[
https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143535#comment-16143535
]
Takanobu Asanuma commented on HDFS-11964:
-
Thanks for reviewing and the comment, [~drankye]!
> We could also compare it with the default RS coder.
{{RSRawDecoder#doDecode}} (the default RS coder) also receives unnecessary
input arrays when it preads, but it does like a post fix, and decode blocks
well.
> I wonder if we could get right at the first place.
I agree with you. Unlike {{StatefulStripeReader#prepareDecodeInputs()}},
{{PositionStripeReader#prepareDecodeInputs()}} seems not to generate necessary
and sufficient input arrays. I will investigate it more. Thanks!
> RS-6-3-LEGACY has a decoding bug when it is used for pread
> --
>
> Key: HDFS-11964
> URL: https://issues.apache.org/jira/browse/HDFS-11964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Takanobu Asanuma
> Attachments: HDFS-11964.1.patch
>
>
> TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on
> trunk:
> {code}
> Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<<
> FAILURE! - in
> org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
> testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy)
> Time elapsed: 1.265 sec <<< FAILURE!
> org.junit.internal.ArrayComparisonFailure: arrays first differed at element
> [327680]; expected:<-36> but was:<2>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at org.junit.Assert.assertArrayEquals(Assert.java:305)
> at
> org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
[
https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143530#comment-16143530
]
Yiqun Lin commented on HDFS-12354:
--
I agree with your proposal, and will attach initial patch in two days. Thank
you.
> Ozone: Shuffle container list for datanode BlockDeletingService
> ---
>
> Key: HDFS-12354
> URL: https://issues.apache.org/jira/browse/HDFS-12354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Weiwei Yang
>Assignee: Yiqun Lin
>
> {{BlockDeletingService}} is a per-datanode container block deleting service
> takes in charge of the "real" deletion of ozone blocks. It spawns a worker
> thread per container and delete blocks/chunks from disk as background
> threads. The number of threads currently is throttled by
> {{ozone.block.deleting.container.limit.per.interval}}, but there is a
> potential problem. Containers are sorted so it always fetch same of
> containers, we need to fix this by creating an API in
> {{ContainerManagerImpl}} to get a shuffled list of containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12258) ec -listPolicies should list all policies in system, no matter it's enabled or disabled
[ https://issues.apache.org/jira/browse/HDFS-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhou updated HDFS-12258: Attachment: HDFS-12258.06.patch Fix checkstyle issue, the test failures are not related to this patch. Thanks! > ec -listPolicies should list all policies in system, no matter it's enabled > or disabled > --- > > Key: HDFS-12258 > URL: https://issues.apache.org/jira/browse/HDFS-12258 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: Wei Zhou > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-12258.01.patch, HDFS-12258.02.patch, > HDFS-12258.03.patch, HDFS-12258.04.patch, HDFS-12258.05.patch, > HDFS-12258.06.patch > > > ec -listPolicies should list all policies in system, no matter it's enabled > or disabled -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143523#comment-16143523
]
Hadoop QA commented on HDFS-12365:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 48s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12365 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884011/HDFS-12365-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d290676976b1 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 76a156e |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20894/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20894/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20894/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL:
[jira] [Updated] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[
https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-11968:
-
Attachment: HDFS-11968.004.patch
> ViewFS: StoragePolicies commands fail with HDFS federation
> --
>
> Key: HDFS-11968
> URL: https://issues.apache.org/jira/browse/HDFS-11968
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.1
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch,
> HDFS-11968.003.patch, HDFS-11968.004.patch
>
>
> hdfs storagepolicies command fails with HDFS federation.
> For storage policies commands, a given user path should be resolved to a HDFS
> path and
> storage policy command should be applied onto the resolved HDFS path.
> {code}
> static DistributedFileSystem getDFS(Configuration conf)
> throws IOException {
> FileSystem fs = FileSystem.get(conf);
> if (!(fs instanceof DistributedFileSystem)) {
> throw new IllegalArgumentException("FileSystem " + fs.getUri() +
> " is not an HDFS file system");
> }
> return (DistributedFileSystem)fs;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12367) Ozone: Too many open files error while running corona
[
https://issues.apache.org/jira/browse/HDFS-12367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143518#comment-16143518
]
Weiwei Yang commented on HDFS-12367:
Looks like there are some resource leaks? :P
> Ozone: Too many open files error while running corona
> -
>
> Key: HDFS-12367
> URL: https://issues.apache.org/jira/browse/HDFS-12367
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, tools
>Reporter: Weiwei Yang
>
> Too many open files error keeps happening to me while using corona, I have
> simply setup a single node cluster and run corona to generate 1000 keys, but
> I keep getting following error
> {noformat}
> ./bin/hdfs corona -numOfThreads 1 -numOfVolumes 1 -numOfBuckets 1 -numOfKeys
> 1000
> 17/08/28 00:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 17/08/28 00:47:42 INFO tools.Corona: Number of Threads: 1
> 17/08/28 00:47:42 INFO tools.Corona: Mode: offline
> 17/08/28 00:47:42 INFO tools.Corona: Number of Volumes: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Buckets per Volume: 1.
> 17/08/28 00:47:42 INFO tools.Corona: Number of Keys per Bucket: 1000.
> 17/08/28 00:47:42 INFO rpc.OzoneRpcClient: Creating Volume: vol-0-05000, with
> wwei as owner and quota set to 1152921504606846976 bytes.
> 17/08/28 00:47:42 INFO tools.Corona: Starting progress bar Thread.
> ...
> ERROR tools.Corona: Exception while adding key: key-251-19293 in bucket:
> bucket-0-34960 of volume: vol-0-05000.
> java.io.IOException: Exception getting XceiverClient.
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:156)
> at
> org.apache.hadoop.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:122)
> at
> org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.getFromKsmKeyInfo(ChunkGroupOutputStream.java:289)
> at
> org.apache.hadoop.ozone.client.rpc.OzoneRpcClient.createKey(OzoneRpcClient.java:487)
> at
> org.apache.hadoop.ozone.tools.Corona$OfflineProcessor.run(Corona.java:352)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: failed to create a child event loop
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
> at
> com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
> at
> org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:144)
> ... 9 more
> Caused by: java.lang.IllegalStateException: failed to create a child event
> loop
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
> at
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44)
> at
> io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
> at org.apache.hadoop.scm.XceiverClient.connect(XceiverClient.java:76)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:151)
> at
> org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:145)
> at
> com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> ... 12 more
> Caused by: io.netty.channel.ChannelException: failed to open a new selector
> at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
> at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> at
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
> ...
[jira] [Created] (HDFS-12367) Ozone: Too many open files error while running corona
Weiwei Yang created HDFS-12367:
--
Summary: Ozone: Too many open files error while running corona
Key: HDFS-12367
URL: https://issues.apache.org/jira/browse/HDFS-12367
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone, tools
Reporter: Weiwei Yang
Too many open files error keeps happening to me while using corona, I have
simply setup a single node cluster and run corona to generate 1000 keys, but I
keep getting following error
{noformat}
./bin/hdfs corona -numOfThreads 1 -numOfVolumes 1 -numOfBuckets 1 -numOfKeys
1000
17/08/28 00:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
17/08/28 00:47:42 INFO tools.Corona: Number of Threads: 1
17/08/28 00:47:42 INFO tools.Corona: Mode: offline
17/08/28 00:47:42 INFO tools.Corona: Number of Volumes: 1.
17/08/28 00:47:42 INFO tools.Corona: Number of Buckets per Volume: 1.
17/08/28 00:47:42 INFO tools.Corona: Number of Keys per Bucket: 1000.
17/08/28 00:47:42 INFO rpc.OzoneRpcClient: Creating Volume: vol-0-05000, with
wwei as owner and quota set to 1152921504606846976 bytes.
17/08/28 00:47:42 INFO tools.Corona: Starting progress bar Thread.
...
ERROR tools.Corona: Exception while adding key: key-251-19293 in bucket:
bucket-0-34960 of volume: vol-0-05000.
java.io.IOException: Exception getting XceiverClient.
at
org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:156)
at
org.apache.hadoop.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:122)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.getFromKsmKeyInfo(ChunkGroupOutputStream.java:289)
at
org.apache.hadoop.ozone.client.rpc.OzoneRpcClient.createKey(OzoneRpcClient.java:487)
at
org.apache.hadoop.ozone.tools.Corona$OfflineProcessor.run(Corona.java:352)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.IllegalStateException: failed to create a child event loop
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at
org.apache.hadoop.scm.XceiverClientManager.getClient(XceiverClientManager.java:144)
... 9 more
Caused by: java.lang.IllegalStateException: failed to create a child event loop
at
io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
at
io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
at
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61)
at
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52)
at
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44)
at
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
at org.apache.hadoop.scm.XceiverClient.connect(XceiverClient.java:76)
at
org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:151)
at
org.apache.hadoop.scm.XceiverClientManager$2.call(XceiverClientManager.java:145)
at
com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
... 12 more
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:128)
at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
at
io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
at
io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
... 25 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:130)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:69)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
[jira] [Updated] (HDFS-12366) Ozone: Refactor KSM metadata class names to avoid confusion
[
https://issues.apache.org/jira/browse/HDFS-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12366:
---
Status: Patch Available (was: Open)
> Ozone: Refactor KSM metadata class names to avoid confusion
> ---
>
> Key: HDFS-12366
> URL: https://issues.apache.org/jira/browse/HDFS-12366
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Trivial
> Attachments: HDFS-12366-HDFS-7240.001.patch
>
>
> Propose to rename 2 classes in package {{org.apache.hadoop.ozone.ksm}}
> * MetadataManager -> KSMMetadataManager
> * MetadataManagerImpl -> KSMMetadataManagerImpl
> this is to avoid confusions with ozone metadata store classes, such as
> {{MetadataKeyFilters}}, {{MetadataStore}} and {{MetadataStoreBuilder}} etc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12366) Ozone: Refactor KSM metadata class names to avoid confusion
[
https://issues.apache.org/jira/browse/HDFS-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12366:
---
Attachment: HDFS-12366-HDFS-7240.001.patch
> Ozone: Refactor KSM metadata class names to avoid confusion
> ---
>
> Key: HDFS-12366
> URL: https://issues.apache.org/jira/browse/HDFS-12366
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Trivial
> Attachments: HDFS-12366-HDFS-7240.001.patch
>
>
> Propose to rename 2 classes in package {{org.apache.hadoop.ozone.ksm}}
> * MetadataManager -> KSMMetadataManager
> * MetadataManagerImpl -> KSMMetadataManagerImpl
> this is to avoid confusions with ozone metadata store classes, such as
> {{MetadataKeyFilters}}, {{MetadataStore}} and {{MetadataStoreBuilder}} etc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12366) Ozone: Refactor KSM metadata class names to avoid confusion
Weiwei Yang created HDFS-12366:
--
Summary: Ozone: Refactor KSM metadata class names to avoid
confusion
Key: HDFS-12366
URL: https://issues.apache.org/jira/browse/HDFS-12366
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Trivial
Propose to rename 2 classes in package {{org.apache.hadoop.ozone.ksm}}
* MetadataManager -> KsmMetadataManager
* MetadataManagerImpl -> KsmMetadataManagerImpl
this is to avoid confusions with ozone metadata store classes, such as
{{MetadataKeyFilters}}, {{MetadataStore}} and {{MetadataStoreBuilder}} etc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12366) Ozone: Refactor KSM metadata class names to avoid confusion
[
https://issues.apache.org/jira/browse/HDFS-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12366:
---
Description:
Propose to rename 2 classes in package {{org.apache.hadoop.ozone.ksm}}
* MetadataManager -> KSMMetadataManager
* MetadataManagerImpl -> KSMMetadataManagerImpl
this is to avoid confusions with ozone metadata store classes, such as
{{MetadataKeyFilters}}, {{MetadataStore}} and {{MetadataStoreBuilder}} etc.
was:
Propose to rename 2 classes in package {{org.apache.hadoop.ozone.ksm}}
* MetadataManager -> KsmMetadataManager
* MetadataManagerImpl -> KsmMetadataManagerImpl
this is to avoid confusions with ozone metadata store classes, such as
{{MetadataKeyFilters}}, {{MetadataStore}} and {{MetadataStoreBuilder}} etc.
> Ozone: Refactor KSM metadata class names to avoid confusion
> ---
>
> Key: HDFS-12366
> URL: https://issues.apache.org/jira/browse/HDFS-12366
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Trivial
>
> Propose to rename 2 classes in package {{org.apache.hadoop.ozone.ksm}}
> * MetadataManager -> KSMMetadataManager
> * MetadataManagerImpl -> KSMMetadataManagerImpl
> this is to avoid confusions with ozone metadata store classes, such as
> {{MetadataKeyFilters}}, {{MetadataStore}} and {{MetadataStoreBuilder}} etc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12357) Let NameNode to bypass external attribute provider for special user
[
https://issues.apache.org/jira/browse/HDFS-12357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143514#comment-16143514
]
Hadoop QA commented on HDFS-12357:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 465 unchanged - 0 fixed = 470 total (was 465) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
48s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Write to static field
org.apache.hadoop.hdfs.server.namenode.FSDirectory.usersToBypassExtAttrProvider
from instance method new
org.apache.hadoop.hdfs.server.namenode.FSDirectory(FSNamesystem, Configuration)
At FSDirectory.java:from instance method new
org.apache.hadoop.hdfs.server.namenode.FSDirectory(FSNamesystem, Configuration)
At FSDirectory.java:[line 371] |
| Failed junit tests | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
||
[jira] [Updated] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12365:
---
Status: Patch Available (was: Open)
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL: https://issues.apache.org/jira/browse/HDFS-12365
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12365-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a key in ozone with corona (this delegates the call to
> OzoneRpcClient), e.g
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs corona -numOfThreads 1
> -numOfVolumes 1 -numOfBuckets 1 -numOfKeys 1
> {code}
> 2. Run listVolume
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs oz -listVolume
> http://localhost:9864 -user wwei
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-31437",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-38900",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {code}
> Note, the time displayed in {{createdOn}} are both incorrect {{Thu, 01 Jan
> 1970 00:00:00 GMT}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12365:
---
Attachment: HDFS-12365-HDFS-7240.001.patch
Attached a pretty straight forward fix, it only adds a line to add
{{createdOn}} time while creating the request. Please kindly review. Thanks.
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL: https://issues.apache.org/jira/browse/HDFS-12365
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12365-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a key in ozone with corona (this delegates the call to
> OzoneRpcClient), e.g
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs corona -numOfThreads 1
> -numOfVolumes 1 -numOfBuckets 1 -numOfKeys 1
> {code}
> 2. Run listVolume
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs oz -listVolume
> http://localhost:9864 -user wwei
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-31437",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-38900",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {code}
> Note, the time displayed in {{createdOn}} are both incorrect {{Thu, 01 Jan
> 1970 00:00:00 GMT}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
