[jira] [Commented] (HDFS-11183) Document XOR policy XOR-2-1-64k support in HDFS
[ https://issues.apache.org/jira/browse/HDFS-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898856#comment-15898856 ] SammiChen commented on HDFS-11183: -- Hi, [~andrew.wang], I see you have mentioned XOR-2-1-64K policy in HDFSErasureCoding.md in last check-in for HDFS-11427. So I'm going to close this JIRA. Agree? > Document XOR policy XOR-2-1-64k support in HDFS > --- > > Key: HDFS-11183 > URL: https://issues.apache.org/jira/browse/HDFS-11183 > Project: Hadoop HDFS > Issue Type: Task >Affects Versions: 3.0.0-alpha2 >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-must-do > > A new EC policy XOR-2-1-64k is added in HDFS. This task is to update HDFS > related documents about how to use this policy -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9705) Refine the behaviour of getFileChecksum when length = 0
[
https://issues.apache.org/jira/browse/HDFS-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SammiChen updated HDFS-9705:
Attachment: HDFS-9705-v7.patch
The previous comment is kind of misleading. When the crcType is "NULL" which
means no checksum for files, the execution will go through the "default" branch
of {makeFinalResult} function.
> Refine the behaviour of getFileChecksum when length = 0
> ---
>
> Key: HDFS-9705
> URL: https://issues.apache.org/jira/browse/HDFS-9705
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Kai Zheng
>Priority: Minor
> Attachments: HDFS-9705-v1.patch, HDFS-9705-v2.patch,
> HDFS-9705-v3.patch, HDFS-9705-v4.patch, HDFS-9705-v5.patch,
> HDFS-9705-v6.patch, HDFS-9705-v7.patch
>
>
> {{FileSystem#getFileChecksum}} may accept {{length}} parameter and 0 is a
> valid value. Currently it will return {{null}} when length is 0, in the
> following code block:
> {code}
> //compute file MD5
> final MD5Hash fileMD5 = MD5Hash.digest(md5out.getData());
> switch (crcType) {
> case CRC32:
> return new MD5MD5CRC32GzipFileChecksum(bytesPerCRC,
> crcPerBlock, fileMD5);
> case CRC32C:
> return new MD5MD5CRC32CastagnoliFileChecksum(bytesPerCRC,
> crcPerBlock, fileMD5);
> default:
> // If there is no block allocated for the file,
> // return one with the magic entry that matches what previous
> // hdfs versions return.
> if (locatedblocks.size() == 0) {
> return new MD5MD5CRC32GzipFileChecksum(0, 0, fileMD5);
> }
> // we should never get here since the validity was checked
> // when getCrcType() was called above.
> return null;
> }
> {code}
> The comment says "we should never get here since the validity was checked"
> but it does. As we're using the MD5-MD5-X approach, and {{EMPTY--CONTENT}}
> actually is a valid case in which the MD5 value is
> {{d41d8cd98f00b204e9800998ecf8427e}}, so suggest we return a reasonable value
> other than null. At least some useful information in the returned value can
> be seen, like values from block checksum header.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-9705) Refine the behaviour of getFileChecksum when length = 0
[
https://issues.apache.org/jira/browse/HDFS-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898838#comment-15898838
]
SammiChen edited comment on HDFS-9705 at 3/7/17 6:45 AM:
-
The previous comment is kind of misleading. When the crcType is "NULL" which
means no checksum for files, the execution will go through the "default" branch
of {{makeFinalResult}} function.
was (Author: sammi):
The previous comment is kind of misleading. When the crcType is "NULL" which
means no checksum for files, the execution will go through the "default" branch
of {makeFinalResult} function.
> Refine the behaviour of getFileChecksum when length = 0
> ---
>
> Key: HDFS-9705
> URL: https://issues.apache.org/jira/browse/HDFS-9705
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Kai Zheng
>Priority: Minor
> Attachments: HDFS-9705-v1.patch, HDFS-9705-v2.patch,
> HDFS-9705-v3.patch, HDFS-9705-v4.patch, HDFS-9705-v5.patch,
> HDFS-9705-v6.patch, HDFS-9705-v7.patch
>
>
> {{FileSystem#getFileChecksum}} may accept {{length}} parameter and 0 is a
> valid value. Currently it will return {{null}} when length is 0, in the
> following code block:
> {code}
> //compute file MD5
> final MD5Hash fileMD5 = MD5Hash.digest(md5out.getData());
> switch (crcType) {
> case CRC32:
> return new MD5MD5CRC32GzipFileChecksum(bytesPerCRC,
> crcPerBlock, fileMD5);
> case CRC32C:
> return new MD5MD5CRC32CastagnoliFileChecksum(bytesPerCRC,
> crcPerBlock, fileMD5);
> default:
> // If there is no block allocated for the file,
> // return one with the magic entry that matches what previous
> // hdfs versions return.
> if (locatedblocks.size() == 0) {
> return new MD5MD5CRC32GzipFileChecksum(0, 0, fileMD5);
> }
> // we should never get here since the validity was checked
> // when getCrcType() was called above.
> return null;
> }
> {code}
> The comment says "we should never get here since the validity was checked"
> but it does. As we're using the MD5-MD5-X approach, and {{EMPTY--CONTENT}}
> actually is a valid case in which the MD5 value is
> {{d41d8cd98f00b204e9800998ecf8427e}}, so suggest we return a reasonable value
> other than null. At least some useful information in the returned value can
> be seen, like values from block checksum header.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11158) Ozone: TestDatanodeStateMachine is failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-11158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898797#comment-15898797 ] Anu Engineer commented on HDFS-11158: - +1, I will add another test case to make sure we can launch the state machine in the thread. It is more important to get a consistent working test case without race conditions. > Ozone: TestDatanodeStateMachine is failing intermittently > - > > Key: HDFS-11158 > URL: https://issues.apache.org/jira/browse/HDFS-11158 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11158-HDFS-7240.001.patch > > > Failed in HDFS-11138 patch v6 run -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11451) Ozone: Add protobuf definitions for container reports
[ https://issues.apache.org/jira/browse/HDFS-11451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898788#comment-15898788 ] Anu Engineer commented on HDFS-11451: - [~xyao] verbally approved checking the patch v4 since jenkins is not working and all tests pass locally. Committing based on that. > Ozone: Add protobuf definitions for container reports > - > > Key: HDFS-11451 > URL: https://issues.apache.org/jira/browse/HDFS-11451 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11451-HDFS-7240.001.patch, > HDFS-11451-HDFS-7240.002.patch, HDFS-11451-HDFS-7240.003.patch, > HDFS-11451-HDFS-7240.004.patch > > > Container reports are send from datanodes describing the state of the > container. Since a full container report can be as large as 2 MB, the > container reports are send by datanode only when SCM approves it. > In this change, datanode informs the SCM that it has a container report > ready. SCM based on load will send a command to datanode to send the actual > report. The protobuf classes and plumbing required for that change is part of > this patch. This whole container reports handling will be broken into > multiple JIRAs to make it easy to code review. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11158) Ozone: TestDatanodeStateMachine is failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-11158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer reassigned HDFS-11158: --- Assignee: Weiwei Yang (was: Anu Engineer) > Ozone: TestDatanodeStateMachine is failing intermittently > - > > Key: HDFS-11158 > URL: https://issues.apache.org/jira/browse/HDFS-11158 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Weiwei Yang > Attachments: HDFS-11158-HDFS-7240.001.patch > > > Failed in HDFS-11138 patch v6 run -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11508) Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state
[
https://issues.apache.org/jira/browse/HDFS-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-11508:
-
Status: Patch Available (was: Open)
> Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket
> is in TIME_WAIT state
> ---
>
> Key: HDFS-11508
> URL: https://issues.apache.org/jira/browse/HDFS-11508
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11508.001.patch
>
>
> Bind can fail in SimpleTCPServer & Portmap because socket is in TIME_WAIT
> state.
> Socket options should be changed here to use the setReuseAddress option.
> {noformat}
> 2017-03-05 08:37:13,283 INFO oncrpc.SimpleUdpServer
> (SimpleUdpServer.java:run(73)) - Started listening to UDP requests at port
> 4242 for Rpc program: mountd at localhost:4242 with workerCount 1
> 2017-03-05 08:37:13,298 FATAL mount.MountdBase
> (MountdBase.java:startTCPServer(85)) - Failed to start the TCP server.
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> 0.0.0.0/0.0.0.0:4242
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
> at org.apache.hadoop.oncrpc.SimpleTcpServer.run(SimpleTcpServer.java:86)
> at org.apache.hadoop.mount.MountdBase.startTCPServer(MountdBase.java:83)
> at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:98)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-03-05 08:37:13,304 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
> Exiting with status 1
> 2017-03-05 08:37:13,308 INFO nfs3.Nfs3Base (LogAdapter.java:info(45)) -
> SHUTDOWN_MSG:
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11508) Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state
[
https://issues.apache.org/jira/browse/HDFS-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-11508:
-
Attachment: HDFS-11508.001.patch
Tested this patch manually by starting NFS gateway and portmap multiple times
> Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket
> is in TIME_WAIT state
> ---
>
> Key: HDFS-11508
> URL: https://issues.apache.org/jira/browse/HDFS-11508
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11508.001.patch
>
>
> Bind can fail in SimpleTCPServer & Portmap because socket is in TIME_WAIT
> state.
> Socket options should be changed here to use the setReuseAddress option.
> {noformat}
> 2017-03-05 08:37:13,283 INFO oncrpc.SimpleUdpServer
> (SimpleUdpServer.java:run(73)) - Started listening to UDP requests at port
> 4242 for Rpc program: mountd at localhost:4242 with workerCount 1
> 2017-03-05 08:37:13,298 FATAL mount.MountdBase
> (MountdBase.java:startTCPServer(85)) - Failed to start the TCP server.
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> 0.0.0.0/0.0.0.0:4242
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
> at org.apache.hadoop.oncrpc.SimpleTcpServer.run(SimpleTcpServer.java:86)
> at org.apache.hadoop.mount.MountdBase.startTCPServer(MountdBase.java:83)
> at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:98)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
> at
> org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-03-05 08:37:13,304 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
> Exiting with status 1
> 2017-03-05 08:37:13,308 INFO nfs3.Nfs3Base (LogAdapter.java:info(45)) -
> SHUTDOWN_MSG:
> /
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11465) Rename BlockType#CONTIGUOUS to BlockType#REPLICATED
[ https://issues.apache.org/jira/browse/HDFS-11465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-11465: -- Status: Patch Available (was: Open) > Rename BlockType#CONTIGUOUS to BlockType#REPLICATED > --- > > Key: HDFS-11465 > URL: https://issues.apache.org/jira/browse/HDFS-11465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Yuanbo Liu > Attachments: HDFS-11465.001.patch > > > HDFS-10759 introduced the a BlockType enum to INodeFile, with possible values > CONTIGUOUS or STRIPED. > Since HDFS-8030 wants to implement "contiguous EC", CONTIGUOUS isn't an > appropriate name. I propose we rename CONTIGUOUS to REPLICATED for clarity. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11465) Rename BlockType#CONTIGUOUS to BlockType#REPLICATED
[ https://issues.apache.org/jira/browse/HDFS-11465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanbo Liu updated HDFS-11465: -- Attachment: HDFS-11465.001.patch upload v1 patch for this JIRA, basically there are two changes here: * rename BlockType#CONTIGUOUS to BlockType#REPLICATED * rename BlockInfoContiguous.java to BlockInfoReplicated.java > Rename BlockType#CONTIGUOUS to BlockType#REPLICATED > --- > > Key: HDFS-11465 > URL: https://issues.apache.org/jira/browse/HDFS-11465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Yuanbo Liu > Attachments: HDFS-11465.001.patch > > > HDFS-10759 introduced the a BlockType enum to INodeFile, with possible values > CONTIGUOUS or STRIPED. > Since HDFS-8030 wants to implement "contiguous EC", CONTIGUOUS isn't an > appropriate name. I propose we rename CONTIGUOUS to REPLICATED for clarity. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11497) Ozone: Fix Container test regression.
[ https://issues.apache.org/jira/browse/HDFS-11497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11497: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~xyao] Thanks for the reviews. I have committed this to the feature branch. > Ozone: Fix Container test regression. > -- > > Key: HDFS-11497 > URL: https://issues.apache.org/jira/browse/HDFS-11497 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11497-HDFS-7240.001.patch > > > One the recent commits caused a regression in the container HB path. This > JIRA fixes it. Thanks to [~vagarychen] for bringing it to my attention. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898769#comment-15898769
]
Masatake Iwasaki commented on HDFS-11499:
-
Thanks for working on this, [~lukmajercak] and [~manojg].
While testing the 03 patch, the added
testDecommissionWithOpenFileAndDatanodeFailing intermittently timeouts waiting
for decomission. I'm looking into the cause.
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.03.patch,
> HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898770#comment-15898770
]
Surendra Singh Lilhore commented on HDFS-10838:
---
Thanks [~arpitagarwal] for review and commit ...
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, HDFS-10838.006.patch, NN_UI.png,
> NN_UI_relative_time_minutes.png, NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11497) Ozone: Fix Container test regression.
[ https://issues.apache.org/jira/browse/HDFS-11497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898763#comment-15898763 ] Anu Engineer commented on HDFS-11497: - None of these failures are reproducible on local machine, and it looks like Jenkins machines ran out of memory. So I am going to commit this fix. > Ozone: Fix Container test regression. > -- > > Key: HDFS-11497 > URL: https://issues.apache.org/jira/browse/HDFS-11497 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-11497-HDFS-7240.001.patch > > > One the recent commits caused a regression in the container HB path. This > JIRA fixes it. Thanks to [~vagarychen] for bringing it to my attention. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-11499:
--
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 3.0.0-alpha2, 2.7.3, 2.7.2, 2.7.1
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.03.patch,
> HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Majercak updated HDFS-11499:
--
Status: Open (was: Patch Available)
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 3.0.0-alpha2, 2.7.3, 2.7.2, 2.7.1
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.03.patch,
> HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11508) Fix bind failure in SimpleTCPServer & Portmap where bind fails because socket is in TIME_WAIT state
Mukul Kumar Singh created HDFS-11508:
Summary: Fix bind failure in SimpleTCPServer & Portmap where bind
fails because socket is in TIME_WAIT state
Key: HDFS-11508
URL: https://issues.apache.org/jira/browse/HDFS-11508
Project: Hadoop HDFS
Issue Type: Bug
Components: nfs
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Bind can fail in SimpleTCPServer & Portmap because socket is in TIME_WAIT state.
Socket options should be changed here to use the setReuseAddress option.
{noformat}
2017-03-05 08:37:13,283 INFO oncrpc.SimpleUdpServer
(SimpleUdpServer.java:run(73)) - Started listening to UDP requests at port 4242
for Rpc program: mountd at localhost:4242 with workerCount 1
2017-03-05 08:37:13,298 FATAL mount.MountdBase
(MountdBase.java:startTCPServer(85)) - Failed to start the TCP server.
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:4242
at
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at org.apache.hadoop.oncrpc.SimpleTcpServer.run(SimpleTcpServer.java:86)
at org.apache.hadoop.mount.MountdBase.startTCPServer(MountdBase.java:83)
at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:98)
at
org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
at
org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-03-05 08:37:13,304 INFO util.ExitUtil (ExitUtil.java:terminate(124)) -
Exiting with status 1
2017-03-05 08:37:13,308 INFO nfs3.Nfs3Base (LogAdapter.java:info(45)) -
SHUTDOWN_MSG:
/
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11502) dn.js set datanode UI to window.location.hostname, it should use jmx "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get tag.hostname
[
https://issues.apache.org/jira/browse/HDFS-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898730#comment-15898730
]
Hadoop QA commented on HDFS-11502:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11502 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856430/HDFS-11502.001.patch |
| Optional Tests | asflicense |
| uname | Linux d535ef6d5eb9 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1441398 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/18610/artifact/patchprocess/whitespace-eol.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18610/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> dn.js set datanode UI to window.location.hostname, it should use jmx
> "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get
> tag.hostname
> --
>
> Key: HDFS-11502
> URL: https://issues.apache.org/jira/browse/HDFS-11502
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.2, 2.7.3
> Environment: all
>Reporter: Jeffrey E Rodriguez
>Assignee: Jeffrey E Rodriguez
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2
>
> Attachments: HDFS-11502.001.patch
>
>
> Datanode UI calls dn.js which loads properties for datanode..
> dn.js sets datanode UI to window.location.hostname, it should use jmx
> "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get
> tag.hostname.
> The issue is that if we proxy datanode UI we would get the proxy as the
> hostname because UI code uses window.location.hostname instead of the actual
> datanode hostname.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11502) dn.js set datanode UI to window.location.hostname, it should use jmx "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get tag.hostname
[ https://issues.apache.org/jira/browse/HDFS-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey E Rodriguez updated HDFS-11502: Fix Version/s: 2.7.3 3.0.0-alpha1 3.0.0-alpha2 2.8.0 Status: Patch Available (was: In Progress) > dn.js set datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname > -- > > Key: HDFS-11502 > URL: https://issues.apache.org/jira/browse/HDFS-11502 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.3, 2.7.2 > Environment: all >Reporter: Jeffrey E Rodriguez >Assignee: Jeffrey E Rodriguez > Fix For: 2.8.0, 3.0.0-alpha2, 3.0.0-alpha1, 2.7.3 > > Attachments: HDFS-11502.001.patch > > > Datanode UI calls dn.js which loads properties for datanode.. > dn.js sets datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname. > The issue is that if we proxy datanode UI we would get the proxy as the > hostname because UI code uses window.location.hostname instead of the actual > datanode hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11502) dn.js set datanode UI to window.location.hostname, it should use jmx "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get tag.hostname
[ https://issues.apache.org/jira/browse/HDFS-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey E Rodriguez updated HDFS-11502: Attachment: HDFS-11502.001.patch Patch fixes Datanode html UI so we use datahostname tag.Hostname from JvmMetrics jmx instead of relying on window.location.hostname. window.location.hostname which in a case where we proxy datanode UI would return host of proxy instead. Since we want the real hostname of datanode it may be preferable to use actual datanode jmx data. > dn.js set datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname > -- > > Key: HDFS-11502 > URL: https://issues.apache.org/jira/browse/HDFS-11502 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.2, 2.7.3 > Environment: all >Reporter: Jeffrey E Rodriguez >Assignee: Jeffrey E Rodriguez > Attachments: HDFS-11502.001.patch > > > Datanode UI calls dn.js which loads properties for datanode.. > dn.js sets datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname. > The issue is that if we proxy datanode UI we would get the proxy as the > hostname because UI code uses window.location.hostname instead of the actual > datanode hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-11502) dn.js set datanode UI to window.location.hostname, it should use jmx "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get tag.hostname
[ https://issues.apache.org/jira/browse/HDFS-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-11502 started by Jeffrey E Rodriguez. --- > dn.js set datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname > -- > > Key: HDFS-11502 > URL: https://issues.apache.org/jira/browse/HDFS-11502 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.2, 2.7.3 > Environment: all >Reporter: Jeffrey E Rodriguez >Assignee: Jeffrey E Rodriguez > > Datanode UI calls dn.js which loads properties for datanode.. > dn.js sets datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname. > The issue is that if we proxy datanode UI we would get the proxy as the > hostname because UI code uses window.location.hostname instead of the actual > datanode hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898700#comment-15898700
]
Yiqun Lin commented on HDFS-11499:
--
You are right, I misreading for this. +1 for the patch. Pending Jenkins.
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.03.patch,
> HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10983) OIV tool should make an EC file explicit
[
https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-10983:
--
Attachment: HDFS-10983.03.patch
[~andrew.wang],
Removed the explicit replication field write for Striped files in FSImage at
{{FSImageFormatPBINode#buildPermissionStatus}}. And extended the
{{FSImageFormatPBINode#loadINodeFile}} with an assert to verify that FSImage
doesn't have replication field set for Striped files. With this new
verification {{TestFSImage#testSupportBlockGroup}} fails at FSImage load for EC
Files when the replication fields are additionally set.
{code}
@@ -332,6 +332,7 @@ private INodeFile loadINodeFile(INodeSection.INode n) {
assert n.getType() == INodeSection.INode.Type.FILE;
..
BlockType blockType = PBHelperClient.convert(f.getBlockType());
LoaderContext state = parent.getLoaderContext();
boolean isStriped = f.hasErasureCodingPolicyID();
+ assert ((!isStriped) || (isStriped && !f.hasReplication()));
Short replication = (!isStriped ? (short) f.getReplication() : null);
{code}
Currently, FSImage Loaders and Savers are not exposed properly for testing and
having an external test for verifying the fsimage inode_section image contents
seems not feasible. So, extending the {{loadINodeFile()}} with assert like few
other asserts in the same file. Please let me know if this is ok ?
> OIV tool should make an EC file explicit
>
>
> Key: HDFS-10983
> URL: https://issues.apache.org/jira/browse/HDFS-10983
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-10983.01.patch, HDFS-10983.02.patch,
> HDFS-10983.03.patch
>
>
> The OIV tool's webhdfs interface does not print if a file is striped or not.
> Also, it prints the file's EC policy ID as replication factor, which is
> inconsistent to the output of a typical webhdfs call to the cluster, which
> always shows replication factor of 0 for EC files.
> Not just webhdfs, but delimiter output does not print if a file is stripped
> or not either.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11502) dn.js set datanode UI to window.location.hostname, it should use jmx "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get tag.hostname
[ https://issues.apache.org/jira/browse/HDFS-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula reassigned HDFS-11502: --- Assignee: Jeffrey E Rodriguez > dn.js set datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname > -- > > Key: HDFS-11502 > URL: https://issues.apache.org/jira/browse/HDFS-11502 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.2, 2.7.3 > Environment: all >Reporter: Jeffrey E Rodriguez >Assignee: Jeffrey E Rodriguez > > Datanode UI calls dn.js which loads properties for datanode.. > dn.js sets datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname. > The issue is that if we proxy datanode UI we would get the proxy as the > hostname because UI code uses window.location.hostname instead of the actual > datanode hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898651#comment-15898651
]
Manoj Govindassamy commented on HDFS-11499:
---
[~linyiqun],
Thanks for the review. I was using the below {{writeFile}} method version,
where the last param is the file replication factor and not block count. The
method in turn creates a total of 2 blocks for the file with the provided repl
factor. Is the comment still wrong ? please let me know.
{code}
static protected void writeFile(FileSystem fileSys, Path name, int repl)
throws IOException {
writeFile(fileSys, name, repl, 2);
}
{code}
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.03.patch,
> HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11071) Ozone: SCM: Move SCM config keys to ScmConfig
[
https://issues.apache.org/jira/browse/HDFS-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898634#comment-15898634
]
Weiwei Yang commented on HDFS-11071:
There seems to be some issue on Jenkins
{quote}
FATAL: command execution failed
java.io.IOException: Backing channel 'H0' is disconnected
{quote}
I've run {{test-patch}} locally that I did not see so many issues. Will trigger
another jenkins job if necessary.
> Ozone: SCM: Move SCM config keys to ScmConfig
> -
>
> Key: HDFS-11071
> URL: https://issues.apache.org/jira/browse/HDFS-11071
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Weiwei Yang
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-11071-HDFS-7240.001.patch
>
>
> Move SCM keys from OzoneConfigKeys.Ozone to SCMConfigKeys.SCM *
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11446) TestMaintenanceState#testWithNNAndDNRestart fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-11446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11446:
-
Attachment: HDFS-11446-branch-2.patch
HDFS-11446.003.patch
Reupload the patch to trigger the Jenkins.
> TestMaintenanceState#testWithNNAndDNRestart fails intermittently
>
>
> Key: HDFS-11446
> URL: https://issues.apache.org/jira/browse/HDFS-11446
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11446.001.patch, HDFS-11446.002.patch,
> HDFS-11446.003.patch, HDFS-11446-branch-2.patch
>
>
> The test {{TestMaintenanceState#testWithNNAndDNRestart}} fails in trunk. The
> stack info(
> https://builds.apache.org/job/PreCommit-HDFS-Build/18423/testReport/ ):
> {code}
> java.lang.AssertionError: expected null, but was: for block BP-1367163238-172.17.0.2-1487836532907:blk_1073741825_1001:
> expected 3, got 2
> ,DatanodeInfoWithStorage[127.0.0.1:42649,DS-c499e6ef-ce14-428b-baef-8cf2a122b248,DISK],DatanodeInfoWithStorage[127.0.0.1:40774,DS-cc484c09-6e32-4804-a337-2871f37b62e1,DISK],pending
> block # 1 ,under replicated # 0 ,>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.hdfs.TestMaintenanceState.testWithNNAndDNRestart(TestMaintenanceState.java:731)
> {code}
> The failure seems due to pending block has not been replicated. We can bump
> the retry times since sometimes the cluster would be busy. Also we can use
> {{GenericTestUtils#waitFor}} to simplified the current compared logic.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11446) TestMaintenanceState#testWithNNAndDNRestart fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-11446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11446:
-
Attachment: (was: HDFS-11446-branch-2.patch)
> TestMaintenanceState#testWithNNAndDNRestart fails intermittently
>
>
> Key: HDFS-11446
> URL: https://issues.apache.org/jira/browse/HDFS-11446
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11446.001.patch, HDFS-11446.002.patch
>
>
> The test {{TestMaintenanceState#testWithNNAndDNRestart}} fails in trunk. The
> stack info(
> https://builds.apache.org/job/PreCommit-HDFS-Build/18423/testReport/ ):
> {code}
> java.lang.AssertionError: expected null, but was: for block BP-1367163238-172.17.0.2-1487836532907:blk_1073741825_1001:
> expected 3, got 2
> ,DatanodeInfoWithStorage[127.0.0.1:42649,DS-c499e6ef-ce14-428b-baef-8cf2a122b248,DISK],DatanodeInfoWithStorage[127.0.0.1:40774,DS-cc484c09-6e32-4804-a337-2871f37b62e1,DISK],pending
> block # 1 ,under replicated # 0 ,>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.hdfs.TestMaintenanceState.testWithNNAndDNRestart(TestMaintenanceState.java:731)
> {code}
> The failure seems due to pending block has not been replicated. We can bump
> the retry times since sometimes the cluster would be busy. Also we can use
> {{GenericTestUtils#waitFor}} to simplified the current compared logic.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11446) TestMaintenanceState#testWithNNAndDNRestart fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-11446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-11446:
-
Attachment: (was: HDFS-11446.003.patch)
> TestMaintenanceState#testWithNNAndDNRestart fails intermittently
>
>
> Key: HDFS-11446
> URL: https://issues.apache.org/jira/browse/HDFS-11446
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11446.001.patch, HDFS-11446.002.patch
>
>
> The test {{TestMaintenanceState#testWithNNAndDNRestart}} fails in trunk. The
> stack info(
> https://builds.apache.org/job/PreCommit-HDFS-Build/18423/testReport/ ):
> {code}
> java.lang.AssertionError: expected null, but was: for block BP-1367163238-172.17.0.2-1487836532907:blk_1073741825_1001:
> expected 3, got 2
> ,DatanodeInfoWithStorage[127.0.0.1:42649,DS-c499e6ef-ce14-428b-baef-8cf2a122b248,DISK],DatanodeInfoWithStorage[127.0.0.1:40774,DS-cc484c09-6e32-4804-a337-2871f37b62e1,DISK],pending
> block # 1 ,under replicated # 0 ,>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotNull(Assert.java:664)
> at org.junit.Assert.assertNull(Assert.java:646)
> at org.junit.Assert.assertNull(Assert.java:656)
> at
> org.apache.hadoop.hdfs.TestMaintenanceState.testWithNNAndDNRestart(TestMaintenanceState.java:731)
> {code}
> The failure seems due to pending block has not been replicated. We can bump
> the retry times since sometimes the cluster would be busy. Also we can use
> {{GenericTestUtils#waitFor}} to simplified the current compared logic.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898628#comment-15898628
]
Yiqun Lin commented on HDFS-11499:
--
Great work, everyone!
Only one nit for the v03 patch:
{code}
+
+Path openFile = new Path("/testClosingFileInMaintenance.dat");
+// Lets write 2 blocks of data to the openFile
+writeFile(getCluster().getFileSystem(), openFile, (short) 3);
+
{code}
The comment seems not accurate, here we write three block replica to openFile ,
right? [~manojg]. Or this should be replaced by "two more blocks".
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.03.patch,
> HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10983) OIV tool should make an EC file explicit
[
https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898592#comment-15898592
]
Manoj Govindassamy commented on HDFS-10983:
---
Thanks for the review [~andrew.wang].
If we don't set replication explicitly for Striped blocks, the field is 0 by
default and anyone reading this replication field from the FSImage would be
surprised. So, am explicitly setting it to DEFAULT_REPL_FOR_STRIPED_BLOCKS.
Example, in {{PBImageXMLWriter#dumpINodeFile}}, the previous code was just read
the replication field and dump in xml as is, which could be wrong for Striped
blocks. So, I later changed to read the file type and dump replication param
accordingly. Though this was fixed, i was worried about similar readers not
caring about striped files and just reading replication param.
{{FSImageFormatPBInode}}
{code}
public static INodeSection.INodeFile.Builder buildINodeFile(
INodeFileAttributes file, final SaverContext state) {
INodeSection.INodeFile.Builder b = INodeSection.INodeFile.newBuilder()
.setAccessTime ...
if (file.isStriped()) {
b.setReplication(INodeFile.DEFAULT_REPL_FOR_STRIPED_BLOCKS);
b.setErasureCodingPolicyID(file.getErasureCodingPolicyID());
} else {
b.setReplication(file.getFileReplication());
}
{code}
> OIV tool should make an EC file explicit
>
>
> Key: HDFS-10983
> URL: https://issues.apache.org/jira/browse/HDFS-10983
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-10983.01.patch, HDFS-10983.02.patch
>
>
> The OIV tool's webhdfs interface does not print if a file is striped or not.
> Also, it prints the file's EC policy ID as replication factor, which is
> inconsistent to the output of a typical webhdfs call to the cluster, which
> always shows replication factor of 0 for EC files.
> Not just webhdfs, but delimiter output does not print if a file is stripped
> or not either.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11314) Enforce set of enabled EC policies on the NameNode
[ https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898588#comment-15898588 ] Kai Zheng commented on HDFS-11314: -- Thanks for your clarifying and consideration, Andrew. Yes renaming the getters should work. > Enforce set of enabled EC policies on the NameNode > -- > > Key: HDFS-11314 > URL: https://issues.apache.org/jira/browse/HDFS-11314 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11314.001.patch, HDFS-11314.002.patch, > HDFS-11314.003.patch, HDFS-11314.004.patch, HDFS-11314.005.patch, > HDFS-11314.006.patch > > > Filing based on discussion in HDFS-8095. A user might specify a policy that > is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster > only has 10 nodes. The NN should only allow the client to choose from a > pre-approved list determined by the cluster administrator. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11502) dn.js set datanode UI to window.location.hostname, it should use jmx "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get tag.hostname
[ https://issues.apache.org/jira/browse/HDFS-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898587#comment-15898587 ] Jeffrey E Rodriguez commented on HDFS-11502: - Thanks I would do so > dn.js set datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname > -- > > Key: HDFS-11502 > URL: https://issues.apache.org/jira/browse/HDFS-11502 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.2, 2.7.3 > Environment: all >Reporter: Jeffrey E Rodriguez > > Datanode UI calls dn.js which loads properties for datanode.. > dn.js sets datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname. > The issue is that if we proxy datanode UI we would get the proxy as the > hostname because UI code uses window.location.hostname instead of the actual > datanode hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9705) Refine the behaviour of getFileChecksum when length = 0
[
https://issues.apache.org/jira/browse/HDFS-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898584#comment-15898584
]
Hadoop QA commented on HDFS-9705:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 89
unchanged - 1 fixed = 89 total (was 90) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFSOutputSummer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-9705 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856226/HDFS-9705-v6.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0d53ef231060 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5e74196 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18601/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18601/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18601/console
[jira] [Commented] (HDFS-11505) Do not enable any erasure coding policies by default
[ https://issues.apache.org/jira/browse/HDFS-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898583#comment-15898583 ] Kai Zheng commented on HDFS-11505: -- Thanks Andrew for your thought and it makes sense. The client-side config similar to dfs.replication to specify a default policy is a great idea and I guess we can do it later given some real user requirement that incurs much need of setting policy per file. > Do not enable any erasure coding policies by default > > > Key: HDFS-11505 > URL: https://issues.apache.org/jira/browse/HDFS-11505 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-must-do > > As discussed on HDFS-11314, administrators need to choose the correct set of > EC policies based on cluster size and desired fault-tolerance properties. > This means we should not enable any EC policies by default, since any default > value could be incorrect. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11499:
--
Attachment: HDFS-11499.03.patch
[~lukmajercak],
Attached v03 patch to have the same test timeout as the other tests and also
fixed checkstyle issues. Please take a look.
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.03.patch,
> HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11502) dn.js set datanode UI to window.location.hostname, it should use jmx "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get tag.hostname
[ https://issues.apache.org/jira/browse/HDFS-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898579#comment-15898579 ] Yuanbo Liu commented on HDFS-11502: --- [~jeffreyr97] If you want to work on it, please send a email to hadoop-dev and apply for "assign to myself" permission. Currently we don't have the privilege to assign JIRAs. > dn.js set datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname > -- > > Key: HDFS-11502 > URL: https://issues.apache.org/jira/browse/HDFS-11502 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.2, 2.7.3 > Environment: all >Reporter: Jeffrey E Rodriguez > > Datanode UI calls dn.js which loads properties for datanode.. > dn.js sets datanode UI to window.location.hostname, it should use jmx > "qry=Hadoop:service=DataNode,name=RpcDetailedActivityForPort8010" to get > tag.hostname. > The issue is that if we proxy datanode UI we would get the proxy as the > hostname because UI code uses window.location.hostname instead of the actual > datanode hostname. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11314) Enforce set of enabled EC policies on the NameNode
[
https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-11314:
---
Attachment: HDFS-11314.006.patch
Thanks for reviewing Kai! Regarding the naming, I was trying to differentiate
between:
# The set of all possible policies
# The set of enabled policies
I reused "system" to refer to the first set, while using "enabled" to refer to
the second set. I forgot that "system" specifically refers to the hardcoded
policies, so let's make the names more general as you say.
I shortened your proposal a little bit if that's okay, named the getters
{{getPolicyByName}} and {{getPolicyByID}}.
> Enforce set of enabled EC policies on the NameNode
> --
>
> Key: HDFS-11314
> URL: https://issues.apache.org/jira/browse/HDFS-11314
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11314.001.patch, HDFS-11314.002.patch,
> HDFS-11314.003.patch, HDFS-11314.004.patch, HDFS-11314.005.patch,
> HDFS-11314.006.patch
>
>
> Filing based on discussion in HDFS-8095. A user might specify a policy that
> is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster
> only has 10 nodes. The NN should only allow the client to choose from a
> pre-approved list determined by the cluster administrator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898571#comment-15898571
]
Lukas Majercak commented on HDFS-11499:
---
[~manojg], yes
TestDecommission#testDecommissionWithOpenFileAndDatanodeFailing(). It will wait
for all three DNs to be decommissioned right, and the log you showed is just 1
of them. I would suggest increasing it to 360sec yes. It finishes in 30seconds~
on my machine, similar to
TestDecommission#testDeadNodeCountAfterNamenodeRestart, which has 360sec
timeout.
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11314) Enforce set of enabled EC policies on the NameNode
[ https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11314: --- Attachment: HDFS-11314.005.patch New patch to address checkstyle errors. > Enforce set of enabled EC policies on the NameNode > -- > > Key: HDFS-11314 > URL: https://issues.apache.org/jira/browse/HDFS-11314 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11314.001.patch, HDFS-11314.002.patch, > HDFS-11314.003.patch, HDFS-11314.004.patch, HDFS-11314.005.patch > > > Filing based on discussion in HDFS-8095. A user might specify a policy that > is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster > only has 10 nodes. The NN should only allow the client to choose from a > pre-approved list determined by the cluster administrator. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11314) Enforce set of enabled EC policies on the NameNode
[
https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898546#comment-15898546
]
Kai Zheng commented on HDFS-11314:
--
I agree it's much safer to have admin configure and ensure the valid policies.
The list should only affect the choices that users would choose for other
files/folders, and incur no impact for existing erasure coded files/folders.
Thanks [~andrew.wang] for the great work!
One question and minor, noticed pretty many changes like follows and wondered
why. If we would allow user defined policies in future, due to whatever
considerations like new codec, new option even for existing codec and etc., I
guess the more general {{getPolicyByPolicyID}} name would be better.
{code}
-return ErasureCodingPolicyManager.getPolicyByPolicyID(
+return ErasureCodingPolicyManager.getSystemPolicyByID(
{code}
> Enforce set of enabled EC policies on the NameNode
> --
>
> Key: HDFS-11314
> URL: https://issues.apache.org/jira/browse/HDFS-11314
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11314.001.patch, HDFS-11314.002.patch,
> HDFS-11314.003.patch, HDFS-11314.004.patch
>
>
> Filing based on discussion in HDFS-8095. A user might specify a policy that
> is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster
> only has 10 nodes. The NN should only allow the client to choose from a
> pre-approved list determined by the cluster administrator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11505) Do not enable any erasure coding policies by default
[
https://issues.apache.org/jira/browse/HDFS-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898544#comment-15898544
]
Andrew Wang commented on HDFS-11505:
Hi Kai,
Your understanding is correct. However, I think the vast majority of EC writes
will happen by inheriting the policy from a parent directory at create time, so
usage will be transparent. It's likely that only command-line users will be
calling the setPolicy API, so they'll (hopefully) get nice error messages about
what's an invalid or disabled policy and how to enable policies. As a reference
point, I don't think we've seen a similar need for the storage policy APIs.
Apps like HBase or Hive have often depended on admins setting up
directory-level policies (e.g. storage policies, encryption).
However, if you think having a default policy is important, we could add a
client-side config similar to {{dfs.replication}} to specify a default policy.
Alternatively, this could also be set in NN configuration and queried via
getServerDefaults. As per above though, I'd hope that dir-level policies are
the best fit for users who want a simple way of using EC.
> Do not enable any erasure coding policies by default
>
>
> Key: HDFS-11505
> URL: https://issues.apache.org/jira/browse/HDFS-11505
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Manoj Govindassamy
> Labels: hdfs-ec-3.0-must-do
>
> As discussed on HDFS-11314, administrators need to choose the correct set of
> EC policies based on cluster size and desired fault-tolerance properties.
> This means we should not enable any EC policies by default, since any default
> value could be incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11314) Enforce set of enabled EC policies on the NameNode
[
https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898541#comment-15898541
]
Hadoop QA commented on HDFS-11314:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 8 new + 771 unchanged - 6 fixed = 779 total (was 777) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11314 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856372/HDFS-11314.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 0568dc0418e4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5e74196 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18599/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18599/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18599/co
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898537#comment-15898537
]
Manoj Govindassamy commented on HDFS-11499:
---
[~lukmajercak],
Are you referring to the timeout in
TestDecommission#testDecommissionWithOpenFileAndDatanodeFailing() which was
part of the patch v01 ? In the patch v02 I added Maintenance State related
test. Not sure, if extending the timeout for the failed test is going to solve
the problem. Because, the nodes didn't move to DECOMMISSIONED state as the test
is expecting .
{noformat}
2017-03-06 23:33:49,462 [Thread-782] INFO hdfs.AdminStatesBaseTest
(AdminStatesBaseTest.java:waitNodeState(342)) - Waiting for node
127.0.0.1:33069 to change state to Decommissioned current state: Decommission
In Progress
2017-03-06 23:33:49,462 [Thread-782] INFO hdfs.AdminStatesBaseTest
(AdminStatesBaseTest.java:waitNodeState(342)) - Waiting for node
127.0.0.1:33069 to change state to Decommissioned current state: Decommission
In Progress
[test timeout]
2017-03-06 23:33:49,486 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:shutdown(1951)) - Shutting down the Mini HDFS Cluster
{noformat}
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[ https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11482: -- Attachment: HDFS-11482.003.patch v001 and v002 patch were missing the change to one more file. Fixed in v003 patch. Also, there is potential race condition that can cause the code to loop forever when choosing a random node. This issue also exists in current NetworkTopology, tracked in HDFS-11507 > Add storage type demand to into DFSNetworkTopology#chooseRandom > --- > > Key: HDFS-11482 > URL: https://issues.apache.org/jira/browse/HDFS-11482 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11482.001.patch, HDFS-11482.002.patch, > HDFS-11482.003.patch > > > HDFS-11450 adds storage type info into network topology, with on this info we > may change chooseRandom to take storage type requirement as parameter, only > checking subtrees required storage type available. This way we avoid blindly > picking up nodes that are not applicable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11505) Do not enable any erasure coding policies by default
[ https://issues.apache.org/jira/browse/HDFS-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898529#comment-15898529 ] Kai Zheng commented on HDFS-11505: -- So assume some policies are enabled by the configuration, will users have to specify some one when create a file/folder or assign/set policy to it later? I'm asking this because, I'm wondering, for most applications/users they might not want to bother to query/check the enabled policies first to pick up some to specify. If users don't have to or the policy parameter is optional, I guess we could use the first one in the enabled list. Thoughts? > Do not enable any erasure coding policies by default > > > Key: HDFS-11505 > URL: https://issues.apache.org/jira/browse/HDFS-11505 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-must-do > > As discussed on HDFS-11314, administrators need to choose the correct set of > EC policies based on cluster size and desired fault-tolerance properties. > This means we should not enable any EC policies by default, since any default > value could be incorrect. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898524#comment-15898524
]
Lukas Majercak commented on HDFS-11499:
---
Looks like the test timed out, [~manojg], do you mind increasing the timeout to
360sec (based on the other tests in TestDecommission) and submitting a patch?
Thanks!
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11507) NetworkTopology#chooseRandom may run into a dead loop due to race condition
Chen Liang created HDFS-11507:
-
Summary: NetworkTopology#chooseRandom may run into a dead loop due
to race condition
Key: HDFS-11507
URL: https://issues.apache.org/jira/browse/HDFS-11507
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Chen Liang
Assignee: Chen Liang
{{NetworkTopology#chooseRandom()}} works as:
1. counts the number of available nodes as {{availableNodes}},
2. checks how many nodes are excluded, deduct from {{availableNodes}}
3. if {{availableNodes}} still > 0, then there are nodes available.
4. keep looping to find that node
But now imagine, in the meantime, the actually available nodes got removed in
step 3 or step 4, and all remaining nodes are excluded nodes. Then, although
there are no more nodes actually available, the code would still run as
{{availableNodes}} > 0, and then it would keep getting excluded node and loop
forever, as
{{if (excludedNodes == null || !excludedNodes.contains(ret))}}
will always be false.
We may fix this by expanding the while loop to also include the
{{availableNodes}} calculation. Such that we re-calculate {{availableNodes}}
every time it fails to find an available node.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11071) Ozone: SCM: Move SCM config keys to ScmConfig
[
https://issues.apache.org/jira/browse/HDFS-11071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898519#comment-15898519
]
Hadoop QA commented on HDFS-11071:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
5s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240 has
98 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
5 unchanged - 0 fixed = 12 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}142m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestFileStatus |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.TestEncryptionZonesWithHA |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.TestNameNodeXAttr |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.ozone.web.TestOzoneWebAccess |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.server.datanode.TestCachingStrategy |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.cli.TestXAttrCLI |
| | hadoop.ozone
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898518#comment-15898518
]
Hudson commented on HDFS-10838:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11358 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11358/])
HDFS-10838. Last full block report received time for each DN should be (arp:
rev b5adc5c3011f111f86d232cb33ec522547f68a95)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeInfo.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, HDFS-10838.006.patch, NN_UI.png,
> NN_UI_relative_time_minutes.png, NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6984) In Hadoop 3, make FileStatus serialize itself via protobuf
[
https://issues.apache.org/jira/browse/HDFS-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-6984:
Attachment: HDFS-6984.004.patch
> In Hadoop 3, make FileStatus serialize itself via protobuf
> --
>
> Key: HDFS-6984
> URL: https://issues.apache.org/jira/browse/HDFS-6984
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Labels: BB2015-05-TBR
> Attachments: HDFS-6984.001.patch, HDFS-6984.002.patch,
> HDFS-6984.003.patch, HDFS-6984.004.patch, HDFS-6984.nowritable.patch
>
>
> FileStatus was a Writable in Hadoop 2 and earlier. Originally, we used this
> to serialize it and send it over the wire. But in Hadoop 2 and later, we
> have the protobuf {{HdfsFileStatusProto}} which serves to serialize this
> information. The protobuf form is preferable, since it allows us to add new
> fields in a backwards-compatible way. Another issue is that already a lot of
> subclasses of FileStatus don't override the Writable methods of the
> superclass, breaking the interface contract that read(status.write) should be
> equal to the original status.
> In Hadoop 3, we should just make FileStatus serialize itself via protobuf so
> that we don't have to deal with these issues. It's probably too late to do
> this in Hadoop 2, since user code may be relying on the existing FileStatus
> serialization there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[
https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898493#comment-15898493
]
Hadoop QA commented on HDFS-11482:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
34s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 34s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 47s{color} | {color:orange} root: The patch generated 2 new + 51 unchanged -
0 fixed = 53 total (was 51) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
15s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11482 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856373/HDFS-11482.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e08333756f58 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5e74196 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/1860
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898491#comment-15898491
]
Hadoop QA commented on HDFS-11499:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 146 unchanged - 0 fixed = 150 total (was 146) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestFileAppend3 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| Timed out junit tests | org.apache.hadoop.hdfs.server.namenode.TestEditLog |
| | org.apache.hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | org.apache.hadoop.hdfs.server.namenode.TestEditLogAutoroll |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11499 |
| GITHUB PR | https://github.com/apache/hadoop/pull/199 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 278985d535e8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d9dc444 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18598/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18598/artifact/patchprocess/patch-
[jira] [Commented] (HDFS-11477) Simplify file IO profiling configuration
[
https://issues.apache.org/jira/browse/HDFS-11477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898478#comment-15898478
]
Hadoop QA commented on HDFS-11477:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 11s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11477 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855960/HDFS-11477.003.patch |
| Optional Tests | asflicense mvnsite compile javac javadoc mvninstall
unit findbugs checkstyle |
| uname | Linux 4b91f7864c20 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d9dc444 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18597/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache
[jira] [Commented] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[
https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898476#comment-15898476
]
Hadoop QA commented on HDFS-11482:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
40s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 40s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 47s{color} | {color:orange} root: The patch generated 2 new + 51 unchanged -
0 fixed = 53 total (was 51) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
48s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11482 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856373/HDFS-11482.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 73ebee8cdb84 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5e74196 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/1860
[jira] [Updated] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-10838:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
2.9.0
Status: Resolved (was: Patch Available)
Committed this to trunk and branch-2. Thanks for the contribution
[~surendrasingh]!
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, HDFS-10838.006.patch, NN_UI.png,
> NN_UI_relative_time_minutes.png, NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898471#comment-15898471
]
Arpit Agarwal edited comment on HDFS-10838 at 3/7/17 12:44 AM:
---
Committed this to trunk and branch-2. Thanks for the contribution
[~surendrasingh]!
(verified that the TestBalancer failure did not repro locally).
was (Author: arpitagarwal):
Committed this to trunk and branch-2. Thanks for the contribution
[~surendrasingh]!
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, HDFS-10838.006.patch, NN_UI.png,
> NN_UI_relative_time_minutes.png, NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898443#comment-15898443
]
Hadoop QA commented on HDFS-10838:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 1s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
430 unchanged - 2 fixed = 432 total (was 432) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10838 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856072/HDFS-10838.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 78709fe36bcb 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d9dc444 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/18596/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18596/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.
[jira] [Commented] (HDFS-10983) OIV tool should make an EC file explicit
[ https://issues.apache.org/jira/browse/HDFS-10983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898441#comment-15898441 ] Andrew Wang commented on HDFS-10983: Looks good overall, thanks for revving Manoj! One question though, in FSImageFormatPBInode, is it necessary to set the replication factor for an EC file? I commented this line out and ran a few EC unit tests successfully. It'd be nice to unit test this to make sure we aren't writing extraneous fields to the FSImage. > OIV tool should make an EC file explicit > > > Key: HDFS-10983 > URL: https://issues.apache.org/jira/browse/HDFS-10983 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Wei-Chiu Chuang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-10983.01.patch, HDFS-10983.02.patch > > > The OIV tool's webhdfs interface does not print if a file is striped or not. > Also, it prints the file's EC policy ID as replication factor, which is > inconsistent to the output of a typical webhdfs call to the cluster, which > always shows replication factor of 0 for EC files. > Not just webhdfs, but delimiter output does not print if a file is stripped > or not either. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11498) Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0
[ https://issues.apache.org/jira/browse/HDFS-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898416#comment-15898416 ] Hudson commented on HDFS-11498: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11357 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11357/]) HDFS-11498. Make RestCsrfPreventionHandler and WebHdfsHandler compatible (wang: rev 5e74196ede9bfc20eb6d6fe3aa6a0e5c47a40fdd) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/RestCsrfPreventionFilterHandler.java * (edit) hadoop-project/pom.xml > Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0 > --- > > Key: HDFS-11498 > URL: https://issues.apache.org/jira/browse/HDFS-11498 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HDFS-11498.001.patch, HDFS-11498.branch-2.001.patch > > > Per discussion in HADOOP-13866, it looks like we can change 2.8.0 back to > exposing Netty 4.0, but still be ABI compatible with Netty 4.1 for users like > HBase that want to swap out the version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[
https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11482:
--
Comment: was deleted
(was: I was looking into what would happen when chooseRandom and add/remove are
being called simultaneously. I'm inclined to believe that as long as the
{{DFSNetworkTopology}} object is synchronized by the caller when calling
chooseRandom and add/remove, there should be no problem. But any comments are
more than welcome.)
> Add storage type demand to into DFSNetworkTopology#chooseRandom
> ---
>
> Key: HDFS-11482
> URL: https://issues.apache.org/jira/browse/HDFS-11482
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11482.001.patch, HDFS-11482.002.patch
>
>
> HDFS-11450 adds storage type info into network topology, with on this info we
> may change chooseRandom to take storage type requirement as parameter, only
> checking subtrees required storage type available. This way we avoid blindly
> picking up nodes that are not applicable.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[
https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898412#comment-15898412
]
Chen Liang commented on HDFS-11482:
---
I was looking into what would happen when chooseRandom and add/remove are being
called simultaneously. I'm inclined to believe that as long as the
{{DFSNetworkTopology}} object is synchronized by the caller when calling
chooseRandom and add/remove, there should be no problem. But any comments are
more than welcome.
> Add storage type demand to into DFSNetworkTopology#chooseRandom
> ---
>
> Key: HDFS-11482
> URL: https://issues.apache.org/jira/browse/HDFS-11482
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11482.001.patch, HDFS-11482.002.patch
>
>
> HDFS-11450 adds storage type info into network topology, with on this info we
> may change chooseRandom to take storage type requirement as parameter, only
> checking subtrees required storage type available. This way we avoid blindly
> picking up nodes that are not applicable.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11497) Ozone: Fix Container test regression.
[
https://issues.apache.org/jira/browse/HDFS-11497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898404#comment-15898404
]
Hadoop QA commented on HDFS-11497:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.server.datanode.TestDataNodeMXBean |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.TestDataTransferKeepalive |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.server.namenode.ha.TestQuotasWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.namenode.TestAddBlockRetry |
| | hadoop.security.TestRefreshUserMappings |
| | hadoop.tracing.TestTraceAdmin |
| | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.TestGenericRefresh |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
| | hadoop.hdfs.TestWriteConfigurationToDFS |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockId |
| | hadoop.hdfs.server.datanode.TestIncrementalBrVariations |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| | hadoop.hdfs.TestDFSStartupVersions |
| | hadoop.hdfs.server.namenode.TestDefau
[jira] [Updated] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[ https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11482: -- Attachment: HDFS-11482.002.patch > Add storage type demand to into DFSNetworkTopology#chooseRandom > --- > > Key: HDFS-11482 > URL: https://issues.apache.org/jira/browse/HDFS-11482 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11482.001.patch, HDFS-11482.002.patch > > > HDFS-11450 adds storage type info into network topology, with on this info we > may change chooseRandom to take storage type requirement as parameter, only > checking subtrees required storage type available. This way we avoid blindly > picking up nodes that are not applicable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11314) Enforce set of enabled EC policies on the NameNode
[ https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11314: --- Attachment: HDFS-11314.004.patch Here's a patch to fix the TestOIV failure, had to enable an EC policy. > Enforce set of enabled EC policies on the NameNode > -- > > Key: HDFS-11314 > URL: https://issues.apache.org/jira/browse/HDFS-11314 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11314.001.patch, HDFS-11314.002.patch, > HDFS-11314.003.patch, HDFS-11314.004.patch > > > Filing based on discussion in HDFS-8095. A user might specify a policy that > is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster > only has 10 nodes. The NN should only allow the client to choose from a > pre-approved list determined by the cluster administrator. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[ https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11482: -- Status: Patch Available (was: Open) > Add storage type demand to into DFSNetworkTopology#chooseRandom > --- > > Key: HDFS-11482 > URL: https://issues.apache.org/jira/browse/HDFS-11482 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11482.001.patch > > > HDFS-11450 adds storage type info into network topology, with on this info we > may change chooseRandom to take storage type requirement as parameter, only > checking subtrees required storage type available. This way we avoid blindly > picking up nodes that are not applicable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11482) Add storage type demand to into DFSNetworkTopology#chooseRandom
[ https://issues.apache.org/jira/browse/HDFS-11482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11482: -- Attachment: HDFS-11482.001.patch > Add storage type demand to into DFSNetworkTopology#chooseRandom > --- > > Key: HDFS-11482 > URL: https://issues.apache.org/jira/browse/HDFS-11482 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11482.001.patch > > > HDFS-11450 adds storage type info into network topology, with on this info we > may change chooseRandom to take storage type requirement as parameter, only > checking subtrees required storage type available. This way we avoid blindly > picking up nodes that are not applicable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11498) Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0
[ https://issues.apache.org/jira/browse/HDFS-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11498: --- Release Note: This JIRA sets the Netty 4 dependency to 4.0.23. This is an incompatible change for the 3.0 release line, as 3.0.0-alpha1 and 3.0.0-alpha2 depended on Netty 4.1.0.Beta5. > Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0 > --- > > Key: HDFS-11498 > URL: https://issues.apache.org/jira/browse/HDFS-11498 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HDFS-11498.001.patch, HDFS-11498.branch-2.001.patch > > > Per discussion in HADOOP-13866, it looks like we can change 2.8.0 back to > exposing Netty 4.0, but still be ABI compatible with Netty 4.1 for users like > HBase that want to swap out the version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11498) Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0
[ https://issues.apache.org/jira/browse/HDFS-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11498: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 2.8.0 Status: Resolved (was: Patch Available) Thanks again for the reviews Akira and Jason, after backporting the revert of HDFS-8377, the patch here applied cleanly. I've committed to all the branches back through branch-2.8.0, we can resume discussion on the Netty 4.1 upgrade over at HADOOP-13866. > Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0 > --- > > Key: HDFS-11498 > URL: https://issues.apache.org/jira/browse/HDFS-11498 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HDFS-11498.001.patch, HDFS-11498.branch-2.001.patch > > > Per discussion in HADOOP-13866, it looks like we can change 2.8.0 back to > exposing Netty 4.0, but still be ABI compatible with Netty 4.1 for users like > HBase that want to swap out the version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11376) Revert HDFS-8377 Support HTTP/2 in datanode
[ https://issues.apache.org/jira/browse/HDFS-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11376: --- Affects Version/s: (was: 2.9.0) 2.8.0 Target Version/s: 3.0.0-alpha3 (was: 2.9.0, 3.0.0-alpha3) Fix Version/s: (was: 2.9.0) 2.8.0 > Revert HDFS-8377 Support HTTP/2 in datanode > --- > > Key: HDFS-11376 > URL: https://issues.apache.org/jira/browse/HDFS-11376 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.8.0, 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Xiao Chen > Fix For: 2.8.0, 3.0.0-alpha3 > > > Tracking JIRA so this revert shows up in the release notes. See discussion on > HADOOP-13866. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11376) Revert HDFS-8377 Support HTTP/2 in datanode
[ https://issues.apache.org/jira/browse/HDFS-11376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898346#comment-15898346 ] Andrew Wang commented on HDFS-11376: Per discussion on HDFS-11498, I've backported this to branch-2.8 and branch-2.8.0 for inclusion in 2.8.0. > Revert HDFS-8377 Support HTTP/2 in datanode > --- > > Key: HDFS-11376 > URL: https://issues.apache.org/jira/browse/HDFS-11376 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.8.0, 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Xiao Chen > Fix For: 2.8.0, 3.0.0-alpha3 > > > Tracking JIRA so this revert shows up in the release notes. See discussion on > HADOOP-13866. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11498) Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0
[ https://issues.apache.org/jira/browse/HDFS-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898323#comment-15898323 ] Andrew Wang commented on HDFS-11498: Yea, sorry for only testing on branch-2, I assumed it'd be a smooth backport to 2.8 as well. +1 on Jason's recommendation. I'll work on backporting HDFS-11376, then I'll apply this JIRA (HDFS-11498) everywhere. Given that there are two +1s on this pretty straightforward change, I'll go ahead and commit everything if I can do it with minimal cherry-pick conflicts. > Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0 > --- > > Key: HDFS-11498 > URL: https://issues.apache.org/jira/browse/HDFS-11498 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-11498.001.patch, HDFS-11498.branch-2.001.patch > > > Per discussion in HADOOP-13866, it looks like we can change 2.8.0 back to > exposing Netty 4.0, but still be ABI compatible with Netty 4.1 for users like > HBase that want to swap out the version. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898290#comment-15898290
]
Lukas Majercak commented on HDFS-11499:
---
LGTM
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-4934) add symlink support to WebHDFS server side
[ https://issues.apache.org/jira/browse/HDFS-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-4934: - Assignee: (was: Andrew Wang) > add symlink support to WebHDFS server side > -- > > Key: HDFS-4934 > URL: https://issues.apache.org/jira/browse/HDFS-4934 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.3.0 > Environment: followup on HADOOP-8040 >Reporter: Alejandro Abdelnur > > follow up on HADOOP-8040 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-4933) add symlink support to WebHDFS to HTTP REST API & client filesystem
[ https://issues.apache.org/jira/browse/HDFS-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-4933: - Assignee: (was: Andrew Wang) > add symlink support to WebHDFS to HTTP REST API & client filesystem > --- > > Key: HDFS-4933 > URL: https://issues.apache.org/jira/browse/HDFS-4933 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur > > follow up on HADOOP-8040 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-4935) add symlink support to HttpFS server side
[ https://issues.apache.org/jira/browse/HDFS-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-4935: - Assignee: (was: Andrew Wang) > add symlink support to HttpFS server side > - > > Key: HDFS-4935 > URL: https://issues.apache.org/jira/browse/HDFS-4935 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 > Environment: followup on HADOOP-8040 >Reporter: Alejandro Abdelnur > > follow up on HADOOP-8040 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11499:
--
Attachment: HDFS-11499.02.patch
[~lukmajercak], [~jojochuang], [~linyiqun],
Attached v02 patch to address the following. Can you please take a look at the
patch.
* {{BlockManager#commitOrCompleteLastBlock()}} to consider entering_maintenance
replicase along with decommissioning replicas for usable replicas.
* Added unit test testFileCloseAfterEnteringMaintenance to
{{TestMaintenanceState}} based on the test given by Yiqun. Without fix, the
test fails at file close.
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.02.patch, HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11487) Ozone: Exclude container protobuf files from findbugs check
[ https://issues.apache.org/jira/browse/HDFS-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11487: Resolution: Fixed Hadoop Flags: Reviewed Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~yuanbo] Thank you for the contribution. I have committed this patch to the feature branch. > Ozone: Exclude container protobuf files from findbugs check > --- > > Key: HDFS-11487 > URL: https://issues.apache.org/jira/browse/HDFS-11487 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Yuanbo Liu > Labels: newbie > Fix For: HDFS-7240 > > Attachments: HDFS-11487-HDFS-7240.001.patch > > > We moved the container protobuf definitions from hdfs project to hdfs-client. > This is to track that we need to add an exclusion to hdfs-client findbugs xml > to ignore those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11487) Ozone: Exclude container protobuf files from findbugs check
[ https://issues.apache.org/jira/browse/HDFS-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898259#comment-15898259 ] Anu Engineer commented on HDFS-11487: - +1, i will commit this shortly. > Ozone: Exclude container protobuf files from findbugs check > --- > > Key: HDFS-11487 > URL: https://issues.apache.org/jira/browse/HDFS-11487 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Yuanbo Liu > Labels: newbie > Fix For: HDFS-7240 > > Attachments: HDFS-11487-HDFS-7240.001.patch > > > We moved the container protobuf definitions from hdfs project to hdfs-client. > This is to track that we need to add an exclusion to hdfs-client findbugs xml > to ignore those files. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11505) Do not enable any erasure coding policies by default
[ https://issues.apache.org/jira/browse/HDFS-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy reassigned HDFS-11505: - Assignee: Manoj Govindassamy (was: Andrew Wang) > Do not enable any erasure coding policies by default > > > Key: HDFS-11505 > URL: https://issues.apache.org/jira/browse/HDFS-11505 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-must-do > > As discussed on HDFS-11314, administrators need to choose the correct set of > EC policies based on cluster size and desired fault-tolerance properties. > This means we should not enable any EC policies by default, since any default > value could be incorrect. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11506) Move ErasureCodingPolicyManager#getSystemDefaultPolicy to test code
[ https://issues.apache.org/jira/browse/HDFS-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy reassigned HDFS-11506: - Assignee: Manoj Govindassamy (was: Andrew Wang) > Move ErasureCodingPolicyManager#getSystemDefaultPolicy to test code > --- > > Key: HDFS-11506 > URL: https://issues.apache.org/jira/browse/HDFS-11506 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Manoj Govindassamy > Labels: hdfs-ec-3.0-nice-to-have > > In HDFS-11416, we removed usages of getSystemDefaultPolicy in production > code. Let's move it to a test package to make sure no production code calls > it in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[
https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898251#comment-15898251
]
Arpit Agarwal commented on HDFS-10838:
--
Thanks [~surendrasingh]. +1 pending Jenkins.
HDFS Pre-commit runs appear to be busted right now due to a bad node, we should
retrigger the Jenkins runs when the node is fixed.
> Last full block report received time for each DN should be easily discoverable
> --
>
> Key: HDFS-10838
> URL: https://issues.apache.org/jira/browse/HDFS-10838
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ui
>Reporter: Arpit Agarwal
>Assignee: Surendra Singh Lilhore
> Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch,
> HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch,
> HDFS-10838.005.patch, HDFS-10838.006.patch, NN_UI.png,
> NN_UI_relative_time_minutes.png, NN_UI_relative_time.png
>
>
> It should be easy for administrators to discover the time of last full block
> report from each DataNode.
> We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin
> -report}}, or both.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898250#comment-15898250
]
Lukas Majercak commented on HDFS-11499:
---
[~manojg], I don't really mind, let's go for the second option? Also, +
[~mingma], do you want to check this? Seems like it is related to your
HDFS-9390.
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11506) Move ErasureCodingPolicyManager#getSystemDefaultPolicy to test code
Andrew Wang created HDFS-11506: -- Summary: Move ErasureCodingPolicyManager#getSystemDefaultPolicy to test code Key: HDFS-11506 URL: https://issues.apache.org/jira/browse/HDFS-11506 Project: Hadoop HDFS Issue Type: Improvement Components: erasure-coding Affects Versions: 3.0.0-alpha3 Reporter: Andrew Wang Assignee: Andrew Wang In HDFS-11416, we removed usages of getSystemDefaultPolicy in production code. Let's move it to a test package to make sure no production code calls it in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11314) Enforce set of enabled EC policies on the NameNode
[
https://issues.apache.org/jira/browse/HDFS-11314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898246#comment-15898246
]
Andrew Wang commented on HDFS-11314:
Hi Rakesh, thanks for reviewing!
bq. I'd also prefer keep the list empty and lets users configure the right
values based on their topology.
Great, I filed HDFS-11505. I'll handle this after we get this one in.
bq. Could you please point me the way Namenode handles this during restarts
and read from edit log. It would be good to add test case if not added.
I extended TestErasureCodingPolicies#testBasicSetECPolicy, little snippets here:
{code}
+// Verify that policies are successfully loaded even when policies
+// are disabled
+cluster.getConfiguration(0).set(
+DFSConfigKeys.DFS_NAMENODE_EC_POLICIES_ENABLED_KEY, "");
+cluster.restartNameNodes();
+cluster.waitActive();
{code}
{code}
+// Also check loading disabled EC policies from fsimage
+fs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER);
+fs.saveNamespace();
+fs.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_LEAVE);
+cluster.restartNameNodes();
{code}
bq. Whats your opinion to provide a facility to dynamically/reconfigure
enabling the policies with out needing to restart Namenode?
I think changing the set of EC policies is a pretty rare operation, so it's not
that important.
One related nice work item would be doing this for the replication throttles,
since that's an issue even now.
> Enforce set of enabled EC policies on the NameNode
> --
>
> Key: HDFS-11314
> URL: https://issues.apache.org/jira/browse/HDFS-11314
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Blocker
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11314.001.patch, HDFS-11314.002.patch,
> HDFS-11314.003.patch
>
>
> Filing based on discussion in HDFS-8095. A user might specify a policy that
> is not appropriate for the cluster, e.g. a RS (10,4) policy when the cluster
> only has 10 nodes. The NN should only allow the client to choose from a
> pre-approved list determined by the cluster administrator.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11487) Ozone: Exclude container protobuf files from findbugs check
[
https://issues.apache.org/jira/browse/HDFS-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898239#comment-15898239
]
Hadoop QA commented on HDFS-11487:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11487 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856240/HDFS-11487-HDFS-7240.001.patch
|
| Optional Tests | asflicense xml |
| uname | Linux 2d1bf10b2882 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 0951726 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18592/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Exclude container protobuf files from findbugs check
> ---
>
> Key: HDFS-11487
> URL: https://issues.apache.org/jira/browse/HDFS-11487
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yuanbo Liu
> Labels: newbie
> Fix For: HDFS-7240
>
> Attachments: HDFS-11487-HDFS-7240.001.patch
>
>
> We moved the container protobuf definitions from hdfs project to hdfs-client.
> This is to track that we need to add an exclusion to hdfs-client findbugs xml
> to ignore those files.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11445) FSCK shows overall health stauts as corrupt even one replica is corrupt
[
https://issues.apache.org/jira/browse/HDFS-11445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898236#comment-15898236
]
Jing Zhao commented on HDFS-11445:
--
Thanks for working on this, [~brahmareddy]!
I think you just find a scenario that the following inconsistency happens:
{code:title=BlockManager#createLocatedBlock}
NumberReplicas numReplicas = countNodes(blk);
final int numCorruptNodes = numReplicas.corruptReplicas();
final int numCorruptReplicas = corruptReplicas.numCorruptReplicas(blk);
if (numCorruptNodes != numCorruptReplicas) {
LOG.warn("Inconsistent number of corrupt replicas for "
+ blk + " blockMap has " + numCorruptNodes
+ " but corrupt replicas map has " + numCorruptReplicas);
}
{code}
I also did some debugging using your unit test. Looks like the root cause for
this inconsistency is: {{BlockInfo#setGenerationStampAndVerifyReplicas}} may
remove a datanode storage from the block's storage list, but still leave the
storage in the CorruptReplicasMap.
This inconsistency later can be fixed automatically, e.g., by a full block
report. But maybe we should consider using
{{BlockManager#removeStoredBlock(BlockInfo, DatanodeDescriptor)}} to remove all
the records related to the block-dn pair.
> FSCK shows overall health stauts as corrupt even one replica is corrupt
> ---
>
> Key: HDFS-11445
> URL: https://issues.apache.org/jira/browse/HDFS-11445
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-11445.patch
>
>
> In the following scenario,FSCK shows overall health status as corrupt even
> it's has one good replica.
> 1. Create file with 2 RF.
> 2. Shutdown one DN
> 3. Append to file again.
> 4. Restart the DN
> 5. After block report, check Fsck
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11505) Do not enable any erasure coding policies by default
Andrew Wang created HDFS-11505: -- Summary: Do not enable any erasure coding policies by default Key: HDFS-11505 URL: https://issues.apache.org/jira/browse/HDFS-11505 Project: Hadoop HDFS Issue Type: Improvement Components: erasure-coding Affects Versions: 3.0.0-alpha3 Reporter: Andrew Wang Assignee: Andrew Wang As discussed on HDFS-11314, administrators need to choose the correct set of EC policies based on cluster size and desired fault-tolerance properties. This means we should not enable any EC policies by default, since any default value could be incorrect. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11196) Ozone: Improve logging and error handling in the container layer
[
https://issues.apache.org/jira/browse/HDFS-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898231#comment-15898231
]
Hadoop QA commented on HDFS-11196:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-11196 does not apply to HDFS-7240. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856024/HDFS-11196-HDFS-7240.003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18589/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Improve logging and error handling in the container layer
>
>
> Key: HDFS-11196
> URL: https://issues.apache.org/jira/browse/HDFS-11196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
> Attachments: HDFS-11196-HDFS-7240.001.patch,
> HDFS-11196-HDFS-7240.002.patch, HDFS-11196-HDFS-7240.003.patch
>
>
> Improve logging and error handling in container layer.
> * With this change Storage Containers return StorageContainerException.
> * Precondition checks fail with a human readable error.
> * All failed requests are logged with traceID in the dispatcher.
> * Returns proper error codes for corresponding failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11196) Ozone: Improve logging and error handling in the container layer
[
https://issues.apache.org/jira/browse/HDFS-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898232#comment-15898232
]
Hadoop QA commented on HDFS-11196:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-11196 does not apply to HDFS-7240. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856024/HDFS-11196-HDFS-7240.003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18590/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Improve logging and error handling in the container layer
>
>
> Key: HDFS-11196
> URL: https://issues.apache.org/jira/browse/HDFS-11196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
> Attachments: HDFS-11196-HDFS-7240.001.patch,
> HDFS-11196-HDFS-7240.002.patch, HDFS-11196-HDFS-7240.003.patch
>
>
> Improve logging and error handling in container layer.
> * With this change Storage Containers return StorageContainerException.
> * Precondition checks fail with a human readable error.
> * All failed requests are logged with traceID in the dispatcher.
> * Returns proper error codes for corresponding failures.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898222#comment-15898222
]
Manoj Govindassamy commented on HDFS-11499:
---
yes [~lukmajercak], we need to additionally count in the ENTERING MAINTENANCE
nodes as well. I am adding a new test based on the one given by [~linyiqun] in
HDFS-11486 in TestMaintenanceState to cover this case.
[~lukmajercak]/[~jojochuang], Shall I merge all the fix and test patches along
with mine and post a complete patch covering both HDFS-11499 and HDFS-11486 ?
Or shall I submit v02 patch for this HDFS-11499 alone with maintenance state
included. Your suggestion please ?
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898211#comment-15898211
]
Lukas Majercak commented on HDFS-11499:
---
Hi [~jojochuang], [~manojg]
So shall we count in the replicas on maintenance nodes as well? Can we need to
add a test case to cover this/modify the one in the patch?
Thanks
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11500) Java Unlimited Cryptography Extension doesn't work
[ https://issues.apache.org/jira/browse/HDFS-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898191#comment-15898191 ] Jeff W edited comment on HDFS-11500 at 3/6/17 9:42 PM: --- Run command: curl -k -v -u nn/hdfs-nn1.cloud.local@CLOUD.LOCAL:password --negotiate https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 Will got the same error information, it must be the Jetty server configuration issue. was (Author: jeff...@yahoo.com): Run command: curl -k -v --negotiate https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 Will got the same error information, it must be the Jetty server configuration issue. > Java Unlimited Cryptography Extension doesn't work > -- > > Key: HDFS-11500 > URL: https://issues.apache.org/jira/browse/HDFS-11500 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node, namenode >Affects Versions: 2.7.3 > Environment: Centos 7, Java 1.8, HDFS HA >Reporter: Jeff W > Labels: security > Attachments: log_from_journal_node.txt, log_from_name_node.txt > > > I'm setting up my HDFS cluster with JQM service, and enabled Kerberos > authentication. replaced JCE to unlimited extension, but I still got the > following error information from the Journal node: > - > 2017-03-04 15:31:39,566 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for nn/hdfs-nn1.cloud.local@CLOUD.LOCAL (auth:KERBEROS) > 2017-03-04 15:31:39,585 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for nn/hdfs-nn1.cloud.local@CLOUD.LOCAL > (auth:KERBEROS) for protocol=interface > org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol > ... > 2017-03-04 15:31:40,345 WARN > org.apache.hadoop.security.authentication.server.AuthenticationFilter: > Authentication exception: GSSException: Failure unspecified at GSS-API level > (Mechanism level: Invalid argument (400) - Cannot find key of appropriate > type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96) > > > IPC authentication for Namenode looks good, but following with > AuthenticationFilter exception came from HTTP authentication, as I'm able to > see the same error message appears in Namenode's log: > - > 2017-03-04 15:31:40,402 ERROR > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception > initializing > https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 > java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > Authentication failed, status: 403, message: GSSException: Failure > unspecified at GSS-API level (Mechanism level: Invalid argument (400) - > Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with > HMAC SHA1-96) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456) > at java.security.AccessController.doPrivileged(Native Method) > ... > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > Authentication failed, status: 403, message: GSSException: Failure > unspecified at GSS-API level (Mechanism level: Invalid argument (400) - > Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with > HMAC SHA1-96) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274) > at > org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:461) > ... 30 more > 2017-03-04 20:06:51,612 ERROR > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Got error reading > edit log input stream > https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2e
[jira] [Comment Edited] (HDFS-11500) Java Unlimited Cryptography Extension doesn't work
[ https://issues.apache.org/jira/browse/HDFS-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898191#comment-15898191 ] Jeff W edited comment on HDFS-11500 at 3/6/17 9:41 PM: --- Run command: curl -k -v --negotiate https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 Will got the same error information, it must be the Jetty server configuration issue. was (Author: jeff...@yahoo.com): Run command: curl -k -v --negotiate https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 Will got same error information, it must be the Jetty server configuration issue. > Java Unlimited Cryptography Extension doesn't work > -- > > Key: HDFS-11500 > URL: https://issues.apache.org/jira/browse/HDFS-11500 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node, namenode >Affects Versions: 2.7.3 > Environment: Centos 7, Java 1.8, HDFS HA >Reporter: Jeff W > Labels: security > Attachments: log_from_journal_node.txt, log_from_name_node.txt > > > I'm setting up my HDFS cluster with JQM service, and enabled Kerberos > authentication. replaced JCE to unlimited extension, but I still got the > following error information from the Journal node: > - > 2017-03-04 15:31:39,566 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for nn/hdfs-nn1.cloud.local@CLOUD.LOCAL (auth:KERBEROS) > 2017-03-04 15:31:39,585 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for nn/hdfs-nn1.cloud.local@CLOUD.LOCAL > (auth:KERBEROS) for protocol=interface > org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol > ... > 2017-03-04 15:31:40,345 WARN > org.apache.hadoop.security.authentication.server.AuthenticationFilter: > Authentication exception: GSSException: Failure unspecified at GSS-API level > (Mechanism level: Invalid argument (400) - Cannot find key of appropriate > type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96) > > > IPC authentication for Namenode looks good, but following with > AuthenticationFilter exception came from HTTP authentication, as I'm able to > see the same error message appears in Namenode's log: > - > 2017-03-04 15:31:40,402 ERROR > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception > initializing > https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 > java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > Authentication failed, status: 403, message: GSSException: Failure > unspecified at GSS-API level (Mechanism level: Invalid argument (400) - > Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with > HMAC SHA1-96) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456) > at java.security.AccessController.doPrivileged(Native Method) > ... > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > Authentication failed, status: 403, message: GSSException: Failure > unspecified at GSS-API level (Mechanism level: Invalid argument (400) - > Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with > HMAC SHA1-96) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274) > at > org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:461) > ... 30 more > 2017-03-04 20:06:51,612 ERROR > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Got error reading > edit log input stream > https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648; > failing over
[jira] [Commented] (HDFS-11500) Java Unlimited Cryptography Extension doesn't work
[ https://issues.apache.org/jira/browse/HDFS-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898191#comment-15898191 ] Jeff W commented on HDFS-11500: --- Run command: curl -k -v --negotiate https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 Will got same error information, it must be the Jetty server configuration issue. > Java Unlimited Cryptography Extension doesn't work > -- > > Key: HDFS-11500 > URL: https://issues.apache.org/jira/browse/HDFS-11500 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node, namenode >Affects Versions: 2.7.3 > Environment: Centos 7, Java 1.8, HDFS HA >Reporter: Jeff W > Labels: security > Attachments: log_from_journal_node.txt, log_from_name_node.txt > > > I'm setting up my HDFS cluster with JQM service, and enabled Kerberos > authentication. replaced JCE to unlimited extension, but I still got the > following error information from the Journal node: > - > 2017-03-04 15:31:39,566 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for nn/hdfs-nn1.cloud.local@CLOUD.LOCAL (auth:KERBEROS) > 2017-03-04 15:31:39,585 INFO > SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: > Authorization successful for nn/hdfs-nn1.cloud.local@CLOUD.LOCAL > (auth:KERBEROS) for protocol=interface > org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol > ... > 2017-03-04 15:31:40,345 WARN > org.apache.hadoop.security.authentication.server.AuthenticationFilter: > Authentication exception: GSSException: Failure unspecified at GSS-API level > (Mechanism level: Invalid argument (400) - Cannot find key of appropriate > type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96) > > > IPC authentication for Namenode looks good, but following with > AuthenticationFilter exception came from HTTP authentication, as I'm able to > see the same error message appears in Namenode's log: > - > 2017-03-04 15:31:40,402 ERROR > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception > initializing > https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 > java.io.IOException: > org.apache.hadoop.security.authentication.client.AuthenticationException: > Authentication failed, status: 403, message: GSSException: Failure > unspecified at GSS-API level (Mechanism level: Invalid argument (400) - > Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with > HMAC SHA1-96) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456) > at java.security.AccessController.doPrivileged(Native Method) > ... > Caused by: > org.apache.hadoop.security.authentication.client.AuthenticationException: > Authentication failed, status: 403, message: GSSException: Failure > unspecified at GSS-API level (Mechanism level: Invalid argument (400) - > Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with > HMAC SHA1-96) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274) > at > org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77) > at > org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:214) > at > org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) > at > org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:461) > ... 30 more > 2017-03-04 20:06:51,612 ERROR > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Got error reading > edit log input stream > https://jn1.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648; > failing over to edit log > https://jn2.cloud.local:8481/getJournal?jid=hadoop1&segmentTxId=1&storageInfo=-63%3A1598677718%3A0%3ACID-2eecd392-dae7-480b-a867-5e0295c78648 > > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: > got premature end-of-file at txid 0; expected file to go up to 2 > at > org.apache
[jira] [Commented] (HDFS-11496) Ozone: Merge with trunk needed a ignore duplicate entry in pom file due to shading
[
https://issues.apache.org/jira/browse/HDFS-11496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898138#comment-15898138
]
Anu Engineer commented on HDFS-11496:
-
[~busbey] That makes sense, I think we did add netty dependency. Thanks for
your suggestions, I will fix this before merging the way you are suggesting.
> Ozone: Merge with trunk needed a ignore duplicate entry in pom file due to
> shading
> --
>
> Key: HDFS-11496
> URL: https://issues.apache.org/jira/browse/HDFS-11496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>
> The trunk merge needed a hack in the
> hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml to ignore
> netty files. Not sure that is the right thing to do, so tracking here if we
> need to revert this change before we merge.
> {code}
> ~/a/hadoop> git diff
> hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
> ─
> modified: hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
> ─
> @ pom.xml:106 @
> *
>
>
>
>
>
>
>
>
>
>
> io.netty
> netty
> *
>
>
>
>
>
> {code}
> [~andrew.wang] or [~busbey] Would one of you care to comment if this is the
> right thing to do ? Thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11496) Ozone: Merge with trunk needed a ignore duplicate entry in pom file due to shading
[
https://issues.apache.org/jira/browse/HDFS-11496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898103#comment-15898103
]
Sean Busbey commented on HDFS-11496:
I would guess that the Ozone branch added netty as a dependency for something
that is transitively included in minicluster. using maven's dependency tree
should show you where the dependency comes in.
> Ozone: Merge with trunk needed a ignore duplicate entry in pom file due to
> shading
> --
>
> Key: HDFS-11496
> URL: https://issues.apache.org/jira/browse/HDFS-11496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>
> The trunk merge needed a hack in the
> hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml to ignore
> netty files. Not sure that is the right thing to do, so tracking here if we
> need to revert this change before we merge.
> {code}
> ~/a/hadoop> git diff
> hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
> ─
> modified: hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
> ─
> @ pom.xml:106 @
> *
>
>
>
>
>
>
>
>
>
>
> io.netty
> netty
> *
>
>
>
>
>
> {code}
> [~andrew.wang] or [~busbey] Would one of you care to comment if this is the
> right thing to do ? Thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11499) Decommissioning stuck because of failing recovery
[
https://issues.apache.org/jira/browse/HDFS-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898100#comment-15898100
]
Manoj Govindassamy commented on HDFS-11499:
---
Sure [~jojochuang]. I can give you a patch for the same, soon.
> Decommissioning stuck because of failing recovery
> -
>
> Key: HDFS-11499
> URL: https://issues.apache.org/jira/browse/HDFS-11499
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
>Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
>Reporter: Lukas Majercak
>Assignee: Lukas Majercak
> Labels: blockmanagement, decommission, recovery
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11499.patch
>
>
> Block recovery will fail to finalize the file if the locations of the last,
> incomplete block are being decommissioned. Vice versa, the decommissioning
> will be stuck, waiting for the last block to be completed.
> {code:xml}
> org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException):
> Failed to finalize INodeFile testRecoveryFile since blocks[255] is
> non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...
> {code}
> The fix is to count replicas on decommissioning nodes when completing last
> block in BlockManager.commitOrCompleteLastBlock, as we know that the
> DecommissionManager will not decommission a node that has UC blocks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11163) Mover should move the file blocks to default storage once policy is unset
[
https://issues.apache.org/jira/browse/HDFS-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898092#comment-15898092
]
Chris Nauroth commented on HDFS-11163:
--
The {{FsServerDefaults}} class is annotated {{Public}}, so let's maintain the
existing constructor signature and add a new constructor that supports passing
default storage policy ID. The old constructor can delegate to the new
constructor with default storage policy ID of 0. That also would remove the
need to change {{FtpConfigKeys}} and {{LocalConfigKeys}} in this patch.
The logic is looking good to me, but I'd still like a second opinion review
before committing anything.
> Mover should move the file blocks to default storage once policy is unset
> -
>
> Key: HDFS-11163
> URL: https://issues.apache.org/jira/browse/HDFS-11163
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11163-001.patch, HDFS-11163-002.patch,
> HDFS-11163-003.patch, HDFS-11163-004.patch, HDFS-11163-005.patch,
> temp-YARN-6278.HDFS-11163.patch
>
>
> HDFS-9534 added new API in FileSystem to unset the storage policy. Once
> policy is unset blocks should move back to the default storage policy.
> Currently mover is not moving file blocks which have zero storage ID
> {code}
> // currently we ignore files with unspecified storage policy
> if (policyId == HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11496) Ozone: Merge with trunk needed a ignore duplicate entry in pom file due to shading
[
https://issues.apache.org/jira/browse/HDFS-11496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898035#comment-15898035
]
Anu Engineer commented on HDFS-11496:
-
[~busbey] Thanks for your comment. I agree that changing invariants is a bad
idea. That is why I filed this JIRA to get your expert comments. I will try to
fix it the way you are suggesting. Before I do that, I wanted to ask you why
this is not happening in trunk and why this change is needed in Ozone branch,
any guesses to where the root issue might be ?
> Ozone: Merge with trunk needed a ignore duplicate entry in pom file due to
> shading
> --
>
> Key: HDFS-11496
> URL: https://issues.apache.org/jira/browse/HDFS-11496
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>
> The trunk merge needed a hack in the
> hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml to ignore
> netty files. Not sure that is the right thing to do, so tracking here if we
> need to revert this change before we merge.
> {code}
> ~/a/hadoop> git diff
> hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
> ─
> modified: hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml
> ─
> @ pom.xml:106 @
> *
>
>
>
>
>
>
>
>
>
>
> io.netty
> netty
> *
>
>
>
>
>
> {code}
> [~andrew.wang] or [~busbey] Would one of you care to comment if this is the
> right thing to do ? Thanks in advance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
