[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Status: Patch Available (was: Open) > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353-2.patch, > HDFS-12353-3.patch, HDFS-12353-4.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Status: Open (was: Patch Available) > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353-2.patch, > HDFS-12353-3.patch, HDFS-12353-4.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Attachment: HDFS-12353-4.patch > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353-2.patch, > HDFS-12353-3.patch, HDFS-12353-4.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[
https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164178#comment-16164178
]
Brahma Reddy Battula commented on HDFS-12381:
-
bq.Apparently there is yet one more method addition to ClientProtocol related
to EC.
Yes, it's after HDFS-12381. we need to rebase.
*Compilation Errors for reference*
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project h
adoop-hdfs: Compilation failure: Compilation failure:
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[69,39] cannot find symbol
[ERROR] symbol: class BlocksStats
[ERROR] location: package org.apache.hadoop.hdfs.protocol
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[79,39] cannot find symbol
[ERROR] symbol: class ECBlockGroupsStats
[ERROR] location: package org.apache.hadoop.hdfs.protocol
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1882,10] cannot find symbol
[ERROR] symbol: class ECBlockGroupsStats
[ERROR] location: class
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1894,10] cannot find symbol
[ERROR] symbol: class BlocksStats
[ERROR] location: class
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[138,8]
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer is not abstract
and doe
s not override abstract method getECBlockGroupStats() in
org.apache.hadoop.hdfs.protocol.ClientProtocol
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1881,3] method does not override or implement a
method from a supertype
[ERROR]
/D:/HDFS-10467/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/rou
ter/RouterRpcServer.java:[1893,3] method does not override or implement a
method from a supertype
{noformat}
> [Documentation] Adding configuration keys for the Router
>
>
> Key: HDFS-12381
> URL: https://issues.apache.org/jira/browse/HDFS-12381
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Fix For: HDFS-10467
>
> Attachments: HDFS-12381-HDFS-10467.000.patch,
> HDFS-12381-HDFS-10467.001.patch
>
>
> Adding configuration options in tabular format.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12378) TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar reassigned HDFS-12378:
-
Assignee: Ajay Kumar
> TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
> --
>
> Key: HDFS-12378
> URL: https://issues.apache.org/jira/browse/HDFS-12378
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiao Chen
>Assignee: Ajay Kumar
>Priority: Blocker
>
> Saw on
> https://builds.apache.org/job/PreCommit-HDFS-Build/20928/testReport/org.apache.hadoop.hdfs/TestClientProtocolForPipelineRecovery/testZeroByteBlockRecovery/:
> Error Message
> {noformat}
> Failed to replace a bad datanode on the existing pipeline due to no more good
> datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> {noformat}
> Stacktrace
> {noformat}
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1322)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1388)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1587)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1488)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1470)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1274)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
> {noformat}
> Standard Output
> {noformat}
> 2017-08-30 18:02:37,714 [main] INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:(469)) - starting cluster: numNameNodes=1,
> numDataNodes=3
> Formatting using clusterid: testClusterID
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSEditLog
> (FSEditLog.java:newInstance(224)) - Edit logging is async:false
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(742)) - KeyProvider: null
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(120)) - fsLock is fair: true
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(136)) - Detailed lock hold time metrics
> enabled: false
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(763)) - fsOwner = jenkins (auth:SIMPLE)
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(764)) - supergroup = supergroup
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(765)) - isPermissionEnabled = true
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(776)) - HA Enabled: false
> 2017-08-30 18:02:37,718 [main] INFO common.Util
> (Util.java:isDiskStatsEnabled(395)) -
> dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO
> profiling
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(301)) - dfs.block.invalidate.limit:
> configured=1000, counted=60, effected=1000
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(309)) -
> dfs.namenode.datanode.registration.ip-hostname-check=true
> 2017-08-30 18:02:37,719 [main] INFO blockmanagement.BlockManager
> (InvalidateBlocks.java:printBlockDeletionTime(76)) -
> dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
> 2017-08-30 18:02:37,719 [main] INFO blockmanagement.BlockManager
> (InvalidateBlocks.java:printBlockDeletionTime(82)) - The block deletion will
> start around 2017 Aug 30 18:02:37
> 2017-08-30 18:02:37,719 [main] INFO util.GSet
> (LightWeightGSet.java:computeCapacity(395)) - Computing capacity for map
> BlocksMap
> 2017-08-30 18:02:37,719 [main] INFO util.GSet
>
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[
https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164158#comment-16164158
]
Ming Ma commented on HDFS-10702:
[~csun], the long checkpoint duration could cause the following issue:
* Checkpointer holding {{cpLock}} takes a long time to finish for a large
namespace.
* edit log tailer blocked by {{cpLock}} can't update namespace. Thus the
namespace becomes stale.
* An application deletes a file. StandbyNN receives incremental block report
indicating the blocks have been removed and update its blockmap.
* StandbyNN's stale namespace still has the file but without block locations.
> Add a Client API and Proxy Provider to enable stale read from Standby
> -
>
> Key: HDFS-10702
> URL: https://issues.apache.org/jira/browse/HDFS-10702
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jiayi Zhou
>Assignee: Sean Mackrory
>Priority: Minor
> Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch,
> HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch,
> HDFS-10702.006.patch, HDFS-10702.007.patch, HDFS-10702.008.patch,
> StaleReadfromStandbyNN.pdf
>
>
> Currently, clients must always talk to the active NameNode when performing
> any metadata operation, which means active NameNode could be a bottleneck for
> scalability. One way to solve this problem is to send read-only operations to
> Standby NameNode. The disadvantage is that it might be a stale read.
> Here, I'm thinking of adding a Client API to enable/disable stale read from
> Standby which gives Client the power to set the staleness restriction.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11677) OZone: SCM CLI: Implement get container command
[
https://issues.apache.org/jira/browse/HDFS-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11677:
Labels: OzonePostMerge command-line tocheck (was: command-line ozoneMerge
tocheck)
> OZone: SCM CLI: Implement get container command
> ---
>
> Key: HDFS-11677
> URL: https://issues.apache.org/jira/browse/HDFS-11677
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Chen Liang
> Labels: OzonePostMerge, command-line, tocheck
>
> Implement get container
> {code}
> hdfs scm -container get -o
> {code}
> This command works only against a closed container. If the container is
> closed, then SCM will return the address of the datanodes. The datanodes
> support an API called copyCon- tainer, which returns the container as a tar
> ball.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11735) Ozone: In Ratis, leader should validate ContainerCommandRequestProto before propagating it to followers
[ https://issues.apache.org/jira/browse/HDFS-11735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11735: Labels: OzonePostMerge tocheck (was: ozoneMerge tocheck) > Ozone: In Ratis, leader should validate ContainerCommandRequestProto before > propagating it to followers > --- > > Key: HDFS-11735 > URL: https://issues.apache.org/jira/browse/HDFS-11735 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Labels: OzonePostMerge, tocheck > Attachments: HDFS-11735-HDFS-7240.20170501.patch > > > The leader should use the API provided by HDFS-11734 to check if a > ContainerCommandRequestProto is valid before propagating it to followers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[
https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164146#comment-16164146
]
Hadoop QA commented on HDFS-12353:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 221 unchanged - 0 fixed = 227 total (was 221) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12353 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886785/HDFS-12353-3.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d47986f8a1c5 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 123342c |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/21110/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21110/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21110/testReport/ |
| modules | C:
[jira] [Commented] (HDFS-11139) Ozone: SCM: Handle duplicate Datanode ID
[ https://issues.apache.org/jira/browse/HDFS-11139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164145#comment-16164145 ] Tsz Wo Nicholas Sze commented on HDFS-11139: This seems a general problem in HDFS, not specific to Ozone. How about move this out from Ozone? > Ozone: SCM: Handle duplicate Datanode ID > - > > Key: HDFS-11139 > URL: https://issues.apache.org/jira/browse/HDFS-11139 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Labels: OzonePostMerge, tocheck > Fix For: HDFS-7240 > > > The Datanode ID is used when a data node registers. It is assumed that > datanodes are unique across the cluster. > However due to operator error or other cases we might encounter duplicate > datanode ID. SCM should be able to recognize this and handle in correctly. > Here is a sub-set of datanode scenarios it needs to handle. > 1. Normal Datanode > 2. Copy of a Datanode metadata by operator to another node > 3. A Datanode being renamed - hostname change > 4. Container Reports -- 2 machines with same datanode ID. SCM thinks they are > same node. > 5. Decommission -- we decommission both nodes if IDs are same. > 6. Commands will be send to both nodes. > So it is necessary that SCM identity when a datanode is reusing a datanode ID > that is already used by another node. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11139) Ozone: SCM: Handle duplicate Datanode ID
[ https://issues.apache.org/jira/browse/HDFS-11139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11139: Labels: OzonePostMerge tocheck (was: ozoneMerge tocheck) > Ozone: SCM: Handle duplicate Datanode ID > - > > Key: HDFS-11139 > URL: https://issues.apache.org/jira/browse/HDFS-11139 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Labels: OzonePostMerge, tocheck > Fix For: HDFS-7240 > > > The Datanode ID is used when a data node registers. It is assumed that > datanodes are unique across the cluster. > However due to operator error or other cases we might encounter duplicate > datanode ID. SCM should be able to recognize this and handle in correctly. > Here is a sub-set of datanode scenarios it needs to handle. > 1. Normal Datanode > 2. Copy of a Datanode metadata by operator to another node > 3. A Datanode being renamed - hostname change > 4. Container Reports -- 2 machines with same datanode ID. SCM thinks they are > same node. > 5. Decommission -- we decommission both nodes if IDs are same. > 6. Commands will be send to both nodes. > So it is necessary that SCM identity when a datanode is reusing a datanode ID > that is already used by another node. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11734) Ozone: provide a way to validate ContainerCommandRequestProto
[ https://issues.apache.org/jira/browse/HDFS-11734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11734: Labels: OzonePostMerge tocheck (was: ozoneMerge tocheck) > Ozone: provide a way to validate ContainerCommandRequestProto > - > > Key: HDFS-11734 > URL: https://issues.apache.org/jira/browse/HDFS-11734 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Anu Engineer >Priority: Critical > Labels: OzonePostMerge, tocheck > > We need some API to check if a ContainerCommandRequestProto is valid. > It is useful when the container pipeline is run with Ratis. Then, the leader > could first checks if a ContainerCommandRequestProto is valid before the > request is propagated to the followers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12060) Ozone: OzoneClient: Add list calls
[
https://issues.apache.org/jira/browse/HDFS-12060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164142#comment-16164142
]
Nandakumar commented on HDFS-12060:
---
[~anu], this depends upon HDFS-12385, I have linked it.
> Ozone: OzoneClient: Add list calls
> --
>
> Key: HDFS-12060
> URL: https://issues.apache.org/jira/browse/HDFS-12060
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12060-HDFS-7240.000.patch
>
>
> Support for {{listVolumes}}, {{listBuckets}}, {{listKeys}} in {{OzoneClient}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12378) TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
[
https://issues.apache.org/jira/browse/HDFS-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-12378:
-
Priority: Blocker (was: Major)
> TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk
> --
>
> Key: HDFS-12378
> URL: https://issues.apache.org/jira/browse/HDFS-12378
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiao Chen
>Priority: Blocker
>
> Saw on
> https://builds.apache.org/job/PreCommit-HDFS-Build/20928/testReport/org.apache.hadoop.hdfs/TestClientProtocolForPipelineRecovery/testZeroByteBlockRecovery/:
> Error Message
> {noformat}
> Failed to replace a bad datanode on the existing pipeline due to no more good
> datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> {noformat}
> Stacktrace
> {noformat}
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:51925,DS-274e8cc9-280b-4370-b494-6a4f0d67ccf4,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1322)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1388)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1587)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1488)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1470)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1274)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
> {noformat}
> Standard Output
> {noformat}
> 2017-08-30 18:02:37,714 [main] INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:(469)) - starting cluster: numNameNodes=1,
> numDataNodes=3
> Formatting using clusterid: testClusterID
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSEditLog
> (FSEditLog.java:newInstance(224)) - Edit logging is async:false
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(742)) - KeyProvider: null
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(120)) - fsLock is fair: true
> 2017-08-30 18:02:37,716 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:(136)) - Detailed lock hold time metrics
> enabled: false
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(763)) - fsOwner = jenkins (auth:SIMPLE)
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(764)) - supergroup = supergroup
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(765)) - isPermissionEnabled = true
> 2017-08-30 18:02:37,717 [main] INFO namenode.FSNamesystem
> (FSNamesystem.java:(776)) - HA Enabled: false
> 2017-08-30 18:02:37,718 [main] INFO common.Util
> (Util.java:isDiskStatsEnabled(395)) -
> dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO
> profiling
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(301)) - dfs.block.invalidate.limit:
> configured=1000, counted=60, effected=1000
> 2017-08-30 18:02:37,718 [main] INFO blockmanagement.DatanodeManager
> (DatanodeManager.java:(309)) -
> dfs.namenode.datanode.registration.ip-hostname-check=true
> 2017-08-30 18:02:37,719 [main] INFO blockmanagement.BlockManager
> (InvalidateBlocks.java:printBlockDeletionTime(76)) -
> dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
> 2017-08-30 18:02:37,719 [main] INFO blockmanagement.BlockManager
> (InvalidateBlocks.java:printBlockDeletionTime(82)) - The block deletion will
> start around 2017 Aug 30 18:02:37
> 2017-08-30 18:02:37,719 [main] INFO util.GSet
> (LightWeightGSet.java:computeCapacity(395)) - Computing capacity for map
> BlocksMap
> 2017-08-30 18:02:37,719 [main] INFO util.GSet
> (LightWeightGSet.java:computeCapacity(396))
[jira] [Resolved] (HDFS-12436) TestClientProtocolForPipelineRecovery fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal resolved HDFS-12436.
--
Resolution: Duplicate
Indeed, thanks.
> TestClientProtocolForPipelineRecovery fails in trunk
>
>
> Key: HDFS-12436
> URL: https://issues.apache.org/jira/browse/HDFS-12436
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Arpit Agarwal
>Priority: Blocker
>
> Fails consistently in trunk with the following exception:
> {code}
> Running org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
> Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 71.317 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
> testZeroByteBlockRecovery(org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery)
> Time elapsed: 11.422 sec <<< ERROR!
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1321)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1387)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1586)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1487)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1469)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1273)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12385) Ozone: OzoneClient: Refactoring OzoneClient API
[
https://issues.apache.org/jira/browse/HDFS-12385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164129#comment-16164129
]
Nandakumar commented on HDFS-12385:
---
Thanks [~xyao] for the review.
bq. One question I have is how do we expect the client to handle the dangling
reference inside the volume/bucket/key when the AutoClosable OzoneClient is
closed? Do we need reference counting for the proxy handle before the
OzoneClient can be closed?
Once OzoneClient is closed volume/bucket/key created using that client will
become non usable. I don't think we need to have a reference count for the
proxy. Do you see any issue with this logic?
bq. Line 80: ozone.rest.servers?
Initial idea was to have this property, which points to REST Servers
(DataNodes) so that client knows where to connect. This is no more required if
we are going with cluster discovery API - [comment |
https://issues.apache.org/jira/browse/HDFS-12385?focusedCommentId=16150630=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16150630].
Will remove it.
bq. Line 83: Will"localhost" will work with non-standalone case?
RestProtocol or RestClient is not yet ready, this is just a placeholder to
implement it later. This has to be verified.
bq. Do we have an open ticket for implement the RestProtol class?
As of now there is no open ticket, will create one shortly.
I will update a new patch addressing the review comments.
> Ozone: OzoneClient: Refactoring OzoneClient API
> ---
>
> Key: HDFS-12385
> URL: https://issues.apache.org/jira/browse/HDFS-12385
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12385-HDFS-7240.000.patch, OzoneClient.pdf
>
>
> This jira is for refactoring {{OzoneClient}} API. [^OzoneClient.pdf] will
> give an idea on how the API will look.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12436) TestClientProtocolForPipelineRecovery fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164109#comment-16164109
]
Brahma Reddy Battula commented on HDFS-12436:
-
Dupe of HDFS-12378..?
> TestClientProtocolForPipelineRecovery fails in trunk
>
>
> Key: HDFS-12436
> URL: https://issues.apache.org/jira/browse/HDFS-12436
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Arpit Agarwal
>Priority: Blocker
>
> Fails consistently in trunk with the following exception:
> {code}
> Running org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
> Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 71.317 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
> testZeroByteBlockRecovery(org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery)
> Time elapsed: 11.422 sec <<< ERROR!
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]]).
> The current failed datanode replacement policy is ALWAYS, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1321)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1387)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1586)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1487)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1469)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1273)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
[
https://issues.apache.org/jira/browse/HDFS-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164089#comment-16164089
]
Hadoop QA commented on HDFS-12438:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSShellGenericOptions |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReconstructStripedFile |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12438 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886763/HDFS-12438.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 74d3419e5784 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f4b6267 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/21109/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
[
https://issues.apache.org/jira/browse/HDFS-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164080#comment-16164080
]
Kai Zheng commented on HDFS-12438:
--
Sorry for the late. Maybe we could change it further, ending with
{{dfs.datanode.ec.reconstruction.threads}}?
> Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to
> dfs.datanode.ec.reconstruction.stripedblock.threads
> --
>
> Key: HDFS-12438
> URL: https://issues.apache.org/jira/browse/HDFS-12438
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12438.001.patch
>
>
> We should rename this config key to match other config keys used to size
> thread pools.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12423) Ozone: TopN container choosing policy should ignore containers that has no pending deletion blocks
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12423:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
I just committed this to the feature branch, thanks for the contribution
[~linyiqun].
> Ozone: TopN container choosing policy should ignore containers that has no
> pending deletion blocks
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12423-HDFS-7240.001.patch,
> HDFS-12423-HDFS-7240.002.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164067#comment-16164067
]
Hadoop QA commented on HDFS-12420:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 5 new + 184 unchanged - 1 fixed = 189 total (was 185) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestClusterId |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.namenode.TestGenericJournalConf |
| | hadoop.hdfs.qjournal.TestNNWithQJM |
| | hadoop.hdfs.server.namenode.TestAllowFormat |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestLease |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12420 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886764/HDFS-12420.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d719cb91ece5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f4b6267 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Updated] (HDFS-12423) Ozone: TopN container choosing policy should ignore containers that has no pending deletion blocks
[
https://issues.apache.org/jira/browse/HDFS-12423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-12423:
---
Summary: Ozone: TopN container choosing policy should ignore containers
that has no pending deletion blocks (was: Ozone: TopN container choosing
policy should ignore unnecessary containers)
> Ozone: TopN container choosing policy should ignore containers that has no
> pending deletion blocks
> --
>
> Key: HDFS-12423
> URL: https://issues.apache.org/jira/browse/HDFS-12423
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Labels: ozoneMerge
> Attachments: HDFS-12423-HDFS-7240.001.patch,
> HDFS-12423-HDFS-7240.002.patch
>
>
> TopN container choosing policy should ignore unnecessary containers.
> Currently TopN policy selects specified count of containers but not check the
> number of pending deletion blocks. So there is a chance there will be some
> unnecessary containers being chosen. That is say we will choose some
> containers that don't include any pending deletion blocks. The related output:
> {noformat}
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container c7a85e6e-3528-45d1-8063-cd7fec114545 for block deletion,
> pending deletion blocks num: 1.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 1d163265-8d47-4ed3-845f-f7d3eb569b83 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 21017018-be32-44e6-9f62-e7fc9f6a6021 for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 4cccadc8-ef5e-466d-bd9e-5b9705f8748c for block deletion,
> pending deletion blocks num: 0.
> 17/09/11 02:58:30 INFO impl.TopNOrderedContainerDeletionChoosingPolicy:
> Select container 55b11be9-f16f-4620-a310-07c9be3bbfee for block deletion,
> pending deletion blocks num: 0.
> {noformat}
> We should ignore these containers which pending deletion blocks num is 0,
> this can reduce some unnecessary block-deletion iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11294) libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails
[
https://issues.apache.org/jira/browse/HDFS-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164060#comment-16164060
]
Hadoop QA commented on HDFS-11294:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-8707 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
40s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
42s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
57s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
19s{color} | {color:green} HDFS-8707 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
3s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
3s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}443m 47s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_151. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}571m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_151 Failed CTEST tests |
test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static |
| | test_hdfs_ext_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:3117e2a |
| JIRA Issue | HDFS-11294 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886668/HDFS-11294.HDFS-8707.001.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 3e57f80cbc00 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / 3f92e63 |
| Default Java | 1.7.0_151 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_144
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21096/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_151.txt
|
| JDK v1.7.0_151 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21096/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21096/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails
> -
>
>
[jira] [Updated] (HDFS-12353) Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of datanode reports.
[ https://issues.apache.org/jira/browse/HDFS-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven-wugang updated HDFS-12353: - Attachment: HDFS-12353-3.patch > Modify Dfsuse percent of dfsadmin report inconsistent with Dfsuse percent of > datanode reports. > -- > > Key: HDFS-12353 > URL: https://issues.apache.org/jira/browse/HDFS-12353 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: steven-wugang >Assignee: steven-wugang > Attachments: HDFS-12353-1.patch, HDFS-12353-2.patch, > HDFS-12353-3.patch, HDFS-12353.patch > > > use command "hdfs dfsadmin -report",as follows: > [hdfs@zhd2-3 sbin]$ hdfs dfsadmin -report > Configured Capacity: 157497375621120 (143.24 TB) > Present Capacity: 148541284228197 (135.10 TB) > DFS Remaining: 56467228499968 (51.36 TB) > DFS Used: 92074055728229 (83.74 TB) > DFS Used%: 61.99% > Under replicated blocks: 1 > Blocks with corrupt replicas: 3 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > - > Live datanodes (4): > Name: 172.168.129.1:50010 (zhd2-1) > Hostname: zhd2-1 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23560170107046 (21.43 TB) > Non DFS Used: 609684660058 (567.81 GB) > DFS Remaining: 15204489138176 (13.83 TB) > DFS Used%: 59.84% > DFS Remaining%: 38.62% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 36 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.3:50010 (zhd2-3) > Hostname: zhd2-3 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23463410242057 (21.34 TB) > Non DFS Used: 620079140343 (577.49 GB) > DFS Remaining: 15290854522880 (13.91 TB) > DFS Used%: 59.59% > DFS Remaining%: 38.83% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 30 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.4:50010 (zhd2-4) > Hostname: zhd2-4 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 23908322375185 (21.74 TB) > Non DFS Used: 618808670703 (576.31 GB) > DFS Remaining: 14847212859392 (13.50 TB) > DFS Used%: 60.72% > DFS Remaining%: 37.71% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 38 > Last contact: Fri Aug 25 10:06:50 CST 2017 > Name: 172.168.129.2:50010 (zhd2-2) > Hostname: zhd2-2 > Decommission Status : Normal > Configured Capacity: 39374343905280 (35.81 TB) > DFS Used: 21142153003941 (19.23 TB) > Non DFS Used: 7107518921819 (6.46 TB) > DFS Remaining: 11124671979520 (10.12 TB) > DFS Used%: 53.70% > DFS Remaining%: 28.25% > Configured Cache Capacity: 60 (5.59 GB) > Cache Used: 0 (0 B) > Cache Remaining: 60 (5.59 GB) > Cache Used%: 0.00% > Cache Remaining%: 100.00% > Xceivers: 22 > Last contact: Fri Aug 25 10:06:50 CST 2017 > The first "DFS Used%" value on the top is DFS Used/Present Capacity,but "DFS > Used%" value in other live datanode reports is DFS Used/Configured Capacity. > The two calculation methods are inconsistent,misunderstanding may arise. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12019) Ozone: Misc : Make sure that ozone-site.xml is in etc/hadoop in tarball created from mvn package.
[ https://issues.apache.org/jira/browse/HDFS-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12019: - Summary: Ozone: Misc : Make sure that ozone-site.xml is in etc/hadoop in tarball created from mvn package. (was: Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created from mvn package.) > Ozone: Misc : Make sure that ozone-site.xml is in etc/hadoop in tarball > created from mvn package. > - > > Key: HDFS-12019 > URL: https://issues.apache.org/jira/browse/HDFS-12019 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Trivial > Labels: ozoneMerge > Attachments: HDFS-12019-HDFS-7240.001.patch > > > Right now the tarball that is created by `mvn package` command does not carry > ozone-defaults.xml. We need to make sure this is packaged as part of the > tarball. Thanks to [~xyao] for pointing this out. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12412) Change ErasureCodingWorker.stripedReadPool to cached thread pool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164036#comment-16164036
]
Hadoop QA commented on HDFS-12412:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 415 unchanged - 2 fixed = 415 total (was 417) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestParallelShortCircuitReadUnCached |
| |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
| | hadoop.hdfs.security.TestDelegationToken |
| | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
| | hadoop.hdfs.TestEncryptedTransfer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12412 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886754/HDFS-12412.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux c01dee58fa7f 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 68282c8 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-11294) libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails
[
https://issues.apache.org/jira/browse/HDFS-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164026#comment-16164026
]
Hadoop QA commented on HDFS-11294:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
2s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-8707 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
1s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
10s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
2s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
28s{color} | {color:green} HDFS-8707 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
39s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
47s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}274m 0s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK
v1.7.0_151. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}413m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_151 Failed CTEST tests | test_hdfs_ext_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:3117e2a |
| JIRA Issue | HDFS-11294 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886700/HDFS-11294.HDFS-8707.002.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 955e1287e049 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / 3f92e63 |
| Default Java | 1.7.0_151 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_144
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 |
| CTEST |
https://builds.apache.org/job/PreCommit-HDFS-Build/21101/artifact/patchprocess/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_151-ctest.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21101/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_151.txt
|
| JDK v1.7.0_151 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21101/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21101/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> libhdfs++: Segfault in HA failover if DNS lookup
[jira] [Commented] (HDFS-12412) Change ErasureCodingWorker.stripedReadPool to cached thread pool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164002#comment-16164002
]
Hudson commented on HDFS-12412:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12857 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12857/])
HDFS-12412. Change ErasureCodingWorker.stripedReadPool to cached thread (lei:
rev 123342cd0759ff88801d4f5ab10987f6e3f344b0)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Change ErasureCodingWorker.stripedReadPool to cached thread pool
>
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12412.00.patch, HDFS-12412.01.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12222) Document and test BlockLocation for erasure-coded files
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163998#comment-16163998
]
Huafeng Wang commented on HDFS-1:
-
Thanks [~andrew.wang] for your advice and help!
> Document and test BlockLocation for erasure-coded files
> ---
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-1.001.patch, HDFS-1.002.patch,
> HDFS-1.003.patch, HDFS-1.004.patch, HDFS-1.005.patch,
> HDFS-1.006.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12412) Change ErasureCodingWorker.stripedReadPool to cached thread pool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12412:
-
Resolution: Fixed
Fix Version/s: 3.0.0-beta1
Release Note: Changed {{stripedReadPool}} to unbounded cachedThreadPool.
User should combine {{dfs.datanode.ec.reconstruction.stripedblock.threads}} and
{{dfs.namenode.replication.max-streams}} to tune recovery performance.
Status: Resolved (was: Patch Available)
Thanks for the reviews, [~andrew.wang]
Ran {{TestReconstructStripedFile,TestReadStripedFileWithMissingBlocks}} locally
and passes. The rest seem not relevant. Filed HDFS-12439 for
TestReconstructStripedFile.
Committed to {{trunk}}.
> Change ErasureCodingWorker.stripedReadPool to cached thread pool
>
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12412.00.patch, HDFS-12412.01.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12412) Change ErasureCodingWorker.stripedReadPool to cached thread pool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12412:
-
Summary: Change ErasureCodingWorker.stripedReadPool to cached thread pool
(was: Remove ErasureCodingWorker.stripedReadPool)
> Change ErasureCodingWorker.stripedReadPool to cached thread pool
>
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch, HDFS-12412.01.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12439) TestReconstructStripedFile.testNNSendsErasureCodingTasks fails occasionally
Lei (Eddy) Xu created HDFS-12439:
Summary: TestReconstructStripedFile.testNNSendsErasureCodingTasks
fails occasionally
Key: HDFS-12439
URL: https://issues.apache.org/jira/browse/HDFS-12439
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding
Affects Versions: 3.0.0-alpha4
Reporter: Lei (Eddy) Xu
With error message:
{code}
Error Message
test timed out after 6 milliseconds
Stacktrace
java.lang.Exception: test timed out after 6 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:917)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1199)
at
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.hdfs.DFSTestUtil.writeFile(DFSTestUtil.java:835)
at
org.apache.hadoop.hdfs.TestReconstructStripedFile.writeFile(TestReconstructStripedFile.java:273)
at
org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:461)
at
org.apache.hadoop.hdfs.TestReconstructStripedFile.testNNSendsErasureCodingTasks(TestReconstructStripedFile.java:439)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
[ https://issues.apache.org/jira/browse/HDFS-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12438: --- Status: Patch Available (was: Open) > Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to > dfs.datanode.ec.reconstruction.stripedblock.threads > -- > > Key: HDFS-12438 > URL: https://issues.apache.org/jira/browse/HDFS-12438 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-12438.001.patch > > > We should rename this config key to match other config keys used to size > thread pools. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163970#comment-16163970
]
Hadoop QA commented on HDFS-12412:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 415 unchanged - 2 fixed = 415 total (was 417) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestPread |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.qjournal.TestNNWithQJM |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestQuota |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestReconstructStripedFile |
| Timed out junit tests |
org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData |
| | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail |
| | org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | org.apache.hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA |
| | org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12412 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886518/HDFS-12412.00.patch |
| Optional Tests | asflicense
[jira] [Commented] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
[ https://issues.apache.org/jira/browse/HDFS-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163969#comment-16163969 ] Lei (Eddy) Xu commented on HDFS-12438: -- +1 pending jenkins. Thanks for taking care of this, [~andrew.wang] > Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to > dfs.datanode.ec.reconstruction.stripedblock.threads > -- > > Key: HDFS-12438 > URL: https://issues.apache.org/jira/browse/HDFS-12438 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-12438.001.patch > > > We should rename this config key to match other config keys used to size > thread pools. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12222) Document and test BlockLocation for erasure-coded files
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163968#comment-16163968
]
Hudson commented on HDFS-1:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12856 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12856/])
HDFS-1. Document and test BlockLocation for erasure-coded files. (wang: rev
f4b6267465d139bfdaf75e25761672eaf61d8a11)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BlockLocation.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsLocatedFileStatus.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystemWithECFile.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocatedFileStatus.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/fs/Hdfs.java
> Document and test BlockLocation for erasure-coded files
> ---
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-1.001.patch, HDFS-1.002.patch,
> HDFS-1.003.patch, HDFS-1.004.patch, HDFS-1.005.patch,
> HDFS-1.006.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163963#comment-16163963
]
Ajay Kumar commented on HDFS-12420:
---
[~rushabh.shah],[~arpitagarwal],[~vagarychen] thanks for review. Attaching new
patch with suggested changes.
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage,
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12420) Disable Namenode format when data already exists
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12420:
--
Attachment: HDFS-12420.02.patch
> Disable Namenode format when data already exists
>
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch
>
>
> Disable NameNode format to avoid accidental formatting of Namenode in
> production cluster. If someone really wants to delete the complete fsImage,
> they can first delete the metadata dir and then run {code} hdfs namenode
> -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163958#comment-16163958
]
Andrew Wang commented on HDFS-12412:
I also filed and posted a patch on HDFS-12438 for the config renaming I
mentioned.
> Remove ErasureCodingWorker.stripedReadPool
> --
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch, HDFS-12412.01.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
[ https://issues.apache.org/jira/browse/HDFS-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12438: --- Labels: hdfs-ec-3.0-must-do (was: ) > Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to > dfs.datanode.ec.reconstruction.stripedblock.threads > -- > > Key: HDFS-12438 > URL: https://issues.apache.org/jira/browse/HDFS-12438 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-12438.001.patch > > > We should rename this config key to match other config keys used to size > thread pools. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
[ https://issues.apache.org/jira/browse/HDFS-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12438: --- Hadoop Flags: Incompatible change Release Note: Config key `dfs.datanode.ec.reconstruction.stripedblock.threads.size` has been renamed to `dfs.datanode.ec.reconstruction.stripedblock.threads`. > Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to > dfs.datanode.ec.reconstruction.stripedblock.threads > -- > > Key: HDFS-12438 > URL: https://issues.apache.org/jira/browse/HDFS-12438 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-12438.001.patch > > > We should rename this config key to match other config keys used to size > thread pools. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
[ https://issues.apache.org/jira/browse/HDFS-12438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12438: --- Attachment: HDFS-12438.001.patch Hi [~eddyxu] do you mind doing a quick review? > Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to > dfs.datanode.ec.reconstruction.stripedblock.threads > -- > > Key: HDFS-12438 > URL: https://issues.apache.org/jira/browse/HDFS-12438 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha3 >Reporter: Andrew Wang >Assignee: Andrew Wang > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-12438.001.patch > > > We should rename this config key to match other config keys used to size > thread pools. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12438) Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads
Andrew Wang created HDFS-12438: -- Summary: Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.stripedblock.threads Key: HDFS-12438 URL: https://issues.apache.org/jira/browse/HDFS-12438 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.0.0-alpha3 Reporter: Andrew Wang Assignee: Andrew Wang We should rename this config key to match other config keys used to size thread pools. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
[
https://issues.apache.org/jira/browse/HDFS-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163912#comment-16163912
]
Konstantin Shvachko edited comment on HDFS-12323 at 9/13/17 12:40 AM:
--
+1 on the patch.
But please check [the whitespace
warning|https://builds.apache.org/job/PreCommit-HDFS-Build/21099/artifact/patchprocess/whitespace-eol.txt].
Most of failed tests are flaky, and {{TestClientProtocolForPipelineRecovery}}
is tracked in HDFS-12436.
was (Author: shv):
+1 on the patch.
But please check the whitespace warning.
Most of failed tests are flaky, and {{TestClientProtocolForPipelineRecovery}}
is tracked in HDFS-12436.
> NameNode terminates after full GC thinking QJM unresponsive if full GC is
> much longer than timeout
> --
>
> Key: HDFS-12323
> URL: https://issues.apache.org/jira/browse/HDFS-12323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.4
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: HDFS-12323.000.patch, HDFS-12323.001.patch,
> HDFS-12323.002.patch
>
>
> HDFS-10733 attempted to fix the issue where the Namenode process would
> terminate itself if it had a GC pause which lasted longer than the QJM
> timeout, since it would think that the QJM had taken too long to respond.
> However, it only bumps up the timeout expiration by one timeout length, so if
> the GC pause was e.g. 2x the length of the timeout, a TimeoutException will
> be thrown and the NN will still terminate itself.
> Thanks to [~yangjiandan] for noting this issue as a comment on HDFS-10733; we
> have also seen this issue on a real cluster even after HDFS-10733 is applied.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163953#comment-16163953
]
Andrew Wang commented on HDFS-12412:
+1 LGTM pending precommit, thanks Eddy.
> Remove ErasureCodingWorker.stripedReadPool
> --
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch, HDFS-12412.01.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12222) Document and test BlockLocation for erasure-coded files
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163947#comment-16163947
]
Andrew Wang commented on HDFS-1:
+1 LGTM thanks for working on this Huafeng, will commit shortly.
> Document and test BlockLocation for erasure-coded files
> ---
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-1.001.patch, HDFS-1.002.patch,
> HDFS-1.003.patch, HDFS-1.004.patch, HDFS-1.005.patch,
> HDFS-1.006.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12222) Document and test BlockLocation for erasure-coded files
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-1:
---
Resolution: Fixed
Fix Version/s: 3.0.0-beta1
Status: Resolved (was: Patch Available)
Committed to trunk and branch-3.0, thanks for the contribution Huafeng!
> Document and test BlockLocation for erasure-coded files
> ---
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-1.001.patch, HDFS-1.002.patch,
> HDFS-1.003.patch, HDFS-1.004.patch, HDFS-1.005.patch,
> HDFS-1.006.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode
[
https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163920#comment-16163920
]
Lei (Eddy) Xu commented on HDFS-7859:
-
Thanks for the updates, [~Sammi]. LGTM overall. +1 pending.
Some small issues:
{code}
// loadPolicy()
if (!CodecUtil.hasCodec(policy.getCodecName()) ||
policy.getCellSize() > maxCellSize) {
// If policy is not supported in current system, set the policy state to
// DISABLED;
policy.setState(ErasureCodingPolicyState.DISABLED);
}
{code}
Does it mean that user can still enable this policy later via
{{enablePolicy()}}? It is not necessary to be addressed in this patch, but do
we have a way to guard what policy can be enabled.
> Erasure Coding: Persist erasure coding policies in NameNode
> ---
>
> Key: HDFS-7859
> URL: https://issues.apache.org/jira/browse/HDFS-7859
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: SammiChen
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch,
> HDFS-7859.004.patch, HDFS-7859.005.patch, HDFS-7859.006.patch,
> HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch,
> HDFS-7859.010.patch, HDFS-7859.011.patch, HDFS-7859.012.patch,
> HDFS-7859.013.patch, HDFS-7859.014.patch, HDFS-7859.015.patch,
> HDFS-7859.016.patch, HDFS-7859-HDFS-7285.002.patch,
> HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we
> persist EC schemas in NameNode centrally and reliably, so that EC zones can
> reference them by name efficiently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12222) Document and test BlockLocation for erasure-coded files
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-1:
---
Summary: Document and test BlockLocation for erasure-coded files (was: Add
EC information to BlockLocation)
> Document and test BlockLocation for erasure-coded files
> ---
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-1.001.patch, HDFS-1.002.patch,
> HDFS-1.003.patch, HDFS-1.004.patch, HDFS-1.005.patch,
> HDFS-1.006.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
[
https://issues.apache.org/jira/browse/HDFS-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163912#comment-16163912
]
Konstantin Shvachko commented on HDFS-12323:
+1 on the patch.
But please check the whitespace warning.
Most of failed tests are flaky, and {{TestClientProtocolForPipelineRecovery}}
is tracked in HDFS-12436.
> NameNode terminates after full GC thinking QJM unresponsive if full GC is
> much longer than timeout
> --
>
> Key: HDFS-12323
> URL: https://issues.apache.org/jira/browse/HDFS-12323
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, qjm
>Affects Versions: 2.7.4
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: HDFS-12323.000.patch, HDFS-12323.001.patch,
> HDFS-12323.002.patch
>
>
> HDFS-10733 attempted to fix the issue where the Namenode process would
> terminate itself if it had a GC pause which lasted longer than the QJM
> timeout, since it would think that the QJM had taken too long to respond.
> However, it only bumps up the timeout expiration by one timeout length, so if
> the GC pause was e.g. 2x the length of the timeout, a TimeoutException will
> be thrown and the NN will still terminate itself.
> Thanks to [~yangjiandan] for noting this issue as a comment on HDFS-10733; we
> have also seen this issue on a real cluster even after HDFS-10733 is applied.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[
https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163875#comment-16163875
]
Hadoop QA commented on HDFS-12017:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
27s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}137m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.TestFileAppend3 |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.ozone.container.replication.TestContainerReplicationManager |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.TestEncryptedTransfer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.ozone.web.client.TestKeys |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem ||
[jira] [Commented] (HDFS-12407) Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start
[
https://issues.apache.org/jira/browse/HDFS-12407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163877#comment-16163877
]
Hudson commented on HDFS-12407:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12854 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12854/])
HDFS-12407. Journal node fails to shutdown cleanly if (arp: rev
68282c8eacfedb0298741e6c41ff0b2ec71b018c)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start
> ---
>
> Key: HDFS-12407
> URL: https://issues.apache.org/jira/browse/HDFS-12407
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12407.01.patch, HDFS-12407.02.patch,
> HDFS-12407.03.patch, HDFS-12407.04.patch
>
>
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start.
> Steps to recreate the issue:
> # Change the http port for JournalNodeHttpServerr to some port which is
> already in use
> {code}dfs.journalnode.http-address{code}
> # Start the journalnode. JournalNodeHttpServer start will fail with bind
> exception while journalnode process will continue to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12360) TestLeaseRecoveryStriped.testLeaseRecovery failure
[
https://issues.apache.org/jira/browse/HDFS-12360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu resolved HDFS-12360.
--
Resolution: Duplicate
> TestLeaseRecoveryStriped.testLeaseRecovery failure
> --
>
> Key: HDFS-12360
> URL: https://issues.apache.org/jira/browse/HDFS-12360
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>
> TestLeaseRecoveryStriped.testLeaseRecovery failed:
> {code}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.hdfs.TestLeaseRecoveryStriped
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.808 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.TestLeaseRecoveryStriped
> testLeaseRecovery(org.apache.hadoop.hdfs.TestLeaseRecoveryStriped) Time
> elapsed: 15.509 sec <<< FAILURE!
> java.lang.AssertionError: failed testCase at i=0, blockLengths=[10485760,
> 4194304, 6291456, 10485760, 11534336, 11534336, 6291456, 4194304, 3145728]
> java.io.IOException: Failed: the number of failed blocks = 4 > the number of
> data blocks = 3
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamers(DFSStripedOutputStream.java:393)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.handleStreamerFailure(DFSStripedOutputStream.java:411)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.flushAllInternals(DFSStripedOutputStream.java:1128)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:628)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:564)
> at
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:217)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:164)
> at
> org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:145)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:79)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:48)
> at java.io.DataOutputStream.write(DataOutputStream.java:88)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.writePartialBlocks(TestLeaseRecoveryStriped.java:182)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:158)
> at
> org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> at org.junit.Assert.fail(Assert.java:88)
> at
>
[jira] [Updated] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12412:
-
Attachment: HDFS-12412.01.patch
Thanks for the inputs, [~andrew.wang]. Removed references of
{{dfs.datanode.ec.reconstruction.stripedread.threads}} in the {01} patch.
> Remove ErasureCodingWorker.stripedReadPool
> --
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch, HDFS-12412.01.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12407) Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start
[
https://issues.apache.org/jira/browse/HDFS-12407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-12407:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
I've committed this. The test case failures are all unrelated.
Thanks for the contribution [~ajayydv].
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start
> ---
>
> Key: HDFS-12407
> URL: https://issues.apache.org/jira/browse/HDFS-12407
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12407.01.patch, HDFS-12407.02.patch,
> HDFS-12407.03.patch, HDFS-12407.04.patch
>
>
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start.
> Steps to recreate the issue:
> # Change the http port for JournalNodeHttpServerr to some port which is
> already in use
> {code}dfs.journalnode.http-address{code}
> # Start the journalnode. JournalNodeHttpServer start will fail with bind
> exception while journalnode process will continue to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[
https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12412:
-
Status: Patch Available (was: Open)
> Remove ErasureCodingWorker.stripedReadPool
> --
>
> Key: HDFS-12412
> URL: https://issues.apache.org/jira/browse/HDFS-12412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12412.00.patch
>
>
> In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule
> the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads
> in each recovery task. We only need one of them to throttle the speed of
> recovery process, because each EC recovery task has a fix number of source
> readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the
> speed of EC recovery can be throttled by {{strippedReconstructionPool}} with
> {{xmitsInProgress}}.
> Moreover, keeping {{stripedReadPool}} makes customer difficult to understand
> and calculate the right balance between
> {{dfs.datanode.ec.reconstruction.stripedread.threads}},
> {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and
> {{maxReplicationStreams}}. For example, a small {{stripread.threads}}
> (comparing to which {{reconstruction.threads.size}} implies), will
> unnecessarily limit the speed of recovery, which leads to larger MTTR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12408) Many EC tests fail in trunk
[ https://issues.apache.org/jira/browse/HDFS-12408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-12408. -- Resolution: Done Resolving since HDFS-12417 is committed. That significantly reduced noise in the pre-commit runs. Filed two new blockers for other consistently failing tests - HDFS-12436 and HDFS-12437. > Many EC tests fail in trunk > --- > > Key: HDFS-12408 > URL: https://issues.apache.org/jira/browse/HDFS-12408 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha4 >Reporter: Arpit Agarwal >Priority: Blocker > > Many EC tests are failing in pre-commit runs. e.g. > https://builds.apache.org/job/PreCommit-HDFS-Build/21055/testReport/ > https://builds.apache.org/job/PreCommit-HDFS-Build/21052/testReport/ > https://builds.apache.org/job/PreCommit-HDFS-Build/21048/testReport/ > This is creating a lot of noise in Jenkins runs outputs. We should either fix > or disable these tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12430) Rebasing HDFS-10467 (3)
[
https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163792#comment-16163792
]
Íñigo Goiri commented on HDFS-12430:
The javac failure doesn't show any federation/router error.
The {{TestNamenodeHeartbeat}} unit test runs fine in my machine.
> Rebasing HDFS-10467 (3)
> ---
>
> Key: HDFS-12430
> URL: https://issues.apache.org/jira/browse/HDFS-12430
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12430-HDFS-10467.000.patch
>
>
> HDFS-10467 is broken because of the new methods added to {{ClientProtocol}}
> from HDFS-12269 and HDFS-12218.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12437) TestLeaseRecoveryStriped fails in trunk
Arpit Agarwal created HDFS-12437:
Summary: TestLeaseRecoveryStriped fails in trunk
Key: HDFS-12437
URL: https://issues.apache.org/jira/browse/HDFS-12437
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Arpit Agarwal
Priority: Blocker
Fails consistently for me in trunk with the following call stack.
{code}
TestLeaseRecoveryStriped.testLeaseRecovery:152 failed testCase at i=0,
blockLengths=[5242880, 7340032, 5242880, 8388608, 7340032, 3145728, 9437184,
10485760, 11534336]
java.io.IOException: Failed: the number of failed blocks = 4 > the number of
parity blocks = 3
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamers(DFSStripedOutputStream.java:394)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.handleStreamerFailure(DFSStripedOutputStream.java:412)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.flushAllInternals(DFSStripedOutputStream.java:1264)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:629)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:565)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:217)
at
org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:164)
at
org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:145)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:79)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:48)
at java.io.DataOutputStream.write(DataOutputStream.java:88)
at
org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.writePartialBlocks(TestLeaseRecoveryStriped.java:182)
at
org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.runTest(TestLeaseRecoveryStriped.java:158)
at
org.apache.hadoop.hdfs.TestLeaseRecoveryStriped.testLeaseRecovery(TestLeaseRecoveryStriped.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12436) TestClientProtocolForPipelineRecovery fails in trunk
Arpit Agarwal created HDFS-12436:
Summary: TestClientProtocolForPipelineRecovery fails in trunk
Key: HDFS-12436
URL: https://issues.apache.org/jira/browse/HDFS-12436
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Arpit Agarwal
Priority: Blocker
Fails consistently in trunk with the following exception:
{code}
Running org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
Tests run: 11, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 71.317 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
testZeroByteBlockRecovery(org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery)
Time elapsed: 11.422 sec <<< ERROR!
java.io.IOException: Failed to replace a bad datanode on the existing pipeline
due to no more good datanodes being available to try. (Nodes:
current=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]],
original=[DatanodeInfoWithStorage[127.0.0.1:63722,DS-9befc828-8ff7-4284-8fba-a6c55627ab3d,DISK]]).
The current failed datanode replacement policy is ALWAYS, and a client may
configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'
in its configuration.
at
org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1321)
at
org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1387)
at
org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1586)
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1487)
at
org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1469)
at
org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1273)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:684)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12407) Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start
[
https://issues.apache.org/jira/browse/HDFS-12407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163733#comment-16163733
]
Hadoop QA commented on HDFS-12407:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 17 unchanged - 1 fixed = 17 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestAbandonBlock |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestMissingBlocksAlert |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12407 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886710/HDFS-12407.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e269c18f4cf1 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f1d751b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21102/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21102/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21102/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start
>
[jira] [Commented] (HDFS-12430) Rebasing HDFS-10467 (3)
[
https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163727#comment-16163727
]
Hadoop QA commented on HDFS-12430:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-10467 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 7m
9s{color} | {color:red} root in HDFS-10467 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} | {color:red} hadoop-hdfs in HDFS-10467 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} HDFS-10467 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} hadoop-hdfs in HDFS-10467 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} hadoop-hdfs in HDFS-10467 failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} HDFS-10467 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 324 new + 84 unchanged
- 11 fixed = 408 total (was 95) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new + 9
unchanged - 4 fixed = 9 total (was 13) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.server.federation.router.TestNamenodeHeartbeat |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12430 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886713/HDFS-12430-HDFS-10467.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b115ce78eb5f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10467 / 66dc3d1 |
| Default Java | 1.8.0_144 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/21103/artifact/patchprocess/branch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/21103/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
[jira] [Commented] (HDFS-12407) Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start
[
https://issues.apache.org/jira/browse/HDFS-12407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163716#comment-16163716
]
Arpit Agarwal commented on HDFS-12407:
--
+1 pending Jenkins.
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start
> ---
>
> Key: HDFS-12407
> URL: https://issues.apache.org/jira/browse/HDFS-12407
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12407.01.patch, HDFS-12407.02.patch,
> HDFS-12407.03.patch, HDFS-12407.04.patch
>
>
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start.
> Steps to recreate the issue:
> # Change the http port for JournalNodeHttpServerr to some port which is
> already in use
> {code}dfs.journalnode.http-address{code}
> # Start the journalnode. JournalNodeHttpServer start will fail with bind
> exception while journalnode process will continue to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12424) Datatable sorting on the Datanode Information page in the Namenode UI is broken
[
https://issues.apache.org/jira/browse/HDFS-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163671#comment-16163671
]
Hadoop QA commented on HDFS-12424:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 11m
1s{color} | {color:red} Docker failed to build yetus/hadoop:5af2af1. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12424 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886702/HDFS-12424-branch-2.8.1.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21105/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Datatable sorting on the Datanode Information page in the Namenode UI is
> broken
> ---
>
> Key: HDFS-12424
> URL: https://issues.apache.org/jira/browse/HDFS-12424
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Shawna Martell
>Assignee: Shawna Martell
> Attachments: HDFS-12424.1.patch, HDFS-12424-branch-2.8.1.patch
>
>
> Attempting to sort the "In operation" table by Last contact, Capacity,
> Blocks, or Version can result in unexpected behavior. Sorting by Blocks and
> Version actually sort by entirely different columns, and Last contact and
> Capacity are not being sorted numerically but rather alphabetically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12424) Datatable sorting on the Datanode Information page in the Namenode UI is broken
[ https://issues.apache.org/jira/browse/HDFS-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163664#comment-16163664 ] Daryn Sharp commented on HDFS-12424: Quasi +1. I really don't understand the JS magic but it looks straightforward. All I know is sorting has been completely broken ever since it was added to the new ui. I clicked around on a test NN and certainly seems to work. Any other web experts want to review? Otherwise I'll commit in a few days. > Datatable sorting on the Datanode Information page in the Namenode UI is > broken > --- > > Key: HDFS-12424 > URL: https://issues.apache.org/jira/browse/HDFS-12424 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Shawna Martell >Assignee: Shawna Martell > Attachments: HDFS-12424.1.patch, HDFS-12424-branch-2.8.1.patch > > > Attempting to sort the "In operation" table by Last contact, Capacity, > Blocks, or Version can result in unexpected behavior. Sorting by Blocks and > Version actually sort by entirely different columns, and Last contact and > Capacity are not being sorted numerically but rather alphabetically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout
[
https://issues.apache.org/jira/browse/HDFS-12323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163647#comment-16163647
]
Hadoop QA commented on HDFS-12323:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 8 unchanged - 1 fixed = 9 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
|
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12323 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886698/HDFS-12323.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 453c5891d47e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f1d751b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/21099/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/21099/artifact/patchprocess/whitespace-eol.txt
|
| unit |
[jira] [Commented] (HDFS-12409) Add metrics of execution time of different stages in EC recovery task
[
https://issues.apache.org/jira/browse/HDFS-12409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163641#comment-16163641
]
Hadoop QA commented on HDFS-12409:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886525/HDFS-12409.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1d24ae41575f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f1d751b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21100/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21100/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21100/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add metrics of execution time of different stages in EC recovery task
> -
>
> Key: HDFS-12409
> URL:
[jira] [Updated] (HDFS-12424) Datatable sorting on the Datanode Information page in the Namenode UI is broken
[ https://issues.apache.org/jira/browse/HDFS-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shawna Martell updated HDFS-12424: -- Status: Patch Available (was: Open) > Datatable sorting on the Datanode Information page in the Namenode UI is > broken > --- > > Key: HDFS-12424 > URL: https://issues.apache.org/jira/browse/HDFS-12424 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Shawna Martell >Assignee: Shawna Martell > Attachments: HDFS-12424.1.patch, HDFS-12424-branch-2.8.1.patch > > > Attempting to sort the "In operation" table by Last contact, Capacity, > Blocks, or Version can result in unexpected behavior. Sorting by Blocks and > Version actually sort by entirely different columns, and Last contact and > Capacity are not being sorted numerically but rather alphabetically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12019) Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created from mvn package.
[ https://issues.apache.org/jira/browse/HDFS-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163624#comment-16163624 ] Anu Engineer commented on HDFS-12019: - +1, I will commit this shortly. > Ozone: Misc : Make sure that ozone-defaults.xml to part of tarball created > from mvn package. > > > Key: HDFS-12019 > URL: https://issues.apache.org/jira/browse/HDFS-12019 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh >Priority: Trivial > Labels: ozoneMerge > Attachments: HDFS-12019-HDFS-7240.001.patch > > > Right now the tarball that is created by `mvn package` command does not carry > ozone-defaults.xml. We need to make sure this is packaged as part of the > tarball. Thanks to [~xyao] for pointing this out. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163618#comment-16163618 ] Anu Engineer commented on HDFS-12017: - +1, pending jenkins. > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12017-HDFS-7240.000.patch > > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12017: Status: Patch Available (was: Open) > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12017-HDFS-7240.000.patch > > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12268) Ozone: Add metrics for pending storage container requests
[ https://issues.apache.org/jira/browse/HDFS-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163613#comment-16163613 ] Chen Liang commented on HDFS-12268: --- Thanks [~linyiqun] for updating the patch! A few more minor comments. Basically can we make the name methods consistent with the metric name? Specifically: 1. can we rename methods incrContainerOpsMetrics and decrContainerOpsMetrics to incrPendingContainerOpsMetrics and decrPendingContainerOpsMetrics 2. can we rename variable pendingOpsLatency to containerOpsLatency (because they are no longer pending when their latencies are added) +1 with this fixed. > Ozone: Add metrics for pending storage container requests > - > > Key: HDFS-12268 > URL: https://issues.apache.org/jira/browse/HDFS-12268 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: ozoneMerge > Attachments: HDFS-12268-HDFS-7240.001.patch, > HDFS-12268-HDFS-7240.002.patch, HDFS-12268-HDFS-7240.003.patch, > HDFS-12268-HDFS-7240.004.patch, HDFS-12268-HDFS-7240.005.patch, > HDFS-12268-HDFS-7240.006.patch > > > As storage container async interface has been supported after HDFS-11580, we > need to keep an eye on the queue depth of pending container requests. It can > help us better found if there are some performance problems. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12428) Upgrade JUnit from 4 to 5 in hadoop-hdfs cli
[ https://issues.apache.org/jira/browse/HDFS-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HDFS-12428: - Assignee: Bharat Viswanadham > Upgrade JUnit from 4 to 5 in hadoop-hdfs cli > > > Key: HDFS-12428 > URL: https://issues.apache.org/jira/browse/HDFS-12428 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Ajay Kumar >Assignee: Bharat Viswanadham > > Upgrade JUnit from 4 to 5 in hadoop-hdfs cli -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11096) Support rolling upgrade between 2.x and 3.x
[ https://issues.apache.org/jira/browse/HDFS-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163583#comment-16163583 ] Ray Chiang commented on HDFS-11096: --- FYI, while going through the JACC stuff for YARN-6142, I did gather up these JIRAs at HADOOP-14534, all of which did actually correctly get marked as incompatible. > Support rolling upgrade between 2.x and 3.x > --- > > Key: HDFS-11096 > URL: https://issues.apache.org/jira/browse/HDFS-11096 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rolling upgrades >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Sean Mackrory >Priority: Blocker > Attachments: HDFS-11096.001.patch, HDFS-11096.002.patch > > > trunk has a minimum software version of 3.0.0-alpha1. This means we can't > rolling upgrade between branch-2 and trunk. > This is a showstopper for large deployments. Unless there are very compelling > reasons to break compatibility, let's restore the ability to rolling upgrade > to 3.x releases. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12430) Rebasing HDFS-10467 (3)
[
https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163582#comment-16163582
]
Íñigo Goiri commented on HDFS-12430:
HDFS-12381 showed these errors in HDFS-10467 after rebasing. I'll wait for
Jenkins and commit.
> Rebasing HDFS-10467 (3)
> ---
>
> Key: HDFS-12430
> URL: https://issues.apache.org/jira/browse/HDFS-12430
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12430-HDFS-10467.000.patch
>
>
> HDFS-10467 is broken because of the new methods added to {{ClientProtocol}}
> from HDFS-12269 and HDFS-12218.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12434) Upgrade JUnit from 4 to 5 in hadoop-hdfs tools
[ https://issues.apache.org/jira/browse/HDFS-12434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDFS-12434: - Assignee: Ajay Kumar > Upgrade JUnit from 4 to 5 in hadoop-hdfs tools > -- > > Key: HDFS-12434 > URL: https://issues.apache.org/jira/browse/HDFS-12434 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Ajay Kumar >Assignee: Ajay Kumar > > Upgrade JUnit from 4 to 5 in hadoop-hdfs tools (org.apache.hadoop.tools) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12430) Rebasing HDFS-10467 (3)
[
https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-12430:
---
Description: HDFS-10467 is broken because of the new methods added to
{{ClientProtocol}} from HDFS-12269 and HDFS-12218.
> Rebasing HDFS-10467 (3)
> ---
>
> Key: HDFS-12430
> URL: https://issues.apache.org/jira/browse/HDFS-12430
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Fix For: HDFS-10467
>
> Attachments: HDFS-12430-HDFS-10467.000.patch
>
>
> HDFS-10467 is broken because of the new methods added to {{ClientProtocol}}
> from HDFS-12269 and HDFS-12218.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12432) Upgrade JUnit from 4 to 5 in hadoop-hdfs metrics2.sink
[ https://issues.apache.org/jira/browse/HDFS-12432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDFS-12432: - Assignee: Ajay Kumar > Upgrade JUnit from 4 to 5 in hadoop-hdfs metrics2.sink > --- > > Key: HDFS-12432 > URL: https://issues.apache.org/jira/browse/HDFS-12432 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Ajay Kumar >Assignee: Ajay Kumar > > Upgrade JUnit from 4 to 5 in hadoop-hdfs metrics2 > (org.apache.hadoop.metrics2.sink) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12435) Upgrade JUnit from 4 to 5 in hadoop-hdfs tracing
Ajay Kumar created HDFS-12435: - Summary: Upgrade JUnit from 4 to 5 in hadoop-hdfs tracing Key: HDFS-12435 URL: https://issues.apache.org/jira/browse/HDFS-12435 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ajay Kumar Upgrade JUnit from 4 to 5 in hadoop-hdfs tracing (org.apache.hadoop.tracing) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12434) Upgrade JUnit from 4 to 5 in hadoop-hdfs tools
Ajay Kumar created HDFS-12434: - Summary: Upgrade JUnit from 4 to 5 in hadoop-hdfs tools Key: HDFS-12434 URL: https://issues.apache.org/jira/browse/HDFS-12434 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ajay Kumar Upgrade JUnit from 4 to 5 in hadoop-hdfs tools (org.apache.hadoop.tools) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12430) Rebasing HDFS-10467 (3)
[ https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12430: --- Attachment: HDFS-12430-HDFS-10467.000.patch > Rebasing HDFS-10467 (3) > --- > > Key: HDFS-12430 > URL: https://issues.apache.org/jira/browse/HDFS-12430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: HDFS-12430-HDFS-10467.000.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12430) Rebasing HDFS-10467 (3)
[ https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12430: --- Status: Patch Available (was: Open) > Rebasing HDFS-10467 (3) > --- > > Key: HDFS-12430 > URL: https://issues.apache.org/jira/browse/HDFS-12430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12433) Upgrade JUnit from 4 to 5 in hadoop-hdfs security
Ajay Kumar created HDFS-12433: - Summary: Upgrade JUnit from 4 to 5 in hadoop-hdfs security Key: HDFS-12433 URL: https://issues.apache.org/jira/browse/HDFS-12433 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ajay Kumar Upgrade JUnit from 4 to 5 in hadoop-hdfs security (org.apache.hadoop.security) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12432) Upgrade JUnit from 4 to 5 in hadoop-hdfs metrics2.sink
Ajay Kumar created HDFS-12432: - Summary: Upgrade JUnit from 4 to 5 in hadoop-hdfs metrics2.sink Key: HDFS-12432 URL: https://issues.apache.org/jira/browse/HDFS-12432 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ajay Kumar Upgrade JUnit from 4 to 5 in hadoop-hdfs metrics2 (org.apache.hadoop.metrics2.sink) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[
https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163575#comment-16163575
]
Nandakumar commented on HDFS-12017:
---
Following are the ports that are already in use
|| Port || Property ||
| 8485 | dfs.journalnode.rpc-address |
| 8480 | dfs.journalnode.http-address |
| 8481 | dfs.journalnode.https-address |
| 9820 | dfs.namenode.rpc-address |
| 9860 | ozone.scm.client.port |
| 9861 | ozone.scm.datanode.port |
| 9862 | OZONE_KSM_PORT_DEFAULT |
| 9863 | ozone.scm.block.client.port |
| 9864 | dfs.datanode.http.address |
| 9865 | dfs.datanode.https.address |
| 9866 | dfs.datanode.address |
| 9867 | dfs.datanode.ipc.address |
| 9868 | dfs.namenode.secondary.http-address |
| 9869 | dfs.namenode.secondary.https-address |
| 9870 | dfs.namenode.http-address |
| 9871 | dfs.namenode.https-address |
| 50100 | dfs.namenode.backup.address |
| 50105 | dfs.namenode.backup.http-address |
{{dfs.container.ipc}} is currently using 50011, this patch changes it to 9859.
> Ozone: Container: Move IPC port to 98xx range
> -
>
> Key: HDFS-12017
> URL: https://issues.apache.org/jira/browse/HDFS-12017
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Nandakumar
> Labels: ozoneMerge
> Attachments: HDFS-12017-HDFS-7240.000.patch
>
>
> We use port 50011 port -- This is an old format of choosing port. Hadoop 3
> has already moved to port ranges under 98xx. In fact all of KSM/SCM and
> CBlock is using 98xx range. We should move 50011 to a port under 98xx.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12430) Rebasing HDFS-10467 (3)
[ https://issues.apache.org/jira/browse/HDFS-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-12430: -- Assignee: Íñigo Goiri > Rebasing HDFS-10467 (3) > --- > > Key: HDFS-12430 > URL: https://issues.apache.org/jira/browse/HDFS-12430 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12431) Upgrade JUnit from 4 to 5 in hadoop-hdfs hdfs
Ajay Kumar created HDFS-12431: - Summary: Upgrade JUnit from 4 to 5 in hadoop-hdfs hdfs Key: HDFS-12431 URL: https://issues.apache.org/jira/browse/HDFS-12431 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ajay Kumar Upgrade JUnit from 4 to 5 in hadoop-hdfs hdfs (i.e org.apache.hadoop.hdfs) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12017) Ozone: Container: Move IPC port to 98xx range
[ https://issues.apache.org/jira/browse/HDFS-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandakumar updated HDFS-12017: -- Attachment: HDFS-12017-HDFS-7240.000.patch > Ozone: Container: Move IPC port to 98xx range > - > > Key: HDFS-12017 > URL: https://issues.apache.org/jira/browse/HDFS-12017 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Nandakumar > Labels: ozoneMerge > Attachments: HDFS-12017-HDFS-7240.000.patch > > > We use port 50011 port -- This is an old format of choosing port. Hadoop 3 > has already moved to port ranges under 98xx. In fact all of KSM/SCM and > CBlock is using 98xx range. We should move 50011 to a port under 98xx. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12429) Upgrade JUnit from 4 to 5 in hadoop-hdfs fs
Ajay Kumar created HDFS-12429: - Summary: Upgrade JUnit from 4 to 5 in hadoop-hdfs fs Key: HDFS-12429 URL: https://issues.apache.org/jira/browse/HDFS-12429 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ajay Kumar Upgrade JUnit from 4 to 5 in hadoop-hdfs fs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12430) Rebasing HDFS-10467 (3)
Íñigo Goiri created HDFS-12430: -- Summary: Rebasing HDFS-10467 (3) Key: HDFS-12430 URL: https://issues.apache.org/jira/browse/HDFS-12430 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12428) Upgrade JUnit from 4 to 5 in hadoop-hdfs cli
Ajay Kumar created HDFS-12428: - Summary: Upgrade JUnit from 4 to 5 in hadoop-hdfs cli Key: HDFS-12428 URL: https://issues.apache.org/jira/browse/HDFS-12428 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ajay Kumar Upgrade JUnit from 4 to 5 in hadoop-hdfs cli -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[
https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163559#comment-16163559
]
Íñigo Goiri commented on HDFS-12381:
The error is:
{code}
RouterRpcServer.java:[138,7] error: RouterRpcServer is not abstract and does
not override abstract method getECBlockGroupStats() in ClientProtocol
{code}
Apparently there is yet one more method addition to {{ClientProtocol}} related
to EC.
This is weird because HDFS-10467 was working and I haven't rebased.
I'll post another rebase JIRA for HDFS-10467.
> [Documentation] Adding configuration keys for the Router
>
>
> Key: HDFS-12381
> URL: https://issues.apache.org/jira/browse/HDFS-12381
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Fix For: HDFS-10467
>
> Attachments: HDFS-12381-HDFS-10467.000.patch,
> HDFS-12381-HDFS-10467.001.patch
>
>
> Adding configuration options in tabular format.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12407) Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start
[
https://issues.apache.org/jira/browse/HDFS-12407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12407:
--
Attachment: HDFS-12407.04.patch
removed unused import
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start
> ---
>
> Key: HDFS-12407
> URL: https://issues.apache.org/jira/browse/HDFS-12407
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12407.01.patch, HDFS-12407.02.patch,
> HDFS-12407.03.patch, HDFS-12407.04.patch
>
>
> Journal nodes fails to shutdown cleanly if JournalNodeHttpServer or
> JournalNodeRpcServer fails to start.
> Steps to recreate the issue:
> # Change the http port for JournalNodeHttpServerr to some port which is
> already in use
> {code}dfs.journalnode.http-address{code}
> # Start the journalnode. JournalNodeHttpServer start will fail with bind
> exception while journalnode process will continue to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163554#comment-16163554 ] Íñigo Goiri commented on HDFS-11576: The unit tests seem the ones disabled tests in HDFS-12417 which was committed an hour ago. This fix looks good to me. > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.009.patch, HDFS-11576.010.patch, HDFS-11576.011.patch, > HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12381) [Documentation] Adding configuration keys for the Router
[ https://issues.apache.org/jira/browse/HDFS-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163550#comment-16163550 ] Brahma Reddy Battula commented on HDFS-12381: - There are compilations errors , can you look once? I am on mobile.Errors might not related to this jira. > [Documentation] Adding configuration keys for the Router > > > Key: HDFS-12381 > URL: https://issues.apache.org/jira/browse/HDFS-12381 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: HDFS-10467 > > Attachments: HDFS-12381-HDFS-10467.000.patch, > HDFS-12381-HDFS-10467.001.patch > > > Adding configuration options in tabular format. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[
https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163534#comment-16163534
]
Hadoop QA commented on HDFS-11576:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
56s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.hdfs.TestLeaseRecoveryStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestPipelines |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
\\
\\
||
[jira] [Updated] (HDFS-12427) libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine
[ https://issues.apache.org/jira/browse/HDFS-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12427: --- Attachment: HDFS-12427.HDFS-8707.000.patch First cut at a fix. Can't find a simple reproducer. The bug is fairly rare and shows up when the FS is canceled and destroyed in the middle of putting together RPC packets while under lots of load. > libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine > -- > > Key: HDFS-12427 > URL: https://issues.apache.org/jira/browse/HDFS-12427 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-12427.HDFS-8707.000.patch > > > The lifetime of Request objects is tied to the worker thread(s) in the async > event loop. In the current code there's nothing that prevents a request from > outliving the RpcEngine (bound to FileSystem) while it's waiting for IO. If > the Request, or a task that makes a new request, outlives the RpcEngine it > attempts to dereference a dangling pointer and either crashes or continues to > run with bad data. > Proposed fix is to reference count the RpcEngine via shared_ptr so that > Requests can hold a weak_ptr to it. When a request or RpcConnection > attempting to make a request needs something from the RpcEngine like a call > id number it can promote the weak_ptr to a shared_ptr. If it's unable to > promote because the RpcEngine has been destroyed the Request's handler can be > invoked with an appropriate error message. A weak_ptr must be used rather > than a shared_ptr to avoid reference cycles. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12427) libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine
James Clampffer created HDFS-12427: -- Summary: libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine Key: HDFS-12427 URL: https://issues.apache.org/jira/browse/HDFS-12427 Project: Hadoop HDFS Issue Type: Sub-task Reporter: James Clampffer Assignee: James Clampffer Priority: Critical The lifetime of Request objects is tied to the worker thread(s) in the async event loop. In the current code there's nothing that prevents a request from outliving the RpcEngine (bound to FileSystem) while it's waiting for IO. If the Request, or a task that makes a new request, outlives the RpcEngine it attempts to dereference a dangling pointer and either crashes or continues to run with bad data. Proposed fix is to reference count the RpcEngine via shared_ptr so that Requests can hold a weak_ptr to it. When a request or RpcConnection attempting to make a request needs something from the RpcEngine like a call id number it can promote the weak_ptr to a shared_ptr. If it's unable to promote because the RpcEngine has been destroyed the Request's handler can be invoked with an appropriate error message. A weak_ptr must be used rather than a shared_ptr to avoid reference cycles. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12426) Ozone: BlockManager MBean unregister failure upon shutdown
[ https://issues.apache.org/jira/browse/HDFS-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain reassigned HDFS-12426: -- Assignee: Lokesh Jain > Ozone: BlockManager MBean unregister failure upon shutdown > -- > > Key: HDFS-12426 > URL: https://issues.apache.org/jira/browse/HDFS-12426 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-7240 >Reporter: Xiaoyu Yao >Assignee: Lokesh Jain > > This was found during unit test, seems introduced by HDFS-12005. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12426) Ozone: BlockManager MBean unregister failure upon shutdown
[
https://issues.apache.org/jira/browse/HDFS-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16163512#comment-16163512
]
Xiaoyu Yao commented on HDFS-12426:
---
Stack:
{code}
2017-09-12 12:13:04,176 [main] WARN util.MBeans (MBeans.java:unregister(135))
- Error unregistering Hadoop:service=BlockManager,name=BlockManagerImpl
javax.management.InstanceNotFoundException:
Hadoop:service=BlockManager,name=BlockManagerImpl
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:133)
at
org.apache.hadoop.ozone.scm.block.BlockManagerImpl.close(BlockManagerImpl.java:592)
at org.apache.hadoop.io.IOUtils.cleanupWithLogger(IOUtils.java:278)
at
org.apache.hadoop.ozone.scm.StorageContainerManager.stop(StorageContainerManager.java:672)
at
org.apache.hadoop.ozone.MiniOzoneCluster.shutdown(MiniOzoneCluster.java:192)
at
org.apache.hadoop.ozone.web.client.TestOzoneClient.shutdown(TestOzoneClient.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
2017
{code}
> Ozone: BlockManager MBean unregister failure upon shutdown
> --
>
> Key: HDFS-12426
> URL: https://issues.apache.org/jira/browse/HDFS-12426
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>
> This was found during unit test, seems introduced by HDFS-12005.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
