[jira] [Updated] (HDFS-16860) Upgrade moment.min.js to 2.29.4
[ https://issues.apache.org/jira/browse/HDFS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated HDFS-16860: Component/s: ui > Upgrade moment.min.js to 2.29.4 > --- > > Key: HDFS-16860 > URL: https://issues.apache.org/jira/browse/HDFS-16860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > > Upgrade moment.min.js to 2.29.4 to resolve > https://nvd.nist.gov/vuln/detail/CVE-2022-31129 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16860) Upgrade moment.min.js to 2.29.4
[ https://issues.apache.org/jira/browse/HDFS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated HDFS-16860: Affects Version/s: 3.4.0 > Upgrade moment.min.js to 2.29.4 > --- > > Key: HDFS-16860 > URL: https://issues.apache.org/jira/browse/HDFS-16860 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > > Upgrade moment.min.js to 2.29.4 to resolve > https://nvd.nist.gov/vuln/detail/CVE-2022-31129 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16860) Upgrade moment.min.js to 2.29.4
D M Murali Krishna Reddy created HDFS-16860: --- Summary: Upgrade moment.min.js to 2.29.4 Key: HDFS-16860 URL: https://issues.apache.org/jira/browse/HDFS-16860 Project: Hadoop HDFS Issue Type: Improvement Reporter: D M Murali Krishna Reddy Assignee: D M Murali Krishna Reddy Upgrade moment.min.js to 2.29.4 to resolve https://nvd.nist.gov/vuln/detail/CVE-2022-31129 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16761) Namenode UI for Datanodes page not loading if any data node is down
Krishna Reddy created HDFS-16761: Summary: Namenode UI for Datanodes page not loading if any data node is down Key: HDFS-16761 URL: https://issues.apache.org/jira/browse/HDFS-16761 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.2.2 Reporter: Krishna Reddy Fix For: 3.2.2 Steps to reproduce: - Install the hadoop components and add 3 datanodes - Enable namenode HA - Open Namenode UI and check datanode page - check all datanodes will display - Now make one datanode down - wait for 10 minutes time as heartbeat expires - Refresh namenode page and check Actual Result: It is showing error message "NameNode is still loading. Redirecting to the Startup Progress page." -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16652) Upgrade jquery datatable version references to v1.10.19
[ https://issues.apache.org/jira/browse/HDFS-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated HDFS-16652: Attachment: HDFS-16652.001.patch > Upgrade jquery datatable version references to v1.10.19 > --- > > Key: HDFS-16652 > URL: https://issues.apache.org/jira/browse/HDFS-16652 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Attachments: HDFS-16652.001.patch > > > Upgrade jquery datatable version references in hdfs webapp to v1.10.19 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16652) Upgrade jquery datatable version references to v1.10.19
[ https://issues.apache.org/jira/browse/HDFS-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated HDFS-16652: Status: Patch Available (was: Open) > Upgrade jquery datatable version references to v1.10.19 > --- > > Key: HDFS-16652 > URL: https://issues.apache.org/jira/browse/HDFS-16652 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Attachments: HDFS-16652.001.patch > > > Upgrade jquery datatable version references in hdfs webapp to v1.10.19 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16652) Upgrade jquery datatable version references to v1.10.19
D M Murali Krishna Reddy created HDFS-16652: --- Summary: Upgrade jquery datatable version references to v1.10.19 Key: HDFS-16652 URL: https://issues.apache.org/jira/browse/HDFS-16652 Project: Hadoop HDFS Issue Type: Improvement Reporter: D M Murali Krishna Reddy Assignee: D M Murali Krishna Reddy Upgrade jquery datatable version references in hdfs webapp to v1.10.19 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16562) Upgrade moment.min.js to 2.29.2
[ https://issues.apache.org/jira/browse/HDFS-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy reassigned HDFS-16562: --- Assignee: D M Murali Krishna Reddy > Upgrade moment.min.js to 2.29.2 > --- > > Key: HDFS-16562 > URL: https://issues.apache.org/jira/browse/HDFS-16562 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > > Upgrade moment.min.js to 2.29.2 to resolve > [https://nvd.nist.gov/vuln/detail/CVE-2022-24785] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16562) Upgrade moment.min.js to 2.29.2
[ https://issues.apache.org/jira/browse/HDFS-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated HDFS-16562: Summary: Upgrade moment.min.js to 2.29.2 (was: Upgrade moments.js to 2.29.2) > Upgrade moment.min.js to 2.29.2 > --- > > Key: HDFS-16562 > URL: https://issues.apache.org/jira/browse/HDFS-16562 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Priority: Major > > Upgrade moments.js to 2.29.2 to resolve > [https://nvd.nist.gov/vuln/detail/CVE-2022-24785] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16562) Upgrade moment.min.js to 2.29.2
[ https://issues.apache.org/jira/browse/HDFS-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated HDFS-16562: Description: Upgrade moment.min.js to 2.29.2 to resolve [https://nvd.nist.gov/vuln/detail/CVE-2022-24785] (was: Upgrade moments.js to 2.29.2 to resolve [https://nvd.nist.gov/vuln/detail/CVE-2022-24785]) > Upgrade moment.min.js to 2.29.2 > --- > > Key: HDFS-16562 > URL: https://issues.apache.org/jira/browse/HDFS-16562 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Priority: Major > > Upgrade moment.min.js to 2.29.2 to resolve > [https://nvd.nist.gov/vuln/detail/CVE-2022-24785] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16562) Upgrade moments.js to 2.29.2
D M Murali Krishna Reddy created HDFS-16562: --- Summary: Upgrade moments.js to 2.29.2 Key: HDFS-16562 URL: https://issues.apache.org/jira/browse/HDFS-16562 Project: Hadoop HDFS Issue Type: Improvement Reporter: D M Murali Krishna Reddy Upgrade moments.js to 2.29.2 to resolve [https://nvd.nist.gov/vuln/detail/CVE-2022-24785] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15227) FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks
krishna reddy created HDFS-15227: Summary: FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks Key: HDFS-15227 URL: https://issues.apache.org/jira/browse/HDFS-15227 Project: Hadoop HDFS Issue Type: Bug Reporter: krishna reddy FSCK -upgradedomains is failing for upgradedomains when more than 2 million blocks present in hdfs and write in progress of some blocks "hdfs fsck / -files -blocks -upgradedomains" -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15220) FSCK calls are redirecting to Active NN
[ https://issues.apache.org/jira/browse/HDFS-15220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] krishna reddy updated HDFS-15220: - Description: Run any fsck except (-delete & - move) should go to ONN as it is read operation In below image spikes indicates when it ran fsck / -storagepolicies !screenshot-1.png! was: Run any fsck except -delete & - move should go to ONN as it is read operation !screenshot-1.png! > FSCK calls are redirecting to Active NN > --- > > Key: HDFS-15220 > URL: https://issues.apache.org/jira/browse/HDFS-15220 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: screenshot-1.png > > > Run any fsck except (-delete & - move) should go to ONN as it is read > operation > In below image spikes indicates when it ran fsck / -storagepolicies > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15220) FSCK calls are redirecting to Active NN
[ https://issues.apache.org/jira/browse/HDFS-15220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] krishna reddy updated HDFS-15220: - Description: Run any fsck except -delete & - move should go to ONN as it is read operation (was: Run any fsck except -delete & - move should go to ONN as it is read operationhdfs fsck / -storagepolicies and check the RPC calls for observer) > FSCK calls are redirecting to Active NN > --- > > Key: HDFS-15220 > URL: https://issues.apache.org/jira/browse/HDFS-15220 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: screenshot-1.png > > > Run any fsck except -delete & - move should go to ONN as it is read operation -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15220) FSCK calls are redirecting to Active NN
[ https://issues.apache.org/jira/browse/HDFS-15220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] krishna reddy updated HDFS-15220: - Description: Run any fsck except -delete & - move should go to ONN as it is read operation !screenshot-1.png! was:Run any fsck except -delete & - move should go to ONN as it is read operation > FSCK calls are redirecting to Active NN > --- > > Key: HDFS-15220 > URL: https://issues.apache.org/jira/browse/HDFS-15220 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: screenshot-1.png > > > Run any fsck except -delete & - move should go to ONN as it is read operation > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15220) FSCK calls are redirecting to Active NN
[ https://issues.apache.org/jira/browse/HDFS-15220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] krishna reddy updated HDFS-15220: - Attachment: screenshot-1.png > FSCK calls are redirecting to Active NN > --- > > Key: HDFS-15220 > URL: https://issues.apache.org/jira/browse/HDFS-15220 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: krishna reddy >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: screenshot-1.png > > > Run any fsck except -delete & - move should go to ONN as it is read operation -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15220) FSCK calls are redirecting to Active NN
krishna reddy created HDFS-15220: Summary: FSCK calls are redirecting to Active NN Key: HDFS-15220 URL: https://issues.apache.org/jira/browse/HDFS-15220 Project: Hadoop HDFS Issue Type: Bug Reporter: krishna reddy Run any fsck except -delete & - move should go to ONN as it is read operationhdfs fsck / -storagepolicies and check the RPC calls for observer -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15178) Federation: Add missing FederationClientInterceptor APIs
D M Murali Krishna Reddy created HDFS-15178: --- Summary: Federation: Add missing FederationClientInterceptor APIs Key: HDFS-15178 URL: https://issues.apache.org/jira/browse/HDFS-15178 Project: Hadoop HDFS Issue Type: Improvement Components: federation Reporter: D M Murali Krishna Reddy In FederationClientInterceptor, many API's are not Implemented. * getClusterNodes * getQueueInfo * getQueueUserAcls * moveApplicationAcrossQueues * getNewReservation * submitReservation * listReservations * updateReservation * deleteReservation * getNodeToLabels * getLabelsToNodes * getClusterNodeLabels * getApplicationAttemptReport * getApplicationAttempts * getContainerReport * getContainers * getDelegationToken * renewDelegationToken * cancelDelegationToken * failApplicationAttempt * updateApplicationPriority * signalToContainer * updateApplicationTimeouts * getResourceProfiles * getResourceProfile * getResourceTypeInfo * getAttributesToNodes * getClusterNodeAttributes * getNodesToAttributes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14584) Namenode went down with error "RedundancyMonitor thread received Runtime exception"
[
https://issues.apache.org/jira/browse/HDFS-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868270#comment-16868270
]
krishna reddy commented on HDFS-14584:
--
Hadoop version is 3.1.1
> Namenode went down with error "RedundancyMonitor thread received Runtime
> exception"
> ---
>
> Key: HDFS-14584
> URL: https://issues.apache.org/jira/browse/HDFS-14584
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: krishna reddy
>Assignee: Ayush Saxena
>Priority: Major
>
> *Description: *While removing dead nodes, Namenode went down with error
> "RedundancyMonitor thread received Runtime exception"
> *Environment: *
> Server OS :- UBUNTU
> No. of Cluster Node:- 1NN / 225DN's / 3ZK / 2RM/ 4850 NMs
> total 240 machines, in each machine 21 docker containers (1 DN & 20 NM's)
> *Steps:*
> 1. Total number of containers running state : ~53000
> 2. Because of the load, machine was going to outofMemory and restarting the
> machine and starting all the docker containers including NM's and DN's
> 3. in some point namenode throughs below error while removing a node and NN
> went down.
> {noformat}
> 2019-06-19 05:54:07,262 INFO org.apache.hadoop.net.NetworkTopology: Removing
> a node: /rack-1550/255.255.117.195:23735
> 2019-06-19 05:54:07,263 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> removeDeadDatanode: lost heartbeat from 255.255.117.151:23735,
> removeBlocksFromBlockMap true
> 2019-06-19 05:54:07,281 INFO org.apache.hadoop.net.NetworkTopology: Removing
> a node: /rack-4097/255.255.117.151:23735
> 2019-06-19 05:54:07,282 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> removeDeadDatanode: lost heartbeat from 255.255.116.213:23735,
> removeBlocksFromBlockMap true
> 2019-06-19 05:54:07,290 ERROR
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: RedundancyMonitor
> thread received Runtime exception.
> java.lang.IllegalArgumentException: 247 should >= 248, and both should be
> positive.
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:575)
> at
> org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:552)
> at
> org.apache.hadoop.hdfs.net.DFSNetworkTopology.chooseRandomWithStorageTypeTwoTrial(DFSNetworkTopology.java:122)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:873)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:770)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:712)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:507)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:425)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargets(BlockPlacementPolicyDefault.java:311)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:290)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:143)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.chooseTarget(BlockPlacementPolicy.java:103)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.ReplicationWork.chooseTargets(ReplicationWork.java:51)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReconstructionWorkForBlocks(BlockManager.java:1902)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockReconstructionWork(BlockManager.java:1854)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:4842)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$RedundancyMonitor.run(BlockManager.java:4709)
> at java.lang.Thread.run(Thread.java:748)
> 2019-06-19 05:54:07,296 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1: java.lang.IllegalArgumentException: 247 should >= 248, and both
> should be positive.
> 2019-06-19 05:54:07,298 INFO
> org.apache.hadoop.hdfs.server.common.HadoopAuditLogger.audit:
> process=Namenode operation=shutdown result=invoked
> 2019-06-19 05:54:07,298 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> SHUTDOWN_MSG:
> /**
[jira] [Updated] (HDFS-14584) Namenode went down with error "RedundancyMonitor thread received Runtime exception"
[
https://issues.apache.org/jira/browse/HDFS-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
krishna reddy updated HDFS-14584:
-
Description:
*Description: *While removing dead nodes, Namenode went down with error
"RedundancyMonitor thread received Runtime exception"
*Environment: *
Server OS :- UBUNTU
No. of Cluster Node:- 1NN / 225DN's / 3ZK / 2RM/ 4850 NMs
total 240 machines, in each machine 21 docker containers (1 DN & 20 NM's)
*Steps:*
1. Total number of containers running state : ~53000
2. Because of the load, machine was going to outofMemory and restarting the
machine and starting all the docker containers including NM's and DN's
3. in some point namenode throughs below error while removing a node and NN
went down.
{noformat}
2019-06-19 05:54:07,262 INFO org.apache.hadoop.net.NetworkTopology: Removing a
node: /rack-1550/255.255.117.195:23735
2019-06-19 05:54:07,263 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
removeDeadDatanode: lost heartbeat from 255.255.117.151:23735,
removeBlocksFromBlockMap true
2019-06-19 05:54:07,281 INFO org.apache.hadoop.net.NetworkTopology: Removing a
node: /rack-4097/255.255.117.151:23735
2019-06-19 05:54:07,282 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
removeDeadDatanode: lost heartbeat from 255.255.116.213:23735,
removeBlocksFromBlockMap true
2019-06-19 05:54:07,290 ERROR
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: RedundancyMonitor
thread received Runtime exception.
java.lang.IllegalArgumentException: 247 should >= 248, and both should be
positive.
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at
org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:575)
at
org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:552)
at
org.apache.hadoop.hdfs.net.DFSNetworkTopology.chooseRandomWithStorageTypeTwoTrial(DFSNetworkTopology.java:122)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:873)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:770)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:712)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:507)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:425)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargets(BlockPlacementPolicyDefault.java:311)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:290)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:143)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.chooseTarget(BlockPlacementPolicy.java:103)
at
org.apache.hadoop.hdfs.server.blockmanagement.ReplicationWork.chooseTargets(ReplicationWork.java:51)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReconstructionWorkForBlocks(BlockManager.java:1902)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockReconstructionWork(BlockManager.java:1854)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:4842)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$RedundancyMonitor.run(BlockManager.java:4709)
at java.lang.Thread.run(Thread.java:748)
2019-06-19 05:54:07,296 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1: java.lang.IllegalArgumentException: 247 should >= 248, and both
should be positive.
2019-06-19 05:54:07,298 INFO
org.apache.hadoop.hdfs.server.common.HadoopAuditLogger.audit: process=Namenode
operation=shutdown result=invoked
2019-06-19 05:54:07,298 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down NameNode at namenode/255.255.182.104
/
{noformat}
was:
*Description: *While removing dead nodes, Namenode went down with error
"RedundancyMonitor thread received Runtime exception"
*Environment: *
Server OS :- UBUNTU
No. of Cluster Node:- 1NN / 225DN's / 3ZK / 2RM/ 4850 NMs
total 240 machines, in each machine 21 docker containers (1 DN & 20 NN's)
*Steps:*
1. Total number of containers running state : ~53000
2. Because of the load, machine was going to outofMemory and restarting the
machine and starting all the docker co
[jira] [Created] (HDFS-14584) Namenode went down with error "RedundancyMonitor thread received Runtime exception"
krishna reddy created HDFS-14584:
Summary: Namenode went down with error "RedundancyMonitor thread
received Runtime exception"
Key: HDFS-14584
URL: https://issues.apache.org/jira/browse/HDFS-14584
Project: Hadoop HDFS
Issue Type: Bug
Reporter: krishna reddy
*Description: *While removing dead nodes, Namenode went down with error
"RedundancyMonitor thread received Runtime exception"
*Environment: *
Server OS :- UBUNTU
No. of Cluster Node:- 1NN / 225DN's / 3ZK / 2RM/ 4850 NMs
total 240 machines, in each machine 21 docker containers (1 DN & 20 NN's)
*Steps:*
1. Total number of containers running state : ~53000
2. Because of the load, machine was going to outofMemory and restarting the
machine and starting all the docker containers including NM's and DN's
3. in some point namenode throughs below error while removing a node and NN
went down.
{noformat}
2019-06-19 05:54:07,262 INFO org.apache.hadoop.net.NetworkTopology: Removing a
node: /rack-1550/255.255.117.195:23735
2019-06-19 05:54:07,263 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
removeDeadDatanode: lost heartbeat from 255.255.117.151:23735,
removeBlocksFromBlockMap true
2019-06-19 05:54:07,281 INFO org.apache.hadoop.net.NetworkTopology: Removing a
node: /rack-4097/255.255.117.151:23735
2019-06-19 05:54:07,282 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
removeDeadDatanode: lost heartbeat from 255.255.116.213:23735,
removeBlocksFromBlockMap true
2019-06-19 05:54:07,290 ERROR
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: RedundancyMonitor
thread received Runtime exception.
java.lang.IllegalArgumentException: 247 should >= 248, and both should be
positive.
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at
org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:575)
at
org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:552)
at
org.apache.hadoop.hdfs.net.DFSNetworkTopology.chooseRandomWithStorageTypeTwoTrial(DFSNetworkTopology.java:122)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseDataNode(BlockPlacementPolicyDefault.java:873)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:770)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:712)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:507)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:425)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargets(BlockPlacementPolicyDefault.java:311)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:290)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:143)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.chooseTarget(BlockPlacementPolicy.java:103)
at
org.apache.hadoop.hdfs.server.blockmanagement.ReplicationWork.chooseTargets(ReplicationWork.java:51)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReconstructionWorkForBlocks(BlockManager.java:1902)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeBlockReconstructionWork(BlockManager.java:1854)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:4842)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$RedundancyMonitor.run(BlockManager.java:4709)
at java.lang.Thread.run(Thread.java:748)
2019-06-19 05:54:07,296 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1: java.lang.IllegalArgumentException: 247 should >= 248, and both
should be positive.
2019-06-19 05:54:07,298 INFO
org.apache.hadoop.hdfs.server.common.HadoopAuditLogger.audit: process=Namenode
operation=shutdown result=invoked
2019-06-19 05:54:07,298 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down NameNode at namenode/255.255.182.104
/
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
