[ https://issues.apache.org/jira/browse/HDFS-11300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806162#comment-15806162 ]
Allen Wittenauer commented on HDFS-11300: ----------------------------------------- bq. A solution to fix this issue is to fix the upfront resolution and do it in a lazy way. Or, what most large sites do, deploy DNS caching servers on every node. This greatly improves performance in general. > HDFS Nameservices introduces DNS latency inter-dependencies between namenodes > ----------------------------------------------------------------------------- > > Key: HDFS-11300 > URL: https://issues.apache.org/jira/browse/HDFS-11300 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, hdfs-client > Affects Versions: 2.6.0 > Reporter: Rémy SAISSY > > When using HDFS Nameservices, a DNS outage or strong latency on one of > nameservices will impact any DFSClient or other component which has this > nameservice in its hdfs-site.xml. Even though it doesn't use it. > The issue is due to all nameservices being listed in hdfs-site.xml, and > hadoop code trying to resolve all of them upfront. > To ilustrate the issue, here is the use case we have: > * the 'prod' cluster with its dedicated DNS > * the 'preprod' cluster with its dedicated DNS > * both 'prod' and 'preprod' hdfs-site.xml files have each other nameservices > so that they can use HDFS as follows: > (from prod) hdfs dfs -ls hdfs://preprod/user/j.doe > A DNS outage on the preprod cluster slow down production jobs being scheduled > because of the upfront resolution of the hdfs-site.xml nameservices entries > even though a specific job doesn't use it. > A solution to fix this issue is to fix the upfront resolution and do it in a > lazy way. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org