YARN-5244. Documentation required for DNS Server implementation. Contributed by Jon Maron
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8673f24a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8673f24a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8673f24a Branch: refs/heads/yarn-native-services Commit: 8673f24a4a3f2288f284b3f14713a76ac90f552e Parents: 2b3aff2 Author: Gour Saha <gourks...@apache.org> Authored: Thu Aug 24 19:05:13 2017 -0700 Committer: Jian He <jia...@apache.org> Committed: Wed Aug 30 22:22:30 2017 -0700 ---------------------------------------------------------------------- hadoop-project/src/site/site.xml | 7 +- .../native-services/NativeServicesDiscovery.md | 127 +++++++++++++++++++ .../native-services/NativeServicesIntro.md | 0 .../src/site/resources/images/dns_overview.png | Bin 0 -> 41908 bytes .../resources/images/dns_record_creation.jpeg | Bin 0 -> 51911 bytes .../resources/images/dns_record_removal.jpeg | Bin 0 -> 58041 bytes 6 files changed, 133 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/8673f24a/hadoop-project/src/site/site.xml ---------------------------------------------------------------------- diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index e7af227..837ac3f 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -148,7 +148,12 @@ <item name="Timeline Server" href="hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1"/> <item name="Timeline Service V.2" href="hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API"/> </menu> - + + <menu name="YARN Native Services" inherit="top"> + <item name="Introduction" href="hadoop-yarn/hadoop-yarn-site/native-services/NativeServicesIntro.html"/> + <item name="Native Services Discovery" href="hadoop-yarn/hadoop-yarn-site/native-services/NativeServicesDiscovery.html"/> + </menu> + <menu name="Hadoop Compatible File Systems" inherit="top"> <item name="Aliyun OSS" href="hadoop-aliyun/tools/hadoop-aliyun/index.html"/> <item name="Amazon S3" href="hadoop-aws/tools/hadoop-aws/index.html"/> http://git-wip-us.apache.org/repos/asf/hadoop/blob/8673f24a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md new file mode 100644 index 0000000..4a048af --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md @@ -0,0 +1,127 @@ +# YARN DNS Server +## Introduction + +The YARN DNS Server provides a standard DNS interface to the information posted into the YARN Registry by deployed applications. The DNS service serves the following functions: + +1. **Exposing existing service discovery information via DNS**â - Information provided in +the current YARN service registryâs records will be converted into DNS entries, thus +allowing users to discover information about YARN applications using standard DNS +client mechanisms (for e.g. a DNS SRV Record specifying the hostname and port +number for services). +2. **Enabling Container to IP mappingsâÂ** - Enables discovery of the IPs of containers via +standard DNS lookups. Given the availability of the records via DNS, container +nameÂbased communication will be facilitated (e.g. âcurl +http://myContainer.myDomain.com/endpointâ). + +## Service Properties + +The existing YARN Service Registry is leveraged as the source of information for the DNS Service. + +The following core functions are supported by the DNS Server: + +###Functional properties + +1. Supports creation of DNS records for endÂpoints of the deployed YARN applications +2. Record names remain unchanged during restart of containers and/or applications +3. Supports reverse lookups (name based on IP). +4. Supports security using the standards defined by The Domain Name System Security +Extensions (DNSSEC) +5. Highly available +6. Scalable Â- The service provides the responsiveness (e.g. low latency) required to +respond to DNS queries (timeouts yield attempts to invoke other configured name +servers). + +###Deployment properties + +1. Supports integration with existing DNS assets (e.g. a corporate DNS server) by acting as +a DNS server for a Hadoop cluster zone/domain. The server is not intended to act as a +primary DNS server and does not forward requests to other servers. +2. The DNS Server exposes a port that can receive both TCP and UDP requests per +DNS standards. The default port for DNS protocols is in a restricted, administrative port +range (53), so the port is configurable for deployments in which the service may +not be managed via an administrative account. + +##DNS Record Name Structure + +The DNS names of generated records are composed from the following elements (labels). Note that these elements must be compatible with DNS conventions (see âPreferred Name Syntaxâ in RFC 1035): + +* **domain** -â the name of the cluster DNS domain. This name is provided as a +configuration property. In addition, it is this name that is configured at a parent DNS +server as the zone name for the defined yDNS zone (the zone for which the parent DNS +server will forward requests to yDNS). E.g. yarncluster.com +* **userÂname**â - the name of the application deployer. This name is the simple shortÂname (for +e.g. the primary component of the Kerberos principal) associated with the user launching +the application. As the userÂname is one of the elements of DNS names, it is expected +that this also confirms DNS name conventions (RFC 1035 linked above)  so special translation is performed for names with special characters like hyphens and spaces. +* **application Âname** -â the name of the deployed YARN application. This name is inferred +from the YARN registry path to the application's node. Application name, rather thn application id, was chosen as a way of making it easy for users to refer to humanÂ-readable DNS +names. This obviously mandates certain uniqueness properties on application names. +* **container id** -â the YARN assigned ID to a container (e.g. +container_e3741_1454001598828_01_000004) +* **component Âname** -â the name assigned to the deployed component (for e.g. a master +component). A component is a distributed element of an application or service that is +launched in a YARN container (e.g. an HBase master). One can imagine multiple +components within an application. A component name is not yet a first class concept in +YARN, but is a very useful one that we are introducing here for the sake of yDNS +entries. Many frameworks like MapReduce, Slider already have component Ânames +(though, as mentioned, they are not yet supported in YARN in a first class fashion). +* **api** -â the api designation for the exposed endpoint + +###Notes about DNS Names + +* In most instances, the DNS names can be easily distinguished by the number of +elements/labels that compose the name. The clusterâs domain name is always the last +element. After that element is parsed out, reading from right to left, the first element +maps to the application Âuser and so on. Wherever it is not easily distinguishable, naming conventions are used to disambiguate the name Âusing a prefix such as +âcontainerÂâ or suffix such as âapiâ. For example, an endpoint published as a +management endpoint will be referenced with the name *managementÂ-api.griduser.yarncluster.comâ*. +* Unique application Âname (per user) is not currently supported/guaranteed by YARN, but +it is supported by frameworks such as Apache Slider. The yDNS service currently +leverages the last element of the ZK path entry for the application as an +application name. These application names have to be unique for a given user.â + +##DNS Server Functionality + +The primary functions of the DNS service are illustrated in the following diagram: + + + + +###DNS record creation +The following figure illustrates at slightly greater detail the DNS record creation and registration sequence (NOTE: service record updates would follow a similar sequence of steps, +distinguished only by the different event type): + + + +###DNS record removal +Similarly, record removal follows a similar sequence + + + +(NOTE: The DNS Zone requires a record as an argument for the deletion method, thus +requiring similar parsing logic to identify the specific records that should be removed). + +###DNS Service initialization +* The DNS service initializes both UDP and TCP listeners on a configured port. As +noted above, the default port of 53 is in a restricted range that is only accessible to an +account with administrative privileges. +* Subsequently, the DNS service listens for inbound DNS requests. Those requests are +standard DNS requests from users or other DNS servers (for example, DNS servers that have the +YARN DNS service configured as a forwarder). + +## Configuration +The YARN DNS server reads its configuration properties from the yarnÂsite.xml file. The following are the DNS associated configuration properties: + +| Name | Description | +| ------------ | ------------- | +| hadoop.registry.dns.enabled | The DNS functionality is enabled for the cluster. Default is false. | +| hadoop.registry.dns.domain-Âname | The domain name for Hadoop cluster associated records. | +| hadoop.registry.dns.bindÂ-address | Address associated with the network interface to which the DNS listener should bind. | +| hadoop.registry.dns.bind-Âport | The port number for the DNS listener. The default port is 53. However, since that port falls in a administratorÂonly range, typical deployments may need to specify an alternate port. | +| hadoop.registry.dns.dnssec.enabled | Indicates whether the DNSSEC support is enabled. Default is false. | +| hadoop.registry.dns.publicÂ-key | The base64 representation of the serverâs public key. Leveraged for creating the DNSKEY Record provided for DNSSEC client requests. | +| hadoop.registry.dns.private-Âkey-Âfile | The path to the standard DNSSEC private key file. Must only be readable by the DNS launching identity. See [dnssecÂ-keygen](https://ftp.isc.org/isc/bind/cur/9.9/doc/arm/man.dnssec-keygen.html) documentation. | +| hadoop.registry.dnsÂ-ttl | The default TTL value to associate with DNS records. The default value is set to 1 (a value of 0 has undefined behavior). A typical value should be approximate to the time it takes YARN to restart a failed container. | +| hadoop.registry.dns.zone-Âsubnet | An indicator of the IP range associated with the cluster containers. The setting is utilized for the generation of the reverse zone name. | +| hadoop.registry.dns.zone-Âmask | The network mask associated with the zone IP range. If specified, it is utilized to ascertain the IP range possible and come up with an appropriate reverse zone name. | +| hadoop.registry.dns.zonesÂ-dir | A directory containing zone configuration files to read during zone initialization. This directory can contain zone master files named *zone-name.zone*. See [here](http://www.zytrax.com/books/dns/ch6/mydomain.html) for zone master file documentation.| http://git-wip-us.apache.org/repos/asf/hadoop/blob/8673f24a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md new file mode 100644 index 0000000..e69de29 http://git-wip-us.apache.org/repos/asf/hadoop/blob/8673f24a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_overview.png ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_overview.png b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_overview.png new file mode 100644 index 0000000..b9e80b9 Binary files /dev/null and b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_overview.png differ http://git-wip-us.apache.org/repos/asf/hadoop/blob/8673f24a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_creation.jpeg ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_creation.jpeg b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_creation.jpeg new file mode 100644 index 0000000..63b2599 Binary files /dev/null and b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_creation.jpeg differ http://git-wip-us.apache.org/repos/asf/hadoop/blob/8673f24a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_removal.jpeg ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_removal.jpeg b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_removal.jpeg new file mode 100644 index 0000000..40d870c Binary files /dev/null and b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/resources/images/dns_record_removal.jpeg differ --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
