[ https://issues.apache.org/jira/browse/CASSANDRA-7431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224527#comment-14224527 ]
Olivier Michallat commented on CASSANDRA-7431: ---------------------------------------------- I'm running into this issue with the Java driver as well (context: if both client and C* are deployed on EC2, the client should use private addresses for C* nodes in the same region, and public addresses for C* nodes in another region -- EC2's DNS resolves the "right" address automatically if you lookup the node's public hostname; in order to get this public hostname, we do a reverse DNS lookup on the IP exposed in {{system.peers.rpc_address}}). When run from an EC2 instance, a reverse lookup with {{InetAddress.getHostName(publicIp)}} works correctly for instances _in another EC2 region_, but fails for instances in the same region. As mentioned by Paulo, it returns the unresolved IP, whereas command-line tools like {{host}} or {{dig}} correctly resolve to the public hostname. I have no explanation as to why it fails with Java, but it appears to be a JDK bug. The lookup via JNDI (as done in Paulo's patch) works, but the fact that we initialize the factory with {{com.sun.jndi.dns.DnsContextFactory}} makes me wonder if this is portable to other JDK implementations. Another approach is to use [dnsjava|http://www.xbill.org/dnsjava/] (that's what they did in Whirr). > Hadoop integration does not perform reverse DNS lookup correctly on EC2 > ----------------------------------------------------------------------- > > Key: CASSANDRA-7431 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7431 > Project: Cassandra > Issue Type: Bug > Components: Hadoop > Reporter: Paulo Motta > Assignee: Paulo Motta > Attachments: 2.0-CASSANDRA-7431.txt > > > The split assignment on AbstractColumnFamilyInputFormat:247 peforms a reverse > DNS lookup of Cassandra IPs in order to preserve locality in Hadoop (task > trackers are identified by hostnames). > However, the reverse lookup of an EC2 IP does not yield the EC2 hostname of > that endpoint when running from an EC2 instance due to the use of > InetAddress.getHostname(). > In order to show this, consider the following piece of code: > {code:title=DnsResolver.java|borderStyle=solid} > public class DnsResolver { > public static void main(String[] args) throws Exception { > InetAddress namenodePublicAddress = InetAddress.getByName(args[0]); > System.out.println("getHostAddress: " + > namenodePublicAddress.getHostAddress()); > System.out.println("getHostName: " + > namenodePublicAddress.getHostName()); > } > } > {code} > When this code is run from my machine to perform reverse lookup of an EC2 IP, > the output is: > {code:none} > ➜ java DnsResolver 54.201.254.99 > getHostAddress: 54.201.254.99 > getHostName: ec2-54-201-254-99.compute-1.amazonaws.com > {code} > When this code is executed from inside an EC2 machine, the output is: > {code:none} > ➜ java DnsResolver 54.201.254.99 > getHostAddress: 54.201.254.99 > getHostName: 54.201.254.99 > {code} > However, when using linux tools such as "host" or "dig", the EC2 hostname is > properly resolved from the EC2 instance, so there's some problem with Java's > InetAddress.getHostname() and EC2. > Two consequences of this bug during AbstractColumnFamilyInputFormat split > definition are: > 1) If the Hadoop cluster is configured to use EC2 public DNS, the locality > will be lost, because Hadoop will try to match the CFIF split location > (public IP) with the task tracker location (public DNS), so no matches will > be found. > 2) If the Cassandra nodes' broadcast_address is set to public IPs, all hadoop > communication will be done via the public IP, what will incurr additional > transference charges. If the public IP is mapped to the EC2 DNS during split > definition, when the task is executed, ColumnFamilyRecordReader will resolve > the public DNS to the private IP of the instance, so there will be not > additional charges. > A similar bug was filed in the WHIRR project: > https://issues.apache.org/jira/browse/WHIRR-128 -- This message was sent by Atlassian JIRA (v6.3.4#6332)