Johannes, NetUtils.addStaticResolution() would be invoked in your client code, correct ? I'm not sure Tez can do anything useful with this, unless we add some other parameters which can be configured by users.
Setting "hadoop.rpc.socket.factory.class.default" should take affect in Tez as well, as long as it's set in the configuration that is used to create the TezClient. I'm not sure why "yarn.ipc.client.factory.class" needs to be changed, but that isn't relevant to Tez in any case. Does the custom socket factory that you use take care of the static mapping ? On Wed, Aug 13, 2014 at 4:56 AM, Johannes Zillmann <[email protected] > wrote: > Hey Sid, > > as a background, for map-reduce we’re configuring > yarn.ipc.client.factory.class > hadoop.rpc.socket.factory.class.default > to an homegrown socket-factory which does the translation between ec2 > internal to external addresses. > > For Tez this does not seem to have any effect, even if i’m using > NetUtils.createSocketAddrForHost() like you suggested. > However, using NetUtils.createSocketAddrForHost() would allow to add the > translation between the addresses through NetUtils.addStaticResolution() as > i figured out. So doing this change would help. > Should i create a ticket for that ? > > Johannes > > > On 13 Aug 2014, at 10:30, Siddharth Seth <[email protected]> wrote: > > > Johannes > > Getting the client to pick the correct IP to use is a little tricky. > Hadoop itself has some utilities for this, which we could try using. Could > you open a jira for this please ?, and we'll need some help trying it out. > > > > If you're building the Tez code base locally - could you try the > following change > > > > TezClientUtils:820 > > Replace > > final InetSocketAddress serviceAddr = new InetSocketAddress(amHost, > amRpcPort); > > with > > final InetSocketAddress serviceAddr = > NetUtils.createSocketAddrForHost(amHost, amRpcPort); > > > > > > On Wed, Aug 13, 2014 at 12:58 AM, Johannes Zillmann < > [email protected]> wrote: > > Hey Hitesh, > > > > so without chaining the hostname of the ec2 instances to their public > dns the log looks like: > > 2014-08-13 03:53:15,310 INFO [ServiceThread:DAGClientRPCServer] > org.apache.tez.dag.api.client.DAGClientServer: Instantiated > DAGClientRPCServer at domU-12-31-39-0F-30-03/10.193.51.241:31000 > > 2014-08-13 03:53:15,332 INFO [IPC Server Responder] > org.apache.hadoop.ipc.Server: IPC Server Responder: starting > > 2014-08-13 03:53:15,336 INFO [IPC Server listener on 50192] > org.apache.hadoop.ipc.Server: IPC Server listener on 50192: starting > > > > and the exception on the client is then: > > com.google.protobuf.ServiceException: java.net.UnknownHostException: > Invalid host name: local host is: (unknown); destination host is: > "domU-12-31-39-0F-30-03":31000; java.net.UnknownHostException; For more > details see: http://wiki.apache.org/hadoop/UnknownHost > > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216) > > at com.sun.proxy.$Proxy37.getAMStatus(Unknown Source) > > at > org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:503) > > at > org.apache.tez.client.TezClient.waitTillReady(TezClient.java:576) > > > > HTH > > Johannes > > > > On 12 Aug 2014, at 22:05, Hitesh Shah <[email protected]> wrote: > > > > > It seems like the AM is binding to the external/public hostname and > not the internal IP. > > > > > > Could you look for this log message in the AM logs: "Instantiated > DAGClientRPCServer at”. This will provide some information as to what the > AM is binding to. > > > > > > thanks > > > — Hitesh > > > > > > On Aug 11, 2014, at 7:43 AM, Johannes Zillmann < > [email protected]> wrote: > > > > > >> Hey guys, > > >> > > >> having a test-infrastructure for Hadoop on ec2. The client sits > usually outside of ec2. > > >> Using plain map-reduce on YARN everything works fine. > > >> Using Tez i run into following exception: > > >> > > >> INFO [2014-07-29 00:09:06.653] [MrPlanRunnerV2] (TezClient.java:507) > - Failed to retrieve AM Status via proxy > > >> com.google.protobuf.ServiceException: > org.apache.hadoop.net.ConnectTimeoutException: Call From > ip-10-73-6-154.ec2.internal/10.73.6.154 to > ec2-54-81-245-144.compute-1.amazonaws.com:60914 failed on socket timeout > exception: org.apache.hadoop.net.ConnectTimeoutException: connect timed > out; For more details see: http: > > >> //wiki.apache.org/hadoop/SocketTimeout > > >> > > >> at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216) > > >> at com.sun.proxy.$Proxy116.getAMStatus(Unknown Source) > > >> at > org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:500) > > >> at > org.apache.tez.client.TezClient.waitTillReady(TezClient.java:586) > > >> > > >> > > >> I could resolve the problem for Tez changing the hostname of the > instances to their public dns’. However, that is causing problems with > other components. > > >> Do you know of any place in Tez which is related to that ? Any tweak > which could make chaining the hostname superfluous ? > > >> > > >> best > > >> Johannes > > > > > > > > >
