Its https://issues.apache.org/jira/browse/TEZ-1422
Johannes On 14 Aug 2014, at 11:53, Siddharth Seth <[email protected]> wrote: > Please create a jira to at least change the call to make use of NetUtils. > > > Beyond this, I believe this is a use case which Tez should support. > > > On Thu, Aug 14, 2014 at 12:59 AM, Johannes Zillmann > <[email protected]> wrote: > Hey Sid, > > On 13 Aug 2014, at 23:18, Siddharth Seth <[email protected]> wrote: > > > Johannes, > > NetUtils.addStaticResolution() would be invoked in your client code, > > correct ? I'm not sure Tez can do anything useful with this, unless we add > > some other parameters which can be configured by users. > > yes i would invoke that independently of Tez. > > > > > > Setting "hadoop.rpc.socket.factory.class.default" should take affect in Tez > > as well, as long as it's set in the configuration that is used to create > > the TezClient. > Didn’t look like this is having an effect. Guess because the ip-resolving > takes place before creating any socket. > > > I'm not sure why "yarn.ipc.client.factory.class" needs to be changed, > > but that isn't relevant to Tez in any case. > > Does the custom socket factory that you use take care of the static mapping > > ? > > Yes. When creating a socket to an address it checks a map for the address > translation. > > Johannes > > > > > > > > > On Wed, Aug 13, 2014 at 4:56 AM, Johannes Zillmann > > <[email protected]> wrote: > > Hey Sid, > > > > as a background, for map-reduce we’re configuring > > yarn.ipc.client.factory.class > > hadoop.rpc.socket.factory.class.default > > to an homegrown socket-factory which does the translation between ec2 > > internal to external addresses. > > > > For Tez this does not seem to have any effect, even if i’m using > > NetUtils.createSocketAddrForHost() like you suggested. > > However, using NetUtils.createSocketAddrForHost() would allow to add the > > translation between the addresses through NetUtils.addStaticResolution() as > > i figured out. So doing this change would help. > > Should i create a ticket for that ? > > > > Johannes > > > > > > On 13 Aug 2014, at 10:30, Siddharth Seth <[email protected]> wrote: > > > > > Johannes > > > Getting the client to pick the correct IP to use is a little tricky. > > > Hadoop itself has some utilities for this, which we could try using. > > > Could you open a jira for this please ?, and we'll need some help trying > > > it out. > > > > > > If you're building the Tez code base locally - could you try the > > > following change > > > > > > TezClientUtils:820 > > > Replace > > > final InetSocketAddress serviceAddr = new InetSocketAddress(amHost, > > > amRpcPort); > > > with > > > final InetSocketAddress serviceAddr = > > > NetUtils.createSocketAddrForHost(amHost, amRpcPort); > > > > > > > > > On Wed, Aug 13, 2014 at 12:58 AM, Johannes Zillmann > > > <[email protected]> wrote: > > > Hey Hitesh, > > > > > > so without chaining the hostname of the ec2 instances to their public dns > > > the log looks like: > > > 2014-08-13 03:53:15,310 INFO [ServiceThread:DAGClientRPCServer] > > > org.apache.tez.dag.api.client.DAGClientServer: Instantiated > > > DAGClientRPCServer at domU-12-31-39-0F-30-03/10.193.51.241:31000 > > > 2014-08-13 03:53:15,332 INFO [IPC Server Responder] > > > org.apache.hadoop.ipc.Server: IPC Server Responder: starting > > > 2014-08-13 03:53:15,336 INFO [IPC Server listener on 50192] > > > org.apache.hadoop.ipc.Server: IPC Server listener on 50192: starting > > > > > > and the exception on the client is then: > > > com.google.protobuf.ServiceException: java.net.UnknownHostException: > > > Invalid host name: local host is: (unknown); destination host is: > > > "domU-12-31-39-0F-30-03":31000; java.net.UnknownHostException; For more > > > details see: http://wiki.apache.org/hadoop/UnknownHost > > > at > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216) > > > at com.sun.proxy.$Proxy37.getAMStatus(Unknown Source) > > > at > > > org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:503) > > > at > > > org.apache.tez.client.TezClient.waitTillReady(TezClient.java:576) > > > > > > HTH > > > Johannes > > > > > > On 12 Aug 2014, at 22:05, Hitesh Shah <[email protected]> wrote: > > > > > > > It seems like the AM is binding to the external/public hostname and not > > > > the internal IP. > > > > > > > > Could you look for this log message in the AM logs: "Instantiated > > > > DAGClientRPCServer at”. This will provide some information as to what > > > > the AM is binding to. > > > > > > > > thanks > > > > — Hitesh > > > > > > > > On Aug 11, 2014, at 7:43 AM, Johannes Zillmann > > > > <[email protected]> wrote: > > > > > > > >> Hey guys, > > > >> > > > >> having a test-infrastructure for Hadoop on ec2. The client sits > > > >> usually outside of ec2. > > > >> Using plain map-reduce on YARN everything works fine. > > > >> Using Tez i run into following exception: > > > >> > > > >> INFO [2014-07-29 00:09:06.653] [MrPlanRunnerV2] (TezClient.java:507) - > > > >> Failed to retrieve AM Status via proxy > > > >> com.google.protobuf.ServiceException: > > > >> org.apache.hadoop.net.ConnectTimeoutException: Call From > > > >> ip-10-73-6-154.ec2.internal/10.73.6.154 to > > > >> ec2-54-81-245-144.compute-1.amazonaws.com:60914 failed on socket > > > >> timeout exception: org.apache.hadoop.net.ConnectTimeoutException: > > > >> connect timed out; For more details see: http: > > > >> //wiki.apache.org/hadoop/SocketTimeout > > > >> > > > >> at > > > >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216) > > > >> at com.sun.proxy.$Proxy116.getAMStatus(Unknown Source) > > > >> at > > > >> org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:500) > > > >> at > > > >> org.apache.tez.client.TezClient.waitTillReady(TezClient.java:586) > > > >> > > > >> > > > >> I could resolve the problem for Tez changing the hostname of the > > > >> instances to their public dns’. However, that is causing problems with > > > >> other components. > > > >> Do you know of any place in Tez which is related to that ? Any tweak > > > >> which could make chaining the hostname superfluous ? > > > >> > > > >> best > > > >> Johannes > > > > > > > > > > > > > > > >
