Hey Sid, On 13 Aug 2014, at 23:18, Siddharth Seth <[email protected]> wrote:
> Johannes, > NetUtils.addStaticResolution() would be invoked in your client code, correct > ? I'm not sure Tez can do anything useful with this, unless we add some other > parameters which can be configured by users. yes i would invoke that independently of Tez. > > Setting "hadoop.rpc.socket.factory.class.default" should take affect in Tez > as well, as long as it's set in the configuration that is used to create the > TezClient. Didn’t look like this is having an effect. Guess because the ip-resolving takes place before creating any socket. > I'm not sure why "yarn.ipc.client.factory.class" needs to be changed, but > that isn't relevant to Tez in any case. > Does the custom socket factory that you use take care of the static mapping ? Yes. When creating a socket to an address it checks a map for the address translation. Johannes > > > > On Wed, Aug 13, 2014 at 4:56 AM, Johannes Zillmann <[email protected]> > wrote: > Hey Sid, > > as a background, for map-reduce we’re configuring > yarn.ipc.client.factory.class > hadoop.rpc.socket.factory.class.default > to an homegrown socket-factory which does the translation between ec2 > internal to external addresses. > > For Tez this does not seem to have any effect, even if i’m using > NetUtils.createSocketAddrForHost() like you suggested. > However, using NetUtils.createSocketAddrForHost() would allow to add the > translation between the addresses through NetUtils.addStaticResolution() as i > figured out. So doing this change would help. > Should i create a ticket for that ? > > Johannes > > > On 13 Aug 2014, at 10:30, Siddharth Seth <[email protected]> wrote: > > > Johannes > > Getting the client to pick the correct IP to use is a little tricky. Hadoop > > itself has some utilities for this, which we could try using. Could you > > open a jira for this please ?, and we'll need some help trying it out. > > > > If you're building the Tez code base locally - could you try the following > > change > > > > TezClientUtils:820 > > Replace > > final InetSocketAddress serviceAddr = new InetSocketAddress(amHost, > > amRpcPort); > > with > > final InetSocketAddress serviceAddr = > > NetUtils.createSocketAddrForHost(amHost, amRpcPort); > > > > > > On Wed, Aug 13, 2014 at 12:58 AM, Johannes Zillmann > > <[email protected]> wrote: > > Hey Hitesh, > > > > so without chaining the hostname of the ec2 instances to their public dns > > the log looks like: > > 2014-08-13 03:53:15,310 INFO [ServiceThread:DAGClientRPCServer] > > org.apache.tez.dag.api.client.DAGClientServer: Instantiated > > DAGClientRPCServer at domU-12-31-39-0F-30-03/10.193.51.241:31000 > > 2014-08-13 03:53:15,332 INFO [IPC Server Responder] > > org.apache.hadoop.ipc.Server: IPC Server Responder: starting > > 2014-08-13 03:53:15,336 INFO [IPC Server listener on 50192] > > org.apache.hadoop.ipc.Server: IPC Server listener on 50192: starting > > > > and the exception on the client is then: > > com.google.protobuf.ServiceException: java.net.UnknownHostException: > > Invalid host name: local host is: (unknown); destination host is: > > "domU-12-31-39-0F-30-03":31000; java.net.UnknownHostException; For more > > details see: http://wiki.apache.org/hadoop/UnknownHost > > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216) > > at com.sun.proxy.$Proxy37.getAMStatus(Unknown Source) > > at > > org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:503) > > at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:576) > > > > HTH > > Johannes > > > > On 12 Aug 2014, at 22:05, Hitesh Shah <[email protected]> wrote: > > > > > It seems like the AM is binding to the external/public hostname and not > > > the internal IP. > > > > > > Could you look for this log message in the AM logs: "Instantiated > > > DAGClientRPCServer at”. This will provide some information as to what the > > > AM is binding to. > > > > > > thanks > > > — Hitesh > > > > > > On Aug 11, 2014, at 7:43 AM, Johannes Zillmann <[email protected]> > > > wrote: > > > > > >> Hey guys, > > >> > > >> having a test-infrastructure for Hadoop on ec2. The client sits usually > > >> outside of ec2. > > >> Using plain map-reduce on YARN everything works fine. > > >> Using Tez i run into following exception: > > >> > > >> INFO [2014-07-29 00:09:06.653] [MrPlanRunnerV2] (TezClient.java:507) - > > >> Failed to retrieve AM Status via proxy > > >> com.google.protobuf.ServiceException: > > >> org.apache.hadoop.net.ConnectTimeoutException: Call From > > >> ip-10-73-6-154.ec2.internal/10.73.6.154 to > > >> ec2-54-81-245-144.compute-1.amazonaws.com:60914 failed on socket timeout > > >> exception: org.apache.hadoop.net.ConnectTimeoutException: connect timed > > >> out; For more details see: http: > > >> //wiki.apache.org/hadoop/SocketTimeout > > >> > > >> at > > >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:216) > > >> at com.sun.proxy.$Proxy116.getAMStatus(Unknown Source) > > >> at > > >> org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:500) > > >> at > > >> org.apache.tez.client.TezClient.waitTillReady(TezClient.java:586) > > >> > > >> > > >> I could resolve the problem for Tez changing the hostname of the > > >> instances to their public dns’. However, that is causing problems with > > >> other components. > > >> Do you know of any place in Tez which is related to that ? Any tweak > > >> which could make chaining the hostname superfluous ? > > >> > > >> best > > >> Johannes > > > > > > > > >
