I did test my change: https://github.com/radekg/spark/commit/b21aae1468169ee0a388d33ba6cebdb17b895956#diff-0c89b4a60c30a7cd2224bb64d93da942R125 It seems to do what I want it to do, however, not quite sure about the overall impact.
I’d appreciate if someone who knows the NettyRpcEnv details could point to some documentation. Kind regards, Radek Gruchalski [email protected] (mailto:[email protected]) (mailto:[email protected]) de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Thursday, 26 November 2015 at 20:45, Rad Gruchalski wrote: > Dear all, > > I am currently looking at modifying NettyRpcEnv for this PR: > https://github.com/apache/spark/pull/9608 > The functionality which I’m trying to achieve is the following: if there is a > configuration property spark.driver.advertisedPort, make executors reply to > advertisedPort instead of spark.driver.port. This would enable NettyRpcEnv > work correctly on Mesos with Docker bridge networking (something what is > working spot on for AkkaRcpEnv). > > I’ve spent some time looking at the new NettyRpcEnv and I think I know what > is happening but seeking for confirmation. > > The master RpcEndpointAddress appears to be shipped to the executor as part > of a serialized message, when the executors are requested, inside of the > NettyRpcEndpointRef. In order to make my PR work, I need to change the > RcpEndpointAddress shipped to the executors on the master. > I think the best candidate is: > > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala#L125 > > Am I correct that this value becomes the RcpEndpointAddress? If so, my naive > implementation would be like this: > > if (server != null) RpcAddress(host, > conf.getInt(“spark.driver.advertisedPort”, server.getPort())) else null > > however, I am not sure what impact such change would have, about to test my > changes as soon as the code from master branch builds. > > If that is not the correct place to make such change, what would be the best > place to investigate? Is there an overview of NettyRcpEnv available anywhere? > > > > > > > > > > > Kind regards, > Radek Gruchalski > [email protected] (mailto:[email protected]) > (mailto:[email protected]) > de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/) > > Confidentiality: > This communication is intended for the above-named person and may be > confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor must > you copy or show it to anyone; please delete/destroy and inform the sender > immediately. > > > >
