Hi Kevin,
One thing that might help you in the meantime, while we work on a better
interface for all this...
On Thu, May 21, 2015 at 5:21 PM, Kevin Markey <kevin.mar...@oracle.com>
wrote:
> Making *yarn.Client* private has prevented us from moving from Spark
> 1.0.x to Spark 1.2 or 1.3 despite many alluring new features.
>
Since you're not afraid to use private APIs, and to avoid using ugly
reflection hacks, you could abuse the fact that private things in Scala are
not really private most of the time. For example (trimmed to show just
stuff that might be interesting to you):
# javap -classpath
/opt/cloudera/parcels/CDH/jars/spark-assembly-1.3.0-cdh5.4.0-hadoop2.6.0-cdh5.4.0.jar
org.apache.spark.deploy.yarn.Client
Compiled from "Client.scala"
public class org.apache.spark.deploy.yarn.Client implements
org.apache.spark.Logging {
...
public org.apache.hadoop.yarn.client.api.YarnClient
org$apache$spark$deploy$yarn$Client$$yarnClient();
public void run();
public
org.apache.spark.deploy.yarn.Client(org.apache.spark.deploy.yarn.ClientArguments,
org.apache.hadoop.conf.Configuration, org.apache.spark.SparkConf);
public
org.apache.spark.deploy.yarn.Client(org.apache.spark.deploy.yarn.ClientArguments,
org.apache.spark.SparkConf);
public
org.apache.spark.deploy.yarn.Client(org.apache.spark.deploy.yarn.ClientArguments);
}
So it should be easy to write a small Java wrapper around this. No less
hacky than relying on the "private-but-public" code of before.
--
Marcelo
