That works when it is launched from same process - which is
unfortunately not our case :-)
- Mridul
On Sun, May 10, 2015 at 9:05 PM, Manku Timma <manku.tim...@gmail.com> wrote:
> sc.applicationId gives the yarn appid.
>
> On 11 May 2015 at 08:13, Mridul Muralidharan <mri...@gmail.com> wrote:
>>
>> We had a similar requirement, and as a stopgap, I currently use a
>> suboptimal impl specific workaround - parsing it out of the
>> stdout/stderr (based on log config).
>> A better means to get to this is indeed required !
>>
>> Regards,
>> Mridul
>>
>> On Sun, May 10, 2015 at 7:33 PM, Ron's Yahoo!
>> <zlgonza...@yahoo.com.invalid> wrote:
>> > Hi,
>> > I used to submit my Spark yarn applications by using
>> > org.apache.spark.yarn.deploy.Client api so I can get the application id
>> > after I submit it. The following is the code that I have, but after
>> > upgrading to 1.3.1, the yarn Client class was made into a private class. Is
>> > there a particular reason why this Client class was made private?
>> > I know that there’s a new SparkSubmit object that can be used, but
>> > it’s not clear to me how I can use it to get the application id after
>> > submitting to the cluster.
>> > Thoughts?
>> >
>> > Thanks,
>> > Ron
>> >
>> > class SparkLauncherServiceImpl extends SparkLauncherService {
>> >
>> > override def runApp(conf: Configuration, appName: String, queue:
>> > String): ApplicationId = {
>> > val ws = SparkLauncherServiceImpl.getWorkspace()
>> > val params = Array("--class", //
>> > "com.xyz.sparkdb.service.impl.AssemblyServiceImpl", //
>> > "--name", appName, //
>> > "--queue", queue, //
>> > "--driver-memory", "1024m", //
>> > "--addJars",
>> > getListOfDependencyJars(s"$ws/ledp/le-sparkdb/target/dependency"), //
>> > "--jar",
>> > s"file:$ws/ledp/le-sparkdb/target/le-sparkdb-1.0.3-SNAPSHOT.jar")
>> > System.setProperty("SPARK_YARN_MODE", "true")
>> > System.setProperty("spark.driver.extraJavaOptions",
>> > "-XX:PermSize=128m -XX:MaxPermSize=128m
>> > -Dsun.io.serialization.extendedDebugInfo=true")
>> > val sparkConf = new SparkConf()
>> > val args = new ClientArguments(params, sparkConf)
>> > new Client(args, conf, sparkConf).runApp()
>> > }
>> >
>> > private def getListOfDependencyJars(baseDir: String): String = {
>> > val files = new
>> > File(baseDir).listFiles().filter(!_.getName().startsWith("spark-assembly"))
>> > val prependedFiles = files.map(x => "file:" + x.getAbsolutePath())
>> > val result = ((prependedFiles.tail.foldLeft(new
>> > StringBuilder(prependedFiles.head))) {(acc, e) => acc.append(",
>> > ").append(e)}).toString()
>> > result
>> > }
>> > }
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
