I figured out JOB id returned from sparkAppHandle.getAppId(), is unique ApplicationId which looks like these:
for Local mode Spark env: Local-1477184581895 For Distributed Spark mode: Application_1477504900821_0005 ApplicationId represents the globally unique identifier for an application. The globally unique nature of the identifier is achieved by using the cluster timestamp i.e. start-time of the ResourceManager along with a monotonically increasing counter for the application. On Sat, Oct 22, 2016 at 5:18 PM Elkhan Dadashov <elkhan8...@gmail.com> wrote: > I found answer regarding logging in the JavaDoc of SparkLauncher: > > "Currently, all applications are launched as child processes. The child's > stdout and stderr are merged and written to a logger (see > <code>java.util.logging</code>)." > > One last question. sparkAppHandle.getAppId() - does this function > return org.apache.hadoop.mapred.*JobID* which makes it easy tracking in > Yarn ? Or is appId just the Spark app name we assign ? > > If it is JobID, then even if the SparkLauncher handle goes away, by > talking directly to the cluster manager, i can get Job details. > > Thanks. > > On Sat, Oct 22, 2016 at 4:53 PM Elkhan Dadashov <elkhan8...@gmail.com> > wrote: > > Thanks, Marcelo. > > One more question regarding getting logs. > > In previous implementation of SparkLauncer we could read logs from : > > sparkLauncher.getInputStream() > sparkLauncher.getErrorStream() > > What is the recommended way of getting logs and logging of Spark execution > while using sparkLauncer#startApplication() ? > > Thanks. > > On Tue, Oct 18, 2016 at 3:07 PM Marcelo Vanzin <van...@cloudera.com> > wrote: > > On Tue, Oct 18, 2016 at 3:01 PM, Elkhan Dadashov <elkhan8...@gmail.com> > wrote: > > Does my map task need to wait until Spark job finishes ? > > No... > > > Or is there any way, my map task finishes after launching Spark job, and > I > > can still query and get status of Spark job outside of map task (or > failure > > reason, if it has failed) ? (maybe by querying Spark job id ?) > > ...but if the SparkLauncher handle goes away, then you lose the > ability to track the app's state, unless you talk directly to the > cluster manager. > > > I guess also if i want my Spark job to be killed, if corresponding > delegator > > map task is killed, that means my map task needs to stay alive, so i > still > > have SparkAppHandle reference ? > > Correct, unless you talk directly to the cluster manager. > > -- > Marcelo > >
