Default implementation is to add milliseconds. For mesos it is framework-id. If you are using mesos, you can assume that your framework id used to register your app is same as app-id. As you said, you have a system application to schedule spark jobs, you can keep track of framework-ids submitted by your application, you can use the same info.
On Fri, Sep 2, 2016 at 6:29 PM, Amit Shanker <amit.shank...@gmail.com> wrote: > Currently Spark sets current time in Milliseconds as the app Id. Is there > a way one can pass in the app id to the spark job, so that it uses this > provided app id instead of generating one using time? > > Lets take the following scenario : I have a system application which > schedules spark jobs, and records the metadata for that job (say job > params, cores, etc). In this system application, I want to link every job > with its corresponding UI (history server). The only way I can do this is > if I have the app Id of that job stored in this system application. And the > only way one can get the app Id is by using the > SparkContext.getApplicationId() function - which needs to be run from > inside the job. So, this make it difficult to convey this piece of > information from spark to a system outside spark. > > Thanks, > Amit Shanker >
