How about this: 1. You create a primary key in your custom system. 2. Schedule the job with custom primary name as the job name. 3. After setting up spark context (inside the job) get the application id. Then save the mapping of App Name & AppId from spark job to your custom database, through some web service.
On Sun, Sep 4, 2016 at 12:30 AM, Raghavendra Pandey < raghavendra.pan...@gmail.com> wrote: > Default implementation is to add milliseconds. For mesos it is > framework-id. If you are using mesos, you can assume that your framework id > used to register your app is same as app-id. > As you said, you have a system application to schedule spark jobs, you can > keep track of framework-ids submitted by your application, you can use the > same info. > > On Fri, Sep 2, 2016 at 6:29 PM, Amit Shanker <amit.shank...@gmail.com> > wrote: > >> Currently Spark sets current time in Milliseconds as the app Id. Is there >> a way one can pass in the app id to the spark job, so that it uses this >> provided app id instead of generating one using time? >> >> Lets take the following scenario : I have a system application which >> schedules spark jobs, and records the metadata for that job (say job >> params, cores, etc). In this system application, I want to link every job >> with its corresponding UI (history server). The only way I can do this is >> if I have the app Id of that job stored in this system application. And the >> only way one can get the app Id is by using the >> SparkContext.getApplicationId() function - which needs to be run from >> inside the job. So, this make it difficult to convey this piece of >> information from spark to a system outside spark. >> >> Thanks, >> Amit Shanker >> > > -- Best Regards, Ayan Guha
