[ https://issues.apache.org/jira/browse/PIG-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615262#comment-15615262 ]
Adam Szita commented on PIG-5052: --------------------------------- Are we looking for a way to set this value to be equal with the job group ID created in SparkLauncher.launchPig() below? {code} public PigStats launchPig(PhysicalPlan physicalPlan, String grpName, PigContext pigContext) throws Exception ... // Set a unique group id for this query, so we can lookup all Spark job // ids // related to this query. jobGroupID = UUID.randomUUID().toString(); sparkContext.setJobGroup(jobGroupID, "Pig query to Spark cluster", false); {code} Also there is an appID available for a given sparkContext, which can be queried as: {code} String appId = sparkContext.getConf().getAppId(); {code} > Initialize MRConfiguration.JOB_ID in spark mode correctly > --------------------------------------------------------- > > Key: PIG-5052 > URL: https://issues.apache.org/jira/browse/PIG-5052 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: liyunzhang_intel > Fix For: spark-branch > > > currently, we initialize MRConfiguration.JOB_ID in SparkUtil#newJobConf. > we just set the value as a random string. > {code} > jobConf.set(MRConfiguration.JOB_ID, UUID.randomUUID().toString()); > {code} > We need to find a spark api to initiliaze it correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)