[ https://issues.apache.org/jira/browse/PIG-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433199#comment-13433199 ]
Bill Graham commented on PIG-2872: ---------------------------------- I ran into issues with this as well. See PIG-2870 and the attached patch. Does Wonderdog work if you apply this patch? > StoreFuncInterface.setStoreLocation get's a copy of a Configuration object > -------------------------------------------------------------------------- > > Key: PIG-2872 > URL: https://issues.apache.org/jira/browse/PIG-2872 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: 0.11 > Environment: Pig trunk, Hadoop 0.20.205 with Kerberos, ElasticSearch > trunk, Wonderdog trunk > Reporter: Evert Lammerts > > When an implementation of StoreFuncInterface.setStoreLocation is called from > JobControlCompiler.getJob, it is passed a copy of the Configuration that will > be used for the Job that will be submitted: > {code:title=JobControlCompiler.java} > sFunc.setStoreLocation(st.getSFile().getFileName(), new > org.apache.hadoop.mapreduce.Job(nwJob.getConfiguration())); > {code} > When a new org.apache.hadoop.mapreduce.Job is created it creates a copy of > the Configuration object, as far as I know. Thus anything added to the > Configuration object in the implementation of setStoreLocation will not be > included in the Configuration of nwJob in JobControlCompiler.getJob. > I notice this goes wrong in Wonderdog, which needs to include the > Elasticsearch configuration file in the DistributedCache. It is added to > mapred.cache.files through setStoreLocation, but this setting doesn't make it > back into the Job returned by JobControlCompiler.getJob, and is therefore > never localized. > This might be intentional semantics within Pig, but I'm not familiar enough > with StoreFuncs to know whether it is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira