[
https://issues.apache.org/jira/browse/PIG-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804628#comment-13804628
]
Daniel Dai commented on PIG-3441:
---------------------------------
[~bhooshan], I need to make sure we are talking about the same thing. Here is
the thing I know should work:
1. If you are embedding Pig in Java, you can pass a conf object to PigServer:
{code}
Configuration conf = ..... // make conf whatever you want, include
Configuration.addDefaultResource()
PigServer pServer = new PigServer(ExecType.LOCAL, conf); // conf will be
materialized and pass to Pig
{code}
Pig will take the configuration you pass
2. Sequence of the config:
Pig currently will use its predefined config file to override the conf object
you pass to PigServer, so if those config files contains the entry in your conf
object, it will get overridden. PIG-3135 address this issue by setting a flag
not override those entries, but this requires conf object contains every
entries needed (include those should be in core-site, hdfs-site, mapred-site)
So in your case, if you use Pig embedding, you can instantiate the conf object
and pass to Pig. If you are not using Pig embedding, then things could be
different. Let me know if that is the case.
> Allow Pig to use default resources from Configuration objects
> -------------------------------------------------------------
>
> Key: PIG-3441
> URL: https://issues.apache.org/jira/browse/PIG-3441
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11.1
> Reporter: Bhooshan Mogal
> Attachments: PIG-3441_1.patch, PIG-3441.patch
>
>
> Pig currently ignores parameters from configuration files added statically to
> Configuration objects as Configuration.addDefaultResource(filename.xml).
> Consider the following scenario -
> In a hadoop FileSystem driver for a non-HDFS filesystem you load properties
> specific to that FileSystem in a static initializer block in the class that
> extends org.apache.hadoop.fs.Filesystem for your FileSystem like below -
> {code}
> class MyFileSystem extends FileSystem {
> static {
> Configuration.addDefaultResource("myfs-default.xml");
> Configuration.addDefaultResource("myfs-site.xml");
> }
> }
> {code}
> Interfaces like the Hadoop CLI, Hive, Hadoop M/R can find configuration
> parameters defined in these configuration files as long as they are on the
> classpath.
> However, Pig cannot find parameters from these files, because it ignores
> configuration files added statically.
> Pig should allow users to specify if they would like pig to read parameters
> from resources loaded statically.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
