Cheolsoo Park created PIG-3145:
----------------------------------
Summary: Parameters in core-site.xml and mapred-site.xml are not
correctly substituted
Key: PIG-3145
URL: https://issues.apache.org/jira/browse/PIG-3145
Project: Pig
Issue Type: Bug
Reporter: Cheolsoo Park
Assignee: Cheolsoo Park
To reproduce the issue, please do the following:
# Parameterize the address of name node in core-site.xml.
{code}
<property>
<name>fs.default.name</name>
<value>hdfs://${foo}:8020</value>
</property>
{code}
# Set the value of "foo" via -D option.
{code}
export PIG_OPTS="-Dfoo=mr1-0.cheolsoo.com"
{code}
# Pig fails with the following error.
{code}
2013-01-28 18:54:02,786 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: hdfs://${foo}:8020
2013-01-28 18:54:02,805 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. null
Details at logfile: /home/cheolsoo/pig-cdh/pig_1359428042522.log
{code}
Note that the parameter $\{foo\} in core-site.xml is not expanded. This is
because the addresses of name node and job tracker are read directly from
core-site.xml instead of reading via Configuration.get().
{code:title=HExecutionEngine.java}
// properties is Java Properties
cluster = properties.getProperty(JOB_TRACKER_LOCATION);
nameNode = properties.getProperty(FILE_SYSTEM_LOCATION);
{code}
Replacing these lines with Configuration.get() fixes the issue.
{code:title=HExecutionEngine.java}
// jc is Hadoop Configuration
cluster = jc.get(JOB_TRACKER_LOCATION);
nameNode = jc.get(FILE_SYSTEM_LOCATION);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
