[
https://issues.apache.org/jira/browse/CRUNCH-600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Bernstein updated CRUNCH-600:
----------------------------------
Description:
The credentials attached to a job are lost when CrunchOutputs builds a new Job
in getJob.
For example, the following scenario fails:
{code:title=MyTarget.java}
class MyTarget implements MapReduceTarget {
void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String
name) {
job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
CrunchOutputs.addNamedOutput(job, name,
FormatBundle.forOutput(MyOutputFormat.class), Object.class);
}
//..
}
{code}
{code:title=MyOutputFormat.java}
class MyOutputFormat extends OutputFormat<Object,Object> {
public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext
context) throws IOException, InterruptedException {
Assert.equals("secret".getBytes(),
context.getCredentials().getSecretKey("myPassword"));
}
}
{code}
This happens because CrunchOutputs.getJob copies the baseConf using the base
Configuration class and the base Configuration class doesn't have a notion of
Credentials, so the credentials get dropped.
This issue only manifests itself when using the local runner. In an actual
hadoop cluster the credentials will be inherited from UserGroupInformation.
was:
The credentials attached to a job are lost when CrunchOutputs builds a new Job
in getJob.
For example, the following scenario fails:
{code:title=MyTarget.java}
class MyTarget implements MapReduceTarget {
void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String
name) {
job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
CrunchOutputs.addNamedOutput(job, name,
FormatBundle.forOutput(MyOutputFormat.class), Object.class);
}
//..
}
{code}
{code:title=MyOutputFormat.java}
class MyOutputFormat extends OutputFormat<Object,Object> {
public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext
context) throws IOException, InterruptedException {
Assert.equals("secret".getBytes(),
context.getCredentials().getSecretKey("myPassword"));
}
}
{code}
This happens because CrunchOutputs.getJob copies the baseConf using the base
Configuration class and the base Configuration class doesn't have a notion of
Credentials, so the credentials get dropped.
This is issue only manifests itself when using the local runner. In an actual
hadoop cluster the credentials will be inherited from UserGroupInformation.
> Credentials are lost in local runner mode
> -----------------------------------------
>
> Key: CRUNCH-600
> URL: https://issues.apache.org/jira/browse/CRUNCH-600
> Project: Crunch
> Issue Type: Bug
> Reporter: Igor Bernstein
> Priority: Minor
> Attachments: CRUNCH-600.patch
>
>
> The credentials attached to a job are lost when CrunchOutputs builds a new
> Job in getJob.
> For example, the following scenario fails:
> {code:title=MyTarget.java}
> class MyTarget implements MapReduceTarget {
> void configureForMapReduce(Job job, PType<?> ptype, Path outputPath, String
> name) {
> job.getCredentials().addSecretKey("myPassword", "secret".getBytes());
> CrunchOutputs.addNamedOutput(job, name,
> FormatBundle.forOutput(MyOutputFormat.class), Object.class);
> }
> //..
> }
> {code}
> {code:title=MyOutputFormat.java}
> class MyOutputFormat extends OutputFormat<Object,Object> {
> public RecordWriter<Object, Object> getRecordWriter(TaskAttemptContext
> context) throws IOException, InterruptedException {
> Assert.equals("secret".getBytes(),
> context.getCredentials().getSecretKey("myPassword"));
> }
> }
> {code}
> This happens because CrunchOutputs.getJob copies the baseConf using the base
> Configuration class and the base Configuration class doesn't have a notion of
> Credentials, so the credentials get dropped.
> This issue only manifests itself when using the local runner. In an actual
> hadoop cluster the credentials will be inherited from UserGroupInformation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
