Each mapper works on only one file split, which is either from file1 or
file2 in your case. So the value for map.input.file gives you the exact
information you need.
Runping
On 10/23/08 11:09 AM, Steve Gao [EMAIL PROTECTED] wrote:
Thanks, Amogh. But my case is slightly different. The
Sorry for the email. Thanks for any help or hint.
I am using Hadoop Streaming. The input are multiple files.
Is there a way to get the current filename in mapper?
For example:
$HADOOP_HOME/bin/hadoop \
jar $HADOOP_HOME/hadoop-streaming.jar \
-input file1 \
Thanks, Amogh. But my case is slightly different. The command line inputs are 2
files: file1 and file2. I need to tell in the mapper which line is from which
file:
#In mapper
while (STDIN){
//how to tell the current line is from file1 or file2?
}
-jobconfs map.input.file param does not help
I guess one trick you can do without the help of hadoop is to encode the file
identifier inside the file itself. For example, each line of file1 could start
with 1'space''content of the original line'.
- Original Message
From: Steve Gao [EMAIL PROTECTED]
To:
On Wed, Oct 22, 2008 at 18:55, Steve Gao [EMAIL PROTECTED] wrote:
I am using Hadoop Streaming. The input are multiple files.
Is there a way to get the current filename in mapper?
Streaming map tasks should have a map_input_file environment
variable like the following:
I am using Hadoop Streaming. The input are multiple files.
Is there a way to get the current filename in mapper?
For example:
$HADOOP_HOME/bin/hadoop \
jar $HADOOP_HOME/hadoop-streaming.jar \
-input file1 \
-input file2 \
-output myOutputDir \
-mapper mapper \
-reducer
