[I'm starting a new thread because there is sufficiently new/weird info but
I've attached an initial thread in case that might be useful.]
I'm running Hadoop 0.19.0. I have an input file consisting of several lines
that each contain a name and URL. The idea is pretty simple: download the
contents found at the URL and store them in S3. Each download can take quite
a while (up to an hour) so I've set mapred.task.timeout=0. I'm using
streaming with Ruby mappers. I'd like each line to be handled by one map. In
the thread below it was suggested that I use NInputLineFormat so that all
available map tasks will be utilized; I had found that with my single input
file some tasks would be allocated but others would sit idly. I have no need
for the reduce function and hence have set
-jobconf mapred.reduce.tasks=0
With NInputLineFormat I noticed some weird behavior. First, directories were
being created with numerical names as opposed to the meaningful names that I
was expecting. It was suggested that the directory names were the
corrsponding key (offsets within the line) but the names don't seem to
correspond to offset values.
The second weirdness is as follows. Eventually I get the directories I'm
expecting with meaningful names but it appears that after stopping Hadoop
the directories still exist, then disappear and then re-appear as I refresh
the S3 Firefox tool or do a check via the Hadoop command shell. I'm
expecting a directory structure as follows:
user/hadoop/base
.../base/hadoopInput
.../base/alt1
.../base/alt2
.../base/alt3
...
Below I've used the Hadoop command shell to check what directories exist.
Note that I called the following three commands within a period of seconds.
[had...@storm url_download]$ *hadoop dfs -ls base*
Found 1 items
drwxrwxrwx - 0 1969-12-31 18:00 /user/hadoop/base/hadoopInput
[had...@storm url_download]$ *hadoop dfs -ls base*
Found 4 items
drwxrwxrwx - 0 1969-12-31 18:00 /user/hadoop/base/alt1
drwxrwxrwx - 0 1969-12-31 18:00 /user/hadoop/base/alt2
drwxrwxrwx - 0 1969-12-31 18:00 /user/hadoop/base/alt3
drwxrwxrwx - 0 1969-12-31 18:00 /user/hadoop/base/hadoopInput
[had...@storm url_download]$ *hadoop dfs -ls base/alt1*
Found 3 items
-rwxrwxrwx 1 505 2009-02-10 09:22 /user/hadoop/base/alt1/file.asx
-rwxrwxrwx 1 56880 2009-02-10 09:22 /user/hadoop/base/alt1/file.txt
-rwxrwxrwx 1 68989 2009-02-10 09:22 /user/hadoop/base/alt1/cfg.xml
S3 Firefox is giving similar results; I just did a refresh of
/user/hadoop/base that listed only hadoopInput followed by a second refresh
that listed the other subdirectories.
Any clues on how this could be? Perhaps there is a leftover process still
running?
Thanks,
John
On Mon, Feb 2, 2009 at 9:38 PM, Amareshwari Sriramadasu <
[email protected]> wrote:
> S D wrote:
>
>> Thanks for your response. I'm using version 0.19.0 of Hadoop.
>> I tried your suggestion. Here is the line I use to invoke Hadoop
>>
>> hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-0.19.0-streaming.jar \\
>> -input /user/hadoop/hadoop-input/inputFile.txt \\
>> -output /user/hadoop/hadoop-output \\
>> -mapper map-script.sh \\
>> -file map-script.sh \\
>> -file additional-script.rb \\ # Called by map-script.sh
>> -file utils.rb \\
>> -file env.sh \\
>> -file aws-s3-credentials-file \\ # For permissions to use AWS::S3
>> -jobconf mapred.reduce.tasks=0 \\
>> -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat
>>
>> Everything works fine if the -inputformat switch is not included but when
>> I
>> include it I get the following message:
>> ERROR streaming.StreamJob: Job not Successful!
>> and a Runtime exception shows up in the jobtracker log:
>> PipeMapRed.waitOutputThreads(): subprocess failed with code 1
>>
>> My map functions read each line of the input file and create a directory
>> (one for each line) on Hadoop (in our case S3 Native) in which
>> corresponding
>> data is produced and stored. The name of the created directories are based
>> on the contents of the corresponding line. When I include the -inputformat
>> line above I've noticed that instead of the directories I'm expecting
>> (named
>> after the data found in the input file), the directories are given
>> seemingly
>> arbitrary numeric names; e.g., when the input file contained four lines of
>> data, the directories were named: 0, 273, 546 and 819.
>>
>>
>>
> LineRecordReader reads "line" as VALUE and the KEY is offset in the file.
> Looks like your directories are getting named with KEY. But I don't see
> any reason for that, because it is working fine with TextInputFormat (both
> TextInFormat and NLineInputFormat use LineRecordReader.)
>
> -Amareshwari
>
> Any thoughts?
>>
>> John
>>
>> On Sun, Feb 1, 2009 at 11:00 PM, Amareshwari Sriramadasu <
>> [email protected]> wrote:
>>
>>
>>
>>> Which version of hadoop are you using?
>>>
>>> You can directly use -inputformat
>>> org.apache.hadoop.mapred.lib.NLineInputFormat for your streaming job. You
>>> need not include it in your streaming jar.
>>> -Amareshwari
>>>
>>>
>>> S D wrote:
>>>
>>>
>>>
>>>> Thanks for your response Amereshwari. I'm unclear on how to take
>>>> advantage
>>>> of NLineInputFormat with Hadoop Streaming. Is the idea that I modify the
>>>> streaming jar file (contrib/streaming/hadoop-<version>-streaming.jar) to
>>>> include the NLineInputFormat class and then pass a command line
>>>> configuration param to indicate that NLineInputFormat should be used? If
>>>> this is the proper approach, can you point me to an example of what kind
>>>> of
>>>> param should be specified? I appreciate your help.
>>>>
>>>> Thanks,
>>>> SD
>>>>
>>>> On Thu, Jan 29, 2009 at 10:49 PM, Amareshwari Sriramadasu <
>>>> [email protected]> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> You can use NLineInputFormat for this, which splits one line (N=1, by
>>>>> default) as one split.
>>>>> So, each map task processes one line.
>>>>> See
>>>>>
>>>>>
>>>>> http://hadoop.apache.org/core/docs/r0.19.0/api/org/apache/hadoop/mapred/lib/NLineInputFormat.html
>>>>>
>>>>> -Amareshwari
>>>>>
>>>>> S D wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have a clarifying question about Hadoop streaming. I'm new to the
>>>>>> list
>>>>>> and
>>>>>> didn't see anything posted that covers my questions - my apologies if
>>>>>> I
>>>>>> overlooked a relevant post.
>>>>>>
>>>>>> I have an input file consisting of a list of files (one per line) that
>>>>>> need
>>>>>> to be processed independently of each other. The duration for
>>>>>> processing
>>>>>> each file is significant - perhaps an hour each. I'm using Hadoop
>>>>>> streaming
>>>>>> without a reduce function to process each file and save the results
>>>>>> (back
>>>>>> to
>>>>>> S3 native in my case). To handle to long processing time of each file
>>>>>> I've
>>>>>> set mapred.task.timeout=0 and I have a pretty straight forward Ruby
>>>>>> script
>>>>>> reading from STDIN:
>>>>>>
>>>>>> STDIN.each_line do |line|
>>>>>> # Get file from contents of line
>>>>>> # Process file (long running)
>>>>>> end
>>>>>>
>>>>>> Currently I'm using a cluster of 3 workers in which each worker can
>>>>>> have
>>>>>> up
>>>>>> to 2 tasks running simultaneously. I've noticed that if I have a
>>>>>> single
>>>>>> input file with many lines (more than 6 given my cluster), then not
>>>>>> all
>>>>>> workers will be allocated tasks; I've noticed two workers being
>>>>>> allocated
>>>>>> one task each and the other worker sitting idly. If I split my input
>>>>>> file
>>>>>> into multiple files (at least 6) then all workers will be immediately
>>>>>> allocated the maximum number of tasks that they can handle.
>>>>>>
>>>>>> My interpretation on this is fuzzy. It seems that Hadoop streaming
>>>>>> will
>>>>>> take
>>>>>> separate input files and allocate a new task per file (up to the
>>>>>> maximum
>>>>>> constraint) but if given a single input file it is unclear as to
>>>>>> whether
>>>>>> a
>>>>>> new task is allocated per file or line. My understanding of Hadoop
>>>>>> Java
>>>>>> is
>>>>>> that (unlike Hadoop streaming) when given a single input file, the
>>>>>> file
>>>>>> will
>>>>>> be broken up into separate lines and the maximum number of map tasks
>>>>>> will
>>>>>> automagically be allocated to handle the lines of the file (assuming
>>>>>> the
>>>>>> use
>>>>>> of TextInputFormat).
>>>>>>
>>>>>> Can someone clarify this?
>>>>>>
>>>>>> Thanks,
>>>>>> SD
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>