Hi
Under /mapred/local there are directories like
"attempt_200904262046_0026_m_02_0"
Each of these directories contains files of format: intermediate.1
intermediate.2 intermediate.3 intermediate.4 intermediate.5
There are many directories in this format. All these correspond to
killed task
The problem is not going away but i have a lead
Here is a detailed diagnostic of what goes wrong:
1) 2 cluster setup on Ubuntu machines (data node and tasktracker is running
on both...)
2) The reducer tries to look for a non existent file as described below:
The problem happens only on my mach
Hi
What happens when the node rejoins then ?
- The replication level of several blocks increases ?
- The old replicas are removed in favor of new replicas ? (Or the new
ones)
regards
Piotr
2009/4/27 Stas Oskin
> Thanks.
>
> 2009/4/27 Koji Noguchi
>
> > http://hadoop.apache.org/core/
Hi,
It seems related with https://issues.apache.org/jira/browse/HADOOP-4654.
On Tue, Apr 28, 2009 at 4:01 PM, Sandhya E wrote:
> Hi
>
> Under /mapred/local there are directories like
> "attempt_200904262046_0026_m_02_0"
> Each of these directories contains files of format: intermediate.1
> i
Cloudera's Distribution for Hadoop is based on Hadoop 0.18.3 but includes a
backport of HADOOP-2536. You could switch to this distribution instead.
Otherwise, download the 18-branch patch from issue HADOOP-2536 (
http://issues.apache.org/jira/browse/hadoop-2536) and apply it to your local
copy and
Should also say, the link to CDH is http://www.cloudera.com/hadoop
- Aaron
On Tue, Apr 28, 2009 at 5:06 PM, Aaron Kimball wrote:
> Cloudera's Distribution for Hadoop is based on Hadoop 0.18.3 but includes a
> backport of HADOOP-2536. You could switch to this distribution instead.
> Otherwise, do
HI I have applied a small patch for version 0.20 to my old 0.19.1...
After i ran the ant tar I found 3 directories libhdfs and libcodio and c++
were missing from th tared build. Where do you get those from?
I cant really use 0.20 because of massive library changes... So if some one
can help me out
Hi Sandhya,
Which version of HADOOP are you using? There could be
directories in mapred/local, pre 0.17. Now, there should not be any such
directories.
From version 0.17 onwards, the attempt directories will be present only
at mapred/local/taskTracker/jobCache// . If you are
seeing the dire
Hi Amareshwari
We are on 0.18 version. I verified from jobtracker website that not
all killed tasks have left overs in mapred/local. Also there are some
tasks that were successful have left their tmp folders in mapred/local
Can you please give some pointers on how to debug it further.
Regards
S
Again, where are you seeing the attemptid directories? are they at
mapred/local/ or at
mapred/local/taskTracker/jobCache//.
If you are seeing files at mapred/local/, then it is bug.
Please raise a jira and attach tasktracker logs if possible.
If not, mapred/local/taskTracker/jobCache// directori
Hi all,
I am writing an application in which I create a forked process to execute a
specific Map/Reduce job. The problem is that when I try to read the output
stream of the forked process I get nothing and when I execute the same job
manually it starts printing the output I am expecting. For cl
If I understand correctly - Hadoop forms a general purpose cluster on
which you can execute jobs?
We have a Java data processing application here that follows the
Producer -> Consumer pattern. It has been written with threading as a
concern from the start using java.util.concurrent.Callable.
At
Razen Al Harbi wrote:
Hi all,
I am writing an application in which I create a forked process to execute a
specific Map/Reduce job. The problem is that when I try to read the output
stream of the forked process I get nothing and when I execute the same job
manually it starts printing the outpu
Hi,
Is that command available for all nodes? Did you try as below? ;)
Process proc = rt.exec("/bin/hostname");
..
output.collect(hostname, disk usage);
On Tue, Apr 28, 2009 at 6:13 PM, Razen Al Harbi wrote:
> Hi all,
>
> I am writing an application in which I create a forked process to execute
Attempt directories are in /mapred/local
I grep'd for one of the attempt that has left over in mapred/local in
tasktracker logs:
09/04/27 21:07:19 INFO mapred.TaskTracker: LaunchTaskAction:
attempt_200902120108_44218_r_00_0
09/04/27 21:07:29 INFO mapred.TaskTracker:
attempt_200902120108_44218_
hello hadoop users,
Recently I had a chance to lead a team building a log-processing system that
uses Hadoop and MySQL. The system's goal was to process the incoming
information as quickly as possible (real time or near real time), and make it
available for querying in MySQL. I thought it woul
Vishal Ghawate wrote:
Hi,
I want to store the contents of all the client machine(datanode)of hadoop
cluster to centralized machine
with high storage capacity.so that tasktracker will be on the client machine
but the contents are stored on the
centralized machine.
Can anybody he
Have a look at the instructions on
http://wiki.apache.org/hadoop/HowToRelease under the "Building"
section. It tells you which environment settings and Ant targets you
need to set.
Tom
On Tue, Apr 28, 2009 at 9:09 AM, Sid123 wrote:
>
> HI I have applied a small patch for version 0.20 to my old 0
Hi Ankur
Nice share , btw whats your query behavior ?
I'm asking because if the query is simple or could be
transform/normalized , you could try output to HBase directly?
Yi-Kai
hello hadoop users,
Recently I had a chance to lead a team building a log-processing system that uses Hadoop and M
Hi.
I think one needs to run balancer in order to clean out the redundant
blocks.
Can anyone confirm this?
Regards.
2009/4/28 Piotr
> Hi
>
> What happens when the node rejoins then ?
>
> - The replication level of several blocks increases ?
> - The old replicas are removed in favor of new
Hi.
Any idea if the getDiskStatus() function requires superuser rights?
Or it can work for any user?
Thanks.
2009/4/9 Aaron Kimball
> You can insert this propery into the jobconf, or specify it on the command
> line e.g.: -D hadoop.job.ugi=username,group,group,group.
>
> - Aaron
>
> On We
Hi,
That's really cool. It seems that Hadoop could work with SQL DBs like
Mysql with bearable time. I thought when inserting data to Mysql, the
expense of communication was always a big problem. You got a method to
reduce the expense.
Using Distribution Databases like HBase is another good choice
Aaron Kimball wrote:
I'm not aware of any documentation about this particular use case for
Hadoop. I think your best bet is to look into the JNI documentation about
loading native libraries, and go from there.
- Aaron
You could also try
1. Starting the main processing app as a process on the m
Each document processing is independent and can be processed
parallelly, so that part could be done in a map reduce job.
Now whether it suits this use case depends on rate at which new
URI's are discovered for processing and acceptable delay in processing
of a document. The way I see it you can bat
Hi Adam,
It seems that producers and consumer work parallelly, so you can use
Hadoop to process your application. But the problem is the expense of
commucation with DB. You can refer to Ankur's thread with subject
'Hadoop / MySQL'.
Regards,
On Tue, Apr 28, 2009 at 6:05 PM, Adam Retter wrote:
> Are you interested in building such a system?
I would be interested in using such a system, but otherwise I am afraid
that I do not have the time resources available to be involved in such a
project. Sorry.
Registered Office: 7 Abbey Court, Eagle Way, Sowton, Exeter, Devon, EX2 7HY
Registered
> Each document processing is independent and can be processed
> parallelly, so that part could be done in a map reduce job.
> Now whether it suits this use case depends on rate at which new
> URI's are discovered for processing and acceptable delay in processing
> of a document. The way I see it
Hi,
[Ankur]: How can make sure this happens?
-- show processlist is how we spot it... literally it takes hours on
our set up so easy to find.
So we ended up with 2 DBs
- DB1 we insert to, prepare and do batch processing
- DB2 serving the read only web app
Periodically we dump the DB1, point the
Sorry, that was not meant to be sent to the list...
On Tue, Apr 28, 2009 at 3:27 PM, tim robertson
wrote:
> Hi,
>
> [Ankur]: How can make sure this happens?
> -- show processlist is how we spot it... literally it takes hours on
> our set up so easy to find.
>
> So we ended up with 2 DBs
> - DB1
I remember reading an article last year about something similar done
by Rackspace.
They went through various iterations of their logging system and
encountered similar
scaling issues with MySQL.
In the end they started using Hadoop, Lucene and Solr:
"
How Rackspace Now Uses MapReduce and
Thanks for sharing sounds like a nice system - I always advise people to
avoid direct SQL inserts for batch jobs / large amounts of data and use
MySQL's optimized LOAD utility like you did. Same goes for Oracle...
Nothing brings a DB server to its knees like a ton of individual inserts on
indexed
Warning: derailing a bit into MySQL discussion below, but I think enough
people have similar use cases that it's worth discussing this even though
it's gotten off-topic.
2009/4/28 tim robertson
>
> So we ended up with 2 DBs
> - DB1 we insert to, prepare and do batch processing
> - DB2 serving th
Hello users,
I am trying to program a hadoop powered web application with Eclipse as IDE.
Now I have both hadoop perspective and java ee perspective. I wonder if I
can have these two together, that I can use the mapper and producers in my
servlet. Anyone has experience about this? I will apprec
George,
I haven't used the Hadoop perspective in Eclipse so I can't help with that
specifically but map/reduce is a batch process (and can be long running). In
my experience, I've written servlets that write to HDFS and then have a
background process perform the map/reduce. They can both run in b
Excellent discussion. Thank you Todd.
You're forgiven for being off topic (at least by me).
:)
Bill
-Original Message-
From: Todd Lipcon [mailto:t...@cloudera.com]
Sent: Tuesday, April 28, 2009 2:29 PM
To: core-user
Subject: Re: Hadoop / MySQL
Warning: derailing a bit into MySQL discus
HDFS is designed with Hadoop in mind, so there are certain advantages (e.g.
performance, reliability, and ease of use) to using HDFS for Hadoop.
However, it's not required. For example, when you run Hadoop in standalone
mode, it just uses the file system on your local machine. When you run it on
Am
Thanks for the reply,
-Steve:
I know that I can use the JobClient to run or submit jobs; however, for the
time being I need to exec the job as a separate process.
-Edward:
The forked job is not executed from witin a map or reduce so I dont need to
do data collection.
It seems for some reason t
Registration and more information:
http://www.meetup.com/CIO-IT-Executives/calendar/10266376/
Limited seats - register now!
The master in my cluster crashed, the dfs/mapred java processes are
still running on the slaves. What should I do next? I brought the master
back up and ran stop-mapred.sh and stop-dfs.sh and it said this:
slave1.test.com: no tasktracker to stop
slave1.test.com: no datanode to stop
Not sure wha
Hi.
I'm writing streaming based tasks that involves running thousands of
mappers, after that I want to put all these outputs into small number (say
30) output files mainly so that disk space will be used more efficiently,
the way I'm doing it right now is using /bin/cat as reducer and setting
n
Hi all!
I have the large String and I want to write it into the file in HDFS.
(The large string has >100.000 lines.)
Current, I use method copyBytes of class org.apache.hadoop.io.IOUtils.
But the copyBytes request the InputStream of content. Therefore, I have
to convert the String to InputStre
Why not read the output result after job done? And, if you wanted see
the log4j log, you need to set the stdout option to log4jproperties.
On Wed, Apr 29, 2009 at 4:35 AM, Razen Alharbi wrote:
>
> Thanks for the reply,
>
> -Steve:
> I know that I can use the JobClient to run or submit jobs; howev
It may be simpler to just have a post processing step that uses something
like multi-file input to aggregate the results.
As a complete sideways thinking solution, I suspect you have far more map
tasks than you have physical machines,
instead of writing your output via output.collect, your tasks c
There has to be a simpler way :)
On Tue, Apr 28, 2009 at 9:22 PM, jason hadoop wrote:
> It may be simpler to just have a post processing step that uses something
> like multi-file input to aggregate the results.
>
> As a complete sideways thinking solution, I suspect you have far more map
> task
How about new InputStreamReader( new StringReader( String ), "UTF-8" )
replace UTF-8 with an appropriate charset.
On Tue, Apr 28, 2009 at 7:47 PM, nguyenhuynh.mr wrote:
> Hi all!
>
>
> I have the large String and I want to write it into the file in HDFS.
>
> (The large string has >100.000 lines.
Hello,
Thank you to Cloudera for providing an AMI for Hadoop. Would it be
possible to include R in the Cloudera yum repositories or better still
can the i386 and x86-64 AMIs be updated to have R pre-installed?
If yes (thanks!) I think yum -y install R installs R built with
--shared-libs (which all
Thanks for the interest in the Cloudera AMIs, Saptarshi. We're trying to
keep Cloudera-specific discussion out of the Apache forums to respect those
who may not want to follow along. For Cloudera-specific requests, please use
our Get Satisfaction forum at http://getsatisfaction.com/cloudera.
Thank
Where did you get the large string? Can't you generate the string one
line per time and append it to local files, then upload to HDFS when
finished?
On Wed, Apr 29, 2009 at 10:47 AM, nguyenhuynh.mr
wrote:
> Hi all!
>
>
> I have the large String and I want to write it into the file in HDFS.
>
> (T
Adam Retter wrote:
>
> So I don't have to use HDFS at all when using Hadoop?
The input URI list has to be stored in HDFS. Each mapper will work on
a sublist of URIs depending on the no of maps set in job.
- Sharad
Wang Zhong wrote:
> Where did you get the large string? Can't you generate the string one
> line per time and append it to local files, then upload to HDFS when
> finished?
>
> On Wed, Apr 29, 2009 at 10:47 AM, nguyenhuynh.mr
> wrote:
>
>> Hi all!
>>
>>
>> I have the large String and I want to
jason hadoop wrote:
> How about new InputStreamReader( new StringReader( String ), "UTF-8" )
> replace UTF-8 with an appropriate charset.
>
>
> On Tue, Apr 28, 2009 at 7:47 PM, nguyenhuynh.mr
> wrote:
>
>
>> Hi all!
>>
>>
>> I have the large String and I want to write it into the file in HDFS.
51 matches
Mail list logo