Trying to track down exactly what's happening.
Right now I'm getting this (see below).
The setup documentation for 2.4 could definitely be better. Probably with
a sample/working config. Looks like too much of this is left up as an
exercise to the user.
2014-05-23 21:20:30,652 INFO
What's the best way to debug yarn container issues?
I was going to try to tweak the script but it gets deleted after the job
fails.
Looks like I'm having an issue with the classpath.. I'm getting a basic
hadoop NCDFE on startup so I think it just has a broken class path.
but of course I need to
This just bit me… spent half a day figuring it out! :-(
The only way I was able to debug it was with
./bin/container-executor --checksetup
Once that stopped complaining my jobs were working ok.
this shouldn't have taken that much time… initial setup documentation could
be seriously improved.
I have relaxed it even further so now it is 775
kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /
Found 1 items
drwxrwxr-x - hdfs supergroup 0 2013-04-29 15:43 /
But I still get this error:
2013-04-30 07:43:02,520 FATAL
the
permission to 775 so that the group would also have write permission but
that didn't seem to help.
From: Mohammad Tariq [mailto:donta...@gmail.com]
Sent: Tuesday, April 30, 2013 8:20 AM
To: Kevin Burton
Subject: Re: Permission problem
user?ls shows hdfs and the log says mapred..
Warm
for hadoop hdfs and mr. Ideas?
From: Kevin Burton [mailto:rkevinbur...@charter.net]
Sent: Tuesday, April 30, 2013 8:31 AM
To: user@hadoop.apache.org
Cc: 'Mohammad Tariq'
Subject: RE: Permission problem
That is what I perceive as the problem. The hdfs file system was created
with the user 'hdfs
AM, Kevin Burton rkevinbur...@charter.net
wrote:
To further complicate the issue the log file in
(/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
owned by mapred:mapred and the name of the file seems to indicate some other
lineage (hadoop,hadoop). I am out of my
namehadoop.tmp.dir/name
value/data/hadoop/tmp/hadoop-${user.name}/value
descriptionHadoop temporary folder/description
/property
From: Arpit Gupta [mailto:ar...@hortonworks.com]
Sent: Tuesday, April 30, 2013 9:48 AM
To: Kevin Burton
Cc: user@hadoop.apache.org
Subject: Re: Permission
or set mapred.system.dir to
/tmp/mapred/system in your mapred-site.xml.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Apr 30, 2013, at 7:55 AM, Kevin Burton rkevinbur...@charter.net
wrote:
In core-site.xml I have:
property
namefs.default.name/name
value hdfs
[mailto:ar...@hortonworks.com]
Sent: Tuesday, April 30, 2013 10:48 AM
To: Kevin Burton
Cc: user@hadoop.apache.org
Subject: Re: Permission problem
It looks like hadoop.tmp.dir is being used both for local and hdfs
directories. Can you create a jira for this?
What i recommended is that you create
I have a simple MapReduce job that I am trying to get to run on my cluster.
When I run it I get:
13/04/30 11:27:45 INFO mapreduce.Cluster: Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
mapreduce.jobtracker.address configuration value for
To be clear when this code is run with 'java -jar' it runs without
exception. The exception occurs when I run with 'hadoop jar'.
From: Kevin Burton [mailto:rkevinbur...@charter.net]
Sent: Tuesday, April 30, 2013 11:36 AM
To: user@hadoop.apache.org
Subject: Can't initialize cluster
I have
HADOOP_MAPRED_HOME in your hadoop-env.sh file and re-run the job. See
if it helps.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Tue, Apr 30, 2013 at 10:10 PM, Kevin Burton rkevinbur...@charter.net
wrote:
To be clear when this code is run with 'java -jar' it runs
- Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
/blokquote
On Mon, Apr 29, 2013 at 10:21 AM, Kevin Burton rkevinbur...@charter.net
wrote:
I looked at the link you provided and found the Ubuntu is one of the
“supported platforms
Thank you the HDFS system seems to be up. Now I am having a problem with
getting the JobTracker and TaskTracker up. According to the logs on the
JobTracker mapred doesn't have write permission to /. I am not clear on what
the permissions should be.
Anyway, thank you.
On Apr 29, 2013, at 4:30
It is '/'?
On Apr 29, 2013, at 5:09 PM, Mohammad Tariq donta...@gmail.com wrote:
make it 755.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Tue, Apr 30, 2013 at 3:30 AM, Kevin Burton rkevinbur...@charter.net
wrote:
Thank you the HDFS system seems to be up
and job tracker.
Regards,
Sudhakara.st
On Sat, Apr 27, 2013 at 2:52 AM, Kevin Burton rkevinbur...@charter.net
wrote:
It is hdfs://devubuntu05:9000. Is this wrong? Devubuntu05 is the name of
the host where the NameNode and JobTracker should be running. It is also
the host where I am
?
Thanks again.
Kevin
From: Ted Xu [mailto:t...@gopivotal.com]
Sent: Friday, April 26, 2013 10:49 PM
To: user@hadoop.apache.org
Subject: Re: Warnings?
Hi Kevin,
Please see my comments inline,
On Sat, Apr 27, 2013 at 11:24 AM, Kevin Burton rkevinbur...@charter.net
wrote
It is hdfs://devubuntu05:9000. Is this wrong? Devubuntu05 is the name of the
host where the NameNode and JobTracker should be running. It is also the host
where I am running the M/R client code.
On Apr 26, 2013, at 4:06 PM, Rishi Yadav ri...@infoobjects.com wrote:
check core-site.xml and see
Answers below.
From: Omkar Joshi [mailto:ojo...@hortonworks.com]
Sent: Friday, April 26, 2013 7:15 PM
To: user@hadoop.apache.org
Subject: Re: M/R Staticstics
Have you enabled security?
No
can you share the output for your hdfs?
bin/hadoop fs -ls /
kevin@devUbuntu05:~$ hadoop
Is the native library not available for Ubuntu? If so how do I load it?
Can I tell which key is off? Since I am just starting I would want to be as up
to date as possible. It is out of date probably because I copied my examples
from books and tutorials.
The main class does derive from Tool.
I notice that in some beginning texts on starting a Hadoop MapReduce job
sometimes JobClient/JobConf is used and sometimes Job/Configuration is used.
I have yet to see anyone comment on the features/benefits of either set of
methods. Could someone comment on their preferred method for starting a
I execute the line:
sqoop import --connect
'jdbc:sqlserver://nbreports:1433;databaseName=productcatalog' --username
USER --password PASSWORD --table CatalogProducts
And I get the following output:
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME
Thanks for sharing. I'd love to play with it, do you have a
README/user-guide for systat?
Not a ton but I could write some up...
Basically I modeled it after vmstat/iostat on Linux.
http://sebastien.godard.pagesperso-orange.fr/documentation.html
The theory is that most platforms have
One key point I wanted to mention for Hadoop developers (but then check out
the announcement).
I implemented a version of sysstat (iostat, vmstat, etc) in Peregrine and
would be more than happy to move it out and put it in another dedicated
project.
, Kevin Burton burtona...@gmail.comwrote:
We've discussed 'push' v/s 'pull' shuffle multiple times and each time
turned away due to complexities in MR1. With MRv2 (YARN) this would be much
more doable.
Ah gotcha. This is what I expected as well. It would be interesting
to see a list
The current hadoop implementation shuffles directly to disk and then those
disk files are eventually requested by the target nodes which are
responsible for doing the reduce() on the intermediate data.
However, this requires more 2x IO than strictly necessary.
If the data were instead shuffled
On Tue, Dec 20, 2011 at 4:53 PM, Todd Lipcon t...@cloudera.com wrote:
The advantages of the pull based shuffle is fault tolerance - if you
shuffle to the reducer and then the reducer dies, you have to rerun
*all* of the earlier maps in the push model.
you would have the same situation if you
We've discussed 'push' v/s 'pull' shuffle multiple times and each time
turned away due to complexities in MR1. With MRv2 (YARN) this would be much
more doable.
Ah gotcha. This is what I expected as well. It would be interesting to
see a list of changes like this in MR1 vs MR2 to see what
Is it possible to connect the output of one map reduce job so that it is the
input to another map reduce job.
Basically… then reduce() outputs a key, that will be passed to another map()
function without having to store intermediate data to the filesystem.
Kevin
--
Founder/CEO Spinn3r.com
14, 2011 at 9:38 AM, Kevin Burton bur...@spinn3r.com wrote:
You can already do this with the JAR file format… if you load a resource
via path it uses the class loader system to find it in all available jars.
Kevin
On Wed, Sep 14, 2011 at 9:24 AM, Steve Lewis lordjoe2...@gmail.comwrote:
When
OK so it sounds from the group that there are still reasons to provide
rope in ZK to enable algorithms like leader election.
Couldn't ZK ship higher level interfaces for leader election, mutexes,
semapores, queues, barriers, etc instead of pushing this on developers?
Then the remaining APIs,
:) . We havent had the bandwidth to
provide such interfaces for zookeeper. It would be great to have all such
recipes as a part of contrib package of zookeeper.
mahadev
On 1/9/09 11:44 AM, Kevin Burton bur...@spinn3r.com wrote:
OK so it sounds from the group that there are still reasons
On Wed, Jan 7, 2009 at 9:25 AM, Benjamin Reed br...@yahoo-inc.com wrote:
This is the behavior we had when we first implemented the API, and in every
case where people used the information there was a bug. it is virtually
impossible to use correctly. In general I'm all for giving people rope,
:
The version of Jute we use is really an ancient version of recordio
ser/deser library in hadoop. We do want to move to some
better(versioned/fast/well accepted) ser/deser library.
mahadev
On 1/7/09 12:08 PM, Kevin Burton bur...@spinn3r.com wrote:
Ah... you think it was because it was empty
I have an event watching a file... and if I restart the server I get this:
onConnect
onData path: /foo, version: 4, data: '2333'
onDisconnect
onConnect
onData path: /foo, version: 4, data: '2333'
It re-issues the same version of the file. I can of course watch for this in
my code but it seems
are stored locally on a particlar an server, etc)
On Jan 5, 2009, at 12:03 AM, Kevin Burton bur...@spinn3r.com wrote:
I'm not observing this behavior... if I shutdown the zookeeper server my
client doesn't reconnect and I get a disconnect event followed by
eventual
session expiration.
Which
Shutting down my zookeeper server yields this on my client.
Continual disconnect events.
Shouldn't only one be issued? The second one is not a state change.
WatchedEvent: Server state change. New state: Disconnected
WatchedEvent: Server state change. New state: Disconnected
WatchedEvent:
Because watches are one time triggers and there is latency between getting
the
event and sending a new request to get a watch you cannot reliably see
every
change that happens to a node in ZooKeeper. Be prepared to handle the case
where
the znode changes multiple times between getting the
39 matches
Mail list logo
