I've fixed the problem.
The reason namenode won't start was that I accidentally started the cluster
with root account.
This somehow changed the ownership of some hadoop-related files(ex: log
files, and hadoop.tmp.dir/dfs/name/current/edits) from hadoop:hadoop to
root:root.
After I fixed the
I run a 512 node hadoop cluster. Yesterday I moved 30Gb of compressed data from
a NFS mounted partition by running on the namenode
hadoop fs -copyFromLocal /mnt/data/data1 /mnt/data/data2 mnt/data/data3
hdfs:/data
When the job completed the local disk on the namenode was 40% full ( Most of
Hi all,
to create a RecordReader in new API, we needs a TaskAttemptContext object,
which
seems to me the RecordReader should only be created on each split that has been
assigned a task ID. However, I want to do a centralized sampling and create
record reader on some splits before the job is
when you copy files and have a local datanode - first copy will end up
there.
Just stop datanode at the node from which you copy files, and they will end
up on random nodes.
Also don't run datanode at the same machine as namenode.
-Original Message-
From: Raj V [mailto:rajv...@yahoo.com]
I am a complete newbie to hadoop. I'm running a job on 19 Amazon Elastic
MapReduce servers and am trying to understand two separate issues.
1) The job is ending with an error ERROR
org.apache.pig.tools.grunt.GruntParser - ERROR 6015: During execution,
encountered a Hadoop error. I do not
Hi,
EMR has a live debug option in the panel, you will find the logs there as well.
Ankit
-Original Message-
From: grabbler [mailto:twiza...@gmail.com]
Sent: Thursday, August 05, 2010 2:37 PM
To: core-u...@hadoop.apache.org
Subject: Problems running hadoop on Amazon Elastic MapReduce
We are looking to enable LZO compression of the map outputs on our
Cloudera 0.20.1 cluster. It seems there are various sets of
instructions available and I am curious what your thoughts are
regarding which one would be best for our Hadoop distribution and OS
(Ubuntu 8.04 64-bit). In particular,
Please take questions on Cloudera Distro to their internal lists.
On Aug 5, 2010, at 3:52 PM, Bobby Dennett wrote:
We are looking to enable LZO compression of the map outputs on our
Cloudera 0.20.1 cluster. It seems there are various sets of
instructions available and I am curious what your
Bobby,
We're working hard to make compression easier, the biggest hurdle
currently is the licensing issues around the LZO codec libs (GPL,
which is not compatible with ASF bsd-style license).
Outside of making the changes to the mapred-site.xml file, with your
setup would do you view as the
Hi Josh,
No real pain points... just trying to investigate/research the best
way to create the necessary libraries and jar files to support LZO
compression in Hadoop. In particular, there are the 2 repositories
to build from and I am trying to find out if one should be used over
the other. For
On Thu, Aug 5, 2010 at 4:52 PM, Bobby Dennett bdenn...@gmail.com wrote:
Hi Josh,
No real pain points... just trying to investigate/research the best
way to create the necessary libraries and jar files to support LZO
compression in Hadoop. In particular, there are the 2 repositories
to build
As part of our experimentation, the plan is to pull 4 slave nodes out of a
8-slave/1-master cluster. With replication factor set to 3, I thought
losing half of the cluster may be too much for hdfs to recover. Thus I
copied out all relevant data from hdfs to local disk and reconfigure the
12 matches
Mail list logo
