Hi,
I don't have the "dfs.hosts.exclude" property defined, but I still get
the error "There are 2 datanode(s) running and 2 node(s) are excluded in
this operation." when I run the distcp command. Any help?
296868864 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013
Any suggestion to fix this problem?
On 08/28/2013 12:09 PM, Jitendra Yadav wrote:
Hi,
Also can you please share the dfs heath check report of your cluster?
Thanks
On Wed, Aug 28, 2013 a
296868864 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013
Any suggestion to fix this problem?
On 08/28/2013 12:09 PM, Jitendra Yadav wrote:
Hi,
Also can you please share the dfs heath check report of your cluster?
Thanks
On Wed, Aug 28, 2013 a
Hi,
I am having some difficulty in copy data between 2 HDFS filesystems in
Amazon EC2.I want to try to use distcp2 command to see if I can.
- Where is the distcp2 command in yarn?
- Is it possible to copy data between HDFS using SSL?
- Has anyone copied data between HDFS filesystems in 2 di
is it possible to set the number of reduce tasks in the wordcount
example when I launch the job by command line?
Thanks
In yarn 2.0.5, where I set this?
On 09/25/2013 11:16 PM, Shahab Yunus wrote:
Have you tried setting *mapred.reduce.tasks *property?
Regards,
Shahab
On Wed, Sep 25, 2013 at 6:01 PM, xeon <mailto:xeonmailingl...@gmail.com>> wrote:
is it possible to set the number of reduce tas
When I execute the wordcount example I get this error:
2013-09-29 01:20:47,557 WARN [main] org.apache.hadoop.mapred.Task: Failure asking whether task can
commit: java.io.IOException: Failed on local exception: java.io.IOException: Response is null.;
Host Details : local host is:
"pc18.hadoopm
Hi,
I want execute the wordcount in yarn with compression enabled with a dir
with several files, but for that I must compress the input.
dir1/file1.txt
dir1/file2.txt
dir1/file3.txt
dir1/file4.txt
dir1/file5.txt
1 - Should I compress the whole dir or each file in the dir?
2 - Should I use g
ient.java:1199)
Caused by: java.io.FileNotFoundException: File does not exist:
/user/xeon/job_1383556901244_0002
--
Thanks,
I am trying to run an example, but I get a PriviledgedActionException.
How should I configure the hdfs-site.xml?
2013-12-16 13:43:22,892 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:ec2-user (auth:SIMPLE)
cause:java.io.IOException: File /tmp/l
ogs/ec2-
Is it possible to access the YARN webpages in a text browser?
--
Thanks,
Hi,
I would like to know if the MRv2 provide the following commands through
the bash command line:
- get the number of jobs running?
- get the percentage of job completion of jobs?
- get the number of jobs that are waiting to be submitted?
--
Thanks,
Hi,
Is it possible that jobs submitted stay waiting before starting to run?
Is there a command that list the jobs that are submitted and are waiting
to start to run?
--
Thanks,
Hi,
I want to set the number of map tasks in the Wordcount example. Is is
possible to set this variable in MRv2?
Thanks,
Hi,
1 - I installed Hadoop MRv2 in VirtualMachines. When the jobs are
running, I try to list them with "hadoop jobs -list", but it takes lots
of time for the command being executed. This happens because of the
performance of the VM. I just wonder how it works with big machines.
Does anyone have an
Hi,
I want to copy data between hdfs in different hosts using webhdfs in
hadoop MRv2. The destination host has the user and group as root and
supergroup.
WorkGroup0010:~/# hdfs dfs -ls /
14/04/12 16:42:18 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... u
nd restart mapreduce & hdfs daemons
dfs.permissions
false
--
Thanks,
Sergey
On 12/04/14 20:52, xeon wrote:
Hi,
I want to copy data between hdfs in different hosts using webhdfs in
hadoop MRv2. The destination host has the user and group as root and
supergroup.
WorkGroup0010:~/
Hi,
I am trying to copy data between HDFS that are located away from each
other, and when I run the distcp command, I get the errors below in the
namenode and the datanode of the target hosts. What is happening?
The 2 mapreduce runtime are running in a VLAN. The host are physically
distant,
13/2014 11:08 AM, xeon wrote:
Hi,
I am trying to copy data between HDFS that are located away from each
other, and when I run the distcp command, I get the errors below in
the namenode and the datanode of the target hosts. What is happening?
The 2 mapreduce runtime are running in a VLAN. Th
site. Any suggestion to fix this error?
WorkGroup$ hdfs dfs -copyFromLocal setup.py
'hdfs://WorkGroup0010:9000/' <<- this command works
WorkGroup$ hadoop distcp hdfs://WorkGroup:9000/wiki
hdfs://WorkGroup0010:9000/wiki <<- this command doesn'twork
On 04/13
Is it possible to access the YARN webpages in a text browser?
Hi,
Is it possible that jobs submitted stay waiting before starting to run?
Is there a command that list the jobs that are submitted and are waiting to
start to run?
--
Thanks,
I am running an wordcount example it MRv2, but I get this error in a
Datanode. It looks that it is a problem in the network between the Namenode
and the Datanode, but I am not sure.
What is this error? How can I fix this problem?
2014-01-03 16:46:29,319 INFO
org.apache.hadoop.hdfs.server.datanode
When I try to launch the namenode and the datanode in MRv2, the datanode
can't connect to the namenode, giving me the error below. I also put the
core-site file that I use below.
The Firewall in the hosts is disabled. I don't have excluded nodes defined.
Why the datanodes can't connect to the name
I am trying to launch the datanodes in Hadoop MRv2, and I get the error
below. I looked to Hadoop conf files and the /etc/hosts and everything
looks ok. What is wrong in my configuration?
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException:
Datanode denied communication with nameno
I am trying to run an example and I get the following error:
HadoopMaster-nh:~# /root/Programs/hadoop/bin/hdfs dfs -count /wiki
OpenJDK 64-Bit Server VM warning: You have loaded library
/root/Programs/hadoop-2.0.5-alpha/lib/native/libhadoop.so.1.0.0 which might
have disabled stack guard. The VM wi
I am trying to set Hadoop MapReduce (MRv2) behind the NAT, but when I try
to connect the Datanode, I get the error below.
The hosts have 2 interfaces, one with a private address and another with
the NAT address. To access the host with SSH, I must use an external IP,
that NAT server will redirect.
Hi
I want to have a job that copies the map output, or the reduce output to
another hdfs. Is is possible?
E.g., the job runs in cluster 1 and takes the input from this cluster.
Then, before the job finishes, it copies the map output or the reduce
output to the hdfs in the cluster 2.
Thanks,
Hi,
I don't understand this part of your answer: "read the other as a
side-input directly by creating a client.".
If I consider both inputs through the InputFormat, this means that a job
will contain both input path in its configuration, and this is enough to
work. So, what is the "other"? Is is
1. I am using these 2 commands below to try to copy data from local disk to
HDFS. Unfortunately these commands are not working, and I don't understand
why they are not working. I have configured HDFS to use WebHDFS
protocol. How I copy data from the local disk to HDFS using WebHDfS
protocol?
xubun
Hi,
This is a snippet of a Gridmix code available in Hadop MapReduce V1, but I
have the following question.
They set `org.apache.hadoop.mapred.SequenceFileInputFormat` and
`org.apache.hadoop.mapred.SequenceFileOutputFormat` as the `inFormat` and
`outFormat` respectively, and it also has `org.apac
With MapReduce v2 (Yarn), the output data that comes out from a map or a
reduce task is saved in the local disk or the HDFS when all the tasks
finish.
Since tasks end at different times, I was expecting that the data were
written as a task finish. For example, task 0 finish and so the output is
wr
I am looking for a way to pause a chained job or a chained task. I want to
do this because I want to validate the output of each map or reduce phase,
or between each job execution. Is it possible to pause the execution of
chained jobs or chained mappers or reducers in MapReduce V2. I was looking
fo
Hi,
I have created a map method that reads the map output of the wordcount
example [1]. This example is away from using the IdentityMapper.class that
MapReduce offers, but this is the only way that I have found to make a
working IdentityMapper for the Wordcount. The only problem is that this
Mappe
I know that it is not possible to suspend and resume mapreduce job, but I
really need to find a workaround. I have looked to the ChainedJobs and to
the CapacityScheduler, but I am really clueless on what to do.
The main goal was to suspend a job when the map tasks finish and the reduce
tasks start
I am trying to implement a mechanism in MapReduce v2 that allows to suspend
and resume a job. I must suspend a job when all the mappers finish, and
resume the job from that point after some time. I do this, because I want
to verify the integrity of the map output before executing the reducers.
I a
I am using Mapreduce v2.
On Aug 25, 2016 8:18 PM, "xeon Mailinglist"
wrote:
> I am trying to implement a mechanism in MapReduce v2 that allows to
> suspend and resume a job. I must suspend a job when all the mappers finish,
> and resume the job from that point after s
thing you can try is to write a map-only job first and then verify the
map out.
On Thu, Aug 25, 2016 at 1:18 PM, xeon Mailinglist wrote:
> I am using Mapreduce v2.
>
> On Aug 25, 2016 8:18 PM, "xeon Mailinglist"
> wrote:
>
> > I am trying to implement a mechanism in
7;s only partially
> dumped to disk. None of the intermediate data goes into HDFS.
>
> Daniel
>
> On Aug 25, 2016 4:10 PM, "xeon Mailinglist"
> wrote:
>
>> But then I need to set identity maps to run the reducers. If I suspend a
>> job after the maps finish,
39 matches
Mail list logo
