Thank you! I realized that, despite I exported the variables in the
scripts, there were a few errors and my desired configuration wasn't being
used (which explained other strange behavior).
However, I'm still getting the same issue with the examples, for instance:
hadoop jar
Hi Viswa,
Sorry for the late reply,
Have you restarted NodeManagers after copying the lzo jars to lib?
Thanks and Regards,
Vinayakumar B
From: Viswanathan J [mailto:jayamviswanat...@gmail.com]
Sent: 06 December 2013 23:32
To: user@hadoop.apache.org
Subject: Compression LZO class not found issue
How many nodes you have?
and if fsck is giving you healthy status no need to worry.
with the replication 10 what i may conclude that you have 10 listed
datanodes so 10 replicated jar files for the job to run.
*Thanks Regards*
∞
Shashwat Shriparv
On Tue, Dec 10, 2013 at 3:50 PM,
Set the class path to where hadoop*Lzo jar file is, and then try..
*Thanks Regards*
∞
Shashwat Shriparv
On Tue, Dec 10, 2013 at 4:00 PM, Vinayakumar B vinayakuma...@huawei.comwrote:
Hi Viswa,
Sorry for the late reply,
Have you restarted NodeManagers after copying the lzo jars
Hadoop default replication factor is 3 and you can configure it in
hdfs-default.xmlhttp://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml,
youu should have one Master (NameNode and 2 Slaves (Data Nodes)
and please follow
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar nirmal.ku...@impetus.co.inwrote:
Hi All,
hadoop dfsadmin -help
DEPRECATED: Use of this script to execute hdfs command is deprecated.
hadoop default replication factor is 3 configured in
hdfs-default.xmlhttp://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml(1
master (namenode) and 2 slaves - data nodes )
useful resource
have you looked at hadoop.security.group.mapping.ldap.* in
hadoop-common/core-default.xmlhttp://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/core-default.xml
additional
What is the recommended way to secure the AM app master web app (if any) in
2.x? The client can obtain the tracking URL from the app report using
appReport.getOriginalTrackingUrl() and bypass the RM proxy. In case of POST
request, which is not currently supported by the proxy, that is the only
10 copies for those job.jar and split are controlled by
mapred.submit.replication property at job init level.
On Mon, Dec 9, 2013 at 5:20 PM, ch huang justlo...@gmail.com wrote:
more strange , in my HDFS cluster ,every block has three replicas,but i
find some one has ten replicas ,why?
#
Please try
2013/12/10 ch huang justlo...@gmail.com
hi,maillist:
i try assign reduce number in commandline but seems not
useful,i run tera sort like this
# hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
terasort /alex/terasort/1G-input
Hi,
Do you have Hadoop libs properly installed? Does $ hadoop version command
run successfully? If true, then It sounds like some classpath issue...
2013/12/10 Manish Bhoge manishbh...@rocketmail.com
Sent from Rocket Mail via Android
--
* From: * Manish Bhoge
It sounds like the job was successfully submitted to the cluster, but there
as some problem when starting/running AM, so that no progress is made. It
happened to me once, when I was playing with YARN on a cluster consisting
of very small machines, and I mis-configured YARN to allocated to AM more
Please have a look at hadoop.security.group.mapping.ldap.* settings as Hardik
Pandya suggests.
=
In advance, just to share our story related to LDAP +
hadoop.security.group.mapping.ldap.*, if you run into the same limitation
as we did:
In many cases hadoop.security.group.mapping.ldap.*
yes,you are right ,thanks
On Wed, Dec 11, 2013 at 7:16 AM, Adam Kawa kawa.a...@gmail.com wrote:
Accidentally, I clicked Sent by a mistake. Plase try:
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
terasort *-Dmapreduce.job.reduces=34* /alex/terasort/1G-input
By default this higher replication level is 10.
is this value can be control via some option or variable? i only hive a
5-worknode cluster,and i think 5 replicas should be better,because every
node can get a local replica.
another question is ,why hdfs fsck check the cluster is healthy and no
Hi ch huang,
It may seem strange, but the fact is,
CorruptBlocks through JMX or http://NNIP:50070/jmxhttp://nnip:50070/jmx means
Number of blocks with corrupt replicas. May not be all replicas are corrupt.
This you can check through jconsole for description.
Where as Corrupt blocks through
thanks for reply, what i do not know is how can i locate the block which
has the corrupt replica,(so i can observe how long the corrupt replica will
be removed and a new health replica replace it,because i get nagios alert
for three days,i do not sure if it is the same corrupt replica cause the
Maybe this can work for you
$ sudo -u hdfs hdfs fsck / -list-corruptfileblocks
?
2013/12/11 ch huang justlo...@gmail.com
thanks for reply, what i do not know is how can i locate the block which
has the corrupt replica,(so i can observe how long the corrupt replica will
be removed and a new
When you identify a file with corrupt block(s), then you can locate the
machines that stores its block by typing
$ sudo -u hdfs hdfs fsck path-to-file -files -blocks -locations
2013/12/11 Adam Kawa kawa.a...@gmail.com
Maybe this can work for you
$ sudo -u hdfs hdfs fsck /
thanks for reply,but if the block just has 1 corrupt replica,hdfs fsck can
not tell you which block of which file has a replica been corrupted,fsck
just useful on all of one block's replica bad
On Wed, Dec 11, 2013 at 10:01 AM, Adam Kawa kawa.a...@gmail.com wrote:
When you identify a file with
So, not knowing much about LDAP, but being very interested in the multiuser
problem on multiuser filesystems, i was excited to see this question Im
researching the same thing at the moment, and it seems obviated by the fact
that :
- the FileSystem API itslef provides implementations for
Hi guys,
Does anyone knows how to ‘capture’ the exception which actually failed the
job running on Mapper or Reducer at runtime? It seems Hadoop is designed to
be fault tolerant that the failed jobs will be automatically rerun for a
certain amount of times and won’t actually expose the real
hi,maillist:
i try to corrupt a block of a file in my benchmark
environment, as the following command i find blk_2504407693800874616_106252
,it's replica on 192.168.10.224 is my target ,but i find all the datadir in
192.168.10.224 ,can not fine the datafile belongs to this replic
hi,maillist:
i run terasort with 16 reducers and 8 reducers,when i double
reducer number, the Shuffled maps is also double ,my question is the job
only run 20 map tasks (total input file is 10,and each file is 100M,my
block size is 64M,so split is 20) why i need shuffle 160 maps in 8
It looks simple, :)
Shuffled Maps= Number of Map Tasks * Number of Reducers
Thanks and Regards,
Vinayakumar B
From: ch huang [mailto:justlo...@gmail.com]
Sent: 11 December 2013 10:56
To: user@hadoop.apache.org
Subject: issue about Shuffled Maps in MR job summary
hi,maillist:
i run
i read the doc, and find if i have 8 reducer ,a map task will output 8
partition ,each partition will be send to a different reducer, so if i
increase reduce number ,the partition number increase ,but the volume on
network traffic is same,why sometime ,increase reducer number will not
decrease job
hi,
I have fixed the error , the code is running fine, but this code just split
the part of the tag.
i want to convert into text format so that i can load them into tables of
hbase and hive.
I have used the DOM Parser but this parser uses File as Object but hdfs
uses FileSystem.
Eg,
File
thanks, Ian, It works!
The problem is that I could APPEND a file, create a directory, but not
CREATE a file, or copy a file. when I try to create, the log at nfs gateway
says: ERROR nfs3.RpcProgramNfs3: Setting file size is not supported when
creating file: test dir fileId:16386
However, if I
Block files are not stored in a flat directory (to avoid FS limits of
max files under a dir). Instead of looking for them right under
finalized, issue a find query with the pattern instead and you
should be able to spot it.
On Wed, Dec 11, 2013 at 9:10 AM, ch huang justlo...@gmail.com wrote:
Hi,
You can check the userlogs directory where the job and attempt logs are
stored. For each attempt you should have a stderr, stdout and syslog file.
The first two hold the program output for each stream (useful for debug
purposes), while the last contains execution details provided by the
31 matches
Mail list logo
