Re: Kerboes--Checksum failed----HDFS IN HA mode

2013-05-08 Thread Subroto
Hi Brahma, Not very sure about the problem but, the following link can show some light: http://stackoverflow.com/questions/8509087/checksum-failed-kerberos-spring-active-directory-2008/13859217#13859217 Cheers, Subroto Sanyal On May 8, 2013, at 6:43 AM, Brahma Reddy Battula wrote: Caused by:

Re: get recent changed files in hadoop

2013-05-08 Thread shashwat shriparv
hadoop dfs -ls|grep date you want to see you can use sort on that what ever you like .. *Thanks Regards* ∞ Shashwat Shriparv On Wed, May 8, 2013 at 6:19 AM, Winston Lin winston...@gmail.com wrote: Any idea to get recent changed file in hadoop? e.g. files created yesterday? fs -ls

Rack Aware Hadoop cluster

2013-05-08 Thread Mohammad Mustaqeem
Hello everyone, I was searching for how to make the hadoop cluster rack-aware and I find out from here http://hadoop.apache.org/docs/r2.0.4-alpha/hadoop-project-dist/hadoop-common/ClusterSetup.html#Hadoop_Rack_Awareness that we can do this by giving property of topology.script.file.name. But

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Leonid Fedotov
You can put this parameter to core-site.xml or hdfs-site.xml It both parsed during the HDFS startup. Leonid On Wed, May 8, 2013 at 6:43 AM, Mohammad Mustaqeem 3m.mustaq...@gmail.comwrote: Hello everyone, I was searching for how to make the hadoop cluster rack-aware and I find out from

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Shahab Yunus
core-site.xml http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml On Wed, May 8, 2013 at 9:43 AM, Mohammad Mustaqeem 3m.mustaq...@gmail.comwrote: Hello everyone, I was searching for how to make the hadoop cluster rack-aware and I find out from here

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Chris Embree
Finally, one I can answer. :) That should be in core-site.xml (unless it's moved from ver 1.x). It needs to be in the configuration for NameNode(s) and JobTracker (Yarn). In 1.x you need to restart NN and JT services for the script to take effect. On Wed, May 8, 2013 at 9:43 AM, Mohammad

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Mohammad Mustaqeem
I have done this and found following error in log - 2013-05-08 18:53:45,221 WARN org.apache.hadoop.net.ScriptBasedMapping: Exception running /home/mustaqeem/development/hadoop-2.0.3-alpha/etc/hadoop/rack.sh 127.0.0.1 org.apache.hadoop.util.Shell$ExitCodeException:

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Chris Embree
Your script has an error in it. Please test your script using both IP Addresses and Names, outside of hadoop. On Wed, May 8, 2013 at 10:01 AM, Mohammad Mustaqeem 3m.mustaq...@gmail.comwrote: I have done this and found following error in log - 2013-05-08 18:53:45,221 WARN

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Leonid Fedotov
Error in script. On Wed, May 8, 2013 at 7:11 AM, Chris Embree cemb...@gmail.com wrote: Your script has an error in it. Please test your script using both IP Addresses and Names, outside of hadoop. On Wed, May 8, 2013 at 10:01 AM, Mohammad Mustaqeem 3m.mustaq...@gmail.com wrote: I have

MapReduce - FileInputFormat and Locality

2013-05-08 Thread Brian C. Huffman
All, I'm trying to understand how the current FileInputFormat implements locality. As far as I can tell, it calculates splits using getSplit and each split will contain the node that hosts the first block of data in that split. Is my understanding correct? Looking at the FileInputFormat

SetIntWritable on reducer issue

2013-05-08 Thread Thamizhannal Paramasivam
Hi, I require clarification on using IntWritable in Set type in Java. Below is my sample code. When I execute reducer with E.x. key=abcd values=12,3,4, then categorySet have any one value from values. My hadoop version is 0.20.2. Any help would be appreciated. public void reduce(Text

Re: SetIntWritable on reducer issue

2013-05-08 Thread Paul Wilkinson
See http://cornercases.wordpress.com/2011/08/18/hadoop-object-reuse-pitfall-all-my-reducer-values-are-the-same/for an explanation of what's going on. Paul On 8 May 2013 16:10, Thamizhannal Paramasivam thamizhanna...@gmail.comwrote: Hi, I require clarification on using IntWritable in Set type

Re: get recent changed files in hadoop

2013-05-08 Thread Chris Nauroth
It sounds like you're interested in something equivalent to the Unix find command for HDFS. HADOOP-8989 tracks adding this feature. There is a patch available, but it's still under review, so the feature isn't available in a release yet. https://issues.apache.org/jira/browse/HADOOP-8989 Chris

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Mohammad Mustaqeem
@chris, I have test it outside. It is working fine. On Wed, May 8, 2013 at 7:48 PM, Leonid Fedotov lfedo...@hortonworks.comwrote: Error in script. On Wed, May 8, 2013 at 7:11 AM, Chris Embree cemb...@gmail.com wrote: Your script has an error in it. Please test your script using both IP

Hadoop Maven Build Fails

2013-05-08 Thread Konstantin Weitz
I'm trying to build the latest svn version of hadoop (r1480394 | 2013-05-08 11:28:22) with maven, but I get many errors of the following kind (this is a small snippet of the output). It appears that something is wrong with protobuf, I have version 2.5.0-3 installed. Any hints on how I can fix

Re: Hadoop Maven Build Fails

2013-05-08 Thread Konstantin Weitz
Using 2.4.1 did the trick. Thank you very much! On Wed, May 8, 2013 at 12:01 PM, Ted Yu yuzhih...@gmail.com wrote: Please use protobuf 2.4.1 Cheers On Wed, May 8, 2013 at 11:59 AM, Konstantin Weitz weitz...@cs.washington.edu wrote: I'm trying to build the latest svn version of hadoop

Re: Rack Aware Hadoop cluster

2013-05-08 Thread Chris Embree
Here is a sample I stole from the web and modified slightly... I think. HADOOP_CONF=/etc/hadoop/conf while [ $# -gt 0 ] ; do nodeArg=$1 exec ${HADOOP_CONF}/rack_info.txt result= while read line ; do ar=( $line ) if [ ${ar[0]} = $nodeArg ] ; then result=${ar[1]} fi

Re: MapReduce - FileInputFormat and Locality

2013-05-08 Thread Vinod Kumar Vavilapalli
I think you misread it. If a given split has only one block, it uses all the locations of that block. If it so happens that a given split has multiple blocks, it uses all the locations of the first block. HTH, +Vinod Kumar Vavilapalli Hortonworks Inc. http://hortonworks.com/ On May 8, 2013,

Re: MapReduce - FileInputFormat and Locality

2013-05-08 Thread Ted Dunning
I think that you just said what the OP said. Your two cases reduce to the same single case that they had. Whether this matters is another question, but it seems like it could in cases where splits != blocks, especially if a split starts near the end of a block which could give an illusion of

Re: get recent changed files in hadoop

2013-05-08 Thread Winston Lin
Thanks so much for all your help. I think I would temporally use ls to get the list and then grep or awk to parse it. Winston On Wed, May 8, 2013 at 9:29 PM, shashwat shriparv dwivedishash...@gmail.com wrote: hadoop dfs -ls|grep date you want to see you can use sort on that what ever

Re: Benefits of Hadoop Distributed Cache

2013-05-08 Thread Harsh J
This has been discussed before, see http://search-hadoop.com/m/xI5AHMD0Vm1 for the previous discussion on this. On Wed, May 8, 2013 at 12:54 AM, Saeed Shahrivari saeed.shahriv...@gmail.com wrote: Would you please tell me why we should use Distributed Cache instead of HDFS? Because HDFS seems

Re: problem building lzo

2013-05-08 Thread Harsh J
What repo of the hadoop-lzo are you using? If you're using a typical Linux OS such as those similar to RedHat/Debian/etc., it is easier to run the hadoop-lzo-packager project mentioned at the same Wiki link. It will hand you ready to install and use packages. Onto your problem though, I think