Re: Run hadoop Map/Reduce app from another machine
Install hadoop on your local machine, copy the configuration files from the remote hadoop culuster server to your local machine(including the hosts file), then you can just submit a *.jar locally as before. 2011/10/5 oleksiy gayduk.a.s...@mail.ru Hello, I'm trying to find a way how to run hadoop MapReduce app from another machine. For instance I have *.jar file with MapReduce app it works ok when I run it from command line for instance using this command: hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output But in situation when I have another server (simple web app) where user can upload jar file, specify configuration for the MapReduce app and so on. And this server should interact with hadoop server. I mean somehow to upload this jar file to the hadoop server and run it with attributes. So, right now I see only one way of how to do this, it's upload jar file to the hadoop server and run remotely command: hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output. So may be hadoop has spatial API for doing this kind of tasks remotely? -- View this message in context: http://old.nabble.com/Run-hadoop-Map-Reduce-app-from-another-machine-tp32595264p32595264.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: hadoop input buffer size
Hi, Hadoop neither read one line each time, nor fetching dfs.block.size of lines into a buffer, Actually, for the TextInputFormat, it read io.file.buffer.size bytes of text into a buffer each time, this can be seen from the hadoop source file LineReader.java 2011/10/5 Mark question markq2...@gmail.com Hello, Correct me if I'm wrong, but when a program opens n-files at the same time to read from, and start reading from each file at a time 1 line at a time. Isn't hadoop actually fetching dfs.block.size of lines into a buffer? and not actually one line. If this is correct, I set up my dfs.block.size = 3MB and each line takes about 650 bytes only, then I would assume the performance for reading 1-4000 lines would be the same, but it isn't ! Do you know a way to find #n of lines to be read at once? Thank you, Mark
TestDFSIO error: libhdfs.so.1 does not exist
Hi all, I am benchmarking a Hadoop Cluster with the hadoop-*-test.jar TestDFSIO but the following error returns: File /usr/hadoop-0.20.2/libhdfs/libhdfs.so.1 does not exist. How to solve this problem? Thanks!
Re: Current available Memory
Thanks a lot! Yang Xiaoliang 2011/2/25 maha m...@umail.ucsb.edu Hi Yang, The problem could be solved using the following link: http://www.roseindia.net/java/java-get-example/get-memory-usage.shtml You need to use other memory managers like the Garbage collector and its finalize method to measure memory accurately. Good Luck, Maha On Feb 23, 2011, at 10:11 PM, Yang Xiaoliang wrote: I had also encuntered the smae problem a few days ago. any one has another method? 2011/2/24 maha m...@umail.ucsb.edu Based on the Java function documentation, it gives approximately the available memory, so I need to tweak it with other functions. So it's a Java issue not Hadoop. Thanks anyways, Maha On Feb 23, 2011, at 6:31 PM, maha wrote: Hello Everyone, I'm using Runtime.getRuntime().freeMemory() to see current memory available before and after creation of an object, but this doesn't seem to work well with Hadoop? Why? and is there another alternative? Thank you, Maha
Re: Current available Memory
I had also encuntered the smae problem a few days ago. any one has another method? 2011/2/24 maha m...@umail.ucsb.edu Based on the Java function documentation, it gives approximately the available memory, so I need to tweak it with other functions. So it's a Java issue not Hadoop. Thanks anyways, Maha On Feb 23, 2011, at 6:31 PM, maha wrote: Hello Everyone, I'm using Runtime.getRuntime().freeMemory() to see current memory available before and after creation of an object, but this doesn't seem to work well with Hadoop? Why? and is there another alternative? Thank you, Maha
