Problems running TestDFSIO to a non-default directory
Hi Konstantin (et al.); A while ago you gave me the following trick to run TestDFSIO to an output directory other than the default- just use -Dtest.build.data=/output/dir to pass the new directory to the executable. I recall this working, but it is failing now under 0.18.1, and looking at it I can't see how it ever worked. The -D option will set the property on the Java virtual machine which runs as a direct child of /bin/hadoop, but I see no way the property would get set on the mapper virtual machines. Should this still work? Thanks, -Joel On Thu, 2008-09-04 at 13:05 -0700, Konstantin Shvachko wrote: Sure. bin/hadoop -Dtest.build.data=/bessemer/welling/hadoop_test/benchmarks/TestDFSIO/ org.apache.hadoop.fs.TestDFSIO -write -nrFiles 2*N -fileSize 360 --Konst Joel Welling wrote: With my setup, I need to change the file directory from /benchmarks/TestDFSIO/io_control to something like /bessemer/welling/hadoop_test/benchmarks/TestDFSIO/io_control . Is there a command line argument or parameter that will do this? Basically, I have to point it explicitly into my Lustre filesystem. -Joel
How to run sort900?
Hi folks; Is there a standard procedure for running the sort900 test? In particular, the timings show time for a verifier, but I can't find where that's implemented. Thanks, -Joel [EMAIL PROTECTED]
Re: reading input for a map function from 2 different files?
Amar, isn't there a problem with your method in that it gets a small result by subtracting very large numbers? Given a million inputs, won't A and B be so much larger than the standard deviation that there aren't enough no bits left in the floating point number to represent it? I just thought I should mention that, before this thread goes in an archive somewhere and some student looks it up. -Joel On Wed, 2008-11-12 at 12:32 +0530, Amar Kamat wrote: some speed wrote: Thanks for the response. What I am trying is to do is finding the average and then the standard deviation for a very large set (say a million) of numbers. The result would be used in further calculations. I have got the average from the first map-reduce chain. now i need to read this average as well as the set of numbers to calculate the standard deviation. so one file would have the input set and the other resultant file would have just the average. Please do tell me in case there is a better way of doing things than what i am doing. Any input/suggestion is appreciated.:) std_dev^2 = sum_i((Xi - Xa) ^ 2) / N; where Xa is the avg. Why dont you use the formula to compute it in one MR job. std_dev^2 = (sum_i(Xi ^ 2) - N * (Xa ^ 2) ) / N; = (A - N*(avg^2))/N For this your map would look like map (key, val) : output.collect(key^2, key); // imagine your input as (k,v) = (Xi, null) Reduce should simply sum over the keys to find out 'sum_i(Xi ^ 2)' and sum over the values to find out 'Xa'. You could use the close() api to finally dump there 2 values to a file. For example : input : 1,2,3,4 Say input is split in 2 groups [1,2] and [4,5] Now there will be 2 maps with output as follows map1 output : (1,1) (4,2) map2 output : (9,3) (16,4) Reducer will maintain the sum over all keys and all values A = sum(key i.e input squared) = 1+ 4 + 9 + 16 = 30 B = sum(values i.e input) = 1 + 2 + 3 + 4 = 10 With A and B you can compute the standard deviation offline. So avg = B / N = 10/4 = 2.5 Hence the std deviation would be sqrt( (A - N * avg^2) / N) = sqrt ((30 - 4*6.25)/4) = *1.11803399 *Using the main formula the answer is *1.11803399* Amar On Mon, Nov 10, 2008 at 4:22 AM, Amar Kamat [EMAIL PROTECTED] wrote: Amar Kamat wrote: some speed wrote: I was wondering if it was possible to read the input for a map function from 2 different files: 1st file --- user-input file from a particular location(path) Is the input/user file sorted? If yes then you can use map-side join for performance reasons. See org.apache.hadoop.mapred.join for more details. 2nd file=--- A resultant file (has just one key,value pair) from a previous MapReduce job. (I am implementing a chain MapReduce function) Can you explain in more detail the contents of 2nd file? Now, for every key,value pair in the user-input file, I would like to use the same key,value pair from the 2nd file for some calculations. Can you explain this in more detail? Can you give some abstracted example of how file1 and file2 look like and what operation/processing you want to do? I guess you might need to do some kind of join on the 2 files. Look at contrib/data_join for more details. Amar Is it possible for me to do so? Can someone guide me in the right direction please? Thanks!
Is there a unique ID associated with each task?
Hi folks; I'm writing a Hadoop Pipes application, and I need to generate a bunch of integers that are unique across all map tasks. If each map task has a unique integer ID, I can make sure my integers are unique by including that integer ID. I have this theory that each map task has a unique identifier associated with some configuration parameter, but I don't know the name of that parameter. Is there an integer associated with each task? If so, how do I get it? While we're at it, is there a way to get the total number of map tasks? Thanks, -Joel
Any examples using Hadoop Pipes with binary SequenceFiles?
Hi folks; I'm interested in reading binary data, running it through some C++ code, and writing the result as binary data. It looks like SequenceFiles and Pipes are the way to do it, but I can't find any examples or docs beyond the API specification. Can someone point me to an example where this is done? Thanks, -Joel
Re: Problems increasing number of tasks per node- really a task management problem!
I think I've found my problem. At some point about a week ago, I must have tried to start new tasktracker processes on my worker nodes without killing the ones that were already there. The new processes died immediately because their sockets were already in use. The old processes then took over their roles, running happily with new JobTrackers and doing tasks as requested. The pid files that are supposed to point to the tasktrackers did not contain their pids, however, and 'bin/stop-mapred.sh' chooses its targets from the pid files. So I could do 'bin/stop-mapred.sh' all day long without killing them. I ended up killing them explicitly one node at a time. These tasktrackers knew the *old* config values that were in force when they were started, so pushing the new values out to the worker nodes had no effect. So. Is there any mechanism for killing 'rogue' tasktrackers? I'm a little surprised that they are killed via their pids rather than by sending them a kill signal via the same mechanism whereby they learn of new work. -Joel [EMAIL PROTECTED] On Tue, 2008-09-23 at 14:29 -0700, Arun C Murthy wrote: On Sep 23, 2008, at 2:21 PM, Joel Welling wrote: Stopping and restarting the mapred service should push the new .xml file out, should it not? I've done 'bin/mapred-stop.sh', No, you need to run 'bin/mapred-stop.sh', push it out to all the machines and then do 'bin/mapred-start.sh'. You do see it in your job's config - but that config isn't used by the TaskTrackers. They use the config in their HADOOP_CONF_DIR; which is why you'd need to push it to all machines. Arun
gridmix on a small cluster?
Hi folks; I'd like to try the gridmix benchmark on my small cluster (3 nodes at 8 cores each, Lustre with IB interconnect). The documentation for gridmix suggests that it will take 4 hours on a 500 node cluster, which suggests it would take me something like a week to run. Is there a way to scale the problem size back? I don't mind the file size too much, but the running time would be excessive if things scale linearly with the number of nodes. Thanks, -Joel
Ordering of records in output files?
Hi folks; I have a simple Streaming job where the mapper produces output records beginning with a 16 character ascii string and passes them to IdentityReducer. When I run it, I get the same number of output files as I have mapred.reduce.tasks . Each one contains some of the strings, and within each file the strings are in sorted order. But there is no obvious ordering *across* the files. For example, I can see where the first few strings in the output went to files 0,1,3,4, and then back to 0, but none of them ended up in file 2. What's the algorithm that determines which strings end up in which files? Is there a way I can change it so that sequentially ordered strings end up in the same file rather than spraying off across all the files? Thanks, -Joel [EMAIL PROTECTED]
Re: Hadoop over Lustre?
Sorry; I'm picking this thread up after a couple day's delay. Setting fs.default.name to the equivalent of file:///path/to/lustre and changing mapred.job.tracker to just a hostname and port does allow mapreduce to start up. However, test jobs fail with the exceptions below. It looks like TaskTracker.localizeJob is looking for job.xml in the local filesystem; I would have expected it to look in lustre. I can't find that particular job.xml anywhere on the system after the run aborts, I'm afraid. I guess it's getting cleaned up. Thanks, -Joel 08/08/28 18:46:07 INFO mapred.FileInputFormat: Total input paths to process : 1508/08/28 18:46:07 INFO mapred.FileInputFormat: Total input paths to process : 1508/08/28 18:46:08 INFO mapred.JobClient: Running job: job_200808281828_0002 08/08/28 18:46:09 INFO mapred.JobClient: map 0% reduce 0% 08/08/28 18:46:12 INFO mapred.JobClient: Task Id : attempt_200808281828_0002_m_00_0, Status : FAILED Error initializing attempt_200808281828_0002_m_00_0: java.io.IOException: file:/tmp/hadoop-welling/mapred/system/job_200808281828_0002/job.xml: No such file or directory at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:216) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:150) at org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:55) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1193) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:668) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1306) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:946) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1343) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2354) 08/08/28 18:46:12 WARN mapred.JobClient: Error reading task outputhttp://foo.psc.edu:50060/tasklog?plaintext=truetaskid=attempt_200808281828_0002_m_00_0filter=stdout 08/08/28 18:46:12 WARN mapred.JobClient: Error reading task outputhttp://foo.psc.edu:50060/tasklog?plaintext=truetaskid=attempt_200808281828_0002_m_00_0filter=stderr On Mon, 2008-08-25 at 14:24 -0700, Konstantin Shvachko wrote: mapred.job.tracker is the address and port of the JobTracker - the main server that controls map-reduce jobs. Every task tracker needs to know the address in order to connect. Do you follow the docs, e.g. that one http://wiki.apache.org/hadoop/GettingStartedWithHadoop Can you start one node cluster? Are there standard tests of hadoop performance? There is the sort benchmark. We also run DFSIO benchmark for read and write throughputs. --Konstantin Joel Welling wrote: So far no success, Konstantin- the hadoop job seems to start up, but fails immediately leaving no logs. What is the appropriate setting for mapred.job.tracker ? The generic value references hdfs, but it also has a port number- I'm not sure what that means. My cluster is small, but if I get this working I'd be very happy to run some benchmarks. Are there standard tests of hadoop performance? -Joel [EMAIL PROTECTED] On Fri, 2008-08-22 at 15:59 -0700, Konstantin Shvachko wrote: I think the solution should be easier than Arun and Steve advise. Lustre is already mounted as a local directory on each cluster machines, right? Say, it is mounted on /mnt/lustre. Then you configure hadoop-site.xml and set property namefs.default.name/name valuefile:///mnt/lustre/value /property And then you start map-reduce only without hdfs using start-mapred.sh By this you basically redirect all FileSystem requests to Lustre and you don't need data-nodes or the name-node. Please let me know if that works. Also it would very interesting to have your experience shared on this list. Problems, performance - everything is quite interesting. Cheers, --Konstantin Joel Welling wrote: 2. Could you set up symlinks from the local filesystem, so point every node at a local dir /tmp/hadoop with each node pointing to a different subdir in the big filesystem? Yes, I could do that! Do I need to do it for the log directories as well, or can they be shared? -Joel On Fri, 2008-08-22 at 15:48 +0100, Steve Loughran wrote: Joel Welling wrote: Thanks, Steve and Arun. I'll definitely try to write something based on the KFS interface. I think that for our applications putting the mapper on the right rack is not going to be that useful. A lot of our calculations are going to be disordered stuff based on 3D spatial relationships like nearest-neighbor finding, so things will be in a random access pattern most of the time. Is there a way to set up the configuration for HDFS so that different datanodes keep their data in different directories? That would be a big help in the short term. yes, but you'd have to push out a different config to each datanode
Re: Hadoop over Lustre?
That seems to have done the trick! I am now running Hadoop 0.18
straight out of Lustre, without an intervening HDFS. The unusual things
about my hadoop-site.xml are:
property
namefs.default.name/name
valuefile:///bessemer/welling/value
/property
property
namemapred.system.dir/name
value${fs.default.name}/hadoop_tmp/mapred/system/value
descriptionThe shared directory where MapReduce stores control
files.
/description
/property
where /bessemer/welling is a directory on a mounted Lustre filesystem.
I then do 'bin/start-mapred.sh' (without starting dfs), and I can run
Hadoop programs normally. I do have to specify full input and output
file paths- they don't seem to be relative to fs.default.name . That's
not too troublesome, though.
Thanks very much!
-Joel
[EMAIL PROTECTED]
On Fri, 2008-08-29 at 10:52 -0700, Owen O'Malley wrote:
Check the setting for mapred.system.dir. This needs to be a path that is on
a distributed file system. In old versions of Hadoop, it had to be on the
default file system, but that is no longer true. In recent versions, the
system dir only needs to be configured on the JobTracker and it is passed to
the TaskTrackers and clients.
Re: Hadoop over Lustre?
So far no success, Konstantin- the hadoop job seems to start up, but fails immediately leaving no logs. What is the appropriate setting for mapred.job.tracker ? The generic value references hdfs, but it also has a port number- I'm not sure what that means. My cluster is small, but if I get this working I'd be very happy to run some benchmarks. Are there standard tests of hadoop performance? -Joel [EMAIL PROTECTED] On Fri, 2008-08-22 at 15:59 -0700, Konstantin Shvachko wrote: I think the solution should be easier than Arun and Steve advise. Lustre is already mounted as a local directory on each cluster machines, right? Say, it is mounted on /mnt/lustre. Then you configure hadoop-site.xml and set property namefs.default.name/name valuefile:///mnt/lustre/value /property And then you start map-reduce only without hdfs using start-mapred.sh By this you basically redirect all FileSystem requests to Lustre and you don't need data-nodes or the name-node. Please let me know if that works. Also it would very interesting to have your experience shared on this list. Problems, performance - everything is quite interesting. Cheers, --Konstantin Joel Welling wrote: 2. Could you set up symlinks from the local filesystem, so point every node at a local dir /tmp/hadoop with each node pointing to a different subdir in the big filesystem? Yes, I could do that! Do I need to do it for the log directories as well, or can they be shared? -Joel On Fri, 2008-08-22 at 15:48 +0100, Steve Loughran wrote: Joel Welling wrote: Thanks, Steve and Arun. I'll definitely try to write something based on the KFS interface. I think that for our applications putting the mapper on the right rack is not going to be that useful. A lot of our calculations are going to be disordered stuff based on 3D spatial relationships like nearest-neighbor finding, so things will be in a random access pattern most of the time. Is there a way to set up the configuration for HDFS so that different datanodes keep their data in different directories? That would be a big help in the short term. yes, but you'd have to push out a different config to each datanode. 1. I have some stuff that could help there, but its not ready for production use yet [1]. 2. Could you set up symlinks from the local filesystem, so point every node at a local dir /tmp/hadoop with each node pointing to a different subdir in the big filesystem? [1] http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf
Re: Hadoop over Lustre?
2. Could you set up symlinks from the local filesystem, so point every node at a local dir /tmp/hadoop with each node pointing to a different subdir in the big filesystem? Yes, I could do that! Do I need to do it for the log directories as well, or can they be shared? -Joel On Fri, 2008-08-22 at 15:48 +0100, Steve Loughran wrote: Joel Welling wrote: Thanks, Steve and Arun. I'll definitely try to write something based on the KFS interface. I think that for our applications putting the mapper on the right rack is not going to be that useful. A lot of our calculations are going to be disordered stuff based on 3D spatial relationships like nearest-neighbor finding, so things will be in a random access pattern most of the time. Is there a way to set up the configuration for HDFS so that different datanodes keep their data in different directories? That would be a big help in the short term. yes, but you'd have to push out a different config to each datanode. 1. I have some stuff that could help there, but its not ready for production use yet [1]. 2. Could you set up symlinks from the local filesystem, so point every node at a local dir /tmp/hadoop with each node pointing to a different subdir in the big filesystem? [1] http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf
Hadoop over Lustre?
Hi folks; I'm new to Hadoop, and I'm trying to set it up on a cluster for which almost all the disk is mounted via the Lustre filesystem. That filesystem is visible to all the nodes, so I don't actually need HDFS to implement a shared filesystem. (I know the philosophical reasons why people say local disks are better for Hadoop, but that's not the situation I've got). My system is failing, and I think it's because the different nodes are tripping over each other when they try to run HDFS out of the same directory tree. Is there a way to turn off HDFS and just let Lustre do the distributed filesystem? I've seen discussion threads about Hadoop with NFS which said something like 'just specify a local filesystem and everything will be fine', but I don't know how to do that. I'm using Hadoop 0.17.2. Thanks, I hope; -Joel Welling [EMAIL PROTECTED]
Problem with installation of 0.17.0: things start but tests fail
I'm new to Hadoop, and am doing an installation of 0.17.1 on a small system. The only thing that I know is unusual about the system is that the underlying filesystem is in fact shared- a single Lustre filesystem is visible to all the nodes, and individual nodes all seem to be storing their blocks in the same directory, under dfs.name.dir/current . I follow the installation instructions on http://hadoop.apache.org/core/docs/current/cluster_setup.html , and everything seems to start up just fine. I then try the test problem from http://hadoop.apache.org/core/docs/current/quickstart.html with: bin/hadoop dfs -put conf input bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+' This results in: 08/08/13 19:18:47 INFO mapred.FileInputFormat: Total input paths to process : 1308/08/13 19:18:47 INFO mapred.JobClient: Running job: job_200808131916_0001 08/08/13 19:18:49 INFO mapred.JobClient: map 0% reduce 0% 08/08/13 19:18:51 INFO mapred.JobClient: map 14% reduce 0% 08/08/13 19:19:05 INFO mapred.JobClient: map 14% reduce 4% 08/08/13 19:19:05 INFO mapred.JobClient: Task Id : task_200808131916_0001_m_02_0, Status : FAILED java.io.IOException: Could not obtain block: blk_-4001091320210055678 file=/user/welling/input/log4j.properties at org.apache.hadoop.dfs.DFSClient $DFSInputStream.chooseDataNode(DFSClient.java:1430) ... and similar errors on a bunch of other files. When I look at the status of a particular file, like log4j.properties, I see: java.io.IOException: Got error in response to OP_READ_BLOCK for file muir03.psc.edu/128.182.99.103:50010:-4001091320210055678 for block -4001091320210055678 However, I do see a block by that name in dfs.name.dir/current ! Is this some sort of configuration issue? Do I need to somehow arrange for the different datanodes to keep their files in separate directories? Thanks, I hope, -Joel Welling [EMAIL PROTECTED]
