Re: [Lustre-discuss] Need help on lustre..
Hi, I think you might be better served with your Hadoop setup by posting to the Hadoop discussion list. Once you have it setup and working, if you run into Lustre related issues, please feel free to post those here. Good luck! -cf On 02/17/2013 04:47 AM, linux freaker wrote: > Great !!! I tried removing entry from mapred-site.xml and it seems to run > well. > > Here are the logs now: > > [code] > [root@alpha hadoop]# bin/hadoop jar hadoop-examples-1.1.1.jar > wordcount /user/ha > doop/hadoop/ > /user/hadoop/hadoop/output > 13/02/17 17:14:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library > 13/02/17 17:14:38 INFO input.FileInputFormat: Total input paths to process : 1 > 13/02/17 17:14:38 WARN snappy.LoadSnappy: Snappy native library not loaded > 13/02/17 17:14:38 INFO mapred.JobClient: Running job: job_local_0001 > 13/02/17 17:14:38 INFO util.ProcessTree: setsid exited with exit code 0 > 13/02/17 17:14:38 INFO mapred.Task: Using ResourceCalculatorPlugin : > org.apache > > .hadoop.util.LinuxResourceCalculatorPlugin@2f74219d > 13/02/17 17:14:38 INFO mapred.MapTask: io.sort.mb = 100 > 13/02/17 17:14:38 INFO mapred.MapTask: data buffer = 79691776/99614720 > 13/02/17 17:14:38 INFO mapred.MapTask: record buffer = 262144/327680 > 13/02/17 17:14:38 INFO mapred.MapTask: Starting flush of map output > 13/02/17 17:14:39 INFO mapred.JobClient: map 0% reduce 0% > 13/02/17 17:14:39 INFO mapred.MapTask: Finished spill 0 > 13/02/17 17:14:39 INFO mapred.Task: Task:attempt_local_0001_m_00_0 > is done. >And is in the process of commiting > 13/02/17 17:14:39 INFO mapred.LocalJobRunner: > 13/02/17 17:14:39 INFO mapred.Task: Task 'attempt_local_0001_m_00_0' done. > 13/02/17 17:14:39 INFO mapred.Task: Using ResourceCalculatorPlugin : > org.apache > > .hadoop.util.LinuxResourceCalculatorPlugin@6d79953c > 13/02/17 17:14:39 INFO mapred.LocalJobRunner: > 13/02/17 17:14:39 INFO mapred.Merger: Merging 1 sorted segments > 13/02/17 17:14:39 INFO mapred.Merger: Down to the last merge-pass, > with 1 segmen >ts left of total size: 79496 bytes > 13/02/17 17:14:39 INFO mapred.LocalJobRunner: > 13/02/17 17:14:39 INFO mapred.Task: Task:attempt_local_0001_r_00_0 > is done. >And is in the process of commiting > 13/02/17 17:14:39 INFO mapred.LocalJobRunner: > 13/02/17 17:14:39 INFO mapred.Task: Task attempt_local_0001_r_00_0 > is allowe >d to commit now > 13/02/17 17:14:39 INFO output.FileOutputCommitter: Saved output of > task 'attempt >_local_0001_r_00_0' to > /user/hadoop/hadoop/output > 13/02/17 17:14:39 INFO mapred.LocalJobRunner: reduce > reduce > 13/02/17 17:14:39 INFO mapred.Task: Task 'attempt_local_0001_r_00_0' done. > 13/02/17 17:14:40 INFO mapred.JobClient: map 100% reduce 100% > 13/02/17 17:14:40 INFO mapred.JobClient: Job complete: job_local_0001 > 13/02/17 17:14:40 INFO mapred.JobClient: Counters: 20 > 13/02/17 17:14:40 INFO mapred.JobClient: File Output Format Counters > 13/02/17 17:14:40 INFO mapred.JobClient: Bytes Written=57885 > 13/02/17 17:14:40 INFO mapred.JobClient: FileSystemCounters > 13/02/17 17:14:40 INFO mapred.JobClient: FILE_BYTES_READ=643420 > 13/02/17 17:14:40 INFO mapred.JobClient: FILE_BYTES_WRITTEN=574349 > 13/02/17 17:14:40 INFO mapred.JobClient: File Input Format Counters > 13/02/17 17:14:40 INFO mapred.JobClient: Bytes Read=139351 > 13/02/17 17:14:40 INFO mapred.JobClient: Map-Reduce Framework > 13/02/17 17:14:40 INFO mapred.JobClient: Map output materialized > bytes=79500 > 13/02/17 17:14:40 INFO mapred.JobClient: Map input records=2932 > 13/02/17 17:14:40 INFO mapred.JobClient: Reduce shuffle bytes=0 > 13/02/17 17:14:40 INFO mapred.JobClient: Spilled Records=11180 > 13/02/17 17:14:40 INFO mapred.JobClient: Map output bytes=212823 > 13/02/17 17:14:40 INFO mapred.JobClient: Total committed heap > usage (bytes)= > 500432896 > 13/02/17 17:14:40 INFO mapred.JobClient: CPU time spent (ms)=0 > 13/02/17 17:14:40 INFO mapred.JobClient: SPLIT_RAW_BYTES=99 > 13/02/17 17:14:40 INFO mapred.JobClient: Combine input records=21582 > 13/02/17 17:14:40 INFO mapred.JobClient: Reduce input records=5590 > 13/02/17 17:14:40 INFO mapred.JobClient: Reduce input groups=5590 > 13/02/17 17:14:40 INFO mapred.JobClient: Combine output records=5590 > 13/02/17 17:14:40 INFO mapred.JobClient: Physical memory (bytes) > snapshot=0 > 13/02/17 17:14:40 INFO mapred.JobClient: Reduce output records=5590 > 13/02/17 17:14:40 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0 > 13/02/17 17:14:40 INFO mapred.JobClient: Map output records=21582 > [root@alpha hadoop]# > > [/code] > > Does it mean hadoop over lustre is working fine? > > On 2/17/13, linux freaker wrote: >> I tried running the below
Re: [Lustre-discuss] Need help on lustre..
Great !!! I tried removing entry from mapred-site.xml and it seems to run well. Here are the logs now: [code] [root@alpha hadoop]# bin/hadoop jar hadoop-examples-1.1.1.jar wordcount /user/ha doop/hadoop/ /user/hadoop/hadoop/output 13/02/17 17:14:37 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/02/17 17:14:38 INFO input.FileInputFormat: Total input paths to process : 1 13/02/17 17:14:38 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/17 17:14:38 INFO mapred.JobClient: Running job: job_local_0001 13/02/17 17:14:38 INFO util.ProcessTree: setsid exited with exit code 0 13/02/17 17:14:38 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache .hadoop.util.LinuxResourceCalculatorPlugin@2f74219d 13/02/17 17:14:38 INFO mapred.MapTask: io.sort.mb = 100 13/02/17 17:14:38 INFO mapred.MapTask: data buffer = 79691776/99614720 13/02/17 17:14:38 INFO mapred.MapTask: record buffer = 262144/327680 13/02/17 17:14:38 INFO mapred.MapTask: Starting flush of map output 13/02/17 17:14:39 INFO mapred.JobClient: map 0% reduce 0% 13/02/17 17:14:39 INFO mapred.MapTask: Finished spill 0 13/02/17 17:14:39 INFO mapred.Task: Task:attempt_local_0001_m_00_0 is done. And is in the process of commiting 13/02/17 17:14:39 INFO mapred.LocalJobRunner: 13/02/17 17:14:39 INFO mapred.Task: Task 'attempt_local_0001_m_00_0' done. 13/02/17 17:14:39 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache .hadoop.util.LinuxResourceCalculatorPlugin@6d79953c 13/02/17 17:14:39 INFO mapred.LocalJobRunner: 13/02/17 17:14:39 INFO mapred.Merger: Merging 1 sorted segments 13/02/17 17:14:39 INFO mapred.Merger: Down to the last merge-pass, with 1 segmen ts left of total size: 79496 bytes 13/02/17 17:14:39 INFO mapred.LocalJobRunner: 13/02/17 17:14:39 INFO mapred.Task: Task:attempt_local_0001_r_00_0 is done. And is in the process of commiting 13/02/17 17:14:39 INFO mapred.LocalJobRunner: 13/02/17 17:14:39 INFO mapred.Task: Task attempt_local_0001_r_00_0 is allowe d to commit now 13/02/17 17:14:39 INFO output.FileOutputCommitter: Saved output of task 'attempt _local_0001_r_00_0' to /user/hadoop/hadoop/output 13/02/17 17:14:39 INFO mapred.LocalJobRunner: reduce > reduce 13/02/17 17:14:39 INFO mapred.Task: Task 'attempt_local_0001_r_00_0' done. 13/02/17 17:14:40 INFO mapred.JobClient: map 100% reduce 100% 13/02/17 17:14:40 INFO mapred.JobClient: Job complete: job_local_0001 13/02/17 17:14:40 INFO mapred.JobClient: Counters: 20 13/02/17 17:14:40 INFO mapred.JobClient: File Output Format Counters 13/02/17 17:14:40 INFO mapred.JobClient: Bytes Written=57885 13/02/17 17:14:40 INFO mapred.JobClient: FileSystemCounters 13/02/17 17:14:40 INFO mapred.JobClient: FILE_BYTES_READ=643420 13/02/17 17:14:40 INFO mapred.JobClient: FILE_BYTES_WRITTEN=574349 13/02/17 17:14:40 INFO mapred.JobClient: File Input Format Counters 13/02/17 17:14:40 INFO mapred.JobClient: Bytes Read=139351 13/02/17 17:14:40 INFO mapred.JobClient: Map-Reduce Framework 13/02/17 17:14:40 INFO mapred.JobClient: Map output materialized bytes=79500 13/02/17 17:14:40 INFO mapred.JobClient: Map input records=2932 13/02/17 17:14:40 INFO mapred.JobClient: Reduce shuffle bytes=0 13/02/17 17:14:40 INFO mapred.JobClient: Spilled Records=11180 13/02/17 17:14:40 INFO mapred.JobClient: Map output bytes=212823 13/02/17 17:14:40 INFO mapred.JobClient: Total committed heap usage (bytes)= 500432896 13/02/17 17:14:40 INFO mapred.JobClient: CPU time spent (ms)=0 13/02/17 17:14:40 INFO mapred.JobClient: SPLIT_RAW_BYTES=99 13/02/17 17:14:40 INFO mapred.JobClient: Combine input records=21582 13/02/17 17:14:40 INFO mapred.JobClient: Reduce input records=5590 13/02/17 17:14:40 INFO mapred.JobClient: Reduce input groups=5590 13/02/17 17:14:40 INFO mapred.JobClient: Combine output records=5590 13/02/17 17:14:40 INFO mapred.JobClient: Physical memory (bytes) snapshot=0 13/02/17 17:14:40 INFO mapred.JobClient: Reduce output records=5590 13/02/17 17:14:40 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0 13/02/17 17:14:40 INFO mapred.JobClient: Map output records=21582 [root@alpha hadoop]# [/code] Does it mean hadoop over lustre is working fine? On 2/17/13, linux freaker wrote: > I tried running the below command but got the below error. > I have not put it into HDFS since Lustre is what I am trying to implement > with. > > [code] > #bin/hadoop jar hadoop-examples-1.1.1.jar wordcount > /user/hadoop/hadoop /user/hadoop-output > > 13/02/17 17:02:50 INFO util.NativeCodeLoader: Loaded the native-hadoop > library > 13/02/17 17:02:50 INFO input.FileInputFormat: Total input paths to process : > 1 > 13/02/17 17:02:50 WARN snappy.LoadSnappy: Snappy native library not loaded
Re: [Lustre-discuss] Need help on lustre..
I tried running the below command but got the below error. I have not put it into HDFS since Lustre is what I am trying to implement with. [code] #bin/hadoop jar hadoop-examples-1.1.1.jar wordcount /user/hadoop/hadoop /user/hadoop-output 13/02/17 17:02:50 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/02/17 17:02:50 INFO input.FileInputFormat: Total input paths to process : 1 13/02/17 17:02:50 WARN snappy.LoadSnappy: Snappy native library not loaded 13/02/17 17:02:50 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/ hadoop-hadoop/mapred/staging/root/.staging/job_201302161113_0004 13/02/17 17:02:50 ERROR security.UserGroupInformation: PriviledgedActionExceptio n as:root cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException: java .io.FileNotFoundException: File file:/tmp/hadoop-hadoop/mapred/staging/root/.sta ging/job_201302161113_0004/job.xml does not exist. at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3731) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3695) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) Caused by: java.io.FileNotFoundException: File file:/tmp/hadoop-hadoop/mapred/st aging/root/.staging/job_201302161113_0004/job.xml does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSys tem.java:397) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem. java:251) at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:406) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3729) ... 12 more org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.FileNotFound Exception: File file:/tmp/hadoop-hadoop/mapred/staging/root/.staging/job_2013021 61113_0004/job.xml does not exist. at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3731) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3695) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) Caused by: java.io.FileNotFoundException: File file:/tmp/hadoop-hadoop/mapred/st aging/root/.staging/job_201302161113_0004/job.xml does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSys tem.java:397) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem. java:251) at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:406) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3729) ... 12 more at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
[Lustre-discuss] Need help on lustre..
Hello, I have 4 machines - 1 MDS, 1 OSS, 2 Linux client. I need to run Hadoop over lustre replacing HDFS. All I have put the setup detail under http://paste.ubuntu.com/1661235/ All I need to know is what I really need for Hadoop, what configuration changes are needed? Please suggest. ___ Lustre-discuss mailing list Lustre-discuss@lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-discuss
