I have few questions regarding the Lustre Filesystem with 1 MDS/MDT on the
same machine, 2 OSS configured with LVM created through LVM(total of 12
OSTs) and 4 Lustre Client running Hadoop(1 namenode and 3 datanode). Hadoop
uses Lustre instead of HDFS.
Question: I have created LVM for OSTs instead
I generally run TestDFSIO for Hadoop running on HDFS as
shown:
hadoop jar hadoop-*test*.jar TestDFSIO -write -nrFiles 10 -fileSize 1000
How to run it on lustre filesystem?
___
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lus
ping 192.168.1.1@o2ib: Input/output error
On Sat, May 4, 2013 at 4:57 PM, linux freaker wrote:
> Hi Lee,
>
> I have one lustreclient1 working fine. I followed it this way:
>
> # lustre_rmmod
>
> #modprobe lustre
> #mount -t lustre 192.168.1.1 at o2ib0:/lustre /mnt/lustr
; --
>
> Brett Lee
>
> Sr. Systems Engineer
>
> Intel High Performance Data Division
>
> ** **
>
> *From:* lustre-discuss-boun...@lists.lustre.org [mailto:
> lustre-discuss-boun...@lists.lustre.org] *On Behalf Of *linux freaker
> *Sent:* Friday, May 03, 20
My MDS and OSS configuration look like:
MDS:
#mount
/dev/mapper/vg00-mdt on /mnt/mdt type lustre (rw)
lctl list_nids
192.168.1.1@o2ib
OSS1 and OSS2:
# lctl list_nids
192.168.1.2@o2ib
cat /proc/fs/lustre/devices
0 UP mgc MGC192.168.1.1@o2ib fe161cab-092e-5a7b-0ac1-6081653d099e 5
1 UP ost
I have 1 MDS, 2 OSS with 6 OSTs each and 4 Lustre Client. I have Hadoop
setup with 1 Namenode and 3 datanode running on lustre clients. While I
have 18GB input data in the form of many small files for wordcount, it took
1 hour time (striping: -1).
I have been running this on Ethernet. I have order
Hi,
I have right now 1 MDS, 2 OSS/OST and 2 LustreClient running. I can run
Wordcount example successfully. Now I want to run bigger application like
BigMapOutput or any other. How to run BigMapOutput?
___
Lustre-discuss mailing list
Lustre-discuss@lists
-- Forwarded message --
From: linux freaker
Date: Monday, March 18, 2013
Subject: Understanding lustre setup ..
To: Colin Faber
On Monday, March 18, 2013, Colin Faber wrote:
> Hi,
>
> On 03/15/2013 10:56 AM, linux freaker wrote:
>>
>> Let me explain in br
Can you please suggest?
On Mon, Mar 18, 2013 at 1:26 PM, linux freaker wrote:
> Hi Colin,
>
> Thanks for the response. So you mean to say I need to install Hadoop
> on just one lustre client.
> Is it enough to setup Hadoop? But how come I will start
> datanode.Where will it r
I installed 1 MDS , 2 OSS/OST and 2 Lustre Client. My MDS shows:
[code]
[root@MDS ~]# lctl list_nids
10.94.214.185@tcp
[root@MDS ~]#
[/code]
On Lustre Client1:
[code]
[root@lustreclient1 lustre]# lfs df -h
UUID bytesUsed Available Use% Mounted on
lustre-MDT_U
e exporting /mnt/lustre to your hadoop
> cluster.
>
> I hope this makes sense =)
>
> -cf
>
>
>
>
> On 03/11/2013 11:33 AM, linux freaker wrote:
>>
>> Hello,
>>
>> I have been reading
>> http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf for s
Hi,
I am trying to run Apache Hadoop project on parallel filesystem like
lustre. I have 1 MDS, 2 OSS/OST and 1 Lustre Client.
My lustre client shows:
Code:
[root@lustreclient1 ~]# lfs df -h
UUID bytesUsed Available Use% Mounted on
lustre-MDT_UUID 4.5G
I installed 1 MDS , 2 OSS/OST and 2 Lustre Client. My MDS shows:
[code]
[root@MDS ~]# lctl list_nids
10.94.214.185@tcp
[root@MDS ~]#
[/code]
On Lustre Client1:
[code]
[root@lustreclient1 lustre]# lfs df -h
UUID bytesUsed Available Use% Mounted on
lus
Hello,
I have been reading
http://wiki.lustre.org/images/1/1b/Hadoop_wp_v0.4.2.pdf for setting up
Hadoop over lustre.
Generally in hadoop setup, we have 1 Namenode and various number of datanodes.
If I want to setup the same keeping Lustre as backend, in the document
it is mentioned that:
"..
snapshot=0
13/02/17 17:14:40 INFO mapred.JobClient: Reduce output records=5590
13/02/17 17:14:40 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
13/02/17 17:14:40 INFO mapred.JobClient: Map output records=21582
[root@alpha hadoop]#
[/code]
Does it mean hadoop over lustre is working
java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
[root@alpha hadoop]#
[/code]
On 2/17/13, linux freaker wrote:
> Hello,
>
> I have 4 machines - 1 MDS, 1 OSS, 2 Linux client. I need to run Hadoop
> over lustre replacing HDFS. All I have put the
Hello,
I have 4 machines - 1 MDS, 1 OSS, 2 Linux client. I need to run Hadoop
over lustre replacing HDFS. All I have put the setup detail under
http://paste.ubuntu.com/1661235/
All I need to know is what I really need for Hadoop, what
configuration changes are needed?
Please suggest.
___
17 matches
Mail list logo
