Data for Testing in Hadoop
Dear all, Designing the architecture is very important for the Hadoop in Production Clusters. We are researching to run Hadoop in Cloud in Individual Nodes and in Cloud Environment ( VM's ). For this, I require some data for testing. Would anyone send me some links for data of different sizes ( 10Gb, 20GB, 30 Gb , 50GB ) . I shall be grateful for this kindness. Thanks & Regards Adarsh Sharma
Re: Data for Testing in Hadoop
How about http://aws.amazon.com/datasets?_encoding=UTF8&jiveRedirect=1 ? Just the first one (WestburyLab USENET corpus) is 40GB. I suspect you can find different formats and data sizes there. Dave Viner On Mon, Jan 3, 2011 at 11:31 PM, Adarsh Sharma wrote: > Dear all, > > Designing the architecture is very important for the Hadoop in Production > Clusters. > > We are researching to run Hadoop in Cloud in Individual Nodes and in Cloud > Environment ( VM's ). > > For this, I require some data for testing. Would anyone send me some links > for data of different sizes ( 10Gb, 20GB, 30 Gb , 50GB ) . > I shall be grateful for this kindness. > > > Thanks & Regards > > Adarsh Sharma > >
Re: Data for Testing in Hadoop
You can use MR to generate the data itself. Checkout GridMix in Hadoop, or PigMix from Pig for examples on general load tests. On Tue, Jan 4, 2011 at 1:01 PM, Adarsh Sharma wrote: > Dear all, > > Designing the architecture is very important for the Hadoop in Production > Clusters. > > We are researching to run Hadoop in Cloud in Individual Nodes and in Cloud > Environment ( VM's ). > > For this, I require some data for testing. Would anyone send me some links > for data of different sizes ( 10Gb, 20GB, 30 Gb , 50GB ) . > I shall be grateful for this kindness. > > > Thanks & Regards > > Adarsh Sharma > > -- Harsh J www.harshj.com
Re: Data for Testing in Hadoop
https://cwiki.apache.org/confluence/display/MAHOUT/Collections All the collections you can imagine. On Tue, Jan 4, 2011 at 12:28 AM, Harsh J wrote: > You can use MR to generate the data itself. Checkout GridMix in > Hadoop, or PigMix from Pig for examples on general load tests. > > On Tue, Jan 4, 2011 at 1:01 PM, Adarsh Sharma > wrote: >> Dear all, >> >> Designing the architecture is very important for the Hadoop in Production >> Clusters. >> >> We are researching to run Hadoop in Cloud in Individual Nodes and in Cloud >> Environment ( VM's ). >> >> For this, I require some data for testing. Would anyone send me some links >> for data of different sizes ( 10Gb, 20GB, 30 Gb , 50GB ) . >> I shall be grateful for this kindness. >> >> >> Thanks & Regards >> >> Adarsh Sharma >> >> > > > > -- > Harsh J > www.harshj.com > -- Lance Norskog goks...@gmail.com
Re: Data for Testing in Hadoop
Also, Amazon offers free public data sets at: http://aws.amazon.com/datasets?_encoding=UTF8&jiveRedirect=1 On Tue, Jan 4, 2011 at 7:28 PM, Lance Norskog wrote: > https://cwiki.apache.org/confluence/display/MAHOUT/Collections > > All the collections you can imagine. > > On Tue, Jan 4, 2011 at 12:28 AM, Harsh J wrote: > > You can use MR to generate the data itself. Checkout GridMix in > > Hadoop, or PigMix from Pig for examples on general load tests. > > > > On Tue, Jan 4, 2011 at 1:01 PM, Adarsh Sharma > wrote: > >> Dear all, > >> > >> Designing the architecture is very important for the Hadoop in > Production > >> Clusters. > >> > >> We are researching to run Hadoop in Cloud in Individual Nodes and in > Cloud > >> Environment ( VM's ). > >> > >> For this, I require some data for testing. Would anyone send me some > links > >> for data of different sizes ( 10Gb, 20GB, 30 Gb , 50GB ) . > >> I shall be grateful for this kindness. > >> > >> > >> Thanks & Regards > >> > >> Adarsh Sharma > >> > >> > > > > > > > > -- > > Harsh J > > www.harshj.com > > > > > > -- > Lance Norskog > goks...@gmail.com >
Re: Data for Testing in Hadoop
On Tuesday 04 January 2011 01:01 PM, Adarsh Sharma wrote: For this, I require some data for testing. Would anyone send me some links for data of different sizes ( 10Gb, 20GB, 30 Gb , 50GB ) . I shall be grateful for this kindness. If you just want random data of a specific size, you can use "dd" on Linux with the /dev/urandom pseudo-file. For example, to generate 10 MiB of random data: dd if=/dev/urandom of=data.bin bs=1024 count=10240 For more structured and "Hadoop-enabled" random data-generation, you can use the data-generator from PigMix2: http://wiki.apache.org/pig/DataGeneratorHadoop https://issues.apache.org/jira/browse/PIG-200 HTH, Ranjit