Hi, friends
I need store the web pages(a huge one) in the MapFile of the hadoop, So i
did use the url as the key, and its type is "text", When writring the
records into the mapfile, it give an error as "out of order", which type
should I choose to represent the key "url", can anyone give me some
Hi Wang,
The keys of MapFile should be in order, so when you add records into
MapFile, you should make sure you insert them in order
Best Regards,
Jeff Zhang
On Wed, Oct 28, 2009 at 4:14 PM, lei wang wrote:
> Hi, friends
> I need store the web pages(a huge one) in the MapFile of the hadoop,
but now, "url" is not in order, must the key be intwritable ? should it be
comparable ?
How to make sure them in order?sort it first?
I just want to insert the pages for random acess by "url ".
On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang wrote:
> Hi Wang,
>
> The keys of MapFile should be in
I do not know why you need use MapFile, could you use SequenceFile instead ?
The MapFile's advantage is its read performance, because it build index on
its keys. So its keys must be in order.
If you really want to use MapFile, you can first write your data to
SequenceFile and then covert it to Ma
If you just want to develop and test Hadoop applications, you can run
Hadoop on your own machine by following the Quick Start instructions
linked at http://hadoop.apache.org/common/docs/r0.20.0/. If you want
to run on a cluster, Amazon is very cheap for small computations
(several dollars p
hi,juff, thanks for your comments.
I did read this book early, I use MapFile to store my web pages for
random access.
First I think the SquenceFile conversion as a solution, howerve, the
problem is that I need append the new pages to the MapFile by minute
or second, so I didn't think SquenceFile
I guess maybe HBase will be fit for you. HBase is a distributed database
built upon Hadoop.
You can use the url as the row key and put other fields into columns.
then you can retrieve the web page through HBase Client API and insert new
web page into it. The performance of HBase 0.20 is good eno
Oh, I have tried hbase in the early.
But I think HDFS may give me a choice.
Thanks.
On Thu, Oct 29, 2009 at 10:16 AM, Jeff Zhang wrote:
> I guess maybe HBase will be fit for you. HBase is a distributed database
> built upon Hadoop.
> You can use the url as the row key and put other fields int
heck out www.HiQube.com or www.pbsgridworks.com
-Original Message-
From: lei wang [mailto:hadoopmaill...@gmail.com]
Sent: Wednesday, October 28, 2009 7:22 PM
To: general@hadoop.apache.org
Subject: Re: Need Help: The problem with text key of MapFile
Oh, I have tried hbase in the early.
B
Hi
There is 1 GB of rdf/owl files that I am executing on EC2. Execution throws
the following exception
---
08/11/19 16:08:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
org.apache.hadoop.fs.s3.S3Except
Hi
There is 1 GB of rdf/owl files that I am executing on EC2. Execution throws
the following exception
---
08/11/19 16:08:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
org.apache.hadoop.fs.s3.S3Except
I cann't understand why you give me two web sites.
On Thu, Oct 29, 2009 at 10:27 AM, Lori Ann Martin wrote:
> heck out www.HiQube.com or www.pbsgridworks.com
>
> -Original Message-
> From: lei wang [mailto:hadoopmaill...@gmail.com]
> Sent: Wednesday, October 28, 2009 7:22 PM
> To: general
12 matches
Mail list logo