Thanks you guys advice! I have to mention more for my use case:
(1) million files to store(2) 99% static, no change once written(3) fast
download, or highly Available (4) cost effective
(5) in future, would like extend a versioning system on the file
of course from administrative point of view,
Ok, i have look at the logs some further and googled every tiny bit of
them, hoping to find an answer out there.
I fear that the following line nails my problem at a big scale:
12/05/22 01:30:21 INFO mapred.ReduceTask: attempt_local_0001_r_00_0
Need another 2 map output(s)
where 0 is
-Original Message-
From: Keith Wiley [mailto:kwi...@keithwiley.com]
Sent: Tuesday, May 22, 2012 9:57 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Storing millions of small files
In addition to the responses already provided, there is another downside to
using hadoop with numerous
Mongo has the best out of box experience of anything, but can be limited in
terms of how far it will scale.
Hbase is a bit tricky to manage if you don't have expertise in managing
Hadoop.
Neither is a great idea if your data objects can be as large as 10MB.
On Wed, May 23, 2012 at 8:30 AM,
