Re: Small File Management

2010-10-26 Thread Ananth Sarathy
Thanks, but that more of a one time use, not ongoing management. Ananth T Sarathy On Tue, Oct 26, 2010 at 12:31 PM, Mark Kerzner wrote: > http://stuartsierra.com/2008/04/24/a-million-little-files > > On Tue, Oct 26, 2010 at 11:28 AM, Ananth Sarathy < > ananth.t.sara...@gmai

Re: Small File Management

2010-10-26 Thread Ananth Sarathy
, Oct 26, 2010 at 1:07 PM, Patrick Angeles wrote: > HBase might fit the bill. > > On Tue, Oct 26, 2010 at 12:28 PM, Ananth Sarathy < > ananth.t.sara...@gmail.com > > wrote: > > > I was wondering if there were any projects out there doing a small file > > management

Small File Management

2010-10-26 Thread Ananth Sarathy
I was wondering if there were any projects out there doing a small file management layer on top of Hadoop? I know that HDFS is primarily for map/reduce but I think companies are going to start using hdfs clusters as storage in the cloud, and i was wondering if any work had been done on this. Ananth

Size of Directory in HDFS

2010-10-20 Thread Ananth Sarathy
I am trying to see how what the total size of all the files within a directory in HDFS is. I try Configuration conf = new Configuration(); Path inFile = new Path( "/MyDir"); FileSystem fs = inFile.getFileSystem(conf); FileStatus status =

Re: Next Release of Hadoop version number and Kerberos

2010-07-07 Thread Ananth Sarathy
gt; Tom > > On Wed, Jul 7, 2010 at 8:09 AM, Ananth Sarathy > wrote: > > > > is the next release of Hadoop going to .21 or .22? I was just wondering, > > cause I am hearing conflicting things about the next release having > Kerberos > > security but loo

Next Release of Hadoop version number and Kerberos

2010-07-07 Thread Ananth Sarathy
is the next release of Hadoop going to .21 or .22? I was just wondering, cause I am hearing conflicting things about the next release having Kerberos security but looking through some past emails, hearing that it was coming in .22. Ananth T Sarathy

Re: HDFS without Consideration for Map and Reduce

2010-07-06 Thread Ananth Sarathy
FS anyway Ananth T Sarathy On Tue, Jul 6, 2010 at 7:26 PM, Allen Wittenauer wrote: > > On Jul 6, 2010, at 1:51 PM, Ananth Sarathy wrote: > > > Yea I know I can use a nas or San. I am not really asking about this as a > > use case on what the best way way to do it is but rather wh

Re: HDFS without Consideration for Map and Reduce

2010-07-06 Thread Ananth Sarathy
, 2010 3:43 PM, "Patrick Angeles" wrote: If all you want is dumb storage for small-ish files, you can always just use NAS or SAN. For the MP3 example, you might want to consider HBase... you can store associated meta-data in column families. On Tue, Jul 6, 2010 at 3:33 PM, Ananth Sar

HDFS without Consideration for Map and Reduce

2010-07-06 Thread Ananth Sarathy
So I am aware of the problem with small files and I have read this article http://www.cloudera.com/blog/2009/02/the-small-files-problem/ I am just wondering if there has been any real change in this? For example's sake, suppose you just want an HDFS Cluster that never does any m/r jobs but would

Re: Questions about SequenceFiles

2010-05-11 Thread Ananth Sarathy
T Sarathy On Tue, May 11, 2010 at 11:04 AM, Owen O'Malley wrote: > On Tue, May 11, 2010 at 7:48 AM, Ananth Sarathy > wrote: > > Ok, how can I report that? > > File a jira on the project that manages the type. I assume it is > Lucene in this case. > > > Also,

Re: Questions about SequenceFiles

2010-05-11 Thread Ananth Sarathy
writable object should have default no-argument > constructor. > > > On Tue, May 11, 2010 at 7:20 AM, Ananth Sarathy > wrote: > > Yeah, no I get that. But when you use the sequence file reader example > from > > The Hadoop The Defintive Guide book page 106 > > &g

Re: Questions about SequenceFiles

2010-05-11 Thread Ananth Sarathy
rocessing better than JavaSerialization. > > Cheers > > On Mon, May 10, 2010 at 6:29 PM, Ananth Sarathy > wrote: > > > My team and I were working with sequence files and were using the > > LuceneDocumentWrapper. But when I try to get the valcall, i get a no such > &

Questions about SequenceFiles

2010-05-10 Thread Ananth Sarathy
My team and I were working with sequence files and were using the LuceneDocumentWrapper. But when I try to get the valcall, i get a no such method exception from the ReflectionUtils, which is caused because it's trying to call a default constructor which doesn't exist for that class. So my questio

CDH2 or Apache Hadoop

2010-02-23 Thread Ananth Sarathy
Just wanted to get the groups general feelings on what the preferred distro is and why? Obviously assuming one didn't have a service agreement with cloudera. Ananth T Sarathy